Chris, Shawn, and Lukas — The Weights & Biases Journey

The three Weights & Biases co-founders (Chris, Shawn, and Lukas) share how the company got started, reflect on the highs and lows, and give advice to first-time entrepreneurs.
Angelica Pan

About this episode

You might know him as the host of Gradient Dissent, but Lukas is also the CEO of Weights & Biases, a developer-first ML tools platform!
In this special episode, the three W&B co-founders — Chris (CVP), Shawn (CTO), and Lukas (CEO) — sit down to tell the company's origin stories, reflect on the highs and lows, and give advice to engineers looking to start their own business.
Chris reveals the W&B tech stack (tl;dr - React + GraphQL + Golang), Shawn shares his favorite product feature (it's a hidden frontend layer), and Lukas explains why it's so important to work with customers that inspire you.

Connect with us

Listen

Apple Podcasts Spotify Google Podcasts YouTube

Timestamps

0:00 Intro
1:29 The stories behind Weights & Biases
7:45 The W&B tech stack
9:28 Looking back at the beginning
11:42 Hallmark moments
14:49 Favorite product features
16:49 Rewriting the W&B backend
18:21 The importance of customer feedback
21:18 How Chris and Shawn have changed
22:35 How the ML space has changed
28:24 Staying positive when things look bleak
32:19 Lukas' advice to new entrepreneurs
35:29 Hopes for the next five years
38:09 Making a paintbot & model understanding
41:30 Biggest bottlenecks in deployment
44:08 Outro
44:38 Bonus: Under- vs overrated technologies

Watch on YouTube

Transcript

Note: Transcriptions are provided by a third-party service, and may contain some inaccuracies. Please submit any corrections to angelica@wandb.com. Thank you!

Intro

Chris:
We had presented to OpenAI, to a group of researchers. I remember I was presenting and half the audience was just looking at their laptops. I left that meeting feeling like no one cares. Then a week later, Woj calls us up and he says, "Hey, we got this problem on the robotics team. Come check this out. Can you help us out?" We go, look, and I remember going there with Shawn. Both Shawn and I are looking at it and we're excited, right? Because we can solve the problem. They're telling us they have a problem and they want us to fix it.
Lukas:
You're listening to Gradient Dissent, a show about machine learning in the real world. I'm your host, Lukas Biewald. This episode is a fun departure from our normal format, where I interview two people in the machine learning space about their work. But these two people happen to be my co-founders of Weights & Biases. Shawn Lewis, our CTO, and Chris Van Pelt, our Corporate Vice-President. I talk to them about questions that have been stewing inside of me for years, like "Why did we start this company?" and "Where is it going?" Honestly, I was surprised and educated by their answers and I hope that you enjoy listening to this episode as much as I enjoyed asking them these hard questions.

The stories behind Weights & Biases

Lukas:
All right. Here we go. Should we just jump right in? How did this company start? I ask this because it's the most common question I get asked if I go on any other podcast or with any candidate. Everyone wants to know how did you start the company. I was kind of realizing you two both have founding stories that's telling the same story, but it may have diverged in its evolution. I would be curious to hear you two's version of the story. Which of you wants to go first?
Chris:
Shawn, I think you should go first.
Shawn:
All right. When I started my career as a software engineer, I was at Google and I was in the platforms team at Google. I joined...this was back in 2006. I worked on all kinds of stuff in the platforms team. The platforms team was responsible for building the data centers and all the machines inside the data centers that Google uses. I wrote a lot of the software that ran in those environments. When you write tests like that, they generate a ton of data. At first, I was writing tests, but then I ended up with just all of this data. Really, where I ended up spending my time was on making tools and data pipelines that would help us understand that data. That's what I realized that I loved doing. I ended up writing all kinds of different tools and data pipelines — and the data's in all these different databases all over Google — and I merged it into once place, and defined the metrics that we used to understand hard drives. And then made these user-facing tools — and when I say user, I mean other people at Google — that they could use to dig in and understand this data. I worked on a lot of stuff like that at Google. It was super fun. Then I eventually left. Then I started this other company called Beep. We built hardware. We built a lot of things. This is kind of tangential to the story so I won't go into it, but we were also in Y Combinator. In that Y Combinator batch was actually Lukas's wife, Noga, who I'm not sure if folks know this, but she's the founder of a company called PicnicHealth. We were in a Y Combinator batch, we ended up sharing an office together in the Mission in San Francisco for a couple of years, and I got to know Lukas that way. We had a hardware component at Beep. We had this hardware lab. Lukas loves robots and he was kind of always tinkering in his garage building robots. He would come up to my hardware lab and poke his head through the door and go, "What are you guys doing in here? Also, my robot's broken," and we'd just start to get to know each other and work on the robot and help him fix it. So we became good friends that way. As Beep wound down — I don't know, I think that was 2016, I think, this is my version of the story — Chris and Lukas were starting to step out of Figure Eight around the same time. The circumstances were really good. Deep learning was starting to take off. We were all really excited about it. I had been thinking about these problems a lot. I loved the chance to work with Lukas and Chris, and we just started hanging out, and talking about it, and jumped on that opportunity, and started building the stuff that I needed to organize that data. It was an exciting time and it still is.
Lukas:
All right. Chris, I think you have a different version of the story. Let's hear it.
Chris:
All right. We got to go back. We got to go back to like 2006, like Shawn. Except in my 2006, I'm coming to San Francisco for the first time and it's to work at an exciting startup. I feel like I've made it. I've been doing web development and trying to advance my career and happened to be into Ruby on Rails, which was really hot and exciting at the time. There was this startup in San Francisco that was using Ruby on Rails and using machine learning to create a smarter, hopefully more relevant search engine. That startup was Powerset and I came up to San Francisco — it was like the beginning of 2007 — and that's actually where I met Lukas. Lukas and I joined Powerset at roughly the same time. I was in the product team interfacing with a whole bunch of the other backend teams to try to create an interface to this exciting new tool. Fast forward about a year or so, and Luke and I decided, "Powerset has been fun, but I think it's time for us to give our own go at creating a Powerset or a startup." We set out to make crowdsourcing more accessible to the enterprise and people that wanted to do it to collect training data to train machine learning models. Way before machine learning was as cool as it is today. Yeah, fast forward about 10 years through creating a startup with Lukas, learning a lot along the way. At the time that we started Weights & Biases, Luke and I's day-to-day responsibilities at the company were winding down. We were asking ourselves, "What is next?" I remember going to Lukas's workshop, where most of his podcasts are recorded and playing with robots. I was really into Lua for a little while there, thinking we could make a cool Lua toolkit for robots. But then it was really Lukas having an internship at OpenAI and actually building models with some of the world's most renowned machine learning researchers and needing tools to help him get his job done that was the initial itch that we were scratching. I love building tools. Luke's like, "Hey, I need a tool to help me build models at OpenAI." I said, "Great. Let me try to whip something up," and just really poured myself into making a very early prototype. Then shortly thereafter, Shawn came into the picture, and I am forever grateful.
Lukas:
Awesome, man. I was getting some messages from Lavanya to pull you both back on the rails while you were talking, but I love the extended cut founding stories. I think mine is like a sentence or two. Sorry, Lavanya. Now she's texting me what the... But, no secrets here.

The W&B tech stack

Lukas:
Actually, this is a good segue into another question for you. Shawn, you've been out on paternity leave. Chris, you've been talking to lots of customers independently from me. People keep asking me this, they're like, "What is the architecture of the Weights & Biases server?" I try to describe it and I realize I honestly have no idea. I know there's MySQL involved and React. Can you give me the several sentence lay of the land? Say I'm an engineering candidate and I just want to know what are we using, how does it all fit together.
Chris:
Okay. All right. We got a single page React application that is our front end. It's just a lot of JavaScript. We load that up into the browser and then it makes requests against our GraphQL backend, which happens to be written in Golang. When a customer wants to run Weights & Biases themselves, we actually deliver all of this — the single-page React app and the GraphQL backend API — in a single Docker container that they could run within their Kubernetes cluster or in a managed Terraform-based deployment that we support. Then the backend persistent stores are super simple. We've got MySQL and an S3-Compatible Object Store or Azure Blob storage or Google Cloud Storage. There's a little Redis in there, but customers generally don't have to worry about that.
Shawn:
It's nice because from early on, we knew that having the potential to go on-prem was really important for our customers because of data privacy concerns, because these datasets are so valuable and have all these other privacy concerns. We really just kept it simple to make that possible and, yeah, those were good early choices.

Looking back at the beginning

Lukas:
Shawn, you wrote a document at one point that was, "What it would take to be a billion-dollar business?" for Weights & Biases. I thought maybe we could pull it up and compare it to what it actually took to be a billion-dollar market cap business. Looking at this document, how do you feel that things have played out? Has anything played out differently than you expected?
Shawn:
The core of the argument was, "We're not sure if we can build better products than everybody else in the space, but we can raise a lot of money." We know we can do that and we are well connected to lots of customers because of Lukas and Chris's background with Figure Eight. So it was very easy for us in the early days to go into a customer with very little to show. We had a demo of the early parts of the product and have good conversations. That's really what it takes to develop good products, is to actually interact with customers who look at the product and give you feedback on either a demo or by actually using the product. The argument was we should rapidly expand into the different parts of the ML pipeline in parallel and leverage those connections and the ability to raise money. We could build a team that could build products in each of these spaces. I would say we didn't quite do that. I think we still have this goal of expanding across the ML pipeline. This early theory that maybe we're not better, we may not be great at building products...I would say this is maybe not so humble of a statement, but I think we built something that users really love and we definitely did it by hiring great engineers and great product people and by talking to these customers a ton and spending lots and lots of time and just having this relentless customer focus. But I also think that somehow at core, there's this magic somewhere in what we're doing at W&B and that we do understand the space and the customers and we turn that money and customer connection into great products. So back then, I was thinking, "Well, we have great products," and looking back now, I feel like we really do. That's really been a cool journey.

Hallmark moments

Lukas:
All right. I have a question for both of you I was wondering about. Was there a moment where you felt like the business was really working or the company was really working? Can you think of a time when you really suddenly felt like that or not? If so, what time was it?
Chris:
I think there's a couple times, but one that really stands out for me is driving down to Palo Alto or Mountain View or wherever we were meeting, and one of our first deals was with Toyota Research Institute. I remember sitting out. I had grabbed lunch with Ari, our first Account Executive. I grabbed lunch. We knew we were going to go into this meeting and present the number that we were going to sell our software for. I had this thought of, "This moment is really important." It's scary. When you're brand new, you don't have many other customers and you go and you say, "Hey, we want to charge this much for our software," I felt like this is make-or-break. We went in. The meeting went really well and we ended up closing. TRI is one of our first customers, which is great. But after that, I mean, I wasn't like, "Okay, now we're set. Next week, we can get to a billion dollar valuation," but that first customer was really big.
Shawn:
Yeah, that was a big one. As a founder, or probably anybody at an early stage startup, you say, "Oh, if we can just do this one thing, then we'll be sure that we made it." Then the next week, you're back to work, and you have to grow some more, and you're always looking at the next thing. But that first customer was a great one. I think for me, what comes to mind is — I think this was maybe around the holidays two years ago — in the earlier stages, even up to, say, 15, 20 people in the company, you kind of have a pretty good sense of everything that's happening. Every deal, I remember being a part of in some form. But there was a moment around the holidays a couple years ago where we have these Friday meetings where we get together and talk about how the week went. It's everybody at the company and somebody says something great that they did. In that particular meeting, I just remember there was somebody on the sales team who had made another sale to a customer I had never talked to. Somebody in the growth team that had found a new growth experiment to do and executed it and actually made numbers change. Somebody in the product team that had done something that I didn't even know they were doing. All of those things came together in one meeting for me. That's when I felt like, "Wow, this company is a lot more than...I can't wrap my arms around it and push everything forward anymore. There is all these great people around me that are doing that." That is an amazing feeling because from there, you add more and more great people and the company continues to go in a good direction. It's bigger than yourself.

Favorite product features

Lukas:
Another question I had for both of you is, is there a favorite feature in the product that you feel really proud of or some way that the product works that you feel like is uniquely great?
Chris:
For me, it's got to be the command line interface. I think it's a very under-appreciated interface to our product. In the early days, I spent an obsessive amount of time on making a whole bunch of command line commands and making it work nicely in Unix. I was piping stuff at one point. I think we've since decided that's not the way most people want to interact with our product, but as a Unix nerd, that's my favorite part for sure.
Shawn:
Maybe something that people don't really see is, there is this layer in the frontend that we made. In W&B, you've got all these different charts on the screen. The architecture of the frontend is that each chart can individually make its own network request to get the data that it needs to show. Those are actually heavy requests because there's millions of data points sometimes if you've logged a lot of data. So we built this cool layer in the frontend, it's like a middleware that watches all of the requests going out. It aggregates them all together so it has a little time delay. It says, "Give me all the requests that happened in the last hundred milliseconds." It does lots of cool handwritten optimizations to figure out how to merge certain kinds of queries together and get all the results at once in a single request and then give that back to the user. A lot of users, you have no idea that's going on, but when you're building UIs, you really want it to be snappy and fast. I hope that I'm not shooting myself in the foot. I'm sure somebody will have a story of Weights & Biases not being snappy and fast. If you do, send it my way and we'll get it fixed. But there is this massive amount of engineering effort that went into that chunk of code to make sure that charts with millions of data points all on the screen at once can be updated really quickly.

Rewriting the W&B backend

Lukas:
Just so people don't think I'm only asking softball questions here, this is something that a couple of candidates have asked me about recently. Have there been product or engineering efforts that we would do a lot differently in hindsight?
Chris:
I mean, I've got one that isn't bad, so it's not fair. When we first started the company, I wrote the backend in Python. It was Python 2.0 because I wanted to use Google App Engine to start things up so I didn't have to do a bunch of DevOps stuff. That quickly stopped scaling, especially because we have a GraphQL backend where things need to happen in parallel. Our first engineer Tom actually rewrote that entire backend in GoLang. When you do a big rewrite like that...we already had TRI as a customer, OpenAI was a heavy user. We had to keep the site up while this was happening. Usually those kinds of exercises can go sideways. You can start to take way longer than you would have anticipated or the ultimate project wouldn't have worked out. That was an example where Tom actually got the project done ahead of time and we're still running pretty much the same backend code to scale up to the tens of thousands of users that we have every day now. It's not a decision I would go back and say, "Not do." Now that I had a good experience doing a rewrite, I certainly wouldn't say we always do a rewrite, but you don't hear it often, our choice to do the rewrite was definitely the right choice and it worked out really well.

The importance of customer feedback

Lukas:
What customer feedback has surprised you the most?
Shawn:
I have an example from early on when we started the company, which was...the very first version of Weights & Biases was more of a command line tool around saving data. There was a little bit of UI. Chris had built this Python library and this UI. The UI was essentially, what it did was it let you log in and set up a place to store the data that the command line tool was saving. But it didn't do a whole lot more than that. I remember Chris added a feature that would just kind of...it also collected the outputs of your training runs, like the logs. Chris added a feature that would look for the specific Keras metrics that it printed over time and he just made a line chart of that. Of course, early on in the startup, what you do is you have a demo or a thing that works you go to customers and you show it to them. All the customers were like, "Yeah, command line tool. Okay, okay. But what's that chart?" They would really focus on that chart. The reason it was surprising to me is because these are programmers and data scientists and people who are really comfortable in Matplotlib and Jupyter notebooks. A thesis you might have is, well, data scientists don't need a tool to create a bunch of charts in their browser because their use cases are going to be so different and they're just comfortable doing it themselves. It was really a big surprise to me that that was the main thing people focused on. We saw that and we said, "Well, let's follow what users want," and we did that. We kept building the UI and making it better and better, and we were able now to have a generic UI that solves lots of different kinds of use cases. But it was surprising that that would be possible for this kind of user.
Lukas:
That's a good founding story right there. Chris, what about you?
Chris:
The most notable user feedback that is top of mind is an early user, Hamel at GitHub, was a heavy user of the tool. I remember one night, Hamel wrote in and said, "Hey, we really want to log HTML," and we were actually able to ship that feature in that same night, which was delighting to Hamel. But Hamel was also...he did not hold back in telling us where we were not being excellent in the UI and was very honest about some pretty serious issues with the system at the time. I remember it breaking my heart as a founder. Like here, I've got someone who is engaged and excited about us, but he's getting frustrated by using our tool. It's the worst possible thing I could imagine. The team really focused and did a lot of hard work to redesign and re-engineer a lot of the problem interfaces that Hamel was running into and ultimately I think it really helped us make a better product.

How Chris and Shawn have changed

Lukas:
How do you think the last four and a half years of running this kind of hyper-growth startup has changed you as a person or changed your perspective on the world?
Chris:
I remember when we first started the company, after Luke and I had been working at CrowdFlower for 10-plus years. I was just so excited to have a blank canvas. It was like, "We can start fresh." There is nothing legacy we need to support. It's just green fields. I tend to think of the company stuff, the process and the management, and all of the things that you need to do to make a company work, historically that didn't interest me that much. I think something has changed, especially with this company. Now I find those things more interesting. Being able to step away from just hacking all the time and actually think about, "Okay, how do we build a culture and how do we mentor and work with the team to ultimately build a better product?" I think those problems are much more interesting for me this time around than they were when we were running CrowdFlower.

How the ML space has changed

Lukas:
Another question I had is, what do you think has changed around us as we've been running this company? Do you feel like customers are different now? Do you feel like the industry is different at all?
Shawn:
We started the company a year or two within when deep learning was first...when AlexNet was first trained or when AlexNet actually showed real results. We really focused on deep learning. I mean, one of our first customers was OpenAI. That's well-known. They're still a great customer of ours and we spend a lot of time building things that were tailored to OpenAI use cases. When you start a company, it's good to make a bet on what you think a growing market will be because you don't want to go into...you can do this, but you don't necessarily want to go into a big established market and just fight with Google and Amazon. It's better to focus maybe on something that's smaller and they won't spend all their resources fighting you on, and then the market grows to the point along with you where all of a sudden, you're this billion-dollar company. You can't just do that. There's some amount of luck, for sure. We had very good timing in starting Weights & Biases. That's a really cool feeling and it's really cool to ride that trend. In doing that, what we've seen is deep learning really took off. I mean, it's applied in every vertical now. Every company has at least a few people who are building deep learning models now. Those teams are constantly growing and we see that in the way that our contracts grow with our customers. We sort of bet that that would happen, but to see it actually happen and to be able to ride that trend, there is no way to really feel what exponential growth is until you're in the middle of it, and that's what it feels like.
Chris:
I remember one moment early in the company that stands out. We had presented to OpenAI, to a group of researchers. I remember I was presenting and half the audience was just looking at their laptops. I left that meeting feeling like no one cares. Then a week later, Woj calls us up and he says, "Hey, we got this problem on the robotics team. Come check this out. Can you help us out?" We go, look, and I remember going there with Shawn. Both Shawn and I are looking at it and we're excited, right? Because we can solve the problem. They're telling us they have a problem and they want us to fix it. After that, Shawn and I, we pulled an all-nighter just cranking out the interface that they wanted and got it to them within a couple days. I remember thinking, "How precious is this relationship with OpenAI, this institution that I really, really admire?" Also, that same feeling that Shawn described of some users saying, "Hey, I have this problem," and we had the power to go back and actually fix that problem for them.
Lukas:
Do you remember the afternoon when they turned on Weights & Biases?
Shawn:
That was another all-nighter.
Chris:
Yeah, it turns out there was some performance problems with my Python backend if I'm recalling correctly.
Shawn:
Well, there were a couple things. We did not anticipate OpenAI's scale, because we're doing the thing that you do as a startup, which is you make an MVP. It doesn't really need to scale. But it turns out our very first customer was one of the largest-scale customers we could have. They were the first person who integrated Weights & Biases into this library that they had that everybody on their robotics team was using to run training code. As soon as they committed that to production, they started sending us a ton of traffic. The site just immediately went down because it's this cobbled together startup website. Actually, the first problem was there was some API limit that we hit on Google because of the way we were making a specific request. Chris might remember what it was. There was no resolution. You can't just call Google and get them to immediately change a limit for you. You actually have to wait for a support case to go through for a number of hours. Of course, now, maybe we're a big enough customer of Google that we could have some influence, but back then, we were just this little startup and we can't go, "But it's our first customer." That doesn't really sway anyone over there. I don't remember how we worked around on that. Do you remember? Did we just wait?
Chris:
We had very smartly designed the Python library to back off when things started failing, so I think the quota resolution got resolved within the retry timeout.
Shawn:
That was one problem. Then later in that evening, of course, we're pumped because we have all this data coming, and it's OpenAI, and it's our first customer. Some other problem cropped up. Again, you might remember this, but it was something where Chris and I, we were up until 5:00 AM that night and Chris was live patching our app engine code. It was Google App Engine at the time, which is this Python auto-scaling platform that's not used as much now. I remember, yeah, we came up with a plan and we live patched the thing. I was like, "Is this going to work?" It did and the traffic started coming in clean and we could see all the data. We were so proud to have our first customer. Then, of course, the next morning, we went to talk to OpenAI and they didn't even notice the hiccup. They were like, "Oh, yeah, cool. Thanks." But, I mean, really, it was working when we went to have that conversation and they started looking at the sort of charts that we had and started giving us that feedback. That's when we got into the feedback cycle. It's important. In the early days, if you have any customer at all and they have a problem, stay up all night and solve their problem. Even if they don't notice it, it's worth it for you to start getting that great customer feedback loop.

Staying positive when things look bleak

Lukas:
What were your darkest moments, specifically?
Chris:
Early on when it was me and Ari, that was the sales team. This was before the pandemic, so we would fly wherever we needed to fly to. Some might think, "Oh, you get to fly to Toronto? That's got to be great." It's not great. You fly and then you go to some hotel and then you go to a meeting where you're just trying to get people to engage with you and learn about the product. There were a couple months there very early on where I felt like, "We can't charge enough for the product, people don't see it as being valuable enough." It can be very demoralizing, especially when you're out there on the front lines of sales and trying to educate and teach people about these concepts that are literally being created as we're iterating on the product.
Shawn:
I think a dark moment for me is — we already talked about this — but when we were first trying to sell to GitHub, we had this user, Hamel, who we talked about earlier. He gave us very direct feedback about how our product sucked. He was totally right. I love building this product and even now, it's hard. I'll take it personally for sure. Even though I know that some of the decisions are bad or there's lots of things that could be improved. When somebody calls it out, it definitely hurts. But we really want that feedback. I'm happy to go through that rollercoaster of emotions to make a better product. Really, that feedback led Weights & Biases to the place it is today. You have to be willing to accept there is lots of bad things. We want to know what they are. We made those decisions. It's my fault. It's Chris' fault. It's probably Lukas' fault and Lavanya's fault to some extent. We can always improve stuff. You take those gut punches in stride and keep making it better.
Lukas:
I have this memory — I wonder if this is an accurate memory or you see it the same way — but I remember thinking of making experiment tracking and doing an offsite where we really built something super custom for TRI, and then going there and showing them a beta of the experiment tracking stuff that we built and having them basically tell us this isn't that interesting and that feeling bad. Then I have this memory of talking to you, Shawn, and I think you were like, "I don't think anyone will pay for experiment tracking." I was kind of thinking, "Yeah, you're probably right." Then I remember talking to you, I was like, "I just need to tell you that I need you to be more positive," which is so funny because I feel like actually you're almost always the optimist. I remember at least thinking to myself, "What I need to communicate is I just need you to be positive even if it's not rational to be positive here because I'm feeling a lot of doubt myself." Is that an accurate memory?
Shawn:
Yeah, yeah. I remember. I remember that. That was before we had that user, that first user who was actually using the thing, and we felt like they were getting value. We kept saying, "What if we build this other thing? Will that get us a user? What if we build this other thing?" We did that for a number of months. It was the early stages of the startup. That was disheartening because it's like, okay, if we could build this other feature tomorrow, but nobody's going to care. That was kind of the mindset I was getting into and you were rightfully calling me out on, well, this is the early stage of a startup, so let's make the next thing until we have that user. It was getting that first user who broke that off. From there, it was, "Hey, I just need this one little tweak. Great, we'll do it," and it was all positive. Maybe not all positive, but more positive from there.

Lukas' advice to new entrepreneurs

Chris:
All right, Luke. If there are other entrepreneurs listening to the podcast and wanting to build a startup that achieves a billion-dollar valuation, what advice as a startup CEO would you give them?
Lukas:
I feel like the advice probably depends on who that person is. Let's picture someone. Who are you thinking of here?
Chris:
All right. It's someone that looks a little like us, right? They're programmers. They're interested in starting a company, but maybe don't have a ton of experience on the business side of things, but they're passionate about the product they're creating.
Lukas:
There's so much advice out there that I think is really good these days. I feel like when we were all starting our companies the first time, being an entrepreneur wasn't a thing. Y Combinator's put out so much good stuff that's really...you forget how not obvious it is to people that you need to make something that people want, right? You can't emphasize that enough, right? I feel like now people kind of know that, which is fantastic. It definitely wasn't obvious to everyone when we were starting or maybe as obvious to us as it should have been. I think the thing that people don't talk about as much as I think they should or the advice that I feel like I can uniquely offer, because it's worked so well for me, is to pick a customer that you really love spending time with. I feel like a lot of these ML startups especially, they totally start from a technology and what's interesting to do with it. That's a bad idea. Everyone kind of knows that's a bad idea. Then they work backwards from a use case that they find interesting and that's maybe...it's an okay idea. The thing that gets lost is that, at least for me, the thing I do as CEO, the thing I have to do all day long is spending time with customers, spending time empathizing with customers, thinking about customers and bringing the customer voice into the company. Given that that's, I think, maybe the most important job as CEO, you should pick a customer that you really like, right? Because you're going to spend so much time with them over the entire arc of your company. Having a specific idea of who that is and making sure you like them, I think, is a really key thing. I remember at CrowdFlower, we tried to sell into different types of customers and so I really felt this. I went to CMO conferences. I contrast that for myself going to NeurIPS and just really enjoying making small talk, enjoying all the details, the things that people say. I also believe it's very powerful for the world and good for the world, but I think even more than that, just on a day-to-day motivation...the impact will sustain you over the long-term, but over the short-term, I think I really appreciate that I'm working with a user base that I really care about and enjoy talking to.

Hopes for the next five years

Shawn:
Lavanya's in the chat here. We're taking questions from the audience.
Lukas:
What's the hope for Weights & Biases in the next five years? I feel like almost it's like a jinx to say that question. I don't even know if I have a good answer. Maybe if you guys want to try first?
Shawn:
Of course, I take it from a product and tools standpoint. I hope that we can build interconnected tools across the ML pipeline that really work well together because they share these common underlying threads or infrastructural pieces or, dare I say, bones? Which is what I like to call them internally and everybody makes fun of me for. To me, it's really important that the data that you collect about your model in production can be used to inform decisions that you make back in the data collection process and the training process. There's so many parts of the ML pipeline, it's hard to build all this stuff. But if you think of the best companies, somebody like Google who's building ML, they've built all this. They've verticalized all of it internally and built it themselves. They've built tools out of other tools. I want to be able to make a platform like that outside of a giant company and give it to the rest of the world and use all the use cases that we encountered to make it better and better, and more general.
Lukas:
I think it'd be really satisfying if Weights & Biases becomes a core part of every ML team's infrastructure and we're really known for making really high quality stuff, really useful stuff, really powerful stuff. I think we're on that trajectory, but I think ML is growing so much that that becomes every company when every company has an ML team, which it seems like we're headed to in the next five years. So I think that's the biggest thing.
Shawn:
If you imagine the company that is in that position, what does it look like internally, I think we have this today. But, I mean, there's folks building data tools and ML tools at lots of companies in the world and doing a great job. I want to get all those people in the same place. People who love building these tools. Maybe there's folks out there who work on these tools and don't really love it and want to switch to something else. That's great too. I mean, you should move through things in your career. But I would love to be surrounded by people who really love this problem and really love the people who work on this problem, these customers, just all together, all of us. Maybe not in an office in the modern world anymore, but distributed around working on these problems together and just building great stuff that users love so they can build ML models that make the world better.

Making a paintbot & model understanding

Lukas:
Here on Gradient Dissent, since you guys are both mega fans, you know this, but for new listeners, we always end with two questions. The penultimate question is what is the most, or an underrated topic in machine learning? Something that you would love to work on if you weren't working on Weights & Biases.
Chris:
l want to make a painting robot. But it uses an actual brush, okay? Not just a plotter or something. It's going to be very complicated. That's not the big world-changing answer maybe you were looking for, but that's what I would be really stoked to pour a year of my life into I think.
Lukas:
Like a paintbot?
Chris:
Yeah, a paintbot. That's right.
Lukas:
Cool, cool.
Shawn:
Underrated. Underrated topic. Maybe we should have started that robot company... This question's funny because we touch so many of the problems in machine learning at Weights & Biases. Not all of them, but we're building tools that are used for all the verticals, so of course we're going to touch them. So maybe I'll say something that we're not explicitly doing, that I think is really important, and I feel like I am actually doing, which is...I think model understanding is critical in the future. Deep learning models are really tricky to understand. It's a research area. It's really dependent on what kind of model you're building, what techniques you might use to figure out "Why did my car make the decision that it did at a specific point in time?" I think as these models get more and more complex, it's more and more important. I want to understand the models. I want to understand the models for the world to be good and I want to understand the models because I think that gives us some understanding into the nature of intelligence and our own decision-making processes. We're not explicitly doing model understanding at Weights & Biases, but we're trying to build tools, and we'll talk more about this over the next six months, I guess, that head in that direction.
Lukas:
I'm tempted to answer this question, but I feel like maybe it's better as a host if I remain mysterious in terms of what I think is the most underrated topic. I will say I love the company that we started and I would not change it. Obviously it was a really good choice to do it. But one thing that we kicked around in the early days... Well, we kicked around two ideas. One I think is a terrible idea, which is a painting drone, which I think would be fun, but probably just terrible idea. But still, it would be really fun. The second idea which I think I would still love to do it, and I think we thought we were the wrong team to do it, but actually I think we might have been a good team to do it, but the timing seems like it might have been bad, which is to build a better simulator to help robot companies simulate what they're doing and then deploy into the real world. I just think that that still kind of needs to exist. There's different takes on it, but it doesn't seem like it's been nailed at all, especially with the physical part of it. I also think that would be a really fun company to do, although a much slower ride. But my answer to the most underrated topic in ML is still a secret.

Biggest bottlenecks in deployment

Lukas:
The final question is when you look at actually all of our customers trying to deploy models successfully, what do you think their biggest pain point is? We always ask guests this who are mostly at these companies trying to do it, but looking out at everyone — maybe we restrict this to people already using Weights & Biases because to people not using somebody, obviously maybe that's their biggest problem — but the people that are already using us, what is the big thing that they run into?
Shawn:
I mean, there is a cop-out answer, which is it's probably hiring.
Lukas:
That might be accurate. Hiring? Yeah. At some point, we should collect some stats on what people say and maybe that will just answer it based on...at least, people we interview. But yeah, what are you guys seeing?
Shawn:
It really varies in our customers because somebody who is building a self-driving car, for example, is they're building 100 models in parallel and with no proof that self-driving cars could still actually exist and work on public streets. I guess, we're getting very close now. As opposed to somebody maybe who's got a bunch of financial data and needs to predict credit scores. In the credit score problem, you actually...model understanding, what I was just talking about, really, really important. You probably need to use something dumber than a deep learning model so that you can actually say why you made a particular credit prediction. Our customers are extremely varied. I hope we're solving a lot of the problems in the model creation side of things. I think that there is a really hard problem of figuring out what models are doing in production and then taking the data from production and integrating it back into the model training process. I hope that we get to work on that problem too, but I bet that a lot of our customers would express that they have challenges there.
Chris:
Yeah, I'll piggyback on Shawn's response and say specifically CI and CD, when it comes to these ML pipelines, just is nowhere close to what we have in the regular software development world. I know we have a lot of exciting things on our roadmap to help with automating all of these steps as a model moves through the pipeline and then comes back to get retrained and understanding how it's performing in production, but I personally look forward to the day that all of this can be automated in a way that doesn't involve people manually running shell scripts, which is often the case today and really unfortunate.

Outro

Lukas:
Awesome. Well, thanks so much, guys. It's been a real pleasure working with you and can't wait for many more years.
Shawn:
You guys too.
Chris:
Can't wait for more podcasts, Luke. You're a fantastic podcast host.
Lukas:
Yeah, we got to bring you back a year from now, see where we're at.
Lukas:
If you're enjoying these interviews and you want to learn more, please click on the link to the show notes in the description where you can find links to all the papers that are mentioned, supplemental material, and a transcription that we worked really hard to produce. So check it out.

Bonus: Under- vs overrated technologies

Lukas:
I thought it'd be fun, since it's a friendly guest obviously, to add some zany new features to this podcast. I thought one section would be..I'm going to name a technology and then you got to immediately say underrated or overrated, and then if you disagree, we can fight it out. What do you think?
Shawn:
Let's go.
Lukas:
Are you ready? Okay. Reinforcement learning.
Shawn:
ls there a middle ground? Wait, is it only under or over?
Lukas:
Yeah, you got to decide.
Shawn:
I love reinforcement learning. Underrated.
Chris:
Yeah, I'll go with Shawn, underrated.
Lukas:
All right, all right. AutoML.
Chris:
Overrated.
Shawn:
Yeah.
Chris:
Come on, Shawn. It's bad for business.
Shawn:
Underrated.
Lukas:
I already forgot what you said, Chris. You think it's overrated?
Chris:
Yeah, yeah.
Lukas:
Wait, why?
Chris:
Because I think it's really important that the ML practitioner is using their own creative powers to make choices about how the model is architected. It's like automation everywhere else. I have mixed feelings about it.
Lukas:
Shawn, you want a quick rebuttal to that?
Shawn:
Yeah, I mean, I'm torn on this one. There is two good answers, but I do think...of course, the technology to train models should evolve over time and things should get smarter. For example, our Sweeps tool does help you find good parameters for a model. I think as a company, what we need to do is as those tools get better and better at automatically building models, there's all kinds of other problems around model building, like getting the right data in the first place, and we need to build those. This space is moving so quickly, and of course AutoML will probably continue to improve, but there's lots of other problems around it to be solved for the practitioners.
Lukas:
All right. Well, I think people should leave comments for which they found more convincing, but definitely bonus points for the Weights & Biases plug in there. I like it. Okay, next one. The singularity.
Chris:
Underrated.
Shawn:
Yup. I'm with Chris. Underrated.
Lukas:
Whoa, whoa. Bold. All right. We'll move on. Okay, ready? Bigtable.
Chris:
Overrated.
Shawn:
I'll go with underrated. Sure.
Lukas:
Whoa, all right. Shawn, maybe you go first.
Shawn:
I think Bigtable...I was at Google when Bigtable was starting to be used. At the time, it's like 2006, Bigtable was really big inside of Google. It enabled all of the technology that we have. The search, and everything else that people were building, Gmail. It didn't exist elsewhere in the world. When I left Google, I saw the world starting to copy Bigtable and became things like NoSQL. It had a huge impact on the world. I think maybe today, raw Bigtable itself, maybe this is what Chris is probably alluding to, is a bit of a challenge. Maybe I'll let Chris take it from there.
Chris:
The reason I said it, because I remember a day early in the founding of this company, when we were trying to figure out where to store our metrics, and you were like, "Bigtable will solve all of our problems. It's perfect for this." It's kind of been a thorn in our side a little bit. I mean, it's done its job well, but you could ask a handful of engineers at Weights & Biases and I bet you most of them would gripe about Bigtable. I think our use case is a little funky for the way we're using it.
Shawn:
Yeah. For large time series where you want to fetch a few million contiguous points at a time, especially in a shared Bigtable cluster that you get from Google, there's some challenges there.
Lukas:
Okay. Python.
Shawn:
Underrated.
Chris:
Yeah, underrated.
Lukas:
Okay. Jupyter.
Shawn:
Underrated.
Chris:
Underrated.
Lukas:
JupyterHub.
Chris:
Underrated.
Shawn:
Underrated. Great.
Lukas:
Kubeflow.
Shawn:
Underrated.
Chris:
Underrated.
Lukas:
All right. All right. SageMaker.
Chris:
Overrated.
Shawn:
Overrated.
Lukas:
Interesting. TensorFlow.
Shawn:
Underrated.
Chris:
Underrated.
Lukas:
Wow, you guys are super aligned. Hard to get some dissent here.
Shawn:
Yeah, we got to live up to your podcast name.
Lukas:
Yeah, yeah. I guess we'll get some other pairs to guess or maybe we could ask other ones the same questions and then see what they say.
Shawn:
I mean, do you differ from us in any of those, Luke?
Lukas:
I mean, some of these technologies, I'm even kind of hazy on what they are. Like I was thinking BigQuery, Bigtable. I was hoping I would learn what the difference is to be honest.