Cade Metz, author of "Genius Makers", on the rise of AI

How Cade got access to the stories behind some of the biggest advancements in AI, and the dynamic playing out between leaders at companies like Google, Microsoft, and Facebook. .
Angelica Pan

Listen on these platforms

Apple Podcasts Spotify Google YouTube Soundcloud
https://soundcloud.com/wandb/cade-metz

Guest bio

Cade Metz is a New York Times reporter covering artificial intelligence, driverless cars, robotics, virtual reality, and other emerging areas. Previously, he was a senior staff writer with Wired magazine and the U.S. editor of The Register, one of Britain’s leading science and technology news sites. His first book, "Genius Makers", tells the stories of the pioneers behind AI.

Connect with Cade

Show Notes

Topics covered

0:00 Sneak peek, intro
3:25 Who is "Genius Makers" for and about?
7:18 *Spoiler alert!* Artificial General Intelligence (AGI)
11:01 How the story continues after the book ends
17:31 Overinflated claims in AGI
23:12 Deep Mind, OpenAI, and AGI
29:02 Outsider perspectives
34:35 Early adopters of ML
38:34 Who gets credit for what?
42:45 Dealing with bias
46:38 Aligning technology with need

Transcript

Note: Transcriptions are provided by a third-party service, and may contain some inaccuracies. Please submit any corrections to angelica@wandb.com. Thank you!
Cade:
AI is a weird field, right? It's this combination of various fields and it's always been like this, right? Since the '50s when the term was coined, it's this blend of Computer Science and Neuroscience and Psychology. That has always been the case and it continues to be the case.
Lukas:
You're listening to Gradient Dissent, a show about machine learning in the real world. And I'm your host, Lukas Biewald.
Lukas:
Cade Metz is a journalist who's been covering technology for the past few decades and he recently wrote a book Genius Makers, which is kind of a historical document up until the present, about artificial intelligence and the people that built the technology behind it. I have so many questions about this book, I can't wait to talk to him.
So, you were the first non-ML practitioner to ever appear on this podcast, so I'm excited to do this. And we might take things in a different direction than normal. But I was really excited to talk to you. I kind of procrastinated on reading your book, and then I actually really enjoyed it. I was kind of afraid that I wouldn't. The name made me a little worried that it might be a bit over the top or something like that. And I also felt like, typically, when you read journalism on topics you know really well, it's hard not to be critical or feel like the person didn't get something exactly right. But then I actually, it kind of reminded me of that show Silicon Valley just in its incredibly accurate details. I feel like I've been in this world of machine learning and I've been in a world of venture capital, which are kind of the two main topics that your book covers. And just all the little anecdotes and details, they really just rang true to me. I felt like you do these things where you explain math, you explain sort of when someone makes fun of somebody for differentiation by faith, what that means or you describe what a TPU does. And you really actually go into technical detail that I'm not even sure, I would necessarily do if I was writing for a mass audience. And I actually think you are remarkably accurate in that. And then you sort of describe these very vivid scenes that just seem, sometimes I feel when you read sort of descriptions after the fact of a scene of getting an acquisition or a fundraiser something, it's like, "I don't think this journalist really is giving us accurate information or transcribe it in a way that just doesn't feel right sometimes." But your book really felt accurate to me. And it was a really interesting lens for me just on a world that I've been sort of adjacent to. Some of the folks have been on our podcast, some of them are customers of ours now, so I know a lot of the characters in your book, but I don't kind of get to know them intimately in the way that you clearly got to know them. So I actually, the question that's kind of dying to ask you, which has really maybe nothing to do with ML is just how did you get so much access and what was your process for researching this book? Because there are some details, I mean, I surprised you got someone to tell you. And it's not you're sort of recanting interviews that you did. It's like somehow you just, it seems you must have actually sat down with Geoff Hinton for a significant amount of time to be able to write this or maybe of some other process that I don't understand.
Cade:
No. I mean, well, I will tell you, but it's really interesting for me, first of all, to hear what you thought you might get, and then also what you might have gotten in the end after reading it. In a way, there was a there was a time when I started working on this book. And really got into it when I realized it was a really dumb idea, because on one sense, your audience hopefully is going to be machine learning professionals and researchers who are really steeped in this stuff. And if you venture too far outside that world, you're going to get them angry with you, and you're going to lose them. But ultimately, the goal of the book should be to have any reader pick this up and enjoy it. And that should be the goal as well. And if you move too far towards the machine learning researcher, you're going to get those people angry and they're not going to take up your book. And the trick becomes to find the sweet spot right in the middle. And that's very difficult. And then on top of that, within the machine learning community, we act like that's a monolithic thing. It's actually this huge spectrum as well. And this is what I get to at the end of the book really is that, you have some people who believe this is just math, and you have other people who, where, "This is something more," right? It's almost like a religion and this is going to create AGI. This is going to create a machine that can do anything the human brain can do. So within the ML community, you had this spectrum where people really, really disagree and the goal and somehow, get all those people interested in your book, it seems a mistake to even try that. But here is what I really believe in and this sort of gets back to your question. Ultimately, this is a book about people. Right? This is a book about some really interesting people. And they are interesting in incredibly different ways. From Geoff Hinton to Demis Hassabis at DeepMind to Jeff Dean, I can go on down the list. Timnit Gebru, who was in the news recently, because she really clashed with people at Google, including Jeff Dean. These are really interesting people, who are religion in a lot of different ways. And ultimately, that's what this book is, right? It's a book about the people. And what I realized is that I if I can just show who these people are and who they are, what their stories are, and how all those stories fit together, then that's what makes it successful. Right? And what that's about, finding what those people are about, ultimately, it's about spending time with them, as you indicated, right? And that takes a lot of doing, right? Some people, because they work for these giant companies, you can't really get at them initially, so you try somebody else and you get some good stories from them. And you go back to the first person, you say, "Hey, I've got this. What else can you tell me?" And you develop in some ways a relationship with them. I tend to, even as you get close to these people, keep a little bit of a distance as a journalist. I think that's important, too because again, you've got to have an objective view and be able to really appreciate and rope in the beliefs and the experiences and the points of view of all these different people. But it's about years and years and years of gathering information and understanding it yourself. And taking it back to people and say, "Can we talk about this more?" And then somewhere along the way, you get them to talk.
Lukas:
Well, I can only speak for myself, but I thought it's a really interesting book. I mean, just, I couldn't put it down once I started.
Cade:
Good to hear.
Lukas:
One thing I was wondering about is the ending I thought was very understated. It sort of ends with Geoff Hinton, who's kind of the main... I mean, I feel like he's almost the main character in your book. Kind of saying, like, "Well, maybe AGI isn't that important." Have a spoiler alert, I guess.
Cade:
Well-
Lukas:
And I thought it doesn't even like, he's kind of like, "Well, would you really want a vacuum cleaner that's..." This is my takeaway. I'm curious if I get it wrong or different than your intention, tell me. But I was thinking, does he really like a vacuum cleaner that could navigate my house and be smart about when to turn on and off and stuff. I mean, it doesn't really exist. And I actually, I think I would want my vacuum cleaner, I think to be reasonably smart. And, bordering on, if it could reason about the world that seems like it could actually be kind of better than a Roomba that, can't yet. I was kind of surprised to Geoff Hinton thinks that and it sort of felt different than what most of the people in your book, were thinking. I was just kind of curious. And then you actually say, you sort of say, but, he kind of invested in some crazy reinforcement learning company. So, maybe he doesn't even really think that. And I'm kind of just left with I wonder like what's the takeaway here? And I was also wondering, what's kind of your takeaway? Because you really noticeably never seem to take a position on this stuff. But I mean, you've been watching the field for decades. I'm sure you have opinions on this.
Cade:
Right. Well, it's interesting that you would, in some ways, have that takeaway, that the ending is understated and sort of questioning AGI almost. My first book review has come out. There's this trade publication called Quercus, which reviews big books. And their takeaway is completely the opposite. If you read the review, it is "Cade Metz is making the case that AGI is possible, right?" And so, it is the completely other end of the spectrum in you and I completely understand why the two of you have come to different conclusions and it makes me happy because that's my aim. My aim is not to judge. My aim is not to make a call. My aim is to show what is going on in the world and what has gone on over the past 10 years. And you have someone Geoff Hinton, who is one of the most respected people in the field. At the same time, there are people who can't stand him because they feel like he has gone too far. There are people who can't stand him because he doesn't go far enough and doesn't say that AGI is around the corner. And you're right, the book ends with him, in a way questioning AGI, but it also ends with him changing? Right? Him embracing some stuff, namely reinforcement learning that he hadn't embraced in the past. And he sees the value there, and he sees it accelerating. And in a way, he's just not going as far as some other people And, I think the book ends with some people who take a very different view and who do think AGI is around the corner. And it states them very explicitly, right? And I think it's about showing where all these people are coming from and letting the reader make their own decision about what's really going to happen.
Lukas:
It's kind of interesting. I mean, one of the things that struck me about your book is you're sort of describing in a historical, like a story and feeling way, something that's like completely in progress. There's a whole bunch of things that happen right after your book stops, right? Timnit and Jeff Dean and all the stuff that happened at Google. And then I was actually thinking it's funny, like Yoshua, I think in your book, he's not really doing a lot of commercial stuff in contrast to some of the other characters. But then I think Element AI, recently kind of sold and that was like a bit of a controversy if that was a good outcome or a bad outcome. And then OpenAI went and became a private company, did you have a sense of like the book needs to stop here or were there other stuff that you kind of wanted to include?
Cade:
Well-
Lukas:
Could there be a sequence to this?
Cade:
There could, but what's interesting about all this stuff you mentioned? And I would argue, almost everything in the book, it is completely in tune with what happens in the book, right? Now, Timnit Gebru is a character in the book and the same things happen in the book, it's just with a different company, right? It's with Amazon. Okay? It's not with Google, then it happens with Google. Yoshua Bengio, he has stayed outside, or more outside of the commercial realm than Hinton and LeCun. But as the book goes into, he dips his toes, certainly, right? In the book, it's his partnership with Microsoft. He's also had one with IBM, and with all these people, it's sort of a balancing act. And I think that's what the book is about is that you have these very idealistic people, whether it's to me, or it's Yoshua, or Hinton. And they all come into contact with these forces that are frankly, much bigger than them. These corporate forces, these government forces, and when that happens, there's going to be conflict. And all those conflicts that have come since I finished the book, it's all happening in the book as well. And OpenAI, how many times have they gone back and forth as far as what they're going to do and what they're not going to do? Are they a not for profit? Are they a public company? Do they believe in withholding the technology or sharing it? Right? These things will continue to go back and forth, but the constant is, is that clash of belief. And then, those corporate forces, which are about money and about attention and promotion. I think that those are the constants. And that's why I really believe in the book is because all those things are just going to continue to play out in the years to come.
Lukas:
I mean, one thing that I really was curious to ask you about, is you kind of set up these kind of dichotomies that you personify, right? Sort of Gary Marcus versus Yann LeCun maybe or like Elon Musk versus Zuckerberg. You can probably what those are. People listening could probably guess what the dichotomies are here.
Cade:
Sure. Sure.
Lukas:
I was curious, where do you land on this stuff, now that you've kind of talked to everybody? Do you feel like for example, like do you feel we are sort of overstating the future progress of AI? It sort of seems like if you take historical view like you're taking, it seems to me like ML has just kind of made this sort of steady incremental progress and people keep moving the goalposts of what it means to do AGI. First you have to win at chess, and then you have to win at Go, and then it's a gift past the Turing test, but then that's even hard enough. And so, for me, when I take a historical view, I sort of imagined steady progress extending out into the future. Then for me, when I look at these algorithms, it sure seems a stretch that they turn into kind of AGI, just with more compute. So, I actually don't even know where I land, but I'm curious where you land on this topic.
Cade:
Well, I think you're right. You have to look at this historically and that's what the book does is that a lot of the claims that are being made now about AGI and this sort of pervading our lives and sort of taking away jobs, all that has been around since the '50s, right? And I showed that in the book and in a way, it's just a repeat of that. Now, that said, there has been a huge amount of progress over the past 10 years, which is what the book really covers, right? We've had a huge amount of progress. What I really believe firmly in as a journalist, particularly as a New York Times reporter, is I feel like, what has happened and what is possible in the public consciousness is way out of whack, right? And a lot of that just has to do with the term Artificial Intelligence, which has been thrown around so much over the past 10 years. That alone gives people a false impression about what is happening and what will happen. And then, frankly, most people writing about this stuff, they for whatever reason, they don't really understand what's going on and they exaggerate, and maybe they exaggerate, consciously, maybe they exaggerate unconsciously, maybe they don't know that they're exaggerating. But if you sit down and you read most of the stuff that's written, you have a false impression. And that is one thing that I really want to, at least in my small corner of the universe, try to correct and show people what is really happening, right? And the fact of the matter is none of us knows what the future is and as much as someone who really believes in AGI might get on this call with us. And get angry at me for not saying AGI is around the corner. The reality and I think the book shows this is that none of us know what the future holds. And when it comes to AGI, it's an argument. It's a religious argument, right? And I show that in the book. People with the same experience, the same knowledge, the same respect across the industry really disagree on this, but... go ahead.
Lukas:
No, no. It's funny. I guess one thing that sort of, it's almost in the water, so I don't think to question it because I kind of swim in it, right? Is why do you think it becomes such a religious argument? Why do you think people feel so passionately frustrated that other people don't agree with them on this particular topic of like is AGI possible or coming or coming soon?
Cade:
Well, I think that the people are just coming from a very different place when they start talking about these things. And one of the things you realize about Silicon Valley is if you're going to be successful, you've got to really believe in what you're doing, right? That's again, either consciously or unconsciously, that's how you attract the money, that's how you attract the talent. That's how you get these things to snowball. Okay? Whether you're building a tiny little app that does something simple or you're trying to build AGI. So, what has happened is that people have taken a rulebook that has worked in Silicon Valley for certain things, let's say Facebook, right? And they're applying it to this notion that they could build a machine that can do anything the human brain can do. So, in their mind, they're just doing what everybody else has done. Right? But AGI is different than Facebook, right? That is a goal that is far, far bigger. And so in their world, they're just doing whatever went around them is doing, has done for the past, however many decades in Silicon Valley. But for someone else, they're taking a huge step, and they just do not see that, right? How can you extrapolate from a machine that can play Go to a machine that can do any anything human brain can do? And if you ask people to describe to you how that's going to happen, right? That's at the very least, that's hard for them to do, describe how that's going to happen, right? The path that they see is a path that they painted very broad strokes. And saying I can build a Facebook, there's a path there to building a social networking app. We know how to do that. We don't know how to do this. And we don't know how to build a self-driving car, right? That alone is an astronomically difficult project that we don't quite know how to complete yet. But people still talk about it in terms like it's already there. And on one level, you see why they do that. But on another level, right? It misleads the public, it misleads people about what's going to happen soon.
Lukas:
So, I guess, I sense that one opinion that you kind of hold is that there are a lot of overinflated claims and therefore, the public does not even have a good sense of what's possible and not possible.
Cade:
That at the very least is true, right? Who knows tomorrow, we may have a new technology that really blows us out of the water. But what we've seen over the past 10 years with this are repeated over inflated claims, just in the sense of they don't get. Think about your mother, right? Or my mother, when they read stories, even in the New York Times over the past few years, where they assume that tomorrow we're going to have cars that can drive by themselves all over the place, right? They can't help but have that assumption, because that's the way it's written about. And journalists write about that way because people Elon Musk and so many others, just say it's around the corner, and they take them at face value, right? So, I think that's really where the problem is that you're misleading the general public. And I do think that that's a real problem, right? At a time when our society is grappling with what is true and what is not, let's make more of an effort to say what is actually possible now and show people what the reality is now. And try to do that in a way that's separate from what might come, right? The reality is now is that self-driving cars aren't up to the task, right?
Lukas:
Well, it's kind of interesting you say that, because, well, I wonder if maybe journalists are at fault then because like, certainly Elan Musk has a pattern of overstated claims, but I think he might be a little bit of an outlier. I mean, you would know better than me, but I feel when I talk to ML researchers, they tend to be fairly understated or almost like well, maybe a little too reticent in their claims and maybe the ones that rise to the top aren't like that. But we've done 30, 40 interviews on this here and I always feel like I'm trying to push people to extrapolate what you're doing, like it seems like big deal. I don't know. When you talk to Geoff Hinton, or actually, let's go way back, right? In your book, you talk about Rosenblatt, and then the New York Times, I think or yeah. It seems like a lot of journalists kind of write about what he's doing, saying it's going to get consciousness soon when he's basically doing like a perception without even a second layer.
Cade:
Exactly.
Lukas:
So, what happened there? Do you think Rosenblatt has a responsibility to communicate better? What's going on? Was he making overinflated claims at that time?
Cade:
Well, he clearly was, right? I mean, he's telling these reporters that we're going to have systems that could walk and talk and recreate themselves and somehow venture into space. Right? And so, the reporter is just going to report that.
Lukas:
Right, right. Okay.
Cade:
And in a lot of ways, it's not that different now. And you talk about Elon Musk be in an outlier, that's true and it's not. Again, you talk about ML researchers that is not a monolithic group. I guess the other thing I want to show people is that even the New York Times has written stories, "AI experts say X," right? Well, AI experts, that's not one group, it's this like spectrum of people. And you got to remember, like DeepMind and OpenAI are founded on the notion that they are going to build AGI and there are people at those companies who really, really believe that. And they're at the top of those companies and they may not be as cavalier as Elon Musk, they may not have the megaphone that he has, but they really believe that. And those are important companies, right? They have a lot of serious research talent, particularly DeepMind has had some really important breakthroughs. Just recently, the CASP contest breakthrough. That's really important research that in some ways it's separate from this notion, they're going after AGI. So, these are important, important labs that are founded on this belief, right? And I've known Demis Hassabis, the co-founder of DeepMind, for a long time now and whatever you think about that belief of AGI, you got to take that guy seriously, right? He has a track record. He is a serious, serious person, and you may have a problem with a lot of the stuff he has done or said, but you have to listen to him, right?
Lukas:
And I mean, similarly, the work coming out of OpenAI, it'd be hard to argue it's not super impressive. So, I feel like some people claim that, said there's a little bit of publicity stunt. But you talk about the robotic hand, manipulating a Rubik's cube and that's really impressive. And maybe the Rubik's Cube makes it more fun, but I still think it's an amazing breakthrough in robotics that-
Cade:
I agree. I completely agree. It's both, right? It is super impressive science on the one hand and it's a stunt. It's both and me as a New York Times reporter, as a book author, my job is to show you that it is both, right? And give a really real sense of what's going on there. It's very easy to see the hand, right? This five-fingered robotic hand solve a Rubik's cube and think, "AGI is going to happen tomorrow, right?" If you're not educated in the field, it is super easy to think that. And so, my job is to say, there is an advance here, right? And you can see it, but there are some chinks in the armor. And the other thing I've seen is that not even everyone at OpenAI is aware of the chinks in the armor, right? And that that hand, while the result is super impressive, there are some caveats there that show you that even the science isn't quite where you think it might be, let alone sort of the stunting nature of it. Right? My point, over and over again, is that these things are complicated.
Lukas:
I guess, maybe this is inserting myself into your story, but I was kind of there throughout it and I couldn't help, but I keep having this thought. I was at the Stanford AI Lab in 2003, 2004, at the sort of like nadir of interest in neural nets and you talked about this in your book. And I felt the Zeitgeist, there was kind of like these neural nets are kind of, the name is too good, we used to support vector machines, not neural nets. That's not serious and it's like these people are just trying to like hype these things and yeah, they sort of work. But they're like, kind of tweaked to the point where they're overfitting and serious people wouldn't make a system called a neural net. And it's been kind of interesting to watch. It turned out that the neural net strategy actually really works. The perceptron of it is like the base thing that that now is used everywhere. And so, I actually kind of feel like maybe the folks I was working with at that time, weren't dreaming enough. I think it's great that engineering kind of when you saw it working really, invested into. But I remember, like you talked about some stories about the skepticism of the progress of neural nets. And I vividly remember that. It's just like everyone says, they have a better algorithm, especially in neural nets, but then they were right. And I kind of wonder if you feel like there's any lessons to that, because it seems so remarkable that something would get all this attention and then sort of like be thought of as bad and then kind of come back is the working technology. I wonder if there's other technologies out there that follow that same path?
Cade:
I think, I mean, it's an incredible story, right? I mean, it's amazing that some people kept worrying on that this stuff.
Lukas:
Amazing, right?
Cade:
And that, again, is at the heart of this book. And it's something that I have always really been amazed by and impressed by, is someone who keeps working on something, even in the face of everyone telling them it's not going to work, right? That is the basis for any good story and that certainly happened here. And it will keep happening. And in fact, in some ways, you've already come full circle, where you have this sort of the let's call them the Gary Marcus crowd, who were saying the same things like, "Neural nets don't do everything these guys say they're going to do. They're limited." And so in a way, they're still fighting the same battle, right? But you're right, there are other technologies that will come along, have already come along that people are skeptical of that, that are going to work in the face of that. And it takes that, right? It takes that belief and that determination and just sort of years and years of hard work to make this stuff do what it's ultimately going to do.
Lukas:
It seems a lot of the characters in your book, I was kind of struck by, I don't have a good stat and I could be wrong, but it seemed a lot of them didn't come from a Computer Science background. It seemed like a remarkable number of them kind of came from Biology and Neuroscience and things like that. Do you have any thoughts on that?
Cade:
I agree. And that's another thing that I'm fascinated by is AI is a weird field, right? It's this combination of various fields and it's always been this right? Since the '50s when the term was coined, it's this blend of Computer Science and Neuroscience and Psychology that has always been the case and it continues to be the case. And this is embodied by again, my main character, Geoff Hinton, right? He is someone who didn't come at this from the Computer Science angle and he's still, like one of the running things in the book is that he loved to downplay his skills as both a computer scientist and a mathematician and he doesn't think of himself as either. He comes at it from that direction and sort of gives this what is really just math, a perspective that you wouldn't necessarily expected to have. And that bothered some people and some people don't understand that perspective that he gives it, but that is how he thinks. And it has a real influence not only on how this field has progressed, but it does have an influence on how people perceive it, right? People don't understand when he and others, as much as they explain it and re-explain it, they don't understand them calling a neural network, a facsimile of the human brain, they don't understand that's just a metaphor in some ways, right? But that's part of the way this field works.
Lukas:
Well, I guess from a historical lens, maybe the takeaway is that being an outsider is an advantage in some ways.
Cade:
Absolutely. Absolutely. And that's sort of the story of Silicon Valley as well, right? But that doesn't mean that just because you're an outsider, that you're going to be right. Not every outsider is right. Some are and some aren't. And I think that's the story of this book, as well.
Lukas:
Probably everyone else is going to ask you this question, but I felt like I had to ask you, do you have any kind of like fun stories that you couldn't fit into the book because they didn't quite fit or any good anecdotes in all the research you were doing?
Cade:
That's a good question. Let me think that over. Most of it's in there to tell you the truth. I mean, like all the good stuff, some of it is just unbelievable. And it took a long time to get and once you have it from one person, you got to get it from another. So, there were a lot of things, right? Including the lead story in the book, in the prologue that I wasn't sure I was going to be able to get in there and thank goodness, I did.
Lukas:
Like with the auction of the company DNN?
Cade:
Exactly, and in particular, the price, right? That was one of the hardest, hardest facts to nail down and the whole process. Yeah?
Lukas:
Yeah, I have to tell you that's the only anecdote in the book, I don't totally believe. It was the one where it's just and maybe it's because it's actually true. It just feels unbelievable, but-
Cade:
It is 100% true and-
Lukas:
Including, to the part that I felt like it might have felt that way to the people involved. But it's hard to believe it actually happened, I guess, is they literally got Google and Baidu to bid at a particular time. They're running a Sotherby auction or something. Are you sure that's true? That's amazing.
Cade:
No, but it's true because I've talked to, I can't tell you the number of people I talked to who were involved in that, directly involved in that. It's absolutely true, and it's-
Lukas:
I guess that's how it goes, right? The thing that's really true is actually unbelievable, right?
Cade:
Exactly. And but like so many parts of that story are amazingly, improbably true, because it encapsulates everything, right? At the very beginning of this movement, let's call it a movement. What is it? The very beginning of this explosion in AI hype, in neural networks starting to work, all the players there, who would be involved are already there, right? From China and Baidu, to Google, to Microsoft to DeepMind is there, right? They're all there in this competition that would play out over the next 10 years. And that whole story came to me in bits and pieces over the course of, it was really months or maybe even years. And as each piece pops into place, you're saying, this sounds too perfect to be true, but you know it's true, because it's coming from multiple people.
Lukas:
As in.
Cade:
And it's verified by multiple people and all the perspectives kind of come together. And some people say, "Well, I won't tell you that." And then you get it from somebody else, and they say, "Okay, yes, it's true," right? That's what's most fun about being a journalist is when you get those nuggets that just show you so much about human nature. And also, just help your story just fit together in ways you never expected. I never expected the book to begin with that, but it had to begin with that because it's just the greatest story.
Lukas:
It's a good start and you go back to it a lot. And yeah, it is a great story. I guess one more just thought that I had reading your book is I hadn't quite had the timeline in my head of when neural nets start taking off, but I feel one thing that's kind of impressive is I feel Elon Musk and Zuckerberg and Larry Page, I feel like they've noticed that neural nets are working really well before most academics even noticed it. I feel like, I was thinking about the timeline, I think about when, I'm not in ML. I'm selling to ML companies for the last 15 years, and I feel like actually they were really early. How did they figure this out?
Cade:
It's remarkable, isn't it? And I think one of the things you can do is contrast the way they reacted. And you can criticize the way they reacted. You can say they went too far, of course, but contrast the way Google and Facebook reacted to the way Microsoft reacted, right? And Microsoft did not jump on it the way that those two other companies did. They didn't see it the way that the leaders of those companies did. Part of the narrative there in my book is that Geoff Hinton was in Microsoft's Lab doing this stuff with speech. And it worked in a way that nobody thought it would work. Nobody in the ML community, nobody at Microsoft, and it works and they're all shocked. They're all blown away, but they don't jump on it the way that Google and Facebook did. That's really, really interesting. And you do wonder, "Is it about the age of the company? Is it about the general area that the company plays in?" Google had a real need for that speech recognition system that Hinton and his students built in a way that Microsoft didn't because it had Android. It had a place to put it. Now, it was also a company that, and this is talking in broad strokes that would take new technologies and put them into play far faster than Microsoft would, especially in those days, right? That's part of it. But in the end, it's a combination of these things, right? It's the way the leaders think. It's the way the company is built, which in some ways is a reflection of the leader. It's about the age of the companies, right? Once these companies get to be a certain size like Microsoft, it becomes harder to jump on, on something. But you see in the book the way that Google jumped on it, and it's astonishing, right? There's a conversation between Larry Page and Alan Eustace, where he says, "You got to bet big on this." And this is, you're right, this is before even the ML community at large really understood what was going on. And Larry Page is telling Alan Eustace to basically bet the farm on it. It's astonishing. It really is.
Lukas:
I guess my takeaway is when I see something working, I'm going to jump on it.
Cade:
But even then, it's unclear where it's going to go, right? It works for speech and then it works for images and that image net is such a big moment. But then people in the ML community are still like, "Is this really going to work with natural language?" I mean, years later, they're saying that. "Is this is really going to work with natural language?" And then it does. These large language models like Google BERT, GPT-3, it really started to work. And there was real doubt there. It's hard to see these things, even when you're close to them. And we could go on down the line Robotics. It's not clear even when this stuff works with multiple different areas, whether it's going to work with the next one.
Lukas:
One theme and it also comes up in your book, of course, because we're talking about academics is sort of who gets credit and who doesn't get credit and where's credit deserved? And actually, one anecdote that I never knew that you have in your book, despite Wojciech being a pretty good friend of mine is that, AlexNet, it was originally called WojNet, is that... do I have it right? I can't believe he never told me that. I feel like if I was him, I would definitely.
Cade:
It's a great story, right? Why do we call it AlexNet? You guys are paying for him. The paper didn't really call it AlexNet. It's like, everybody calls it that. Well, the way it worked was, and this is in the book, in a much more eloquent way. But Google has started to build its own version, basically. And it was Wojciech, who did it. And the way it worked at Google was whoever built the thing you named it after him. And so, that's what they called it. And then Hinton and Krizhevsky and Sutskever show up and they're like, "Why are you calling it that?" Right? It's Krizhevsky, who built the thing. So, they just start calling it that. And that's what propagates all over the community. I think that it's a testament to those guys, right? They're rightfully so in a lot of ways revered in a way and they had some capital, right? But it's also just funny how those things work in the tech community. And sometimes those things are corrected, so to speak. Sometimes you're not, right? And-
Lukas:
Well, what do you think? So, is there someone that stands out to you as kind of not getting the credit they deserve because most of the people that heroes of your book, I think are really, really well-known at least to people listening to this. But do you feel like someone really... do people talk about someone when you interviewed them that that doesn't show up in such a big way?
Cade:
Well, I think, Jurgen Schmidhuber is the classic example, right? He's been written out a lot. He's written about in my book. The reality is-
Lukas:
But I don't know that he comes across so well in your book.
Cade:
Interesting. Okay.
Lukas:
I don't know.
Cade:
I think, well, what I was going to say is, with all of this stuff, it's complicated. Okay? And let's take, let's go, well, before we get to Jurgen, let's start with AlexNet. The reality is, is, although Alex Krizhevsky and Hinton and Ilya Sutskever did the work on that and really made it happen. They are building on the work of Yann LeCun, right? They're using a modified version of his algorithm. And he's building on the work of so many others. Everybody's building on everybody else's work and on some level, they all deserve credit, right? And what Schmidhuber is saying is, "These guys who worked for these very big companies are getting this credit and I'm not," right? And I really like Jurgen and I feel for him. At the same time, he is out there saying, "Give me credit, give me credit," right? And that's part of this, too, right? Some people do that. Some people let the credit come to them, right? And that's going to be viewed in different ways, right? Some people are going to criticize Jurgen for saying, "Give me credit, give me credit." But I know him and you can't help, but feel for him as well. Because the reason that these others have gotten so much credit, in large part is because they had these giant companies behind them, right? And these companies are good at producing and driving narratives, and, some of the narratives that have been out there aren't necessarily true, right? There have been published stuff, a lot of it came from the companies that don't necessarily give the real view of these things. And the real view is that, it's more complicated than you think.
Lukas:
Do you think there's a topic in AI that the press should cover more than they do?
Cade:
I think it's more about and I guess, I'm going back to what I've said before is the press needs to cover this in a different way, right?
Lukas:
And with more skepticism, I guess?
Cade:
With more skepticism and look, it is hard. Again, we're talking about you got to strike the right balance between showing people what's really going on, but not going too deep in the weeds. You don't want to lose people and that's a very hard thing to do. But when it comes to topics, what I will say is that, a lot of people have written about this clash at Google, between Timnit and the company. She's saying that she was fired and some people at Google saying that wasn't the case. And in a way, it's a very specific argument, but I think this is really representative of a much larger clash that is going to have to happen in this field, right? These language models that are being built these giant GPT-3-style language models, they are inherently biased, right? That is just the fact because human language is biased. So, these things train on this enormous amount of text, they're biased and they spew hate speech and other toxic material, that's just that's just the reality. And that's what Timnit and others were saying in the paper, that was an issue at Google. That battle is going to, if these models are going to continue to progress, and they really get out into the world that battle is going to happen. It's going to have to happen and on a much larger scale at so many different companies, right?
Lukas:
And what's the battle like? What are the two visions of the future?
Cade:
Well, on the one hand, you have a company like Microsoft, who put out a much simpler conversational bot years ago now called Tay, right?
Lukas:
Yeah, of course, I remember, yeah.
Cade:
It was rules-based for the most part, chat bot. And it started spewing hate speech and it created this huge backlash, and they took it away. Okay? Microsoft, ostensibly, is going to put GPT-3 out in tandem with OpenAI, that is a clash waiting to happen, right? Microsoft's got to deal with the fact that these things are biased and that's going to offend a lot of people, right? How do you deal with that? That's an open question. It's an open question for Microsoft, for Google, for Facebook, for OpenAI. On the one hand, you have science really progressing and doing amazing things, but you have this problem. It's a problem for a lot of people, right? And some people don't see it as a problem, they just think we need to release this stuff and, get over your issues with the bias and the hate speech, but a lot of people think it's a real problem. And to the extent where, that clash is going to have to happen if those models are going to continue to progress and to get out in the world, right? You got to find a way to deal with that, whether it's technically or by other means, right? And that's why I think that that situation at Google is so important, because it represents something much larger that's going on here. And it's something that the press is going to have to look at, as well as all these companies.
Lukas:
Okay. One more question, why is it so easy to demo a thing that's evocative and so hard to turn that into a complete product that we engage with every day?
Cade:
I think it's, it's just about aligning the technology with the need, okay? That OpenAI Rubik's Cube hand that is not aligned with any need, right? We don't need that. The trick is, is finding where there's real gain and applying it, right? And I think that's where people often sort of missed the point, right? And these neural networks have worked and worked really well, in particular areas, right? They don't work well in other areas. There's all this hype around AI and sort of remaking how your business operates that sort of thing, but that's something different, right? There's not always an alignment. There's an alignment with that DeepMind result, right? That is something that is a real need and they're going after it. And in one sense, they solved it, there's still a lot of work to be done, but that's where-
Lukas:
You're talking about protein folding, right?
Cade:
Protein folding, right? The CASP contest, right? That's something that the world needs and they're going after. GPT-3, it's not hard to be impressed by it, but it's really hard to see where that's going to have the practical application. When you find where it works, it becomes much easier to show people, right? I think the difficulty is often just sort of a misalignment if that makes sense.
Lukas:
Yeah, that totally makes sense. All right. Well, I think that's a good note to end on. Thank you so much. That was a lot of fun. Thanks for answering all my questions.
Cade:
Thank you. Thank you. I'm glad to do it. And really good talking to you as well.
Lukas:
Yeah. Real pleasure.