Chris Mattmann — ML Applications on Earth, Mars, and Beyond
Chris shares some of the incredible work and innovations behind deep space exploration at NASA JPL and reflects on the past, present, and future of machine learning.
Listen on these platforms
Connect with Chris
0:00 Sneak peek, intro
0:52 On Perseverance and Ingenuity
8:40 Machine learning applications at NASA JPL
11:51 Innovation in scientific instruments and data formats
18:26 Data processing levels: Level 1 vs Level 2 vs Level 3
22:20 Competitive data processing
27:38 Kerbal Space Program
30:19 The ideas behind "Machine Learning with Tensorflow, Second Edition"
35:37 The future of MLOps and AutoML
38:51 Machine learning at the edge
Note: Transcriptions are provided by a third-party service, and may contain some inaccuracies. Please submit any corrections to firstname.lastname@example.org. Thank you!
In the next N years, we'll be building, in partnership with ESA, the Fetch Rover, which is more of a couple-tricycle-sized rover that has to drive farther and faster because it's going to have to go pick up all those tubules, make it to a rendezvous point, take those tubules, fly them up out of the Martian atmosphere, into space, to a spacecraft, and then take that spacecraft back to earth. Yes, that's ambitious, but we're NASA JPL.
You're listening to Gradient Dissent, a show about machine learning in the real world. And I'm your host, Lukas Biewald.
Chris Mattmann is Chief Technology and Innovation Officer at NASA Jet Propulsion Laboratory. And he's the author of "Machine Learning with TensorFlow, Second Edition". He recently worked on a number of space missions, including the Mars Rover that just landed on Mars. And I could not be more excited to talk to him about that.
All right. I want to talk to you about your book and about your career, but I saw that you did some work on the recent NASA rover, or involved somehow, and I think probably everyone was watching that in the news and getting excited about it. So I was wondering if you could tell us what work you did and what it felt like to see that on Mars, and how machine learning could help with projects like that?
Yeah. Lukas, I'm really interested in the new rover, it's called Perseverance. Successful landing, February 18th. Entry, descent, and landing. This is a Volkswagen-Bug-sized rover, very similar to the size of the 2012 Curiosity Rover or MSL, Mars Science Laboratory. So, it also necessitated the development of this new entry, descent and landing that we piloted in 2012, which is the sky crane. It's literally a robotic sort of craft that lowers on a crane this rover down onto the surface of Mars for a nice soft landing and so on and so forth. So that was piloted again. It's only the second time that's been used. That was amazing.
And obviously, the 2020 rover, one of the cool parts about, it's got this helicopter, this drone helicopter on it called Ingenuity. We do naming contests throughout the United States with kids in schools and ask them to name the rover and name, in this case, the helicopter, which is really cool. And I think it really symbolizes everybody's feelings, I think, during this pandemic, it's perseverance and also just the humankind ingenuity. But in terms of what we were involved with, there's a couple things.
I'm the Chief Technology and Innovation Officer at NASA JPL. I run the artificial intelligence, analytics and innovation division. We're basically cross-cutting consultants. We do the AI practice, so we consult out to missions, projects and things like that, the cloud practice, and we also have some data visualization and infusion folks working in new ways of tech. A couple of different areas that we helped in 2020. The first was in a concept that we call Drive-By Science. It works like this, we were partnering with the Mars Surface Mobility group in a team led by Hiro Ono there.
And basically, it works like this with Drive-By Science. So earth to Mars, 11 minutes at least, round trip light time, send a command there, get a message back. And so that's a very thin pipe. And so right now, Mars surface operations, even for the Curiosity rover, and also for Perseverance, it uses about 200 images a day to plan what to do the next day, because thin pipe images are expensive, this and that. The other thing that's really important on these rovers are basically, I say these are elephant-sized vehicles with pea-sized brains, unfortunately.
And there's a reason for that. They're running basically a RAD750, which is like an iPhone 1 processor in it. And why are they running such older technology? Well, cosmic radiation. When we put hardware up in space, cosmic radiation does wiggy stuff to the hardware. It flips bits from ones to zeros, zeros to ones. And so, we typically only fly things that are radiation hardened, which pushed us on the technology low-tick instead of the uptick for that. Tomorrow, we'll have high-performance spaceflight computing, feature GPU-like processors that are radiation hardened. And we have some technology demonstrations of that today with things like the Snapdragon, which is on the helicopter, actually. It's running a Qualcomm Snapdragon.
And so we can do that because it's a technology demonstration. It's not critical to the core mission of the rover, and things like that. And so in that feature, when we have big brains on these rovers and assets and things like that, can we run deep learning on board instead of getting 200 images back, sending it across that thin pipe? What if we could give you back a million captions? What if we could run Google show and tell, or an adaptation of that using transfer learning like we've done, which is called SCOTI, for "science terrain captioning"*. We name all of our stuff like Star Trek.
And what if we're running SCOTI on board and we can give you one million captions back? So we call that Drive-By Science. And then another area I'll just mention and then I'll shut up, I'd like to make this a conversation. Another area is, we call it energy-aware optimal auto-navigation. And it's the same type of concept. It's looking out in the distance for the rover, if it sees imagery, if it sees sand, it knows those wheels aren't going to catch as well on it and it's going to use more power. If it sees rocky, it's going to catch the wheels better, it's going to use less power.
So looking at energy-aware optimal auto-navigation using a similar concept. Those are the big things we've been working on.
*SCOTI: Science Captioning of Terrain Images
That's really interesting. So, do you do any machine learning now on the rover? Is that even possible with the hardware you have? And if you have a Snapdragon on the helicopter, it seems like you could do some in that or try to do some. So is there any happening or is it mostly older techniques for now?
Yeah. A lot of it is human in the loop, but there are some elements of autonomy, both in terrain classification. We have been doing a number of work to take newer modern algorithms. The interesting part is "DevOps at the edge" where the edge is Mars. We talk about the edge today in the cloud or in IoT. So it's DevOps. So what you test terrestrially, you've got to make sure we can uplink it, import it to, again, these older devices and in some cases, devices that were deployed almost eight years ago, like Curiosity and things like that. And so we have been working on that. There is an algorithm called SPOC, again Star Trek names, but this is a soil property object classifier. It's like a terrain classifier. And we can run that on the older devices.
Obviously, the tricks with that are, you don't have a GPU, you may have to quantize the models, trade for accuracy and performance and things like that within acceptable bounds. And so a lot of these things are for human subject matter expert review or for mission tactical ops review with human in the loop. The more in the future we can get that out of the loop and more autonomous decisions, we're going to need it. And I'll say one quick example. The next mission is called Mars Sample Return in the program. And the basic idea is this, this big car size rover driving around, Perseverance. One of the things it does is it's coring rocks and it's going to drop tubules of those cored rocks as it drives over the next N years.
In the next N years, we'll be building, in partnership with ESA, the Fetch Rover, which is more of a couple-tricycle-sized rover that has to drive farther and faster because it's going to have to go pick up all those tubules, make it to a rendezvous point, take those tubules, fly them up out of the Martian atmosphere, into space, to a spacecraft, and then take that spacecraft back to Earth. Yes, that's ambitious, but we're NASA JPL. But that whole thing, you need, obviously, more increased autonomy with that 11-minute light time as well as charging and all the other stuff that we got to do on the rovers to basically make sure that they can operate successfully.
Can you push firmware updates at all with the current stuff you have?
Dive into that for me, like updates in where?
I guess could you update the software from Earth or is it like once it's in there, is it like set forever?
Oh yeah. They do update the software from earth, but there's windows of doing that. There's times in the mission life cycle when that's acceptable risk or they'll allow us to do that or things like that. And then there are times when they won't, obviously, during critical mission operations or associated with some science events or things like that. And so, it really depends on the mission life cycle, but we do have the capability to uplink and even to update things. In the past, those mostly have gone well, but sometimes there've been issues with updating, and so they're very reticent to do that a lot. But we do have these technology opportunities to update the assets out in space.
And sometimes they even compete them. They'll issue a proposal solicitation and get the best ideas and then do stuff like that.
Cool. So I guess outside of the rover, what other ML projects are going on at the JPL right now?
Yeah. There's a lot. I like to talk about it in different pocket areas of ML and AI. One of the areas is cybersecurity. We look at signals, we do analysis with like Data Lake or Delta Lake type of partners, where we're getting signals from cyber and they're doing anomalies and stuff like that, detection. Another that's really driven by computer vision is what we talked about, it's the Mars surface, but not just Mars, future lunar missions, small sets, cube sets, what can we do with imagery and computer vision?
Another is basically what we call science planning and scheduling. Basically, the ideas there is like, "Okay, we've got these football-stadium-sized dishes in Madrid, Spain; Canberra, Australia; Goldstone, California. We call that the Deep Space Network. You can imagine these things, they're not just supporting the United States, they're supporting all of our international partners for missions. Everybody must use the DSN because they're just this international asset, this world asset, to communicate in deep space and not everyone has built such infrastructure. And so in any given week, the DSN is massively oversubscribed because missions know possibly months, possibly years ahead of time what their critical events are and when they need tracks on the DSN to track things.
And so you can imagine this very difficult scheduling problem, 80% of which can be solved by traditional AI scheduling and planning. But the last 20% of it basically boils down to managers getting into a room and horse trading. So basically we've been doing a lot of work to learn what those trades are using like deep reinforcement learning, experimenting with quantum computing, looking at Mixed Integer Linear Programming, or MILP, traditional ways of doing that. And doing that in ways where we can apply ML to actually do a couple of things, learn what those moves that the mission managers make, because whenever you ask them, they don't tell you, because to be honest, it's just innate to them.
It's like, "Well, of course I didn't really need six hours on the track on that dish for my mission, we could have lived with four." "Okay. Well, why didn't you tell someone that?" "Well, because you always ask for more than what you can get." And so these types of things. And so the agent has to learn that and they've got to learn how to generate optimal candidate schedules that fulfill like 46 other constraints and other things. So that's another big area. And then finally, I'd be remiss if I didn't mention there's a ton of ML just in science processing related to science data and instruments.
And JPL has a whole science and instruments section and division, that their job is to basically get data off the instrument, do analytics, generate data products, build maps, build decision products, all of these things, help science research. ML is at the cusp of what I would call massive infusion in those areas, lots of experimentation going on and they're at that crux of turning it into ML ops. And that's basically where it is besides IT and business, where we also are doing it with RPA and some of these other areas.
The instrument use case sounded really interesting. Can you give me some concrete examples of that? I'm just totally unfamiliar with that whole space.
Yeah. So imagine JPL minting first-of-a-kind instruments, because that's what we do in earth sciences, space sciences and planetary science. And there's a reason for that, the national labs are supposed to do that work that no other whatever commercial industry or traditional civil servant places can do. And then once we do it, we're supposed to transition then into industry and stuff. And that is very much true for hyperspectral, where the field actually was mainly defined in some ways at JPL, by people like Rob Green and the AVIRIS Spectrometer and things like that, but also in other areas, radar, LIDAR, and things like that.
And so in these instruments, there are all sorts of things, the traditional model for missions at JPL is a phased life cycle where pre-phase A and phase A is like formulation, phase B is like actual, real costing, phase C is where you're building the mission out and you're actually building it. Phase D is like mission launch and whatever. And E is like standard operations. And so associated with that life cycle at each stage and in particular, in phases D and E, besides delivering the mission bits from the instruments, the science data, the engineering data and stuff like that, NASA competes out, typically.
It does a couple of things. It does some directed work, but it also does some competition for basically analytics and ML and things like that, on the analysis. But even during the mission life cycle phase, it's basically, go from voltages, which are basically electrical signals that have measurements buried into them, to geo-calibrated radiances. So radiances is calibrated to say some space on the Earth where you got to map it using orbital parameters, to basically a full physical model, in some cases to extract out from those calibrated, geo calibrated, geo-referenced radiancee what the hell it was measuring.
And that in some cases it's called level two data, and there's a massive amount of it. And in that, in some cases, even in the mission is where some missions stop, and then they compete out the level three, level four product generation, which is basically taking those swaths of instrument actual measurements and mapping it to a geo globally gridded grid, and then doing other stuff and maybe combining other products on it. And so, even in the mission production life cycle, there's an opportunity for ML.
Some people are looking, they say, "Well, can I replace my full physics model with taking geo-calibrated radiances and then mapping them to a map or even to values and measurements, can I say, build a neural network to do that? Can I learn a representation of something with an auto encoder? Can I do, in concept-wise, like a regression or like even a network or CNN to like do value predictions?" And stuff like that. And there's a lot of experimentation during the science mission operations now for doing that. Because obviously, ML has the opportunity to cost much less physically, not require super computers to do some of these things on other specialized computers, but more commercially available, from the cloud and GPUs, TPUs and things like that. And finally... Well, go ahead. You were going to say something,
Oh, I was just wondering, when you're saying compete out these steps of the process, is that something where like, if I want to try to build a model to do this mapping, I could go to a website and get involved? Is this is like Kaggle or is this like... How does that actually work?
This is fabulous. The answer is no today, and there's a reason for it. Actually, this will parlay into the other thing I was going to say. So basically, you get the level two data out. It's massive, it's petabytes in some cases. You always say, people want the level two data, but you really don't want the level two data.
And in some cases, it's so big that there may or may not be a requirement to preserve it because NASA may have made the decision or even other agencies, NOAA, you look at this to basically, well, we could always reproduce this using a big rig processing campaign. What's the minimum bits and level products that we need to store and keep around because there are preservation requirements? And they always ask that question.
And so the answer related to that, to your question, is that, again, you always want the level two products and then you don't because it's too big. Okay. So what are those level two products stored in? They're HDF5, HDF4 with HDF-EOS metadata, they're NetCDF products, GRIB. There's probably a half a dozen archival formats that aren't say machine learning ready, like a big table, that have everything or a multi-dimensional table, SciPy, NumPy, to do it. But there's been a massive work in the Python community and other places to integrate that stuff with... So, believe it or not HDF5 is very popular in machine learning for weights and Keras and all the work Francis and others did in some of these things to do that.
Where did HDF5 come from? It came from NASA and earth science, actually. It was an earth science archival format from investment from NASA, NOAA and the EPA in the HDF group, which spun out as a separate organization to do it. So actually, the storing of matrices, scalars, vectors, named hierarchical representations of them actually came from representing earth science data. The challenge is what you get at that level, again, it's not globally gritted. Some people don't know what to do. Like if you can't do a point in a coordinate reference system and get a value, their heads explode, people, sometimes, because they don't understand these satellites generate these weird U-shaped orbital swaths where the data is only valid at certain times.
Everyone just assumes you can interrogate something and say, "Give me the data and do machine learning." But there's so much processing and level processing that you have to do beyond that. And so what do people do look at? And this was the second thing I was going to talk about. There's also big opportunities. So the science mission sometimes stops at level two data production, but I'm even saying there's ML opportunity there and people are looking at it. But even in the archival, you go to earth science, you go to the DAACs, they are called the distributed active archive centers. These are for earth science, nine places across the US that you can go get data, and you can download it today.
But again, does it stop it at the level two products? Does it stop at the level three? Once you get to level three, we're talking about 100 time data reduction too, because these maps take less because they're interpolated or they're globally averaged, not specific interrogatable values out of points.
Chris, I feel like this is obvious to you, but I want to have a concrete picture in my head of what one of these datasets is and then what the level two version is and level three, and where it actually goes. Just one where I can really picture it.
Yeah. Let's take OCO, which is the Orbiting Carbon Observatory. And it produces a value called XCO2, which is CO2 sources and sinks. It's a column-based measurement of CO2 in the atmosphere. The level two products for OCO, it's actually called the OCO-2 mission, what they look like is kind of like a U-shaped, upside down Bell curve. Say you take a world map and you projected out in our Cartesian space, so you flatten it out to two dimensions, how you see in those maps. And then if you look at the data, just based on how the satellite orbits, first off, there's little data over water, and it's like an upside down U curve. It's almost like a sine wave, so it's like this.
That's the level two data, because the way the satellite is orbiting and when it turns on and when it doesn't to get measurements, because some of these things can't see through clouds or whatever, ends up being this track. That in many cases, independent of the instrument, spectrometer, radar or whatever, just say in the OCO-2 case, is a level two data product. And it might represent, I don't know, depending on the orbit life cycle, 14 days or something like that before you can get the full track across that. It depends on how long it takes to orbit.
So just to make sure I understand. So the satellite's looking down at the earth and measuring the amount of CO2 with some spectral camera or something, and then it's downloading to you the amounts of CO2, but it's only along the like weird linear track that just like its orbit over the earth? But then that track, I guess, is also moving, so you get different measurements in different places on the earth?
Well, that's exactly right. And in your mind, you imagine, "Oh, we fly a mission like OCO2 and it covers a geo-global grid. I ought to be able to go to Russia or Africa, or the United States at any time in any grid cell and get a value, right?"
Well, at level two, you can't, because at level two, you only have values where that satellite saw a data in its orbit. Now, at level three, what people do is they take that Bell curve, the upside down one, and they basically interpolate, or they average it so that you get values in neighboring cells, and you color the cells. It's almost in machine learning terms like a self-organizing map. So you go from basically like a sine wave to a self-organizing map, geo globally grid, where you can interrogate the values at any point in time. And this is the scientific process.
And is that an official level two to level three mapping. How do you compare if two people did different mappings which one is the best one?
Yeah. There you go. And that's a great point, and usually it's controlled by the science team, there are some standards in instrument families, so there's a way to do this typically with spectrometers, there's a way to do it with radars. And then there are other requirements like, what's the precision needed? What type of data density do we need at a particular pixel? What's the resolution? And those are dollars and costs because they translate to mass and power in the instrument, and they also translate to processing time and whatever afterwards, when you get the data. So these are all little knobs that mission managers, science teams, the science requirements for the mission, trade.
But I just want to understand, the part that you're saying you compete out like the mapping from level two to level three here, going from the raw satellite data to the earth data, or is there a further mapping where most of the machine learning happens?
Oh yeah. Great. By definition, if you look at these NASA missions and earth science, very true in planetary too, the archives again, and remember the size from level two, again, 100 times bigger in many cases than level three and level four, the dollars that they go, dollar per bit in preservation, they've got to cut off at some level because they don't have infinite money, but they're supposed to preserve this "forever" in some cases. And so we're talking hundreds of millions of US dollars for investment in these archives just to keep the bits around. And so a lot of the archive systems will say, or some missions will say, "We're only distributing up to the level two products."
Some of them will distribute level three, but they'll have different rolling windows of how long they're be available using their standard algorithms and things like that. Like maybe they don't keep the level three round forever. So now what does NASA do like I was saying with you to compete and to do analytics and really rise the tide in ML and some of these things? Well, what they'll do is they'll say, "Okay, in a particular earth science area," or whatever, they'll say, "Well, we have a number of recurring," and NASA has this thing called research opportunities and space and earth sciences or ROSES.
But these programs in which they release 40 different programs or whatever, to basically write a proposal, compete against other NASA centers or universities or commercial industry to basically do higher order processing, to generate maybe improved level products at the level three, at level four levels, maybe they cost less, that are more accurate, that didn't take as long to do the algorithm for, or as much scientific expertise or knowledge. And those are all the knobs where they'll do that on. It doesn't mean that such algorithms automatically get put into standard level processing or into the archives, but there's the opportunity to do it.
And then some people, this is the beauty, NASA doesn't have to control everything, neither does NOAA, whatever. This creates a market and an opportunity downstream for universities, commercial partners, or whatever, to build better products. And if they do, these could become the standards eventually, and NASA is very happy to do it because they still fulfilled their mission of researching, observing, and making those data products for free to the world.
So it's like, I'll cut you off, and I was really interested in the... You keep saying competed out. Are you saying that the reason that I couldn't try to go build a better mapping and give it to NASA, is that because the data's so big, it would be hard to get, or is there some other bottleneck there?
Well, it's like a combination of both, you might be able to do it because you're Lukas, superpower, have access to massive cloud, whatever, but it's harder for a postdoc or somebody, a K-12 or someone at university undergraduate to be able to get the type of access to basically do this. And it's also part of just the way science occurs. There's this movement as you and I know, so in the context of like ML to the MLOps, lots of people still use Jupyter, I still do everything locally in some cases, they'll just do Jupyter locally to do stuff. But then there is this movement, JupyterHub, but even beyond that, getting stuff out of Jupyter, Python MLOps, frameworks, TensorFlow, PyTorch, whatever.
There's this whole movement again, from the science research for long tail to doing it in a team with DevOps, with all these things, good software engineering, productising, and things like that. The exact same thing exists in the context of science research in fact, many scientists would much rather love pull all the data down to their laptop and crank on it with MV-IDL, or MATLAB, or even Python, because that's what they've learned in atmospheric science. And so there's almost this mentality or paradigm shift that is even undergoing there too. So that's another part in it, Lukas.
And then finally, the last thing I would say is that you also have to basically how... And some of this stuff is self-documenting and some of it isn't, but what assumptions were made at the level two and before era to get to level two, because you've already started potentially to propagate some error bars, even to get to level two, to go from geo-located, physically calibrated radiances to a level-two product, you've already made some assumptions. And so NASA does document those, those are called algorithm theoretical basis documents, and they do make them available, but you also need to dive into some of those to know how to then apply ML to go beyond.
Got it. So it's really just a really hard problem, it's not that there's any resistance to letting people try it, I guess.
Totally. I'd say there's welcomeness for people trying it, it's just a really hard problem, you nailed it.
I'm a little shy of asking this question, and maybe it goes nowhere, but I just want to, because me and my co-founder all love this game called the Kerbal Space Program and we all play it. I wonder if people at JPL are aware of this and if people coming in, I feel like all my instincts about rockets come from this one video game that I got obsessed with a few years ago, does this come up at all?
Not with me. I'll say a lot of JPLers are playing Among Us right now, but that's not Kerbal Space Program. Tell me about it. Tell me, what is it?
Oh man, it's this amazing game where it's very... I think what's fun is it's very self-directed, there's not really a clear goal, but you basically try to build rockets and put them into orbit. I feel like I learned a lot of just how complicated it is to like make a satellite and then try to get a rocket to that satellite and the trade-offs, like you want a lot of thrust at the beginning, but then that could be an inefficient engine. And you realize actually going to a planet with a high gravity and coming back is way, way harder than just orbiting it, for example. So I don't know, I was curious if this...
Because I feel like when I talked to people at places like NASA, I have all these specific questions and then sometimes they're just like, "Oh my God, you must've played that Kerbal Space Program game."
That's awesome. No, well, I'll bring it up. So for your audience and also for you, this isn't always clear and we even did it ourselves just now. I work at NASA, yes, but I'm at the Jet Propulsion Laboratory, which is one of the nine NASA centers. And so typically the NASA centers have different expertises or main things that they do. Big things that JPL does at Pasadena California amongst all the other nine NASA centers is that we run autonomous exploration. We do a lot of autonomous exploration. In fact, we're a Center of Excellence and NASA's only federally funded and research and development center for that, national lab, and the Mars Program. But we don't do very much at all, very little like human space flight.
A lot of the other NASA centers like Marshall, or Johnson, or Kennedy, like mission control or launch pads, that's where you see a lot of that, but our expertise typically is in... So if it's robots and it's deep space, that's usually us.
Well, just for the record, that comes up too a lot in Kerbal Space Program.
Oh, awesome. Well okay, that's it, I'm going to tell my whole team to play this now, Lukas. So we're there. We'll move from Among Us to that.
I really recommend it.
I also wanted to ask you, because this is so impressive, you also wrote a book in your spare time while doing your day job. I was curious what inspired you to write this book. That's great. What inspired you to write this book?
For me, it was almost an appreciation and an underappreciation in a way of the evolution of the field of machine learning. For me, a few years ago, actually it happened right before the pandemic, I would talk to brilliant mensch like you or people on my team, and then they'd be talking to all this machine learning stuff and frameworks, and I'd be like, "Yeah, that's interesting." Heck, like 10, 15 years ago, let's see, well, it's 13 years ago now I graduated with a PhD. I did a minimal amount of maybe what you would call machine learning today. K-means clustering in my dissertation or whatever. I did a little bit of Bayesian inference, but I would say the field of machine learning wasn't back then like it was today.
So I heard everyone talking about this stuff, and I don't need to know stuff materially at that deep level anymore as much, but I said, "Let me go pick up a book." I had written a book about... well, it was in 2010, it was called "Tika in Action". It was the Tika framework, long story short on that one. It's basically, we call it digital babel fish. If you read Hitchhiker's Guide to the Galaxy, you give it, any language you put the babel fish in your ear and out the other end, it's your interpretation. You can understand it. Tika is that for files, you give it any file, it tells you the file type automatically, the mind type, it extracts the text, the metadata and the language.
And basically for your audience, all they need to know about it is, look it up on Wikipedia, it was the key technology to solve the Panama Papers and win the Pulitzer Prize. So I wrote that book in 2010, and I get Manning books sometimes where I talk to them and I can get a book. And there was a book that came out called "Machine Learning with TensorFlow". And I said, "Hey, can I get this book? I want to read it. I want to learn machine learning so I know what the hell my people are talking about." And so I started reading it and I pulled out a pencil, I started drawing matrices, I started really trying to just instead of read it, do it, do the exercises in the book.
And so what I arrived at after nine months, and this was the lead up to 2020 in the pre-pandemic world, what I arrived at was probably 50 Jupyter notebooks, everywhere was a thrown out suggestion in the first edition of the book, like, "Hey, you could try and build a facial identification system." I did it. I rebuilt the VGG face model. I had a publication at Supercomputing. And I basically just had code, Jupyter notebooks and everything, and I was like, "I've got a second edition of this book," because I filled in all the gaps and I added a bunch of new chapters. And so I pitched it to Manning, they loved it and I was away running. And that's how I did the Machine Learning with TensorFlow Second Edition book. And so that's how I got there, so.
That's so awesome. And I saw you made an interesting choice to use TensorFlow V1 in the book, Instead of V2, because it's still in use on supercomputers, is that right? Or what was the thinking there?
Yeah. And V2, for me, I never shied away from this, you know what I mean? Heck, I was on the board of Apache, we were maintaining, what is it, a 25-year-old web server or whatever, so it's not the oldness of the technology for me, part of it was stability. And so what I was finding was that the TensorFlow 2 was changing a lot at the time. And I made the decision in the beginning, I said, "We're going to pin it to 1.15 because that's stable or whatever. And it was still in use that big Supercomputing agency because they hadn't been on the technology uptick and I was writing it a little bit for them. What we ended up doing, and what I promised to Manning, is during the book about midway through or at the end, I would take a look at basically porting every example, every notebook in the book, to TensorFlow 2.
And that's exactly what we did. And let me tell you something in a testament to Google, and I give them credit for this, it took us about two weeks to port the entire book to TensorFlow 2. Myself and a couple of students and folks who literally just donated their time. And so this wasn't a massive undertaking. There was a big paradigm shift mentally in TensorFlow 1 to TensorFlow 2, but I would say 85% of the code from those notebooks is the same because it's all data preparation, making it analytics and machine learning ready, and then doing ROC analysis, ROC receiver operating characteristics area under the curve.
None of that stuff, the beautiful libraries in Python, Matplotlib, NumPy, SciPy, Pandas, all of these things, they didn't change. What changed in the inside was basically how you set up the model for training, how you run the training step, update your gradients if it's a neural networks, all of these things. That part changed, but the other parts didn't.
Scott Penberthy, the head of AI or applied AI at Google basically made this remark in the intro to my book is, "Don't worry about tracking the latest and greatest XYZ API update, these models and the way you build them will stand the test of time, and I agree with them."
Cool. I guess another topic that I think you've been thinking about a lot and you brought up earlier is, ML Ops and open source, are there any projects you're particularly excited about helping just deploy and maintain machine learning models? Are there any that you use at the JPL?
There's a couple, I'm impressed with a lot of the Amazonian things that are coming on, SageMaker Studio and some of the things they're doing. I'm impressed by some of the capabilities that W&B is building, your company, I'm not just saying that too, there seems to be some appetite really to... And for me, my exposure to this was through the DARPA Data-Driven Discovery of Models program, but really looking at parameters, parameter tuning in an automated way, optimizations of that pipeline, and also SME-based, Subject Matter Expert-based, model feedback. And I really think that's where the world's going to go soon in the next, if you look at one, three, five-year timescale.
AutoML is here, if you look at some of the capabilities, Google AutoML, data robot, some of these other things. So really it's going to shift what people are doing, I think, from building models all the time, let a computer put together primitives and score them for you, and whatever. And then start there, give that feedback, change the job that that person is doing to that feedback. And I think it'll make people more optimized and things like that.
Why do you think that hasn't happened already? AutoML has been out for a long time, hyperparameter optimization has been around for quite a long time and good libraries exist, including Weights & Biases has a library that folks use, but it doesn't seem like, I would say, maybe 20% of our users look like they're doing hyperparameter optimization with our stuff or somebody else's stuff. Why do you think it's not more widespread already?
For me, it's a little bit of, in 2018, when I went to a Blockchain Conference at UCLA at the Blockchain Lab, and I sat there and I listened to Ethereum versus, oh God, I forget the other one. Well, obviously there was Bitcoin and then there was EOS and this and that. And I was like, "Oh God, these are the early days that are like IETF, the Internet Engineering Task Force, where everyone was trying to build their specifications of Gopher and this and that. It feels like it's still the Wild West, and there's always dejure competition, which is standards based, getting people together and saying, "This is what thou shall use, thou shall use MapReduce, or thou shall use whatever."
And then there's the defacto with that, is what are people actually using in what they're building? And it feels like the gap between the people that are doing the defacto development and that type of uptick in terms of framework things, aren't meeting the people that are making the framework decisions for the ops and what they're going to double down and invest in. And the closer that does move, and it will happen, I really do believe it'll happen. The pandemic accelerated that in a lot of ways, I think, but I think when those two things move closer, Lukas, I think you'll see it.
Well, that's a good segue, it's two questions that we always end with, and the second last question, which maybe you've answered already, but maybe you could answer with something else is, what's an underrated aspect of machine learning that you feel like people should pay more attention to than they do today?
Actually, I won't pick... I have two choices for that, one thing I could have picked was learning with less labels. That's far out, the zero-shot, one-shot learning, I'll just say, pay attention to that, but the soundbite here for me is ML at the edge. Everyone thinks you can take a machine learning model, put it onto an NVIDIA TX2 or Jetson, and your model is going to perform the same way and it's going to be push button. And that's just bupkis, it doesn't work like that, there's so much engineering involved, you're trading so much at the model. But look, if you look at CES, we're going to move from per capita devices, four to nine right now for people over the next five years to 40, 50, 60, 80 devices per capita.
So these are all going to be running machine learning, ML at the edge. Do you want to know and be involved in what's happening there? If you do, get involved, and also realize it's not push-button model deployment and your models perform a lot differently. And so that's an underappreciated area in my mind, and I want all your smart people, and you, and your audience to focus on that because we need help.
It's funny, that answers the final question that we always ask, which is what is the biggest challenge making machine learning work in the real world or at your job? Is it actually edge deployment and models behaving differently when they're actually in the edge? Is that a fair summary?
Yeah, absolutely. That's a totally fair summary, Lukas. And also where the edge, we definitely do a lot of IoT on campus in clean rooms and other places, and all of this, but also where the edge is Mars.
Do you have any tricks to leave us with in making things work on the edge? Is there any best practices that you've figured out to help with that?
Yeah. One best practice I'll just share with you, and this is our biggest time sink, is don't change the hardware midstream. A different thing that looks compatible is actually vastly different, even within the same product family. So stick to what you got and the computing power you have, and engineer more optimizations there rather than thinking, "Oh, it's just...", because the price point of the hardware, it makes it so attractive, "Oh, I'll just spend another 200 bucks and get a different thing." No, no, no, you'll spend 10 to 50 times that re-engineering your entire pipeline. So don't change the hardware midstream.
Awesome. You've spoken like a real engineer. Good note to end on. Thanks, Chris.
It's good to chat with you.
Back at you. Thanks, Lukas.
Thanks for listening to another episode of Gradient Dissent. Doing these interviews are a lot of fun, and it's especially fun for me when I can actually hear from the people that are listening to these episodes. So if you wouldn't mind leaving a comment and telling me what you think, or starting a conversation, that would make me inspired to do more of these episodes. And also, if you wouldn't mind liking and subscribing, I'd appreciate that a lot.