Chris Padwick — Smart Machines for More Sustainable Farming
Chris explains how Blue River Technology is building smart robots for more sustainable farming by identifying crops and weeds and only spraying the weeds with herbicide.
About this episode
Chris Padwick is Director of Computer Vision Machine Learning at Blue River Technology, a subsidiary of John Deere. Their core product, See & Spray, is a weeding robot that identifies crops and weeds in order to spray only the weeds with herbicide.
Chris and Lukas dive into the challenges of bringing See & Spray to life, from the hard computer vision problem of classifying weeds from crops, to the engineering feat of building and updating embedded systems that can survive on a farming machine in the field. Chris also explains why user feedback is crucial, and shares some of the surprising product insights he's gained from working with farmers.
Connect with Chris:
1:09 How does See & Spray reduce herbicide usage?
9:15 Classifying weeds and crops in real time
17:45 Insights from deployment and user feedback
29:08 Why weed and crop classification is surprisingly hard
37:33 Improving and updating models in the field
40:55 Blue River's ML stack
44:55 Autonomous tractors and upcoming directions
48:05 Why data pipelines are underrated
52:10 The challenges of scaling software & hardware
55:55 Bonus: Transporters and the singularity
Watch on YouTube
Note: Transcriptions are provided by a third-party service, and may contain some inaccuracies. Please submit any corrections to email@example.com. Thank you!
Least what I saw was that people's workflows kind of shifted when they realized that they could use this a little bit more like a scalpel instead of a sledgehammer. If you've got the ability to go in and do sort of targeted delivery to only the things that you care about, then that changes your workflow.
You're listening to Gradient Dissent, a show about machine and learning in the real world. And I'm your host Lukas Biewald.
This is a conversation with Chris Padwick who is Director of Computer Vision and Machine Learning at Blue River, which is a company acquired by John Deere that helps farmers strategically spray pesticides and herbicides to help the environment and help their customers. It's a super cool application of machine learning, because it's so concrete and so unambiguously great for the world, but also super difficult. They run into kind of every edge case you can possibly imagine. It's really fun to talk to someone that's spent so much time working on a single hard problem.
Reducing herbicide usage with See & Spray
Chris, thanks for doing this.
Yeah, absolutely. Thanks for having me, Lukas. This is going to be a lot of fun.
Awesome. You work on some of my favorite ML applications. I'd love to maybe start with an explanation of what you're working on at John Deere.
One of our products that we're working on with John Deere is called See & Spray, and the idea is really interesting. If you look at how a farmer does their workflow right now, there's a bunch of parts to farming. If you think of it as a software stack, you can describe it as a tillage portion of that, where you're preparing the soil, then you're planting the soil, then you're weeding the soil, or you're weeding the plants and you're harvesting.
There's kind of like those four sections. See & Spray targets the weeding section, where what we've done is taken an existing sprayer that folks have, and these are huge machines actually. They have 120-foot spray boom and they're capable of upwards of 800 acres a day in weeding.
These things are — I'm 6'2" and this machine, when I stand beside the wheel, the wheel is taller than my head — so, this is a giant, giant machine. It can do tremendous amounts of productivity, is what people buy this for. The application that we're targeting is instead of spraying your entire field to kill the weeds, you really only need to spray the weeds.
And that's what we built. So, we built a computer vision system with AI and specifically deep learning that does discrimination between crop and weed, and then a robotic system around that, that only targets the weeds.
Is this something that gets pulled behind a tractor? What does this machine actually look like?
Yeah, this is an existing product called a self-propelled sprayer. It's really a purpose-built device. I'll do a little impression of it for you, I should have brought a little toy. But what happens is, it has these spray booms, and I'm actually not flexible enough to do it, but imagine my elbows are pointing exactly forward. When we unfurl the boom, these booms go out and they're 60 feet on each side. We have 98 nozzles actually spaced throughout the boom.
As the farmer goes through the field, what happens is the cameras — we have cameras basically spaced roughly every meter — cameras take pictures as you go through the field. Those pictures go into a machine learning algorithm that we've trained to distinguish a crop from weed. That's a convolutional neural network using deep learning. Then we map those to individual sprayer locations and then try to only spray the weed as we're going over it.
Wow, so you have like, did you say you have like 120 cameras then that unfurl or something like that? Am I doing the math right?
It's a 120-foot boom. And on one of our configurations we've got 36 cameras.
Oh, one every meter I see.
Yeah, that's right.
But then the sprayers are actually at a smaller interval than that. Is each one over one row of crops or are there multiple...how does that work?
Yeah, that's right. So, there's different kind of configurations. It's kind of dictated by what the farmer's doing with the machine.
We do row crops right now. So, three target crops, there's soybean and cotton and corn. Depending where you are in the US, you might grow...if you're in Texas, you might grow cotton at 40-inch rows. In which case you'd sort of take a 20-inch machine where you'd have a sprayer every 20 inches. If you imagine the two rows and they're 40 inches apart, you've got a sprayer there and then a sprayer on the row.
The other common one is 30 inch, which is found a little bit more in the Midwest rather than the south.
The cameras are pointed down at the crops and I guess the cameras are in front of the sprayers. So, they know when the weeds are coming.
Yeah, yeah, that's right. Our geometry is that we actually have a camera that's tilted forward so that we can see the crops and then react to them. We did try to...the most ideal geometry in a side view is if you have something sticking out and then the camera pointing straight down. But unfortunately, anything that sticks out of the boom is something that will probably get torn off when it hits something.
I don't know if you've ever driven something that has like a 60-foot thing sticking out of it on each side. It's really, really hard not to hit stuff. Actually, that was one of our design constraints. We really couldn't...we could put things that come up from the boom, because those are safer, but we couldn't put things that go out from the boom, because that would be a mechanical risk that we would just end up breaking parts and breaking the sprayer.
The benefit here to the farmers is that they use is less, I guess, what do you even call the thing that kills the weeds? Like herbicide?
Yeah. Herbicide. Yeah. Yeah. That's one of the big benefits.
I'll put this problem into perspective for you and try to give you a little bit of an understanding as to why farmers so excited by this product. Let's take an example and say that you're a 5,000-acre cotton farmer. So, you've got 5,000 acres total, maybe spread over multiple fields.
The amount of money that you're going to spend just on herbicides — we actually call this the inputs and so your inputs are fuel for your sprayer. Maybe you have to buy a new sprayer, that could be an input. Depreciation costs that could be an input. But seed costs and also herbicide are one of your really main cost drivers — just in terms of the herbicide, you could be spending up to...it wouldn't be uncommon to spend $150,000 spraying herbicide on your field.
So, if I come to you with a tool like this, the See & Spray system, which is more of a targeted system and say, "Hey, depending on your weed pressure and depending on your farming practices, depending on a few input variables, we actually might be able to save you a lot of money there." Maybe you only have to spray 50% or maybe 70% or maybe 30%. It depends on a lot of factors, but if I could put a dent in that herbicide cost for you, then that's a really interesting proposition, because that's money that you can reinvest in your farm. You can do something else with that money rather than spray herbicide.
That's one of the biggest cost drivers for the farmer. The second kind of cost driver or the second driver is wanting to be more sustainable, like participate in more sustainable farming practices. That's on a lot of farmers' minds of trying to spray less herbicide and be better stewards of the land.
This product goes directly into that use case. So, folks are very excited by those two things. That's the main value propositions for the product.
Classifying weeds and crops in real time
That's super cool. Does it have to then do the inference on computers that live on this device? Do you actually put a whole bunch of computers to process all these images? How does that work?
Yeah, that's exactly right. We've built some custom electronics with John Deere that will survive in the agricultural environment.
It's sort of funny. When we think about, "Okay, where would you put a cluster of computers?" Probably your last choice would be in Texas in 115 degree Fahrenheit heat with lots of dust. That's kind of like not the thing that comes to mind when you're building clusters of computers, but that's actually exactly what we've done.
In the current design, we have these compute units on a machine and they function in literally the worst environment you can imagine. It takes a lot of engineering to make that work.
But yeah, when we first kind of described this workflow to people, they sort of think, "Oh, so you're pushing stuff up to the cloud and doing the inference and sending the result back," and actually we're doing it all on board. Just like an auto driving car, because we just don't have the time, we don't have the latency to be able to do that. We need to make decisions...as soon as we see the weed, we have to react within milliseconds. So, it's all done on the platform.
Is this a hard vision problem? Just to get the accuracy to what you need, is that a big challenge for you?
Yeah, it really has been a hard vision problem.
There's sort of two types of products that we talk about. One of them is called a green-on-brown product and that's out in the market today. It's marketed as See & Spray Select. That's a computer vision-based AI system that is capable of spraying weeds that are in the furrow. If you think of a row crop, you've got like the two rows here and the furrow in the middle. So, this product can spray weeds that are in the furrow.
The next level of that product is the green-on-green product, where now you can say, "Okay, well, I can tell the difference between a weed and a crop, even if it's in the furrow," because your weed can then kind of grow anywhere. It's not only constrained to...anyone who has a garden, you know that weeds will grow wherever they want and they're not constrained to just grow in the furrow.
So, the green-on-green product is sort of the next level of the capability. And it's a really hard vision problem. One of the things that makes it really hard is that it's tough to get labels that are correct for these.
I don't know about you, but it's tough for me to tell the difference between say a Pigweed and a cotton plant that's a certain size or a Velvet Leaf or a Morning Glory. These are weeds that look lot like maybe a cotton plant or maybe a soybean plant at a certain time of its life.
In order for our product to be successful, our labels on that data have to be correct. What we've kind of found through trial and error is that it's pretty easy to find people that will label images for you. But it's actually really, really tough to find people that know the difference between these kinds of weeds and these crops.
That's been kind of our main challenge on this project, assembling what I would call an expert workforce of agronomists, actually, who have — some of them actually have PhD in weed science — and these folks help us develop training materials and help us tell the difference between these different varieties of crop and weed.
That's really, really important as we look at our pipeline and our stack. The thing that we spend probably the most time on is talking about label quality and how to improve it and how to measure it. So, yeah, it's a huge topic for us.
Is that partially because...have weeds evolved to look like plants so that human farmers don't pull them up?
There's some truth to that. I'm going to repeat a story that my agronomist told me, that there's a weed that's called a mimicry weed in rice. What happened is when people started hand weeding rice, all of a sudden, it was sort of a selective process where things that didn't look like rice got weeded out, but things that did look like rice started to survive.
So this mimic weed kind of evolved and it sort of came to the point that it looked so much like rice while it was growing, that a trained person in the field had a roughly 70% chance of telling whether it was rice or not. That's how good the mimicry had gotten. And we definitely do see that in our models too.
We don't officially call it the FBI's most wanted list, but maybe we should call it that. But we do have some weeds that are much more challenging to differentiate. They look a lot like...sometimes we have arguments on the CVML team, like, "Okay, well, what do we think this is?" And some of these cases are pretty ambiguous.
If you know where you planted the plants though, couldn't you say that any plant that's sort of not in the place where you think you planted a plant is a weed or should be sprayed? Why do you need to identify exactly what kind of plant it is?
Oh yeah. That's a really good question. There's kind of two answers to that question.
So, answer number one is that you certainly can...you do have that information, especially if you're using a John Deere planting stack. If you've got John Deere machines at every part of that stack, then you've got information about everything you've done. Specifically with ExactEmerge, it's a technology for planting where you can actually tell precisely where the seeds were planted.
Now, the thing that you don't know is what's emerged. So, you do know like what was planted, but you don't really have a good sense of what's emerged. That's one reason that you do have to do kind of a vision-based approach to this.
The second reason is actually a really kind of a subtle one. When we're talking about herbicides...it's funny, when I started at Blue River, I thought that, "Oh, killing plants is easy. You just spray herbicide on them and they die." And it's actually very, very complicated.
One of the things that's most interesting here is that you've kind of got two species — rough kind of breakdowns of plants — you've got broad leaf and grasses. You actually use different herbicides to go after broad leaf versus grasses.
If I can tell you with my machine learning model that, "Okay, this is a weed and it's a broad leaf weed," then you're going to put something different in your tank mix to actually attack that weed. Similarly, if you have a grass weed...if you try to spray a broad leaf herbicide on your grass weed, it's not going to do anything.
There's the opportunity there for more savings for the customer and more effective weed control by identifying roughly, " Are we dealing with broad leaf or grass?" and the See & Spray is targeting the herbicides directly to the plant that needs them.
Insights from deployment and user feedback
I see. That's very cool. So, how deployed is this? If I went to fields in Texas, would I see this device in use?
Yeah. If you had been in our field season...I guess our field season's still technically going. We're into what we call "fallow" operations right now, which is basically, "Identify any plants that are in a fallow field and spray them."
But between the months of roughly March to say August, that was kind of our main weeding season for soy and cotton and corn. And if you had come to Texas and the Midwest and that area of the US, then yeah, you would've seen this system deployed.
We did something really...kind of a first for Blue River this past year. In previous years, what we'd done is we'd built a machine and then we'd taken it to a grower's field, gotten a cooperating grower, and then we'd operated the machine and kind of done demos to get product feedback.
That worked really well, but it's not the same as a customer actually the operating machine. This year, what we did is we actually handed the keys over — this brand new sprayer that's right off the line, has all the bells and whistles and a bunch of brand new technology — we handed them to growers and gave them a little lesson on how to run it and said, "You actually run this machine and we're just going to sit on the road and kind of watch you do it, and we're not going to interfere with you."
We actually did give customers the ability to run the machine, and the learnings from that were really great. We're actually just sort of still compiling the learnings and bubbling the biggest things that we want to work on up to the top so we can hit the ground running again next year.
Wow. Were there any surprises when you did that?
Yeah. Yeah. Big time.
There was one that I think is really funny. So, on CVML, like computer vision machine learning, we tend to look at the world in a certain way. And that way is like, "Okay, the most important thing for the machine to do is identify the weed and then spray the weed." There's a pretty good reason for that, because what happens to a small weed? Well, it turns into a big weed and that becomes a problem. So, hitting weeds when they're small is something that we think we need to work on.
We worked on that problem very diligently, and we've got a solution that definitely targets the smallest weeds. We do this with a little sensitivity knob on the model. What it's doing there is thresholding the focal loss in our network. We're kind of thresholding that value and then making a decision based on that threshold setting.
As a user, you could target it. So, you could say, "Okay, I want it to go really sensitive and target the smallest of the small weeds, or I can go less sensitive and only kind of care about the big weeds." What we found our customers doing was kind of using this in ways that we hadn't envisioned.
One of those ways that was a total surprise...I spent a bunch of time out in the field this summer working with customers and observing their workflows, and when I came back and said, "Yeah, one of the favorite things to do is to set this thing to be really low sensitivity and then go after only the biggest weeds," and my team's heads explode.
They're like, "They're doing it wrong. That's wrong, they're missing weeds." It was an interesting one, because as the farmers explained it to me, it made a lot more sense. What happens with row crops is when you have...just kind of picture a bare field and then picture putting these crops in rows and then picture the crop emerging.
In that time, when you're actually growing a soybean crop, you do want to target all the weeds, because what will happen is those weeds will compete with your crop and they'll compete for nutrients and it'll reduce your yield. Going pretty aggressively after those weeds in the early — we talk about pre and post. Pre-applications are pre-planting and post are post planting So, the first time you get in after planting is really called your first post pass, if you will — it makes a lot of sense to be aggressive at that early time.
But as the crop starts to grow, what happens is that the crop — if you're successful — the crop will grow faster than the weeds. Because of the spacing of which they're planted, they'll actually start to canopy over. At that point, then they've actually won largely. This is not a generally true statement, but it's almost generally true, that largely your crop has won when it's canopied over.
Interesting. Do you have computer vision people on your team that have deep knowledge about farming? I don't think...that might be a low overlap set of knowledge.
Yeah, it definitely is. I guess that I have the most. I grew up in rural Saskatchewan and we had a small quarter section farm, and I used to ride horses and stuff when I was a kid. I think I've probably got the strongest farming background on the computer vision machine learning team.
But what we tell people is that, hey, you don't have to know a lot about farming to come and work for us. And what you do have to do though, is not be afraid to go out to the field, because we believe that's where we learn the most.
That's been part of kind of Blue River's DNA, I think, for forever essentially. We think, okay, you could go and talk about stuff on the whiteboard and you definitely should do that, but you need to reduce that idea to practice and get into the field as fast as possible so that you can learn.You can blow up your assumptions basically.
That's one of our guiding principles, I guess, at Blue River. One of my friends...we don't like to hire house cats. What that means is like, if you're the sort of person that just likes to kind of sit in their office and work on their problem and not go out to the field, then this probably may not be the greatest place for you. So yeah, no farming knowledge required.
Nice. But I like it. A good customer empathy, I feel the same way with the Weights & Biases engineering team. Have there been any other surprises when you've taken these devices into the fields?
Yeah, there was actually. If you're sitting in the cab, the cab of these machines are just absolutely fantastic. You hop in the cab, and you're sitting in this chair that feels like a fighter jet. Part of the allure of the fighter jet is you've got this joystick that has all these buttons on it and the self-propelled sprayer's exactly the same.
Without a word of a lie, there's like on the order of like 24, 25 buttons on this joystick. They all do something different. It's really fun to get in this thing and like, wow this is really, really cool. We also have a display that you can see. There's a couple of displays. There's one that's kind of sitting here and then one that's kind of up more at your eye level. So, there's two displays that you can look at and you can control the system through the displays.
When we launched the product for customers, we thought that the driving factor would be killing weeds. We said, "Hey, everybody, all the feedback we've heard is that people want to control their weeds. And that's the most important thing. Savings would be kind of second on that list." That's kind of how we came into the season.
Certainly some farmers are like that. And I think initially when they started the machine, when they started using the machine, that was their first concern. Like, "Okay, I'm going to go with high sensitivity. I'm going to kill all my weeds so I get the same weed control as broadcast. And then any savings I get on that are going to be a bonus, but I'm not actually going after savings."
At least what I saw was that people's workflows kind of shifted when they realized that they could use this a little bit more like a scalpel instead of a sledgehammer. If you've got the ability to go in and do sort of targeted delivery to only the things that you care about, then that changes your workflow.
What I saw was that was most interesting is — back to this display —as you're going through the field, it has something we call the applied rate map. It's basically...it's a geospatial map and it shows you what the boom is doing in real time. You can actually see the sprays laid down on this map as you're going over it. It's like a real time measurement of weed pressure, if you will.
I think what I was surprised by was that customers usually don't look at that the display, because it's really boring. If you're just doing a broadcast application, then the applied rate is always the same. It's not really an interesting map other than, "Is a sprayer on?" Or not. But with See & Spray, it's actually a really interesting map, because you can see the patches coming down on each individual spray nozzle.
What I saw that was really cool was growers were looking at that map and then they were looking outside and saying, "Oh yeah, that makes sense. I know I have more weeds in this area of the field," and then they get through another area where it wasn't spraying as much, and "Oh that makes sense. I know that this area isn't this wet, so I don't have as many weeds."
That was really interesting, because their eyes were just kind of glued to this real time mechanism to see, "What's my sprayer doing and what's my weed pressure like?" That was really cool to see folks using that.
Why weed and crop classification is surprisingly hard
How much have your models improved? It sounds like they're over a threshold where it's useful. Do you still feel like there's a ways to go in terms of the quality of detection?
Yeah, that's a great question. So, the models have improved dramatically year over year.
Wow. And that's mainly due to better data labeling?
Yeah. There's kind of two parts to that. Getting smarter with our labels and labeling with a better workforce, which we talked about. Also being more targeted in what we label and really kind of preferring quality over quantity.
When we got into this, a few years ago, we always had in the back of our mind, quantity is really important and specifically diversity is important. The way that we approach this collection of diversity is to try to collect data in every kind of growing condition we can get our hands on.
It's interesting to see how much different the ground looks. When we're kind of talking about soybeans, we might have a picture in our minds of a soybean on dark soil and a really pristine kind of computer vision environment that you could train a model on in 10 minutes and do something.
It turns out not to be true at all. There are so many confounding factors.
One sort of visually confounding factor that's really interesting is, folks are really into no till planting. No till planting is exactly what it sounds like. You just sort of don't do the tillage step and you keep the cover crop that was there last year.
So, let's say you're rotating corn and soybean. You might grow corn this year and then next year you plant soybeans, but you don't actually till it under. You just run your planter through the old dead corn. You have all these stalks sticking up and they're all dead, and then you've got some plants that are emerging that are alive and then you've got weeds.
It's almost like the most confusing computer vision environment you could possibly imagine. That makes it really hard. We've been working really hard on beefing our models up to work in these different situations.
Another really good one is just the soil color and the farming practices. In countries like Brazil, they actually don't really plant on 30-inch rows. They plant sometimes much, much denser than that. That means that you can't really put a sprayer between the rows anymore. So, they actually drive like 45 degrees across the row and kind of kill plants or run over plants with the sprayer.
Another situation that our model has to handle is these different farming practices in different regions.
I guess, as an aside, but how do they then do other... don't they always have to drive some machine over their fields? Why would they put this within...why would they put them so close together that they can't drive machines over the fields without squashing plants?
It's a little bit of a mix of different types of machines.
Oh I see.
Yeah. Sometimes what we see is that...the farming practices kind of in the US, someone described this to me as like, "How would you build a factory?" Well, what you do is you'd mechanize every part of the operation and you'd build machines that do this over and over and over again. You can think of farming the same way, except basically the factory goes to the plants, not the other way around. Excluding kind of vertical farming, which is a different thing.
When you have sort of the John Deere, if you have a full John Deere stack for all this, then you don't really have this problem. You can plant with a compatible spacing with your planter and your weeder and your harvester, all sort of compatible.
But when you mix and match and you don't have that kind of end-to-end solution. You sort of do end up into an interesting area where you've got to make some decisions.
Brazil is a special climate too. It's a lot more humid and they actually kind of grow year round. It's a really interesting one. Soybeans in that kind of environment love to be way dense. So, it's kind of an interesting one, but yeah, that's one of the challenges we have with computer vision of teaching your model to identify weeds really at any kind of orientation.
Yeah. I'm seeing why this is a harder problem than I was imagining. These are really evocative examples.
Does it ever happen that your model has an escape valve or something, or it has some sense that, "You know what? This is too hard. Red alert, I don't want to touch any plants"?
Yeah. We have exactly this system. It's really, in a nutshell, how we respond to dust.
So, you can imagine that dust is a really complicated environment, because there's really two things working against you. Number one, it's almost impossible to get good quality labels on a dusty image. If people can't tell what's in the image, then it's going to be really hard to get a machine to do that too. So, labels are tough to do.
But the other thing that I like to talk about is these models often — and there's research, I think, to back this up. I'm trying to remember the name of the paper that came out, but I think it's called like confidently incorrect — but these machine learning models can be very confidently incorrect.
A really stupid example is if we have an elephant versus giraffe classifier and you show it a rhino, it's going to be confidently incorrect, just kind of by definition. What we've done is we've actually trained...we sort of have an architecture that's a little bit like the Tesla model, where we've got this backbone and then these heads that do different things.
We have this image quality head that tells us our dust probability, and we've trained it to really detect the presence of dust. Once that's above a certain threshold, then we say, "Ah, probably the results from the model are not to be trusted in this scenario. The model might be confidently incorrect." And then what we do is — in our system it's pretty easy, we have what's called a fallback — so we fall back to broadcast.
The idea is if you're not sure what it is in your model, you don't trust the results of the model because it's dusty, then you just turn the sprayer on and that way you make sure you're not missing weeds.
Oh, interesting. So, you're really doing multitask learning. Is that right?
Yeah, that's right. We have a few other kind of image quality related heads that's in our architecture and we're always kind of thinking about adding more as we discover situations that we need to detect.
A good example is implement occlusion. If part of the implement gets into the camera frame, we want to be able to detect that. Doing that as a kind of a detection head off of an encoder backbone is a really very efficient way to do that, because you don't pay much of a runtime. I think we pay — I don't know, the numbers aren't quite in my head — but I think we pay like much less than a millisecond run time for doing a dust classification.
Improving and updating models in the field
How frequently do you update these models? Can you update the model on one of these devices? Would you do that?
Yeah, we do do that. It's going to be up to the farmer, ultimately, how often they update. Really what's going to dictate that is their connectivity.
Some of the farmers that we worked with had really good connectivity and we could push updates to the machine very frequently if we wanted to. Other farmers that we worked with, they're in very remote areas of sort of west Texas and there are no bars. You have to go 20, 30 miles away before you get to one 4G bar.
I think what's going to happen is we're going to see that the folks that have connectivity are going to be updating more often and they're going to be getting kind of the latest and the greatest models. The folks that are not are going to be just updating much, much less often.
Are you able to use the data that the cameras collect in the fields to improve the models?
Yeah, yeah, we do that.
Just like every other company that has a bunch of sensors, you quickly figure out like, okay, well we can't record all the time. Our solution to this is that we do some sort of very sparse triggered recording, as folks are going through the field. Within minutes of them going through field, subject to the connectivity constraint that I talked about, the machines are uploading data so that we can validate the performance of that data.
We like to think of this ML flywheel concept, where...Andrew Ng calls it the virtuous cycle of AI, where you train a model, it does some predictions, you evaluate those predictions, train more models, and your model gets better and better.
We have folks sign a data use agreement that allows us to do this, but this data is a gold mine for us in terms of finding edge conditions in the model.
When you're training machine learning models, you quickly start breaking champagne after you get X number of images in a model, because you're sort of getting to this point of diminishing returns pretty fast. But that doesn't mean your model's working well. That means your model is generally okay. But it might actually really suck in some situations.
So what our metric is, is how many fields are you actually passing our spec on. Instead of aggregating all fields together into a single number we break them out on a per field basis. And our goal is to always drive that number higher.
It really comes into exception management, where you very quickly reach this point where the model's actually doing pretty well, but then it has these notable failures and now your attention shifts to detecting and then addressing these failures. That's what the ultra sparse logging helps us to do, because we can really just get the data from the customers in the fields they're actually trying to work on and we can improve their models.
Blue River's ML stack
You guys are really doing a hard real world application of ML on massive data sizes and you've been doing it for long enough to really be battle hardened, I guess.
I'm really dying to know, what does your ML stack look like? How did you get to it? How did you decide on your ML framework? What are the other tools that are really important to you? Can you talk a little bit about that?
Oh yeah, sure. It's been a really kind of a fun journey. Going back to 2016, we were training our first models in Caffe, because that was the...nobody's heard of Caffe these days, but boy, that was like the tool that everybody used back in 2016. It was way farther ahead than say TensorFlow at the time.
We started with Caffe and we built a system out in Caffe and then we saw some interesting things going with TensorFlow and I suspect this will be kind of a familiar story to hopefully a lot of your listeners.
We moved to TensorFlow in sort of 2018 and then we saw, hey, PyTorch is getting really interesting. So, we moved to PyTorch about a year ago. We've been doing most of our work in PyTorch now.
What caused you to move from TensorFlow to PyTorch? Was it a feature that mattered? What was the driving reason?
It was really adoption. So, we found...our experience with TensorFlow was really before the eager interface. So yeah, prior to eager.
So, this is more than a year ago probably.
Yeah. Yeah, I think it was like 2018 when we were doing PyTorch. It was really before the eager interface and our folks were finding...it follows the papers a little bit too. When folks write a lot of papers and if you go to Papers With Code and you can download the PyTorch thing, that's just a much lower startup cost. So, to some sense it follows that.
Interesting. What else? Do you guys do hyperparameter optimization? Is that important to you? Do you use like a data store for this? What are you using?
Yeah, so in terms of our training stack, PyTorch is kind of our main tool.
We have done a little bit of hyperparameter search. One of the things that's a big challenge for us is that at some level we're stuck between this rock and a hard place. The rock is the accuracy — we want the highest accuracy —but the hard place is also, we need to make this run as fast as possible.
In our system, the speed at which the farmer could drive the machine is directly gated by how fast our inference goes. We have to take great pains to try to pick and engineer a network architecture that's going to satisfy our accuracy, but run in the time constraints that we need.
That actually dictates a lot of our network architecture choices. In terms of our kind of deployment stack, what we...oh, so back to hyper parameters, what that means is that we actually don't have a tremendous number of hyperparameters to search for. It's really just sort of learning rate, learning rate scheduler, and a handful of other things.
We're sort of, by definition...our search space on architecture isn't very large. Some of the things that you would normally do with hyperparameters, we haven't found them to be all that useful for our problem.
Autonomous tractors and upcoming directions
This just came to me, but it seems like a self-driving car is hard, but a self-driving tractor might be pretty easy, especially if someone's just looking at a console instead of the field. Why does a human have to drive the tractor if you can do these really smart things to figure out where the weeds are?
Yeah. You're right on the money there, Lukas, as usual. If you think about, what are the challenges of an auto driving car, they're really large.
Just to pick one example, let's say you're cruising along the freeway and you're going a hundred kilometers an hour — I'm from Canada. So, that's why I think of kilometers — you're going a hundred kilometers an hour and then something happens and you need to slam on the brakes. Is that the right thing to do on the freeway?
The answer is, well, maybe, I don't know. It depends what the problem is.
With tractor automation, it's actually fine, because your field isn't full of tractors going a hundred kilometers an hour, and nobody's going to side swipe you with another tractor. If you see something weird, you could just stop and it's totally fine.
You're right on the money. And it's geofenced, so you could say, okay, well, I'm just going to allow myself to be automated in this area. And there's a lot of it. Yeah, definitely a lot of advantages in terms of our problem space for automation. Huge number of advantages.
Is that something that you might work on or...
You did see that John Deere acquired Bear Flag recently. So, I think you can probably guess that John Deere is very interested in this. I think some more information will be coming.
Awesome. Are there any other farming applications that you're really excited about?
Yeah. We view weeding as almost the beachhead project.
I've got a background in astronomy, so this metaphor kind of makes sense to me. Like, when you build a new telescope that has maybe different modalities or different resolution, all of a sudden you get this data from this new telescope, and then you start answering questions you never even thought to ask before. I kind of feel like the same thing's going to happen with our system.
The fact that you have cameras that are three feet away from plants taking pictures at a high rate of speed and high resolution pictures is really going to open our eyes and change the things that we're doing with the data.
You can actually see the weeds evolve as you spray them. You can go in today, do your sprays. You can look at that map and say, "Okay where did I actually spray the most? And which weeds am I concerned about?" And you can plan your next workflow based on that data.
Why data pipelines are underrated
Cool. Well, we always end with two open ended questions and one doesn't necessarily even need to be about farming. It's kind of your take on what's a topic in machine learning that you think is underappreciated or something that you'd like to dig more into if you had more time in your life.
Oh yeah. I did put a little bit of thought on this one. I think a really underrated aspect of machine learning, there was a paper in, I think it was 2013, it was called Technical Debt in Machine Learning Systems.
It was by Google.
Yeah. You know this paper. Yeah.
The thing that really jumped to me on this paper is, when you think about the whole machine learning pipeline, sort of like 20% of it is the cool, sexy stuff that everybody wants to do. Network design and hyperparameters and all that fun stuff, training on thousands of GPUs. That's all fun and very necessary, but it's about sort of 20% of the problem.
Really, if you don't have a really strong data infrastructure around that, then you actually can't do that part, that 20% part. So, you can't make your product.
Having an excellent data pipeline, I think is kind of underrated. It's something that everybody, I think underestimates when they get into machine learning. It's like, "Okay, all I got to do is take this TensorFlow tutorial and do MNIST and okay, awesome. Now it's just like fine tune a model and I'm off to the races."
If you've got an awesome date of pipeline, then that's kind of largely true. But if you don't, then you got to focus your efforts on building that data pipeline, because that's really probably the most underrated problem in building ML systems in my mind.
Do you have any specific suggestions for someone putting together a data pipeline from approach to even what software they might look at first?
Yeah, that's a great question.
I'll kind of draw parallel to labeling. In 2016 we wanted to do labeling on images and we couldn't really find what we wanted. We kind of had to build our own system based on MTurk to do that. That made sense at the time, because we couldn't sort of look at the marketplace and pick the thing that looked good and then go with it.
Then in 2018, we said, "Okay, well, this exists now, so we could just do this." And that's kind of how we met you with your old company. What I'm seeing is the startup world is a fire with MLOps solutions.
Folks have figured out, wow, these problems are hard and they're hairy and they're nasty. I would encourage folks when they're looking at data pipeline to survey the space. There's new offerings every day out there.
I think you'd be doing yourself a disservice if you didn't at least evaluate them before you embark on your data pipeline journey. I don't know that there's a solution out there that's going to do everything that you want, but there could be something that gets you pretty far, and then you can add your own plugins or something to that effect.
But I would say if you're sort of like clean slate, how do I get started? Well, you should definitely look at some of the software packages that are out there, because this is not really a problem that you necessarily want to solve, unless you absolutely have to.
Well, as someone that makes one of these software packages, I strongly agree with that.
Yeah. I can imagine.
The challenges of scaling software & hardware
I guess that's a good segue to our final question, which is...you guys have had one of the the longest journeys of imagining a solution and then actually getting it working in production. I mean, what's it been like? Eight or nine years maybe, for that full cycle. Where have the biggest bottlenecks or the most surprising bottlenecks been?
Yeah, that's a really, that's a deep one for sure.
Blue River's kind of first computer vision product was the lettuce thinning machine. That operated from 2013 to 2016 in the Salinas valley in Yuma, Arizona. And then we kind of pivoted to do row crops. So, it's kind of when I got involved in the company, is 2016 working on row crops.
Yeah, our journey has kind of been from 2016 to roughly now. The things that are really, I think, hard is when you're talking about building a piece of hardware and a piece of software and ML that work together, that's a really, really hard thing to do.
I think there's definitely a huge difference between kind of building...being really scrappy and building a prototype that is going to work out in the field and get customer feedback. That is a tremendously different proposition from scaling these things out to thousands of machines. I know with John Deere, that's a huge jump.
I think that's kind of like scaling. I was looking at a book, just yesterday, written by Reid Nelson. I think the title was "Scaling". Achieving scale is really a big challenge.
I think it's even more complicated when you've got kind of hardware in the loop, because now instead of just having the software portion or just the ML portion, now you've actually got hardware which necessarily takes a much longer cycle time to improve or fix.
I think the biggest challenge is marrying the two, software and the hardware together, and getting something that drives customer value. That's been the biggest challenge for sure.
Well, congratulations on making such an amazing product that helps farmers and helps the world. It's great to talk to you about it.
Yeah. Thanks a lot. We definitely are excited by this product.
What I'm personally really excited about is, as we scale this thing out to tens of machines and hundreds of machines and thousands of machines, those savings are going to go up proportionally. I've been talking to John Deere about this. I want to get a "Gallons of herbicide saved" on the John Deere website. And it's going to just like keep increasing, going like to this really big number. It's going to be like the national debt, but it'll be like a good number.
Awesome. Well, let me know when that happens. I'll take a look and feel a little bit of pressure too.
Yeah. I'll give you the link.
Awesome. Excellent. Thanks for your time.
Okay. Thanks, Lukas. Take care.
If you're enjoying these interviews and you want to learn more, please click on the link to the show notes in the description where you can find links to all the papers that are mentioned, supplemental material and a transcription that we work really hard to produce. So, check it out.
Bonus: Transporters and the singularity
I guess this doesn't have much to do with machine learning, but it's my podcast so I'm going to ask it anyway. If there was a transporter, like a Star Trek-style transporter that would disintegrate your body and rematerialize it somewhere else, does that seem like a safe thing to use? Would you get into that and use it to transport yourself?
Oh, that's a good one. The answer I'd love to tell you is like, "Oh yeah, awesome. That would be amazing." But yeah, I would never get into that. Just like I would never jump out of a plane. Sometimes you learn things about yourself and the fact is that, yeah, I'm not going to jump out of a plane and I think I'm not going to be an early adopter of the Starship transporter. So, yeah. Sorry, two answers.
Would you be a late adopter? If everyone else was using it, do you think that would convince you?
I think so, yeah. I definitely think I would be...it would have so many advantages that I think it would be almost irresistible not to use it. But yeah, I don't see myself as one of the first people jumping into the transporter. But certainly it would save us so much time and we could just get to London for a meeting like that and then come back. That would be super awesome.
You wouldn't be concerned though, that it's a different person who shows up in London, even though they act like you and look like you?
Yeah. Well, it's interesting I haven't...it's been a while since I've been... I used to do a fair amount of quantum physics and I haven't done very much lately in my current job, but as far as I can remember, there's not really a bound on how your molecules get reassociated.
There's definitely entropy and energy loss in that process. So, I think there are definitely some challenges for sure. And we'll have to see. That'll definitely be a good test, maybe you have to pass some kind of a test when you get out of the transporter to see if you're still you.
Interesting. All right. Okay, here's another sort of fun one that we've been asking guests. What do you think about the singularity? Do you imagine a world where ML gets smarter than us in kind of every way, and we just stop working? Does that seem likely or unlikely to you?
Oh yeah. That's such a great question. I think my thoughts on this have been kind of formed by the Minsky school of thought to some extent. It's interesting that as we get into a more connected society, you can kind of think of intelligence as these nodes that are connected, and information gets shared between the nodes.
I think one thing that's sort of really interesting about that, is that as we put more and more nodes into the network — whether it be sort of Facebook or social media or a computer — as we add more and more nodes we're actually not seeing intelligence emerge out of that. We're in some sense in some of the social media sites, we're seeing the opposite. Sort of like an anti-intelligence almost emerging.
It is sort of an interesting thing, because I don't think anyone would've predicted that 10 years ago that we'd be having these problems all of a sudden. When you put everybody in communication, we have all these problems that kind of popped out of that. I don't think anybody would've predicted that. Certainly I didn't.
In terms of a sentient computer that's going to take over everything. I think we're a ways away from that. Is it a possibility in the future? Yeah, absolutely. How close are we to that? It's pretty tough to say.
I know that with training computer vision models, these things are pretty far from AGI. The teams that I work with and the folks that I interact with are under no illusions that this thing is anything other than a specially trained model to do a certain task. So, they're nowhere close to what I would sort of describe as AGI.
I do think that we have a long way to go before we really have to worry about a Skynet sort of thing, taking over the world. I don't think we're anywhere close to that by any means.