Training a Remote Controlled Car to Drive Itself

With a few hundred dollars, some common tools, and a little patience, you can train an RC car to drive itself. Here's a video showing you how to do just that. .
Justin Tenuto
Whether it’s generating music with OpenAI’s Jukebox project, synthesizing adorable animals with StarGAN, or creating gorgeous lighting effects on static images, we’ve seen our fair share of delightful projects at W&B.
But all of those had one glaring issue: they didn’t involve turning someone’s living room into a racetrack. Thankfully, that problem is now solved.
Yes, that’s our colleague Armand du Parc Locmaria sacrificing some square footage for the sake of training a race car to drive itself.
We’re going to have an additional, more technical piece about the project coming in the next couple weeks, but we wanted to share this video and, of course, the dataset and code in case you were looking to take the project for a test drive (sorry).
There are two other things we felt we should pass on in this interim. First, the general strategy for training your own car and second, a little discovery Armand made during his training that might help you troubleshoot if you do decide to try this out yourself.
So then: how do you train a toy car to drive itself? Armand considered a pair of different strategies. The first was driving the car around with the camera on and using his driving as the training data. In other words, try to make the car behave like he was at the controls (i.e. imitation learning). But he opted for a simpler, more elegant solution for his training technique: take images from the car’s camera and annotate the center of the track. The car’s only real goal is constantly orienting itself towards the center of the track and away from meddlesome objects like couches and walls.
At the outset, the car was driving itself but it was driving itself rather poorly. “It was indeed running into a lot of walls,” was how Armand put it. On some level, that’s expected. After all, models require multiple iterations and retraining. But when he looked at his W&B dashboard and started checking his model’s predictions, he discovered the model wasn’t performing poorly. It was indeed doing well at identifying the center of the track. At this stage, the issue wasn’t his code or his models. It was the hardware.
See, the GPU in this car isn’t exactly the kind of machine you’d train BERT on (if you watched the video, you’ve seen what he was working with). The problem was both that the neural net he trained had too many layers and that the camera’s frame rate was too high. Effectively, that meant that even though his model was making the right predictions, the car wasn’t reacting fast enough. The car was, in essence, choking on too many images.
Discovering that was a relief. It meant that his approach was broadly working and, additionally, he didn't have to collect or annotate any additional training data. He dropped the frame rate and removed a few layers in his NN and the car’s performance improved dramatically. But it’s a good reminder why the more insight you have into your model’s behavior, the more explainable and debuggable it is, the better your projects will turn out.
Thanks for watching and we’ll see you soon with a more technical look at this project!