VIDEO: Teaching a Self-Driving RC Car To Obey the Law. And Then Break It.

Once you teach a car to drive around a track, the next step is obvious: teach it do something far cooler. Made by Justin Tenuto using Weights & Biases
Justin Tenuto
In the past month or so, we’ve shared a few reports and videos from our colleague Armand du Parc Locmaria’s self-driving RC car project. The first version of the driving model was a kind of “lane assist” or “lane centering” system: all it could do was stay on the track, always moving forward. Constantly circling a track is all well and good, but has a kind of sisyphean sadness and boredom. What’s more interesting is teaching a car to both strictly follow the rules of the road and also drive like there’s a nanoscopic baby Vin Diesel behind the wheel.
Which is precisely what Armand did.
Yes, the two videos we’re sharing today are about teaching a self-driving car to obey stop signs and then to do donuts. The videos speak for themselves, but there were a couple interesting takeaways to share for a little color here. Let’s start with the stop signs:
Here, Armand’s goal was to run his car around the same track it was originally trained on but to have it follow the rules of the road by stopping at stop signs. What struck us here was that he used a pre-trained model for this task, coupled with his original model. That original model’s goal was to identify the center of the track and constantly orient itself towards that while this new, pre-trained model simply wants the car to stop when it sees the appropriate signage. Full-size autonomous vehicles also use multiple models that act in concert.
What’s notable here is that this pre-trained model wasn’t trained inside his house––it was trained on real roads. In other words: a wildly different environment. Instead of a background of other cars, buildings, pedestrians, shrubbery, and all the other chaotic inputs of an urban street, we have a living room with a couch and masking tape lane lines. Not to mention that the stop sign itself was only a few inches tall.
But it worked out of the box! Armand didn’t have to finetune that pre-trained model as you might expect. He just had to integrate it with his original driving model.
What was considerably more difficult (and inarguably more metal) was getting this car to do donuts.
The video goes through his approach to the problem, building and testing the OpenAI gym environment for the car, what metrics were actually valuable, how W&B helped, and a whole lot more.
Here though, the interesting bit is that there’s a real difference between just driving around in a circle (boring) and doing donuts (sweet). Armand talks a little about what those differences are and how he could see the neural network’s strategy for doing donuts by observing and correlating rewards with actions in W&B. Plus, you get to see a few very low stakes car crashes, which is never a bad thing.
If you want to see how he debugged his car and see how W&B helped with the project, we recommend you check out this report. And if you have any fun projects you’re working on you think we’d like to showcase, drop a comment below or email me at We’d love to share more cool projects like this one with our community.

Read more about the project

Report Gallery