The Most Important Thing
Have you heard the one about the guy on his hands and knees under the street lamp? His friend found him there.
"What are you up to?"
"Looking for my keys."
"Where did you drop them?"
"Over there by the fence."
"So why are you looking here?"
"Because this is where the light is."
We are naturally drawn toward problems on which we can make measurable progress in minutes or hours. When observable progress takes days or weeks, it’s difficult to maintain motivation. A common strategy for working on big problems is to break them into smaller problems. Then we’re back in the regime of short term rewards, and all is well.
Except that something can be lost in the translation. Short term progress can end up being worse than useless in getting to your larger goals. “Climbing a ladder to get to the moon” is a classic example. You’ll show excellent progress on your quarterly goals right up until you run out of rungs.
In machine learning research this can take the form of working only with benchmark data sets. We can spend entire careers tweaking activation functions, experimenting with optimizers, and exploring architectures for getting small improvements on a handful of data sets. In doing this, we lose sight of the fact that these methods were created to solve practical problems. The constraints of machine learning in organic environments (a.k.a. "in production") can become an annoyance. Robustness to out-of-distribution samples, data quality, and non-stationarity (data characteristics that evolve over time) is hard to measure, but critical to anyone who wants to use machine learning to block spam or drive a car. Ignoring them is a case of looking where the light is.
The way to protect yourself against this is to work on a concrete problem. Have a goal in mind, one that you can easily explain to room full of ten year olds. This is the most important thing, the one weird trick that will ensure your research contributions are substantial. It doesn't have to be a grand goal, like ending poverty or finding a Grand Unified Theory. It just needs to be concrete, like predicting flood levels in your town's river or detecting when a hummingbird visits your backyard feeder.
Your goal doesn't need to be a lifelong quest either. You can change it as often as you want. Learning about a field as immature as machine learning means that there is a lot of wilderness to explore through trial and error. There's nothing lost in changing focus when you hit a dead end or your goal stops sparking your curiosity. Having a direction is the most important thing.
The downside of working on a concrete problem is that you have to learn about a concrete problem. The physical world is hopelessly messy. There are gaps in what we know, and limits to what we can know. Even the notion of ground truth loses its mooring; labels have mistakes, and often they're not exactly the categories we're looking for. On the other hand, there are well-characterized patterns that we can exploit and domain knowledge we can bake into our work to give us a huge head start. Understanding what's known, what's not, and what's in-between can take a lot of time and effort. It's understandable that some would prefer to set this complexity aside in favor of a few well-defined benchmark tasks.
Working on a concrete problem is freeing. Benchmark-driven method research is useful, but it's a pretty crowded arena right now. If you want more space to work, pick an application. The problem is no longer coming up with an idea no one has tried before. The problem becomes choosing between the next hundred ideas you want to try. The space is HUGE. Categorizing images and predicting the next word in a text stream are just two problems. There are a million others. If you want to push the state of the art, pick a problem to solve. Very soon you'll hit the limits of human knowledge, and if you keep pushing, you'll expand them.
Just keep in mind that you're trying to find your keys. You're trying to get to the moon. This is the most important thing.
