The Bandwagon 2.0
Editor's Note: here at Weights & Biases, resurrection of the undead is one of our core competencies. We recently resuscitated Claude Shannon, the founder of information theory, and gave him access to arXiV and Twitter. His skeletal fingers typed out this short paper before Shannon requested that we kindly return him to the grave.
Deep learning has, in the last few years, become something of a scientific bandwagon. Starting as a technical tool for the computer vision engineer, it has received an extraordinary amount of publicity in the popular as well as the scientific press. In part, this has been due to connections with such fashionable fields as computing machines, cybernetics, and automation; and in part, to the novelty of its subject matter. As a consequence, it has perhaps been ballooned to an importance beyond its actual accomplishments. Our fellow scientists in many different fields, attracted by the fanfare and by the new avenues opened to scientific analysis, are using these ideas in their own problems. Applications are being made to biology, psychology, linguistics, fundamental physics, economics, the theory of organization, and many others. In short, deep learning is currently partaking of a somewhat heady draught of general popularity.
Although this wave of popularity is certainly pleasant and exciting for those of us working in the field, it carries at the same time an element of danger. While we feel that deep learning is indeed a valuable tool in providing fundamental insights into the nature of perception and will continue to grow in importance, it is certainly no panacea for the computer vision engineer or, a fortiori, for anyone else. Seldom do more than a few of nature’s secrets give way at one time. It will be all too easy for our somewhat artificial prosperity to collapse overnight when it is realized that the use of a few exciting words like depth, attention, embedding, do not solve all our problems.
What can be done to inject a note of moderation in this situation? In the first place, workers in other fields should realize that the basic results of the subject are aimed in a very specific direction, a direction that is not necessarily relevant to such fields as psychology, economics, and other social sciences. Indeed, the hard core of deep learning is, essentially, a branch of statistics, a strictly predictive system. A thorough understanding of the mathematical foundation and its statistical application is surely a prerequisite to other applications. I personally believe that many of the concepts of deep learning will prove useful in these other fields-and, indeed, some results are already quite promising-but the establishing of such applications is not a trivial matter of translating words to a new domain, but rather the slow tedious process of hypothesis and experimental verification. If, for example, the human being acts in some situations like a convolutional network, this is an experimental and not a mathematical fact, and as such must be tested under a wide variety of experimental situations.
Secondly, we must keep our own house in first class order. The subject of deep learning has certainly been sold, if not oversold. We should now turn our attention to the business of research and development at the highest scientific plane we can maintain. Research rather than exposition is the keynote, and our critical thresholds should be raised. Authors should submit only their best efforts, and these only after careful criticism by themselves and their colleagues. A few first rate research papers are preferable to a large number that are poorly conceived or half-finished. The latter are no credit to their writers and a waste of time to their readers. Only by maintaining a thoroughly scientific attitude can we achieve real progress in deep learning and consolidate our present position.
Coda:
In 1956, Claude Shannon wrote an article called The Bandwagon, decrying the excessive hype around what was then a hot new field, information theory. With the judicious substitution of a few key words, we ended up with an article that's relevant today.
