Supervised learning vs deep learning vs reinforcement learning

Supervised Learning (SL): Learning from examples with known answers.
Deep Learning (DL): Using deep neural networks to extract complex patterns, especially from raw data.
Reinforcement Learning (RL): Learning optimal behaviors by interacting with an environment and receiving feedback.



In this article we'll cover

The landscape: AI, machine learning, and learning paradigms

Supervised Learning: Learns a mapping from inputs to outputs based on labeled examples.
Reinforcement Learning: Learns to make sequences of decisions by maximizing rewards through interaction.

Supervised learning

Classification: Assigns inputs to categories. Example: spam detection or image recognition.
Regression: Predicts continuous values. Example: estimating house prices or forecasting sales.

Source: DatabaseTown

Image recognition: Models learn from millions of labeled images (e.g., "dog," "bicycle") to recognize new images.
Spam filtering: Trains on labeled emails to distinguish between spam and legitimate messages.

Deep Learning

Source

Early layers might detect simple features like edges or colors.
Mid layers might combine these to recognize shapes or textures.
Deeper layers might integrate these shapes to identify complex objects like faces, cars, or specific words.

Large datasets: To capture complex patterns without overfitting.
High compute power: Especially for training, using GPUs or TPUs.

The Transformer revolution and the rise of LLMs



Source

Reinforcement learning



It observes the current state
Chooses an action
Receives a reward
Moves to a new state

Source

Examples of reinforcement learning in action:

AlphaGo (Google DeepMind): Combined supervised learning (learning from expert games) with RL (self-play), enabling it to defeat top human Go players.
ChatGPT with RLHF: After initial training, a reward model trained on human preferences is used to fine-tune the assistant’s responses using RL algorithms like PPO.

Putting reinforcement learning to work: From games to language



Mastering Complex Games (AlphaGo): Google DeepMind's AlphaGo famously defeated the world Go champion. It initially used supervised learning on human expert games to get started. But the real breakthrough came from reinforcement learning, where AlphaGo played millions of games against itself (self-play). By learning purely from the game outcomes (win/loss reward), it discovered strategies beyond human intuition, showcasing RL's power in complex strategic domains.


Making Language Models Helpful (ChatGPT & RLHF): Base LLMs trained via semi-supervised learning are knowledgeable but don't inherently know how to be helpful, harmless assistants aligned with human values. Reinforcement Learning from Human Feedback (RLHF) helps achieve this.
Collect Preferences: Humans rank different responses generated by the LLM for various prompts.
Train Reward Model: This ranking data trains a separate model to predict how humans would rate any response.
Fine-tune with RL: The LLM (agent) generates responses (actions) to prompts (states). It receives rewards from the reward model. Using RL algorithms like Proximal Policy Optimization (PPO), the LLM is fine-tuned to maximize these predicted human preference scores. (Note: This often follows an initial supervised fine-tuning step where the model learns to follow instructions explicitly). RLHF was key to making models like ChatGPT more conversational and aligned.
Other Applications: RL is also used in robotics (learning manipulation, walking), autonomous systems (traffic control, self-driving decisions), resource management (data centers, supply chains), finance (algorithmic trading), and personalization (recommendation systems).

Comparing the approaches

Reinforcement learning vs supervised learning

Feedback: Supervised learning needs explicit input-output labels. Reinforcement learning learns from scalar reward signals obtained through interaction.
Goal: Supervised learning aims for accurate prediction or classification based on a learned mapping. Reinforcement learning aims to find an optimal policy (sequence of actions) to maximize long-term rewards.
Learning Signal: Supervised learning minimizes the error between prediction and true label. Reinforcement learning maximizes cumulative reward.
Data: Supervised learning requires a pre-existing labeled dataset. Reinforcement learning generates its own data through exploration in an environment.
Scenarios: Supervised learning shines where ground truth is available (e.g., image classification, spam detection). Reinforcement learning excels in control problems, game playing, robotics, and sequential decision-making under uncertainty.

Reinforcement learning vs. deep learning

Nature: Reinforcement learning is a learning framework/paradigm focused on decision-making via rewards. Deep learning is a set of techniques using deep neural networks, primarily focused on representation learning and pattern recognition.
Input/Signal: Reinforcement learning learns primarily from sparse rewards experienced during interaction. Deep learning typically learns from large datasets (often labeled or using self-supervision) by minimizing a prediction error based on data patterns.
Goal: Reinforcement learning aims to learn an optimal policy for action selection. Deep learning aims to learn complex mappings or useful data representations.
Core Problem: Reinforcement learning tackles the credit assignment problem (linking actions to delayed rewards). Deep learning tackles the representation learning problem (finding meaningful ways to interpret complex data).

Deep reinforcement learning: Where perception meets decision-making

The Need: Traditional reinforcement learning methods often struggle when the environment's "state" is complex and high-dimensional, like the raw pixels from a video game screen or sensor data from a robot's camera. How does the agent efficiently process this flood of data to understand the situation and make good decisions?
The Solution: Deep reinforcement learning uses deep neural networks as powerful function approximators inside the reinforcement learning loop. The neural network acts as the agent's "brain," processing raw sensory input to understand the state and then outputting decisions (actions or value estimates).
How it Works:
Perception: A deep network (like a CNN for images) takes the raw state observation (e.g., screen pixels) as input.
Understanding: The network's layers automatically extract relevant features to comprehend the current situation.
Decision/Evaluation: The network's output informs the reinforcement learning algorithm, perhaps by estimating the value (expected future reward) of taking each possible action (as in Deep Q-Networks - DQN) or by directly outputting the probability of taking each action (as in policy gradient methods like PPO).

The Atari Breakthrough: Learning from Pixels



The Setup: Imagine an AI agent connected to an Atari emulator. Its only input is the raw pixel data from the game screen (e.g., a 84x84 pixel image) and the current game score. It has no prior knowledge of the game's rules, objectives, or even what objects like paddles, balls, or aliens are.
The Agent: A Deep Q-Network algorithm was used. This combined a CNN (to process the pixels) with a Q-learning reinforcement learning approach. The CNN learned to interpret the visual patterns on the screen, and the Q-learning part learned which joystick actions (left, right, fire, etc.) would lead to higher scores based on the CNN's interpretation.
The Learning: Initially, the agent's actions are random – like a baby flailing at the controls. It might accidentally hit the ball in Breakout or shoot an alien in Space Invaders and see the score increase (a positive reward). Over millions of frames and thousands of gameplay sessions, through trial and error guided by the score changes, the DRL agent learned incredibly effective strategies. In Breakout, it famously learned to tunnel the ball behind the bricks – a strategy known to human players, but discovered entirely autonomously by the AI.
The Result: This single deep reinforcement learning algorithm learned to play dozens of different Atari games, achieving superhuman performance on many of them, purely from pixels and score. This was monumental – demonstrating that an AI could learn complex control policies directly from high-dimensional sensory input without manual feature engineering.



Why deep reinforcement learning is crucial today

Robotics: Enabling robots to learn complex manipulation skills using vision or tactile sensors.
Autonomous Driving: Helping vehicles make better driving decisions based on complex sensor fusion data.
Advanced Game AI: Creating more realistic and challenging opponents or collaborators in complex games (e.g., StarCraft II, Dota 2).
Optimization: Solving complex resource allocation or scheduling problems in simulations.
LLM Alignment: As mentioned, algorithms like Proximal Policy Optimization (PPO), a robust DRL policy gradient method, are core components of the RLHF process used to fine-tune models like ChatGPT to be more helpful and follow instructions based on learned human preferences. While newer techniques like Direct Preference Optimization (DPO) aim to achieve similar alignment goals more directly (often without an explicit reward model and subsequent RL step), the foundation laid by DRL in learning from feedback (human or environmental) remains central to creating aligned AI systems.

Conclusion

We saw how supervised learning acts like a teacher, guiding models with labeled examples to make predictions or classifications.
We explored reinforcement learning as a process of trial-and-error, where agents learn optimal strategies by interacting with an environment and chasing rewards.
And we delved into deep learning, a powerful subset of ML using deep neural networks to automatically extract complex patterns, especially from vast, unstructured data like images and text.