Deep reinforcement learning: Integrating neural networks with RL

 Source

What is deep reinforcement learning?

Source: S. L. Brunton (2021, February 19), Neural Networks for Learning Control Laws

 Source

How do Deep Reinforcement learning agents improve their behavior ?

Agent: The learner (for example, a game AI or robot controller).
Environment: The world in which it operates (such as a traffic network or virtual arena).

State $s_t$: A snapshot of the environment at time t (e.g., camera feed or sensor values).
Action $a_t$: A decision the agent makes (for example, steer left or jump).
Reward $r_{t+1}$: Feedback from the environment indicating success or failure.

Observe state $s_t$
Choose action $a_t$ based on the policy or value estimate.
Execute action and receive reward $r_{t+1}$ and new state
Update neural network parameters via gradient-based methods to improve future estimates.

 Source

The Core Components:

Agent: This is the learner, the AI making decisions (e.g., the self-driving car controller, the game-playing AI).
Environment: This is the world the agent interacts with (e.g., the road network and traffic, the video game world).
State (s): A snapshot of the environment at a particular moment, providing the context for the agent's decision (e.g., current sensor readings, screen pixels). In DRL, this state is often complex and high-dimensional.
Action (a): A choice the agent makes based on the current state (e.g., steer left, accelerate, jump, buy/sell).
Reward (r): A feedback signal from the environment indicating the immediate consequence of the agent's action in that state. It can be positive (a reward for good performance) or negative (a penalty for poor performance).

The Learning Loop (Trial-and-Error):

The agent observes the current state (s) of the environment.
Based on this state, the agent's neural network (its policy or value estimator) decides on an action (a).
The agent performs the action (a).
The environment transitions to a new state (s') and provides a reward (r) back to the agent.
The agent uses this reward signal and the transition information (s, a, r, s') to update its neural network, refining its decision-making process for the future.

Exploration vs Exploitation dilemma

Components and Mathematical Foundations of Reinforcement Learning

Source: S. L. Brunton (2021, February 19), Neural Networks for Learning Control Laws

Policy ($\pi$): This is the agent's strategy or "brain." It defines the agent's behavior. Mathematically, it's a mapping from states to probabilities of selecting each possible action. A deterministic policy maps each state to a single action ($a = \pi(s)$), while a stochastic policy maps each state to a probability distribution over actions ($\pi(a|s) = P(A_t = a | S_t = s)$). In DRL, the policy is often directly represented by the parameters ($\theta$) of a neural network, $\pi_\theta(a|s)$. The goal of learning is essentially to find the optimal policy ($\pi^*$) that maximizes the expected cumulative future reward.
Value Functions: These functions estimate "how good" it is for the agent to be in a particular state, or to take a particular action in a state, under a given policy $\pi$. They are crucial because they allow the agent to look beyond immediate rewards and make decisions based on long-term potential.
The State-Value Function $V^\pi(s)$ is the expected return (sum of discounted future rewards) starting from state $s$ and subsequently following policy $\pi$. Formally: $V^\pi(s) = \mathbb{E}_\pi \left[ \sum_{k=0}^{\infty} \gamma^k r_{t+k+1} | S_t = s \right]$ It answers: "Following policy $\pi$, what is the expected long-term reward from this state $s$?"
The Action-Value Function $Q^\pi(s, a)$ (often called the Q-function) is the expected return starting from state $s$, taking action $a$, and then subsequently following policy $\pi$. Formally: $Q^\pi(s, a) = \mathbb{E}_\pi \left[ \sum_{k=0}^{\infty} \gamma^k r_{t+k+1} | S_t = s, A_t = a \right]$ It answers: "Following policy $\pi$ after taking action $a$ in state $s$, what is the expected long-term reward?" These value functions are interconnected through the famous Bellman equations, which express the value of a state (or state-action pair) in terms of the expected immediate reward plus the discounted value of the successor state(s). This recursive relationship forms the basis for many RL algorithms.

$S$: A finite or infinite set of states.
$A$: A finite or infinite set of actions.
$P$: The state transition probability function, $P(s'|s, a) = P(S_{t+1}=s' | S_t=s, A_t=a)$. This defines the dynamics of the environment.
$R$: The reward function, often defined as the expected immediate reward upon transitioning from state $s$ with action $a$, $R(s, a) = \mathbb{E}[R_{t+1} | S_t=s, A_t=a]$. Sometimes it's defined based on the resulting state as well: $R(s, a, s')$.
$\gamma$: The discount factor ($0 \le \gamma \le 1$). This scalar determines the present value of future rewards. A $\gamma$ close to 0 makes the agent "short-sighted," focusing only on immediate rewards, while a $\gamma$ close to 1 makes it value future rewards highly, essential for long-term planning.

Strengths and Weakness of Deep Reinforcement Learning

Applications of Deep Reinforcement Learning

Conclusion