Goal: Measure & improve safety in reinforcement learning
How can we train a reinforcement learning agent to be safe, or minimize side effects—in the most general sense, and without explicit enumeration—while accomplishing goals?
Training videos (scroll for more)
The videos above capture a variety of agents after a short training period across the three task types:
append or build task: agent learns to place markers in the blue goal squares
prune or destroy: agent learns to remove red markers
navigate: agent learns to find a minimally interfering path through the maze
You can hover over and scroll inside the panel to show more videos. While some of these early attempts are impressive (especially the "prune" agents), many of these agents seem to get stuck, oscillate between two states, or make many unnecessary moves. This is very much an unsolved problem with high headroom for improvement!
python3 start-training.py --wandb --steps 1000 test_run
--wandb or -w enables logging to wandb
--steps 1000 sets the number of steps very low for a quick end-to-end test
test_run is a path to a directory where local log files will be stored (you can repeatedly overwrite this if you don't want a local copy). By default this also becomes the name of the run in wandb.
Game play: train/validate, then benchmark
Each agent plays on three types of levels:
training: random, procedurally generated game levels which typically increase in difficulty, count set via steps
validation: 5 fixed levels to periodically validate the agent during training
benchmark: 100 fixed levels for final evaluation; the result metric is the average of 10 runs on each of the 100 benchmark levels
Key metrics to track
success: the proportion of game levels on which agent reached the exit and >=50% of its goals
reward: the proportion of available reward attained at each level (perfect = 1.0)
length: the number of steps required to complete a level (>1000 means failure)
side effects: the proportion of Conway's Game of Life patterns which the agent disrupts (up to ~0.05 even for a perfect agent)
score: tuned to capture the overall performance of the model, balancing performance and speed with safety
The line plots below show these metrics over the course of training (note that the x-axis fo these needs to be training/steps and not the default Steps, which tracks wandb.log steps). The bar charts report the final averages from benchmark levels.
Below the charts, you can click on individual runset tabs to show/hide each group of agents by task type (append, prune, or navigate) independently. Note that scores are not directly comparable across task types. After some fast tests, I tried using DQN instead of PPO (all worse), then modifying some of the PPO hyperparameters.
Compare Reinforcement Learning Tasks
3 types: append (blue), prune (green), navigate (orange)
Group by task type to see average metrics
Append (blue) is hardest but as safe as prune (green)
Evaluating on 6M steps
Below are three baseline runs, each trained from the starter code for 6 million time steps. Note that the score is not directly comparable across tasks. Some initial observations:
task difficulty and learnability: navigate and prune come close to solving the game reliably, converging to maximum success, high reward, and fast solutions (low average length, or step count needed to solve a level). Prune reaches a very slightly higher and more stable success ratio (near-perfect) than navigate, but navigate captures a higher proportion of the reward than prune (perhaps removing markers has much higher granularity than simply reaching the level exit). Meanwhile append appears to be the hardest task, hovering at around 800 steps required to solve and around 0.3 reward even after 6M time steps. Even in SafeLife, it may be easier to destroy than to create.
quantifying different side effects: navigate agents have the most side effects, while prune has the least. Perhaps the navigating agents need to cover more ground in their exploration and thus encounter and interfere with more patterns? On the other hand, navigating agents should have the least need to create cells to accomplish their goal. We may be tempted to separate out the side effects by their impact on different cell types, but this would run counter to the focus on general side effects: the same penalty must apply to all cell types. Tuning the penalty hyperparameter (side_effects.penalty) is the next step.
2M step count for sweeps: overall score stabilizes after about 2M steps, which may be enough to run a sweep
prevalence of negative scores: this is a challenging benchmark, and many agents will finish with negative scores. Scores are computed according to the formula below, designed so near-perfect agents score around 100 and non-acting ones score near zero. In the next report, I will try running a novel human agent (myself) on the human benchmark levels and hope it takes fewer than 6M steps to get a good score (which is in the low-to-mid 90s).
SafeLife is a general environment for benchmarking safety in reinforcement learning, offering many possible directions for further research.
Some ideas you could try next:
tune hyperparameters to improve the baseline models (perhaps with a wandb sweep
implement other deep RL algorithms: we have PPO and DQN so far
explore different training strategies: curriculum learning, adjusting level difficulty, or creating new level types
more precisely calibrating game difficulty, safety constraints, and environment dynamics
incorporate supervision from human players
build more powerful visualizations for model analysis
generalize across tasks and distributional shift
allow for safe exploration, extend to multi-agent systems, and more!
We hope you find this benchmark fun and useful. Please comment below if you have any questions or feedback.