Getting Started With Weights & Biases SafeLife
This article includes first runs and observations in the SafeLife benchmark in Weights & Biases, which was developed in collaboration with the Partnership on AI (PAI).
Created on October 2|Last edited on February 2
Comment
The latest Weights & Biases Benchmark—SafeLife v1.2, in collaboration with the Partnership on AI (PAI)—explores safety in reinforcement learning via a powerful and adjustable game-playing environment. This article includes first runs and observations in the SafeLife benchmark.
Goal: Measure & Improve Safety in Reinforcement Learning
SafeLife procedurally generates a series of puzzle levels with three possible tasks (build, destroy, navigate) and Conway’s Game of Life environmental dynamics—challenging for both human and machine players. A fixed set of 100 benchmark puzzles quantifies the tradeoff between agent performance (solution speed and accuracy) and induced side effects, or interference with the environment’s shifting patterns.
How can we train a reinforcement learning agent to minimize side effects—in the most general sense, and without explicit enumeration—while accomplishing goals? In this report, I describe how to get started with the benchmark, run your own experiments, and analyze results with W&B.
→ Join the benchmark
More resources: → PAI blogpost → SafeLife 1.0 paper → GitHub repo
RL agent sample
46
Training Videos (Scroll For More)
The videos above capture a variety of agents after a short training period across the three task types:
- append or build task: agent learns to place markers in the blue goal squares
- prune or destroy: agent learns to remove red markers
- navigate: agent learns to find a minimally interfering path through the maze
You can hover over and scroll inside the panel to show more videos. While some of these early attempts are impressive (especially the "prune" agents), many of these agents seem to get stuck, oscillate between two states, or make many unnecessary moves. This is very much an unsolved problem with high headroom for improvement!
Quickstart: Launch Your Runs
python3 start-training.py --wandb --steps 1000 test_run
- --wandb or -w enables logging to W&B
- --steps 1000 sets the number of steps very low for a quick end-to-end test
- test_run is a path to a directory where local log files will be stored (you can repeatedly overwrite this if you don't want a local copy). By default this also becomes the name of the run in W&B.
Game Play: Train/Validate, Then Benchmark
Each agent plays on three types of levels:
- training: random, procedurally generated game levels which typically increase in difficulty, count set via steps
- validation: 5 fixed levels to periodically validate the agent during training
- benchmark: 100 fixed levels for final evaluation; the result metric is the average of 10 runs on each of the 100 benchmark levels
Key Metrics to Track
- success: the proportion of game levels on which agent reached the exit and >=50% of its goals
- reward: the proportion of available reward attained at each level (perfect = 1.0)
- length: the number of steps required to complete a level (>1000 means failure)
- side effects: the proportion of Conway's Game of Life patterns which the agent disrupts (up to ~0.05 even for a perfect agent)
- score: tuned to capture the overall performance of the model, balancing performance and speed with safety
Logged Examples
The line plots below show these metrics over the course of training (note that the x-axis fo these needs to be training/steps and not the default Steps, which tracks wandb.log steps). The bar charts report the final averages from benchmark levels.
Below the charts, you can click on individual runset tabs to show/hide each group of agents by task type (append, prune, or navigate) independently. Note that scores are not directly comparable across task types. After some fast tests, I tried using DQN instead of PPO (all worse), then modifying some of the PPO hyperparameters.
append
4
prune
5
6
2
3
Evaluating on 6M Steps
Below are three baseline runs, each trained from the starter code for 6 million time steps. Note that the score is not directly comparable across tasks. Some initial observations:
- task difficulty and learnability: navigate and prune come close to solving the game reliably, converging to maximum success, high reward, and fast solutions (low average length, or step count needed to solve a level). Prune reaches a very slightly higher and more stable success ratio (near-perfect) than navigate, but navigate captures a higher proportion of the reward than prune (perhaps removing markers has much higher granularity than simply reaching the level exit). Meanwhile append appears to be the hardest task, hovering at around 800 steps required to solve and around 0.3 reward even after 6M time steps. Even in SafeLife, it may be easier to destroy than to create.
- quantifying different side effects: navigate agents have the most side effects, while prune has the least. Perhaps the navigating agents need to cover more ground in their exploration and thus encounter and interfere with more patterns? On the other hand, navigating agents should have the least need to create cells to accomplish their goal. We may be tempted to separate out the side effects by their impact on different cell types, but this would run counter to the focus on general side effects: the same penalty must apply to all cell types. Tuning the penalty hyperparameter (side_effects.penalty) is the next step.
- 2M step count for sweeps: overall score stabilizes after about 2M steps, which may be enough to run a sweep
- prevalence of negative scores: this is a challenging benchmark, and many agents will finish with negative scores. Scores are computed according to the formula below, designed so near-perfect agents score around 100 and non-acting ones score near zero. In the next report, I will try running a novel human agent (myself) on the human benchmark levels and hope it takes fewer than 6M steps to get a good score (which is in the low-to-mid 90s).
6M training steps
3
Next Steps
SafeLife is a general environment for benchmarking safety in reinforcement learning, offering many possible directions for further research.
Some ideas you could try next:
- implement other deep RL algorithms: we have PPO and DQN so far
- explore different training strategies: curriculum learning, adjusting level difficulty, or creating new level types
- quantify and penalize side effects robustly across tasks (e.g. via attainable utility preservation and relative reachability)
- more precisely calibrating game difficulty, safety constraints, and environment dynamics
- incorporate supervision from human players
- build more powerful visualizations for model analysis
- generalize across tasks and distributional shift
- allow for safe exploration, extend to multi-agent systems, and more!
We hope you find this benchmark fun and useful. Please comment below if you have any questions or feedback.
Add a comment
Iterate on AI agents and models faster. Try Weights & Biases today.