Atari RL: A2C vs. PPO vs. DQN on the Arcade Learning Environment

My son Solen and I built this for his 8th grade science project in 2024. It trains three classic deep RL algorithms, A2C, PPO, and DQN, on ten Atari 2600 games under identical conditions and compares how they play. Every agent gets the same preprocessing, the same 20 million training timesteps, the same seed, and the same evaluation protocol, so the differences you see come down to the algorithm.

The project is built on Stable-Baselines3 and a lightly forked RL Baselines3 Zoo. It adds a programmatic training launcher that can fan runs out across a Dask cluster, an evaluation harness that replays trained agents and records per-episode statistics, and the trained A2C and PPO agents themselves.

Contents

• About this project
• Results
• Methodology
• How the code works
• Repository layout
• Getting started
• Trained agents
• Known issues and caveats
• References

About this project

This started as Solen's 8th grade science fair project in early 2024, and the two of us built it together. The science question was easy to state and harder to answer than it looked: if you give three well-known reinforcement learning algorithms the exact same games, the same amount of practice, and the same rules, which one learns to play best?

To answer it we picked ten Atari 2600 games with different styles of play (shooters, maze games, paddle games, and puzzle games), trained A2C, PPO, and DQN on each one for 20 million timesteps, and then had every trained agent play for a long stretch while we recorded its scores. The charts and tables under Results are what came out of that.

The code is the training and evaluation harness we wrote around Stable-Baselines3 and RL Baselines3 Zoo. It is a real experiment pipeline, so the rest of this README documents it like one and anyone can re-run the experiment. Just keep in mind it was made for a middle school science fair: every agent was trained once, with one seed, and the conclusions should be read in that spirit.

Results

Setup at a glance

Algorithms	A2C, PPO, DQN (Stable-Baselines3 2.2.1 implementations)
Games	Asterix, Assault, Asteroids, Breakout, Centipede, MsPacman, Pong, Qbert, Seaquest, SpaceInvaders
Environment	Gymnasium ALE/<Game>-v5 with the SB3 AtariWrapper and 4-frame stacking
Training budget	20,000,000 timesteps per agent, seed 43
During-training eval	5 episodes every 100,000 steps; the best checkpoint is kept as best_model.zip
Post-training eval	best_model.zip replayed with stochastic actions across vectorized environments; every completed game's score, length, and frame count is logged
Tracking	Weights & Biases plus TensorBoard

Headline findings

• The policy-gradient methods usually win. A2C or PPO leads on eight of ten games. DQN is the clear leader only on Pong and roughly ties PPO on Assault.
• A2C is surprisingly strong on high-score games. It dominates Asterix and Centipede by a wide margin and edges out PPO on Qbert and Breakout.
• PPO is the most consistent. It is first or tied for first on seven games and never collapses the way A2C does on Seaquest.
• Seaquest exposes a degenerate A2C policy. Its smoothed score is a flat line at about 1,600 with almost no variance, a classic sign that the agent found a safe, repetitive behavior and stopped exploring.
• Nobody solved Breakout or Pong. Scores of 30 to 45 on Breakout and under +12 on Pong are far below what these algorithms reach with standard NoFrameskip-v4 preprocessing. See the frame-skip note in Known issues and caveats for a likely reason.

Scoreboard

Approximate smoothed episode-score levels read from the evaluation charts below. These are single-seed results and should be treated as indicative, not statistically significant.

Game	A2C	PPO	DQN	Leader
Asterix	~12,000	~4,000	~4,000	A2C
Assault	~2,500	~3,500	~3,500	PPO / DQN
Asteroids	~1,800	~1,800	~600	A2C / PPO
Breakout	~40	~35	~32	A2C (slight)
Centipede	~25,000	~10,000	~5,000	A2C
MsPacman	~4,200	~4,000	~3,200	A2C / PPO
Pong	~-2	~3	~10	DQN
Qbert	~27,500	~26,500	~15,000	A2C (slight)
Seaquest	~1,600	~3,200	~2,300	PPO
SpaceInvaders	~1,050	~1,050	~600	A2C / PPO

Evaluation plots

Each chart shows evaluation episode scores for the three agents on one game. The x-axis is the cumulative ALE frame count of the evaluation run (about 1.6 million frames, corresponding to 100,000 agent steps per environment). Bold lines are Weights & Biases smoothed curves; the faint traces behind them are raw per-episode scores. Scores are true game scores recorded by the Monitor wrapper before reward clipping, and each point is a full game (all lives), not a single life.

Asterix A2C runs roughly three times higher than PPO and DQN, which overlap near 4,000.	Assault PPO and DQN interleave around 3,500. A2C settles about 1,000 points lower.
Asteroids A2C and PPO are indistinguishable at about 1,800. DQN trails at roughly a third of that.	Breakout All three plateau between 30 and 45. A2C edges ahead with occasional spikes, but none learned the high-scoring tunnelling strategy.
Centipede A2C dominates with high variance (20,000 to 30,000). PPO holds about 10,000 and DQN about 5,000.	MsPacman A2C and PPO trade places around 4,000. DQN stays about 1,000 points lower.
Pong DQN is the clear winner at about +10. PPO is modestly positive and A2C never gets above zero. No agent approached the 21-point ceiling.	Qbert A2C and PPO both hold 26,000 to 28,000, with PPO showing deeper dips. DQN sits near 15,000.
Seaquest PPO leads at 3,000 to 3,300. DQN is in the middle. A2C flatlines at about 1,600 with almost no variance.	SpaceInvaders A2C and PPO are neck and neck around 1,000 to 1,100. DQN trails near 600.

Methodology

Environments and preprocessing

All games use the Gymnasium ALE/<Game>-v5 environment IDs provided by ale-py. Each environment is wrapped with the Stable-Baselines3 AtariWrapper, which applies the standard DeepMind preprocessing pipeline:

• random number of no-op actions on reset
• 4-frame action repeat with max-pooling over the last two frames
• episode termination on life loss (training only; evaluation records full games)
• resize to 84x84 grayscale
• reward clipping to {-1, 0, +1} (training only)

Observations are then stacked 4 deep with VecFrameStack so the policy can infer motion. All three algorithms use the SB3 CnnPolicy (the Nature DQN convolutional network).

Hyperparameters

Hyperparameters live in configs/<algo>.yml and are shared across all ten games via the atari: key, which RL Zoo applies to any Atari environment. They are the RL Zoo defaults with the training budget raised to 20 million steps.

Setting	A2C	PPO	DQN
Policy	CnnPolicy	CnnPolicy	CnnPolicy
Timesteps	20,000,000	20,000,000	20,000,000
Parallel envs	16	16	1
Frame stack	4	4	4
Learning rate	SB3 default (7e-4)	2.5e-4, linear decay	1e-4
Entropy coef	0.01	0.01	n/a
Value coef	0.25	0.5	n/a
Rollout length	SB3 default (5)	128 steps x 16 envs	n/a
Epochs / batch	n/a	5 epochs, batch 256	batch 32
Clip range	n/a	0.1, linear decay	n/a
Optimizer	RMSpropTFLike, eps 1e-5	Adam	Adam
Replay buffer	n/a	n/a	100,000
Learning starts	n/a	n/a	1,000,000 steps
Target update	n/a	n/a	every 1,000 steps
Exploration	n/a	n/a	linear 1.0 to 0.01 over first 10% of training

A few things worth noticing:

• A2C uses the TensorFlow-style RMSprop (RMSpropTFLike) to match the original A3C/A2C paper's optimizer behavior. SB3's default PyTorch RMSprop has a different epsilon placement and tends to be less stable on Atari.
• PPO's lin_ prefix tells RL Zoo to decay the learning rate and clip range linearly to zero over training. Both are standard tricks from the PPO paper's Atari experiments.
• DQN waits 1,000,000 steps before learning starts, which is 5% of the training budget spent filling the replay buffer with random play. RL Zoo's default is 100,000.

Training protocol

Each (algorithm, game) pair is one run. For every run the launcher:

1. Builds a TrainArgs record from the shared constants (seed, eval frequency, save frequency, device, Weights & Biases project).
2. Hands it to RL Zoo's ExperimentManager, which reads the config, builds the wrapped environments, constructs the SB3 model, and attaches callbacks.
3. Trains for 20 million timesteps. An EvalCallback runs 5 evaluation episodes every 100,000 steps and overwrites best_model.zip whenever mean reward improves. A CheckpointCallback saves a snapshot every 500,000 steps.
4. Writes the final model, the exact hyperparameters used (config.yml), and the launch arguments (args.yml) into agents/<algo>/ALE-<Game>-v5_1/.

Training was tracked to Weights & Biases (with TensorBoard logs patched through) and ran on cloud GPU instances. The saved models record the environment they were trained in: Python 3.10, PyTorch 2.0.1, Stable-Baselines3 2.2.1, Gymnasium 0.29.1, CUDA enabled.

Evaluation protocol

Post-training evaluation is handled by utils/infer.py, driven from runs/eval/eval.py or the notebooks under runs/eval/. For each agent it:

1. Locates best_model.zip for the (algorithm, game) pair and loads the saved hyperparameters so the test environment matches training.
2. Builds n_envs vectorized environments with the same AtariWrapper and frame stacking, but with the Monitor wrapper on the raw environment so it records unclipped, full-game scores.
3. Runs n_steps agent steps per environment with stochastic action selection (the Atari convention; deterministic argmax policies get stuck in loops).
4. Captures every completed episode's score, length, wall time, and ALE frame counters into an InferLog.
5. Saves the results to data/evals/<ALGO>_<timestamp>/<game>.parquet and optionally logs them to a Weights & Biases evaluation project, which is where the plots above were exported from.

The plotting notebooks in utils/ read the parquet files back and build grouped bar charts of mean score, score standard deviation, mean episode length, and episode counts per algorithm and game using Plotly.

How the code works

Training pipeline

flowchart LR
    A["runs/train_dev.py<br/>games, algos, seed,<br/>paths, wandb settings"] --> B["create_runs()"]
    B --> C{dask_on?}
    C -- no --> D["TrainRun.train()<br/>runs sequentially"]
    C -- yes --> E["dask.delayed tasks<br/>scheduler at localhost:8786"]
    E --> D
    D --> F["TrainArgs<br/>(rl_zoo3 fork)"]
    F --> G["ExperimentManager<br/>reads configs/algo.yml<br/>builds AtariWrapper envs"]
    G --> H["SB3 model<br/>A2C / PPO / DQN"]
    H --> I[("agents/algo/ALE-Game-v5_1/<br/>best_model.zip, evaluations.npz,<br/>monitor csvs, args.yml, config.yml")]
    H --> J["wandb + TensorBoard"]
    classDef default fill:#f4f4f4,stroke:#333,color:#000;

Loading

atari_rl/train.py is the heart of the project:

• create_runs(algos, games, ...) takes lists of algorithm names and game names plus every RL Zoo training option as keyword arguments. It builds the Cartesian product of algorithms and games, creates one TrainRun per pair, and either trains them in order or wraps each in a Dask delayed task keyed by "<ALGO>_<Game>" for parallel execution.
• TrainRun is a dataclass holding the run index, run key, and TrainArgs. Its train() method validates the environment ID against the Gymnasium registry (suggesting the closest match if it is misspelled), seeds everything, initializes a Weights & Biases run if tracking is on, constructs the ExperimentManager, and runs training. If optimize_hyperparameters is set it runs an Optuna study instead.
• get_dask_client() connects to a scheduler on localhost:8786. You start the scheduler and workers yourself; the code only connects.

The TrainArgs dataclass that TrainRun passes to ExperimentManager is the reason this project depends on a fork of RL Zoo. Upstream ExperimentManager expects an argparse.Namespace from its CLI. The fork adds rl_zoo3/args.py with a typed TrainArgs dataclass and changes the ExperimentManager signature to accept it, which is what makes fully programmatic launches like create_runs possible.

Evaluation pipeline

flowchart LR
    A["runs/eval/eval.py<br/>or runs/eval/*.ipynb"] --> B["infer()"]
    B --> C["get_model_path()<br/>finds best_model.zip"]
    B --> D["_create_test_env()<br/>Monitor + AtariWrapper<br/>+ VecFrameStack"]
    C --> E["model.predict loop<br/>n_envs x n_steps<br/>stochastic actions"]
    D --> E
    E --> F["InferLog<br/>one Episode per finished game"]
    F --> G[("data/evals/ALGO_ts/game.parquet")]
    F --> H["wandb eval project"]
    G --> I["utils/create_plots*.ipynb<br/>Plotly bar charts"]
    classDef default fill:#f4f4f4,stroke:#333,color:#000;

Loading

utils/infer.py is a reworked version of RL Zoo's enjoy.py that returns structured data instead of printing it. Its three dataclasses are the output contract:

• Episode: lives remaining, episode frame count, score, length in steps, wall time, and the cumulative ALE frame number at which the episode ended.
• InferLog: one per vectorized environment, accumulates Episode records and exposes scores, lengths, times, frame_numbers, and run_frame_numbers as lists.
• Args: a frozen snapshot of the evaluation settings plus a derived run name and timestamp.

Repository layout

atari-rl/
├── atari_rl/
│   ├── __init__.py          # exports create_runs, get_dask_client
│   └── train.py             # create_runs, TrainRun, Dask client
├── configs/
│   ├── a2c.yml              # shared Atari hyperparameters per algorithm
│   ├── dqn.yml
│   └── ppo.yml
├── runs/
│   ├── train_dev.py         # working training launcher (edit constants, run)
│   ├── dev.py               # CUDA availability check
│   ├── dev.ipynb            # lists registered ALE environment IDs
│   └── eval/
│       ├── eval.py          # batch evaluation across algos and games
│       ├── eval.ipynb       # same, writes parquet + wandb
│       ├── eval_old.ipynb
│       └── algos/           # per-algorithm evaluation notebooks
│           ├── a2c.ipynb
│           ├── dqn.ipynb
│           └── ppo.ipynb
├── utils/
│   ├── infer.py             # evaluation harness (loads agent, plays, logs episodes)
│   ├── create_plots.ipynb   # aggregate parquet results into Plotly charts
│   ├── create_plots_2.ipynb
│   ├── wandb_export.ipynb   # pull run history from the wandb API
│   └── wandb_update.ipynb
├── agents/                  # trained A2C and PPO agents (see below)
│   ├── a2c/ALE-<Game>-v5_1/
│   └── ppo/ALE-<Game>-v5_1/
├── plots/                   # evaluation score charts, one per game
├── scripts/commands.txt     # Docker commands used for the GPU container
├── breakout_eval.ipynb      # quick look at evaluations.npz for Breakout
├── train.py                 # original Colab launcher (stale, see Known issues)
└── requirements.txt

Getting started

Prerequisites

• Python 3.10 (the version the agents were trained and saved with)
• A CUDA-capable GPU is strongly recommended for training; evaluation runs fine on CPU
• Atari ROMs, which are installed automatically by the gymnasium[atari,accept-rom-license] extra

Installation

git clone git@github.com:sequenzia/atari-rl.git
cd atari-rl

python -m venv .venv && source .venv/bin/activate   # or: uv venv && source .venv/bin/activate
pip install -r requirements.txt                      # or: uv pip install -r requirements.txt

# Required: the RL Zoo fork that provides rl_zoo3.args.TrainArgs
pip install git+https://github.com/sequenzia/rl-baselines3-zoo.git

# Optional: SB3 fork with friendlier evaluation log formatting (no functional changes)
pip install git+https://github.com/sequenzia/stable-baselines3.git

The RL Zoo fork is not optional. atari_rl/train.py imports rl_zoo3.args.TrainArgs, which does not exist upstream, and passes it to a modified ExperimentManager. Installing upstream rl_zoo3 will fail at import time.

Note that the repository tracks about 700 MB of trained model weights under agents/, so the initial clone takes a while.

Training

runs/train_dev.py is the working entry point. Edit the constants at the top, then run it:

BASE_DIR = "/path/to/atari-rl"      # absolute path to this checkout

WANDB_PROJECT_NAME = "my-project"   # or set TRACK = False to skip wandb
WANDB_ENTITY = "my-entity"

SEED = 43
SAVE_FREQ = 500000                  # checkpoint every N steps
EVAL_FREQ = 100000                  # evaluate every N steps
EVAL_EPISODES = 5

DEVICE = "cuda"                     # or "cpu"
NUM_THREADS = 1                     # torch threads; 1 is fastest on CPU

GAMES = ["Breakout", "Pong"]        # any ALE game name, without the ALE/ prefix or -v5 suffix
ALGOS = ["ppo"]                     # a2c, ppo, dqn (must have a matching configs/<algo>.yml)

TRACK = True                        # log to wandb
DASK_ON = False                     # run sequentially

python runs/train_dev.py

Each run writes to agents/<algo>/ALE-<Game>-v5_<n>/, where <n> increments if the folder already exists. TensorBoard logs go to logs/tensorboard/.

Parallel training with Dask

With DASK_ON = True, create_runs returns a list of delayed tasks and train_dev.py submits them to a Dask scheduler. Start a scheduler and one or more workers first:

dask scheduler                        # listens on localhost:8786 by default
dask worker localhost:8786 --nworkers 2 --nthreads 1

Each worker trains one run at a time, so with 16 parallel environments per run you will want roughly one worker per GPU or per large CPU allocation.

Programmatic use

create_runs is a regular function, so you can call it from your own script or notebook:

import dask
from atari_rl import create_runs, get_dask_client

runs = create_runs(
    algos=["a2c", "ppo"],
    games=["Breakout", "Qbert"],
    log_folder="agents",
    configs_dir="configs",
    tensorboard_log="logs/tensorboard",
    seed=43,
    eval_freq=100_000,
    eval_episodes=5,
    save_freq=500_000,
    device="cuda",
    track=False,
    dask_on=True,
)

with get_dask_client():
    dask.compute(*runs)

Every keyword that RL Zoo's train.py accepts on the command line is available as a keyword argument, including optimize_hyperparameters, n_trials, sampler, and pruner for Optuna sweeps.

Evaluation

runs/eval/eval.py replays trained agents and collects per-episode statistics. Edit MAIN_PATH, ALGOS, GAMES, N_ENVS, and N_STEPS at the top, then:

python runs/eval/eval.py

The notebooks under runs/eval/ do the same thing but also save each (algorithm, game) result to data/evals/<ALGO>_<timestamp>/<game>.parquet and log to a Weights & Biases evaluation project when WANDB_ON = True. The evaluation runs behind the plots in this README used 5 environments and 100,000 steps per environment.

You can also call the harness directly:

from utils.infer import infer

logs = infer(
    run_key="PPO_Breakout",
    env_id="ALE/Breakout-v5",
    algo="ppo",
    game="Breakout",
    agents_path="agents",
    n_envs=2,
    n_steps=10_000,
    no_render=True,
    device="cpu",
)

for log in logs:
    print(f"env {log.env_idx}: {log.n_episodes} episodes, mean score {sum(log.scores) / len(log.scores):.1f}")

Set no_render=False to watch the agent play in a window.

Plots

utils/create_plots.ipynb and utils/create_plots_2.ipynb glob data/evals/**/*.parquet, concatenate everything into one DataFrame, and produce grouped Plotly bar charts (mean score, normalized mean score, score standard deviation, mean episode length, episode count) by algorithm and game. The per-game line charts in plots/ were exported from the Weights & Biases evaluation project.

Trained agents

The repository ships the trained A2C and PPO agents for all ten games under agents/. Each run directory follows RL Zoo's layout:

agents/ppo/ALE-Breakout-v5_1/
├── best_model.zip           # checkpoint with the best eval mean reward during training
├── ALE-Breakout-v5.zip      # final model at 20M steps
├── evaluations.npz          # timesteps, results, ep_lengths arrays from EvalCallback
├── 0.monitor.csv ... 15.monitor.csv   # per-env training episode logs (reward, length, time)
└── ALE-Breakout-v5/
    ├── args.yml             # exact launch arguments
    └── config.yml           # resolved hyperparameters

Load one with Stable-Baselines3 directly:

from stable_baselines3 import PPO

model = PPO.load("agents/ppo/ALE-Breakout-v5_1/best_model.zip", device="cpu")

Remember to build the observation environment with the same AtariWrapper and 4-frame VecFrameStack the agent was trained with. The infer function in utils/infer.py does this for you by reading config.yml.

The DQN agents were trained on a separate machine and are not included in this repository, although their evaluation results appear in the plots.

Known issues and caveats

• Effective frame skip is 16, not 4. The ALE/*-v5 environments already repeat each action for 4 frames and enable sticky actions (25% chance of repeating the previous action). SB3's AtariWrapper then adds its own 4-frame MaxAndSkipEnv, which SB3 documents as intended for *NoFrameskip-v4 environments. The evaluation plots confirm this: 100,000 agent steps per environment produce about 1.6 million ALE frames. The agents therefore see the game at a coarser time resolution than published Atari results use, which likely explains the low Breakout and Pong scores. All three algorithms were affected equally, so the comparison between them still holds.
• Single seed. Every agent was trained once with seed 43. Deep RL on Atari has high variance across seeds, so treat the scoreboard as indicative rather than conclusive.
• train.py in the repo root is stale. It was written for Google Colab, hardcodes a /content/... path, and calls create_runs with the parameter names envs and return_delayed, which have since been renamed to games and dask_on. Use runs/train_dev.py instead.
• Hardcoded absolute paths. train_dev.py, eval.py, and the notebooks all hardcode a BASE_DIR or MAIN_PATH. Change these to your checkout location before running.
• rl_zoo3 is not in requirements.txt. It must be installed from the fork as described in Installation.
• agents/ is tracked despite being in .gitignore. The committed agents predate or bypass the ignore rule, so new runs you train locally will be ignored by default unless you force-add them.
• DQN agents are absent. Only A2C and PPO weights are included.
• plots/Aserix.png is the Asterix chart; the filename has a typo.

References

• Mnih et al., 2015. Human-level control through deep reinforcement learning (DQN). Nature
• Mnih et al., 2016. Asynchronous Methods for Deep Reinforcement Learning (A3C; A2C is its synchronous variant). arXiv:1602.01783
• Schulman et al., 2017. Proximal Policy Optimization Algorithms (PPO). arXiv:1707.06347
• Machado et al., 2018. Revisiting the Arcade Learning Environment (sticky actions and the v5 protocol). arXiv:1709.06009
• Stable-Baselines3 and RL Baselines3 Zoo by DLR-RM
• Gymnasium and the Arcade Learning Environment by the Farama Foundation
• Weights & Biases for experiment tracking and Dask for distributed execution