How much reward accumulated?
01 / Overview
Teaching an Agent to Play PlayStation
Play. Learn.
Measure.
Measure.
Crash Bandicoot: Warped · PS1 · reinforcement learning
- Reusable PlayStation RL infrastructure · Crash Bandicoot: Warped case study
- Pixels in · button presses out · rewards from game memory
- Build the system, inspect the behavior, test the claims
02 / Overview
5 Adventures, 5 Control Problems
Level 1
Level 2
Level 3
Level 4
Level 5
- Toad Village · Under Pressure · Orient Express
- Bone Yard · Makin’ Waves
- Run, jump and spin · swim, ride and steer · reach the exit
03 / Recorded experiments
Orient Express: A Completed Route
Source 00:02 · 1/2
Recorded OutcomeLevel CompleteFrom the full evaluation
Seed 0 · greedy · level complete46s excerpt · Source 00:02–00:22 → 01:10–01:36
Orient Express · a completed route from orient_express-warm-s0.
Selected excerpts at recorded speed · source-time labels show the edits.
- Demonstration initialization · seed 0
- Watch the jumps, obstacles, and final portal approach
- Recorded outcome: level completion
04 / Environment
Make the Game an Experiment
Restore start→Observe pixels→Choose action→Emulate frames
← next observation · reward · termination ←
Policy input: images. Reward measurement: game memory.
Checkpoint + starting state + controller + configuration = an inspectable experiment
- Pixels guide actions; game memory measures reward
- Configured rewards score game events; each level defines its controller
- Reusable emulator interface · game-specific controls, rewards, and reset logic
05 / Environment
What the Network Actually Sees
t − 3 · oldest
t − 2
t − 1
t · newest
Illustration from successive video frames
- Color frame → grayscale → 84 × 84
- 4 sampled observations provide short motion history
- Illustration below uses a frame from recorded gameplay
06 / Learning
Inside the Visual Network
A Convolution Combines Local Evidence
Local pixels
× learned weights
+ bias
→× learned weights
+ bias
Feature map
Schematic illustration, not measured learned filters or activations
CNN backbone adapted from Mnih et al. (2015) · output heads adapted for PS-Env
- CNN backbone adapted from Mnih et al. (2015), including David Silver
- Deeper layers combine features across larger image regions
- The output assigns a value to each available action
07 / Learning
Double DQN and Dueling
- Our gameplay trials use both: Double DQN targets + dueling action-value heads
- Double DQN: select with the online network, evaluate with the target network
- Dueling: combine state value and action advantages into 1 Q-value per action
08 / Measurement
Reward Needs Inspection
Where did the policy go?
Did it reach the verified ending?
- Game events → reward components
- Return, route coverage and completion answer different questions
- Watch behavior alongside the numbers
09 / Experiments
Experiments
- From Scratch
- Learn from the agent’s own experience.
- Warm Start
- Learn recorded actions first; then use own experience.
- DQN With Demonstrations
- Warm start + examples and imitation loss during training.
- Reverse Curriculum
- Same learner + practice later sections first, then earlier.
- Setup
- 1 CNN trained across levels, with a separate action head for each.
- Training
- DQN with demonstrations; balanced samples from each level.
- Question
- What can the same model do on each level?
- Change
- 1, 2, or 4 active GPUs.
- Hold Fixed
- Global decisions, learning updates, and batch size.
- Measure
- Training time for the same workload.
Gameplay is evaluated separately on each level: completion, progress, and failure mode.
- Training methods change how demonstrations and practice starts are used
- Shared-policy training combines experience across levels
- GPU scaling measures time at a fixed learning workload
10 / Experiments
DQN With Demonstrations
Observation, action, reward, next observation
The agent continues interacting with the game
Sample both sources to update the policy
Behavior-cloning initialization → mixed replay + margin loss → independent evaluation
Adapted from DQfD · Hester et al. (2018) · PS-Env uses behavior-cloning pretraining and a fixed replay mixture.
- Demonstrations generated by automated beam search in the emulator
- Mix recorded transitions with the agent’s own experience
- Reward-based loss + imitation loss on demonstration samples
11 / Experiments
Reverse Curriculum Learning
Restore a saved state later in the route
Success moves the practice frontier toward the start
Evaluate completion from the original start
This adaptation: behavior cloning + DQN learner with demonstrations + reverse curriculum
Inspired by Salimans and Chen (2018) · adapted to local section goals and a DQN learner with demonstrations.
- Demonstration-state curriculum · inspired by Salimans and Chen (2018)
- Practice later sections first, then move the starting frontier earlier
- PS-Env adaptation: local section goals plus full-start episodes
12 / Experiments
Shared Architecture Across Adventures
- 1 shared CNN and 512-unit representation
- Level identifier selects a separate action-value head
- Balance training samples; test the trained model on each level
13 / Recorded experiments
Under Pressure: Progress Without Completion
Source 00:02 · 1/2
Recorded OutcomeDeathFrom the full evaluation
Seed 0 · greedy · death32s excerpt · Source 00:02–00:14 → 01:22–01:42
Source 00:14 · 1/2
Recorded OutcomeDecision LimitFrom the full evaluation
Seed 0 · greedy · decision limit30s excerpt · Source 00:14–00:32 → 09:48–10:00
Source 00:14 · 1/2
Recorded OutcomeDecision LimitFrom the full evaluation
Seed 0 · greedy · decision limit30s excerpt · Source 00:14–00:32 → 09:48–10:00
Under Pressure · an underwater adventure. Reaching the reference-route milestones does not establish level completion.
Selected excerpts at recorded speed · source-time labels show the edits.
- Scratch: movement around rocks and crates; ends in death
- Curriculum and demonstrations: similar machinery-and-cavern route
- Both reach all 8 reference milestones, but not level completion
14 / Recorded experiments
Orient Express: Progress Versus Completion
Source 00:03 · 1/2
Recorded OutcomeDeathFrom the full evaluation
Seed 0 · greedy · death32s excerpt · Source 00:03–00:15 → 01:43–02:03
Source 00:08 · 1/2
Recorded OutcomeLevel CompleteFrom the full evaluation
Seed 0 · greedy · level complete34s excerpt · Source 00:08–00:20 → 01:14–01:36Same recorded trajectory as slide 3
Scratch collects new experience; warm start initializes weights; DQN with demonstrations reuses demonstrations; reverse curriculum changes practice starts.
Selected excerpts at recorded speed · source-time labels show the edits.
- Scratch: moves along the route, but the evaluation ends in death
- Curriculum export: reaches the final portal; completion logged
- Right-hand video repeats the trajectory from slide 3
15 / Shared learning
1 Shared Policy, Different Outcomes
Source 00:12 · 1/2
Recorded OutcomeLevel CompleteFrom the full evaluation
Seed 1 · greedy · level complete30s excerpt · Source 00:12–00:24 → 01:18–01:36Same recorded trajectory as slide 3
Source 00:14 · 1/2
Recorded OutcomeDecision LimitFrom the full evaluation
Same experiment · greedy · decision limit28s excerpt · Source 00:14–00:30 → 09:48–10:00
1 shared encoder, level-specific action heads. 1 level completing does not imply 5-level mastery.
Selected excerpts at recorded speed · source-time labels show the edits.
- Orient Express: completion logged; repeated trajectory from slide 3
- Under Pressure: movement continues, but the decision budget runs out
- Evaluate the shared model separately on each level
16 / Results
Same Workload, Faster Training
Fixed Workload · 200,000 Global Decisions · 7,501 Learning Updates
Mean recorded training interval · 2 seeds · provisioning and separate evaluation excluded
3.32× training speedup · same workload
- Same work: 200,000 global decisions and 7,501 learning updates
- 4 GPUs: 54.7 → 16.5 minutes, about 3.32× faster
- Training time only · same host · 2 seeds per GPU setting
17 / System overview
How Playing Becomes Learning
- Images → action values → button presses
- Experience → training updates → a changed policy
- Save the policy, then test what it learned
18 / Next steps
A Childhood Game, a New Starting Point
Real game interaction, saved policies, and inspectable runs.
3.32× speedup on 4 GPUs for the tested workload.
A separate demonstration-trained policy finished Orient Express.
A working foundation for learning to play further—and measuring every step.
- Built a working platform for repeatable, inspectable gameplay experiments
- Measured 3.32× faster training and recorded an Orient Express completion
- Next: turn this foundation into more reliable play across adventures