# PS-Env — recording script

Audience: technical colleagues with basic machine-learning familiarity. Rehearse once with the clips and pauses; the timer is a pacing aid, not an automatic cutoff.

Open https://psenv.com/recording. Sign in before recording. Use the Speaker window on a second screen, and record only the presentation window. Right/left arrows change sections; F toggles fullscreen. Start the timer when you begin speaking.

## Before recording

- Turn on Do Not Disturb, close unrelated tabs, and test your microphone.
- Use a 1920×1080 display if available. Check that text and videos are readable.
- Each video slide has a Play Clips button: it plays the reviewed excerpts left to right. No manual seeking is needed.
- For the shorter recording, narrate during the clips. The pacing guide is a rehearsal target, not a guaranteed duration.
- Clips retain their recorded speed. Source-time labels disclose cuts; the full recordings remain archived.
- Download the offline kit, which includes this script, 8 clips, level pictures and neural-network diagrams.
- Historic results are curated evidence, separate from the now-empty experiment workspace.

## Run of show

| Time | Section |
|---|---|
| 00:00–00:50 | Teaching an Agent to Play PlayStation |
| 00:50–01:55 | 5 Adventures, 5 Control Problems |
| 01:55–03:20 | Orient Express: A Completed Route |
| 03:20–05:10 | Make the Game an Experiment |
| 05:10–06:20 | What the Network Actually Sees |
| 06:20–07:45 | Inside the Visual Network |
| 07:45–09:10 | Double DQN and Dueling |
| 09:10–09:55 | Reward Needs Inspection |
| 09:55–11:25 | Experiments |
| 11:25–13:10 | DQN With Demonstrations |
| 13:10–14:30 | Reverse Curriculum Learning |
| 14:30–16:00 | Shared Architecture Across Adventures |
| 16:00–18:25 | Under Pressure: Progress Without Completion |
| 18:25–20:15 | Orient Express: Progress Versus Completion |
| 20:15–22:00 | 1 Shared Policy, Different Outcomes |
| 22:00–23:30 | Same Workload, Faster Training |
| 23:30–24:50 | How Playing Becomes Learning |
| 24:50–25:45 | A Childhood Game, a New Starting Point |

## 00:00–00:50 · Teaching an Agent to Play PlayStation

**On screen:** Introduce the project, then advance to the 5 adventures before showing the success video.

Hi, I’m Shehab. **Crash Bandicoot: Warped is a game I played as a kid.** That gives this project a personal connection: I’m coming back to a familiar game, but this time I’m asking whether a learning agent can figure out how to play it.

PS-Env provides **reusable infrastructure for reinforcement learning with PlayStation games**. Crash Bandicoot: Warped is our current implementation and case study. A program looks at images, chooses button presses, and **learns from the consequences**.

There are 2 questions behind the project: can we make learning run efficiently, and does the agent actually play well? First, let’s meet the adventures, then watch the agent play.

## 00:50–01:55 · 5 Adventures, 5 Control Problems

**On screen:** Show the 5 level pictures. Spend about 10 seconds introducing each setting, then explain why they need separate evaluations.

For anyone unfamiliar with the game, you control Crash through routes filled with obstacles, enemies, crates, and collectible Wumpa fruit. You run, jump, and spin in the on-foot sections, while other adventures change how you move and what the controller needs to do. **The aim is to get through the route and reach the level’s exit.**

Before the model, let’s meet the adventures. Toad Village takes us along a village route, with houses, crates and hazards competing for attention. Under Pressure moves underwater: the scene and movement cues feel very different. Orient Express is a fast ride where the timing of a decision matters. Bone Yard brings a prehistoric setting, while Makin’ Waves changes the task to steering a watercraft through a route.

As we watch, look for **progress, reactions, and recovery**. The excerpts run at their recorded speed. Source-time labels show where we cut forward, and the full recordings remain archived.

## 01:55–03:20 · Orient Express: A Completed Route

**On screen:** Play the 46-second excerpt. Introduce the route, let the key jumps and ending play, then state the recorded outcome. Narrate while the clips play to stay within the pacing target.

**Watch the route, then the ending.** The policy follows the wall, jumps across gaps, and passes obstacles. In the later excerpt, it reaches the final approach and the exit portal. This saved evaluation logged a level completion.

The source clock jumps from the opening part of the route to the later part. That is an edit in the excerpt; the action stays at its recorded speed.

This policy was initialized from recorded action examples. The result is a completion for this saved model from its matching start, not proof that it has mastered all adventures.

## 03:20–05:10 · Make the Game an Experiment

**On screen:** Trace the diagram left to right. Pause briefly at reset, observation and reward. Explain that the game runs inside the emulator.

The starting point is an emulator. PS-Env uses the SwanStation libretro core, with a small native wrapper that can press buttons, capture frames, read game memory, and save or restore state. Python exposes a reset-and-step interface to the learner. **The emulator interface is reusable; each game needs its own integration** for controls, rewards, starting states, and completion detection. The results here are all from Crash Bandicoot: Warped.

Reset restores a known starting state. Step applies an action for a configured number of game frames, then returns the next observation, a numerical reward, and whether the attempt has ended. That makes the experiment repeatable: we can compare agents from the same starting point, and we can record exactly which actions they took.

How do we tell the learner what counts as a useful action? **The reward function turns game events into a numerical score.** Depending on the configuration, collecting Wumpa fruit, breaking crates, making new progress, and completing the level can add reward. Losing a life or reaching game over can subtract reward, and a small step penalty can discourage wasting time. The weights and enabled components belong to each experiment’s configuration.

**The policy sees images.** It does not receive Crash’s memory coordinates as a shortcut to choosing actions. We use game memory separately to measure events and compute rewards. This **prevents that privileged-information shortcut**, but does not itself guarantee safety or alignment. We still inspect the policy’s behavior.

Checkpoint plus starting state plus controller plus configuration equals **an inspectable experiment**. A checkpoint alone does not capture the conditions that produced its behavior.

## 05:10–06:20 · What the Network Actually Sees

**On screen:** Point to the color frame and enlarged grayscale image. Explain that the displayed image is a preprocessing illustration, not a learned activation or a claimed training tensor dump.

The game is rendered in color, but the learner uses a smaller representation. We convert a frame to grayscale and resize it to 84 × 84 pixels. That reduces the input size while preserving broad visual structure. The image here illustrates that transformation using a frame extracted from recorded gameplay.

We then stack **4 sampled observations**. A single image may show an obstacle and Crash, but it does not directly tell us how they are moving relative to each other. Changes across the stack provide a short history from which the network can learn motion cues.

These are consecutive sampled observations, not necessarily 4 adjacent game frames. Frame skip determines how much emulated time separates decisions. The input to a batched network call has shape batch by 4 by 84 × 84, and pixel values are scaled before the convolutions.

## 06:20–07:45 · Inside the Visual Network

**On screen:** Trace the architecture from the observation stack to the action-value outputs. Briefly distinguish the shared backbone from the output heads; skip the elementary convolution walkthrough.

Our visual backbone is adapted from the **DQN architecture in Mnih et al., 2015**, with action outputs matched to our controllers and extensions for dueling and shared-policy learning.

The input is a stack of 4 84 × 84 grayscale observations. 3 convolutional layers with ReLU activations reduce spatial resolution while increasing the channel count from 32 to 64. This produces a compact representation with a larger receptive field at each stage.

The final 64 × 7 × 7 tensor is flattened into **3,136 features** and passed to the action-value head. The network outputs 1 estimated Q-value per available action; greedy evaluation selects the highest-valued action.

The convolutional backbone is common to these variants. What changes is the output structure: an ordinary action-value head, dueling value and advantage branches, or adventure-specific heads over a shared representation. **The next slide explains the dueling head and the Double DQN learning target used in our gameplay experiments.**

## 07:45–09:10 · Double DQN and Dueling

**On screen:** Connect both design choices to the gameplay trials first. Then explain the 2 panels; this is background to the experiments, not an ablation of these choices.

**We use both of these in the gameplay experiments shown here.** Our gameplay trials use Double DQN targets with dueling networks, including a dueling head per adventure in the shared model. **We are not comparing Double DQN against dueling.**

2 names in the configuration sound related but describe different design choices. **Double DQN concerns the learning target.** The online network selects which next action looks best, and the target network evaluates that selected action. Separating those roles helps address the tendency to overestimate values when the same noisy estimates both select and evaluate an action.

**Dueling concerns the output architecture.** 1 branch estimates a value for the state, while another estimates relative advantages for the available actions. We combine them by adding the state value and subtracting the mean advantage. The result still contains 1 Q-value per action.

To be clear, **the outputs are values, not button presses or action probabilities**. There is 1 estimated Q-value per available action. The action-selection rule chooses an index, and the controller maps that index to buttons.

## 09:10–09:55 · Reward Needs Inspection

**On screen:** Use the 3 measurement cards. Explain 1 concrete failure: surviving or collecting reward without finishing the route.

But **earning reward is not always playing well**. An agent may repeat a rewarded behavior without finishing. We inspect return, route coverage, and verified completion together with the video.

For example, a policy might stay alive or repeat a rewarded action while making little useful progress. The reward curve can rise even though the behavior we care about has not improved.

Return summarizes the rewards we defined. Route coverage measures progress against a reference. Completion records whether the game actually reached the intended ending. **Those measurements answer different questions.** When they disagree, the recording helps us investigate why.

## 09:55–11:25 · Experiments

**On screen:** Explain the 4 training setups as a progression. Then separate shared-policy training from the GPU timing experiment.

This slide separates **3 experiment types**: training methods, shared-policy training, and GPU scaling.

The 4 training setups build on each other. **From scratch** means no demonstration training: the agent learns from its own interaction. **Warm start** first trains the network to predict demonstrated actions, then continues reinforcement learning using its own experience.

**DQN with demonstrations** adds continued access to the examples. It begins with the same kind of initialization, then mixes demonstration transitions with new experience and applies an imitation loss to the demonstrated actions. The difference is whether demonstrations are used only at the beginning or remain part of training.

**Reverse curriculum learning** adds a practice schedule to that learner. The agent begins from later saved sections and moves earlier as section performance improves, while also practicing full starts. These are cumulative training setups, not 4 unrelated algorithms.

The **shared-policy experiment** changes which levels train the same network. 1 CNN learns from several levels, with a separate action head for each. The recorded shared run uses DQN with demonstrations, and we test each level separately.

The **GPU experiment** changes the number of active GPUs while keeping the global learning workload fixed. Its measurement is training time.

## 11:25–13:10 · DQN With Demonstrations

**On screen:** Explain the plain-language setup, then its relationship to DQfD. Contrast demonstrations used only for initialization with demonstrations used throughout training.

We call this setup **DQN with demonstrations**: the agent learns from rewards while continuing to study recorded examples. This is a descriptive name for our implementation. It draws on **Deep Q-learning from Demonstrations**, or DQfD, by Hester and colleagues, without reproducing the complete published algorithm.

**Who supplies the examples? Automated beam search in the emulator.** It tries alternative action sequences, keeps a limited set of promising surviving branches, and restores save states to explore further. Game-memory information helps score progress and reject failures. Replaying those sequences produces the examples. The search uses game-memory information; the evaluated policy chooses actions from images.

A demonstration contains observations, actions, rewards, and next observations. During reinforcement learning, we sample these recorded transitions alongside experience collected by the agent. A reward-based loss trains action values, and a large-margin imitation loss encourages the demonstrated action to score above alternatives on demonstration samples.

There are implementation differences worth being clear about. Our runs begin with **behavior cloning**: supervised training to predict demonstrated actions. They then use a fixed sampling mixture from separate demonstration and online buffers. Published DQfD uses its own combined pretraining objective and prioritized replay.

At evaluation, the policy chooses actions from images. It is not playing back the recorded buttons. **The experiment asks whether learning from those examples produces useful independent behavior.**

## 13:10–14:30 · Reverse Curriculum Learning

**On screen:** Explain why practice moves backward through demonstration states. Distinguish PS-Env’s local section goals from full-route completion and name the paper as inspiration.

The more precise name here is **reverse curriculum learning from demonstration states**. The idea is related to Salimans and Chen’s work on learning Montezuma’s Revenge from a single demonstration: start practice later in a demonstrated route, then move the starting point backward as the agent improves.

Why reverse? An agent that repeatedly fails near the beginning may never reach later situations often enough to learn them. Restoring a saved state gives it direct access to that practice. Once it succeeds reliably, we move the practice frontier toward the beginning. **The order of practice runs backward; the game still runs forward.**

Our version uses saved route sections and local section goals. The schedule responds to recent section success and also includes full-start episodes. Reaching a section goal is a training success for that section, not a claim that the level is complete.

Unlike the original paper’s setup, we combine this schedule with behavior-cloning initialization and our DQN learner with demonstrations. That is why we call it an adaptation. To isolate the benefit of the curriculum, we would compare against the same learner without section resets.

## 14:30–16:00 · Shared Architecture Across Adventures

**On screen:** Trace the shared feature extractor into the 5 heads. Explain that the level selects a head and that equal sampling does not guarantee equal learning.

This experiment asks whether **1 model can learn across several adventures**. All levels use the same convolutional feature extractor. Its output feeds a shared hidden layer with 512 units. This shared part can learn visual patterns from experience collected across the selected levels.

After that, the network branches into a separate action head for each adventure. A head is the final part that turns the shared representation into action values. The environment supplies the level identifier to select the correct head; the network is not guessing the level from the image.

Why separate heads? The controller choices differ. Toad Village, Under Pressure, and Bone Yard each have 27 configured outputs, Orient Express has 7, and Makin’ Waves has 4. Each dueling head combines a state value with action advantages to produce its action values. **The shared representation is common; the action interface stays specific to the adventure.**

Learning takes equal replay samples from the selected levels and combines their losses into a shared update. The hope is that useful visual features transfer across tasks. The risk is interference: an update that helps 1 adventure may make another worse.

## 16:00–18:25 · Under Pressure: Progress Without Completion

**On screen:** Play 32s, 30s, 30s from left to right. Introduce each clip with its labeled paragraph; let the key action play before the takeaway. Source-time jumps are edits, not uninterrupted gameplay. Narrate while the clips play to stay within the pacing target.

**Left — scratch training.** Crash moves through the open underwater area around rocks and crates, with changes in depth and direction. The full evaluation nevertheless ended in death.

**Middle — reverse curriculum.** The excerpt starts near large machinery, then shows a later rocky passage. The run reached all 8 reference-route milestones, but eventually exhausted its decision budget. That is evidence of reference-route coverage, not a verified level completion.

**Right — DQN with demonstrations.** The opening around the machinery is very similar to the middle clip. In the late excerpt, Crash takes a different path through the same general rocky area. This run also reached 8 of 8 reference milestones and the decision limit.

**There is no clear winner between the middle and right clips here.** Their local movements differ, but their reported coverage and ending agree. The useful contrast is with the scratch attempt’s death, and with the completed route shown earlier: more covered ground does not automatically mean the task is finished.

## 18:25–20:15 · Orient Express: Progress Versus Completion

**On screen:** Play 32s, 34s from left to right. Introduce each clip with its labeled paragraph; let the key action play before the takeaway. Source-time jumps are edits, not uninterrupted gameplay. Narrate while the clips play to stay within the pacing target.

**Left — scratch training.** The policy rides forward and jumps, so at first glance it may look capable. In the selected late segment, it is still negotiating the route; the recorded evaluation ends in death. Its reference-route measurement reached 3 of 12 milestones.

**Right — curriculum export.** Watch the later jumps and the final approach to the portal. This archived evaluation logged completion and 12 of 12 reference milestones. Its training setup included demonstration initialization, continued learning from demonstrations, and section practice.

**This right-hand video is the same source trajectory as slide 3.** It appears again because the archived curriculum export matches that footage. We should not count it as a 2nd distinct visual demonstration of success.

## 20:15–22:00 · 1 Shared Policy, Different Outcomes

**On screen:** Play 30s, 28s from left to right. Introduce each clip with its labeled paragraph; let the key action play before the takeaway. Source-time jumps are edits, not uninterrupted gameplay. Narrate while the clips play to stay within the pacing target.

**Left — Orient Express from the shared experiment.** The excerpt shows route following and a final portal approach, and the archived run logged completion. Its video export matches the successful trajectory already shown on slide 3, so it is repeated footage rather than independent new visual evidence.

**Right — Under Pressure from the same shared experiment.** The opening passes the machinery and the late excerpt stays around a rocky underwater passage. Crash keeps moving locally, but the full attempt ends at the decision limit. It reached the 8 recorded reference milestones without logging level completion.

**The point is that 1 shared model can have different outcomes on different levels.** Its shared visual features and separate action heads do not guarantee equal competence. We therefore report the ending for each level separately, alongside the video, instead of letting 1 success represent the entire model.

## 22:00–23:30 · Same Workload, Faster Training

**On screen:** Read the 1-GPU and 4-GPU times, explain the fixed workload, and finish with the measured training-time speedup.

The question here is **how quickly we can perform the same amount of training**. We compare 1, 2, and 4 active GPUs on the same host, with 2 seeds for each setting.

Each GPU holds a complete model replica; the workers synchronize gradients through NCCL. Every run collects 200,000 decisions in total across its workers. It also performs 7,501 learning updates. These are workload controls, not measures of how intelligent the policy becomes.

The measured training times average 54.7 minutes on 1 GPU, 29.5 on 2, and 16.5 on 4. Dividing the 1-GPU time by the 4-GPU time gives approximately **3.32 times faster**. These are training times, excluding separate provisioning and evaluation.

So this result demonstrates faster execution of the tested learning workload. It does not establish better gameplay, and using more GPUs does not automatically establish lower cost.

## 23:30–24:50 · How Playing Becomes Learning

**On screen:** Read the 3 cards from left to right, then follow the return arrow. Finish with the separate evaluation step.

**The whole system comes down to 1 repeating loop: play, remember, and learn.**

1. Play. The agent sees images from the game. Its neural network estimates a value for each available action; an action-selection rule chooses an action, and the controller turns that choice into button presses. The game responds with a new image and a reward.

2. Remember. We store what the agent saw, what it did, and what happened next. This collection is called replay memory. Keeping these experiences lets the learner revisit earlier attempts instead of learning only from the latest move.

3. Learn. The learner samples those experiences and updates the network’s weights. The updated network then guides later attempts. **That return arrow is the connection between training and playing.**

Underneath the loop is a separate check: save the policy and test it. We record gameplay and measure outcomes to see whether the changes actually helped. **Training changes the model; evaluation tells us what it can do.**

## 24:50–25:45 · A Childhood Game, a New Starting Point

**On screen:** Finish warmly: return to the childhood game, recognize the concrete progress, and invite the next step.

This started with **a game I played as a kid** and a question: could I build an agent that learns to play it?

We now have a working platform that connects the game, visual learning, experiments, and a website for inspecting the results. We measured **3.32 times faster training on 4 GPUs**, and a separate demonstration-trained policy completed an Orient Express route. Those are concrete steps forward.

**The exciting part is that the foundation is now in place.** We can try better training methods and extend the infrastructure to other PlayStation games by adding and validating their game-specific integrations.

I came back to Crash with a different kind of controller: a learning system. We have seen it finish a route. Now I’m excited to see how much further we can take it. Thank you.

## Evidence and wording

The scaling figures use training.elapsed_seconds from the 6 completed overnight-nccl-g{1,2,4}-s{42,43} runs. Each has 200,000 global decisions and 7,501 updates. The mean training intervals are approximately 54.7, 29.5 and 16.5 minutes. Provisioning and separate evaluation are excluded.

The opening Orient Express clip comes from orient_express-warm-s0/result.json: greedy = level_complete. The success is tied to its matching start; do not describe it as robust 5-level mastery. These clips are separate from the 6 scaling runs.

The other clips are greedy evaluations: toad_village-scratch-s0 and bone_yard-scratch-s0 ended at the decision limit; under_pressure-scratch-s0 and makin_waves-replay-s0 ended in death. They illustrate recorded behavior, not additional completion claims. Level pictures are frames extracted from these recordings. The grayscale example illustrates preprocessing; it is not a saved training observation tensor.

The expanded comparisons include scratch, curriculum, DQN from demonstrations and the archived shared-policy experiment. These are selected runs, not a controlled method ranking; reward settings and seeds can differ. Their exact outcomes and sources are in comparison_clips in evidence.json. The machine-readable snapshot and original artifact paths are in evidence.json. The archived run directories remain on the host for inspection.

## Research references

CNN backbone: Mnih et al. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533. https://doi.org/10.1038/nature14236. PS-Env adapts the action outputs and adds optional dueling and shared-policy heads.

Hester et al. (2018). Deep Q-learning from demonstrations. AAAI. https://arxiv.org/abs/1704.03732. PS-Env adapts DQfD with behavior-cloning pretraining, fixed demonstration/online sampling, and a large-margin imitation loss; this is not an exact DQfD reproduction.

Salimans, T., & Chen, R. (2018). Learning Montezuma’s Revenge from a single demonstration. https://arxiv.org/abs/1812.03381. PS-Env adapts the reverse demonstration-state curriculum to local section goals and combines it with its DQN learner with demonstrations.