The Bench
Set up one moment of the game, pick a move, and watch a single update happen with real numbers from a real network. Change anything (the board, the move, a knob) and the whole worked solution recomputes.
Change the rewards
Game points. They are multiplied by 0.01 before training, so a pellet (+10) becomes +0.10. These only affect the Bench.
-
The guess
The live network looks at board S and scores all four moves. Only the move you picked gets graded.
q(S, ) =
-
Peek at the future
Live network picks the best next moveFrozen copy says what it is worthThe game ended on this move, so there is no future to add. v = 0.v =
-
The target
What q should have been, in hindsight: the reward you actually got, plus the discounted value of where you ended up.
-
The gap
Guess against target. This single number is what the whole network gets adjusted to shrink.
gap = loss =
-
Nudge the weights
Backpropagation works out, for every one of the network’s weights, which direction would move q toward the target. Each nudge moves every weight a tiny step that way.
Press a nudge button to watch the guess move toward the target.
Ripple Tank
A simpler world: one cherry worth +10, one ghost worth −10. At the start no square knows anything, so the map is blank. The only true information enters where rewards happen, then leaks backward into the guesses, one step at a time.
Things to try
- Press Sweep a few times. Each sweep grades every square once, against a frozen snapshot of the previous sweep, and the color spreads exactly one square further from the cherry. That is the reward leaking backward.
- Slide future discount down to 0.5. Far squares barely care about the cherry anymore: the glow fades fast with distance.
- Switch to Network and press One step. A single update tints many squares at once, because they share weights. That sharing is why DQN uses a network: it can guess about squares it has never visited.
- After it has learned, move the cherry. Watch the old knowledge get overwritten.
Arcade
Now the whole machine: a brand-new network plays this maze, stores every moment in a replay memory, and learns from random handfuls of it using exactly the update from the Bench. It starts out walking into walls and ghosts. Keep an eye on the exam line under the stats: expect it to start climbing after roughly 20,000 steps, a few minutes on Turbo. Or load the trained brain to skip ahead.
The line is the exam: every 2,500 steps the agent plays the same 10 games with no random moves and no learning, which shows what it has really learned. Dots are practice games, which include random moves. Dashed line: random play.
Knobs
Experiments
Each one changes a knob and marks the moment on the chart so you can see the effect. Run them on a brain that is already learning (or load the trained one) and give each a minute on Turbo.
Four ghosts
The whole 1980 arcade game: the real maze, four ghosts with their own personalities, scatter and chase waves, frightened mode, Cruise Elroy, fruit, lives, and levels that speed up. It is rebuilt rule by rule from the documents that describe the original (listed at the bottom of this station) and drawn in this workshop’s style. A DQN learned to play it with the same update as the Bench, only bigger: a network of about half a million weights, trained for hours.
Each bar is its guess of all the reward still to come after that move, in training units: a dot is worth 0.32 and a lost life −2, and rewards further off count for less.
Ghosts alternate between scattering to their corners and chasing. Each switch makes them turn around. Frightened time pauses the clock.
How the brain learned
The line is the exam: every 250,000 decisions the brain played the same 10 games with no learning and no deliberate random moves, on the same sticky stick it trained with. Dots are practice games: the average of the last 100 finished while learning, random exploration moves included. Dashed line: random play.
Practice runs far below the exam, and not because the brain plays worse there. Each of the 16 training games makes a different share of its moves at random, from 0.07% to 40%, and the most random games end soonest, so they fill about half of the finished games. Replaying the brain on this page that way, the 5 games with 5% or more random moves averaged 8,980 points; the other 11 averaged 30,766.