Think twice / 04
Matchbox Apprentice
A little learner. A handful of beads. What will it remember when its world changes?
How to play
You train a small courier to reach a destination in a nine-room map. It chooses directions by drawing coloured beads from each room's matchbox. Your challenge is to improve its measured deliveries, then discover which learned habits survive a changed destination or a blocked corridor. The courier starts at G and has 24 moves to reach the green destination, initially C. Fresh boxes have three beads for each direction through a wall opening. At G, North and East therefore start equally likely. Drawing a bead chooses a move; the courier never plans ahead.
- Press Test 50 deliveries without learning before training. This is your baseline. The courier starts in G; the green room is its destination. Read both successful deliveries and average moves for successes.
- Press Train 20 deliveries, then test again on the same map. Tap a room to inspect its beads, or replay a journey one move at a time. More beads make a direction more likely, not certain.
- Move the destination or close a corridor while keeping the memory. Test before retraining, then train and compare again. Try the reward and credit controls, or edit beads yourself. A useful finish is a before-and-after result you can explain, including any loss after the world changes.
What to notice
Could a machine learn without a computer? In the 1960s, Donald Michie built MENACE to play noughts and crosses using matchboxes and coloured beads. Each box represented a board position. Each bead represented a possible move. Draw one bead, make that move, and change the beads after the game. The next choice now has different odds.
Imagine a box with five beads for one move, three for another and one for a third. The first move is more likely, but never guaranteed. If a successful game adds beads to a choice, that preference grows. You can open the box and see what the machine has learned. It has no picture of a winning strategy; the bead counts are its memory, a record of choices that tended to earn rewards.
Here, the boxes belong to a courier moving between nine rooms. It starts in G and tries to reach the green destination within 24 moves. It does not look ahead or calculate the shortest route. First test 50 journeys with fresh beads. Then train 20 journeys and run the same test again. Compare arrivals and the moves taken by successful journeys; tap a room to inspect how its choices changed.
Then keep those beads but move the destination. The courier may have become very good at reaching the old goal and worse at reaching the new one. You can also change which parts of a successful journey receive credit: the whole route, the last six choices or only the final move. Which reward teaches a useful habit, and which rewards a detour?
This is a small, inspectable example of reinforcement learning. It also shows why a good training result is not proof that a learner understands the task. Michie's boxes inspired the idea; our courier, nine-room map and reward settings are our own adaptation. Will the courier still know what to do when its destination changes?
Read more: Donald Michie, Experiments on the mechanization of game-learning · Sutton & Barto, Reinforcement Learning.
Deliver the parcel
Start at G · arrive within 24 movesEach room has a matchbox. Draw a bead, follow its direction. Tap a room to inspect its box.
Purple marker = apprentice. Green room = destination. A broken pink corridor is closed.
Test baseline → Train → Test again
First, measure the fresh matchboxes. You can explore in any order.
Start with Test 50 deliveries without learning. Then train and use the same test again.
What did training change?
Both versions face the same 50 test seeds on the same map, with learning switched off. Fresh matchboxes have three beads per direction. This sample is repeatable; it does not guarantee the next delivery.
Detailed test history
| Map / training | Current arrivals | Fresh arrivals | Current avg. moves | Fresh avg. moves |
|---|
Watch a journey or train one delivery
Latest 100 training outcomes on this map: green arrived, pink timed out. Training happens immediately; animation replays the last journey.
Rules & model
How a move happens
The courier starts at G and stops at the destination, or after 24 moves. Its current room has a matchbox with beads for physically possible directions. It draws only from open corridors: six beads make a turn twice as likely as three. Outside walls have no beads. Closed corridors keep their stored beads for when they reopen.
What training changes
On a successful delivery, the chosen reward is added to the last six choices, the whole journey or only the final choice, whichever credit rule you select. Each distinct room/direction gets the reward once per journey, so repeating a loop does not multiply that same reward.
A timeout removes one bead from each distinct choice used. Counts stay between one and sixty for possible directions. That minimum leaves every reopened legal turn a chance, even after failures. Rewards and credit changes affect future training; they do not rewrite existing beads.
Training, watching and testing
Train changes memory immediately. Its animation replays the final journey in that batch. Watch without learning, replay controls and the fifty-delivery test change no beads or training draws. A recorded move shows the draw before any later reinforcement.
The test compares current and fresh matchboxes on the same map with the same fifty test seeds. Average moves counts successful deliveries only; timeouts remain visible in the arrival count. It is a repeatable sample, not a guarantee of the next outcome.
Changing the task
Move the destination or close a corridor to keep the memory but change its usefulness. Fresh matchboxes resets memory, training count and training seed while retaining your map. The shortest-route number is for you: the courier never uses that calculation to choose its moves. All state stays on this page.
Where this began
Donald Michie’s MENACE learned noughts-and-crosses using matchboxes and coloured beads. See his original paper on game-learning. Our nine-room delivery task, bounded bead counts, timeout rule, training-credit controls and changing maps are a new adaptation of that inspectable learning idea.