AI generated
We built a train sim at Extella to see if we could get Qwen 3.5 9B on my Mac to outscore Claude Opus 5.5. We used special designed DSL for writing trip plans and some Python to check and drive them, Qwen’s setup won 6 of 9 runs. Claude was driving from live cab feedback. I work on getting small models to do useful jobs in business software. A lot of those jobs already have rules you can write down and decisions you can check in code. The nice thing about a train is that you can replay the exact same trip and find out what a bad decision actually cost you. Take the Gotthard run. You’ve got a Re 6/6 hauling 900 tonnes over the mountain. Before you leave, you can take a helper engine. That costs 100 points even if you end up doing fine without it. You’ve also got 600 seconds of sand for grip. On wet rail, it matters where you spend it. We gave Qwen six commands to plan the trip. It can take the helper, set a target speed, slow down for a stretch, coast, use sand, or ask the controller to keep to the timetable. This is what two lines of a plan look like: slower 60 from km 93 to km 96 sand from km 42 to km 70 Before departure, we run the plan through the sim using the weather forecast and known track restrictions. If the train gets stuck, Qwen gets a message like “You’ll stall at km 48.1 on wet rail. You need sand or a helper engine.” It can revise the plan twice. If all three attempts fail, we run the one with the best forecast and flag it in the log. Once the train moves, Python works the lever. Every half second it checks the speed, brakes for stops and backs off if the wheels slip. Qwen has already decided where to use sand and how fast it wants to go. It doesn’t get to see the actual grip in advance, or whether there’s a train ahead. The comparison I found most useful was running the same Qwen both ways. We gave it the same briefing and had it write lever positions by kilometre for the whole trip. Over ten Gotthard runs, that averaged −7,686 points. With the trip language, checker and controller, it averaged 852. Across all 30 runs on the German, Swiss and Japanese routes, the planning setup finished every trip without triggering train protection. Qwen writing lever positions triggered it on all 30. For Claude, we recorded a separate set of nine runs, three per route. It saw the cab, chose a lever position and said how long to hold it. Whenever something happened, the sim paused so Claude could react. Same weather, delays and scoring for both drivers. Our Qwen setup averaged 554 points to Claude’s 253, with 1.3 model calls per trip on our side and 60.7 driving decisions per trip on Claude’s. We haven’t tried Qwen with that live cab interface yet. One of the runs Claude won was on the Gotthard. Qwen took the helper and used 2,272 kWh. Claude went without it and used 1,833. That saved Claude enough to win even after train protection stepped in and cost it 150 points. We also ran two plain Python drivers. One of those beat our Qwen setup on average in Germany and Japan. The thing I’d take back to our other work came from reading the rejected plans. Our first version let Qwen say “hold this speed until km X.” It kept picking a reasonable-looking speed and holding it for so long that the train ran late. 28 of 35 rejections were for lateness. On 9 of 30 trips, none of its plans passed. We changed that command so it could ask for a slowdown between two points and gave it one short example. Same model, same weights. Its average score went up by 63 points in Germany, 226 on the Gotthard and 198 in Japan. That’s the part that interests me about using a 9B model for everyday work. We could read its plan, see exactly where it went wrong, change one command and rerun the same trips to see whether we’d helped. You can drive all three routes in the browser. The repo has the sim and the recorded model decisions. If you want to check the numbers, verify.py replays them offline in about ten minutes. No API keys needed. Drive the train · Code and recorded runs