Why Two Robot Cars With the Same Score Can Fail in Opposite Ways
Two robot cars set off on the same small loop of road, each steered by a different piece of software. The first car takes the opening corner and then slows almost to a stop. For the rest of the 30-second test it inches forward at about three centimeters per second, so slowly that it never reaches the stop sign ahead. The second car drives with more confidence, drifts wide on the bends and runs off the road. Across 20 runs, it leaves the track 13 times.
By the usual measure, these two drives are the same. Researchers who compare driving software tend to rank it by a single number, the total reward it earns over a run. The crawling car, steered by a program called TD3, scored −213.78. The crashing car, steered by one called DPW, scored −214.46. On a leaderboard, that is a tie.
We saw this while building Duckietown.jl, a free driving test for software written in the Julia programming language. It is described in our paper by Alfan. The test still reports the score. Alongside it, the test keeps a record of what happened on the road, such as stops made, signs run and crashes, and a fair count of how much computing each program used. The code is on GitHub.
Figure 1. Several programs take turns driving the test loop, seen from the car’s own camera (left) and from above (right). These are demonstration laps from the package’s own renderer. Rendered Duckietown environment courtesy of the Duckietown Project.
A driving test that Julia was missing
Julia has become a popular language for research on decisions under uncertainty, in large part because of a shared toolkit called POMDPs.jl. A researcher describes a problem once, including what the world looks like, which actions are possible and what counts as a good outcome. After that, any decision-making program written for the toolkit can be tried on it without extra glue code. These programs, which researchers call solvers, range from planners that think several steps ahead to systems that learn by trial and error.
The next step was a well-maintained driving test built for that toolkit. The best-known driving tests, including highway-env, CommonRoad, CARLA and nuPlan, are written in Python. Duckietown.jl brings a driving test straight into Julia, so researchers can run their experiments in the same language as their solvers.
Duckietown was a good place to start. It began as a low-cost platform for teaching self-driving, with small robot cars, toy road tiles, traffic signs and rubber ducks standing in for the town’s residents. Its simulator has been used to compare driving methods for close to a decade. Duckietown.jl gives solvers direct access to the driving decisions themselves, so they can focus on how the car should drive.
What the test looks like
The main scenario is called stop and duck. A Duckietown robot car drives around a small square loop made of eight road tiles, where it meets a stop sign and a duck that starts to cross when the car comes close. To pass, the car has to make a real stop at the line, slowing to under two centimeters per second in a short zone in front of it, and then give way to the duck. Rolling through the sign counts as a violation, and hitting the duck ends the run.
The tricky part is timing. The software makes a new decision every 0.2 seconds, but the simulated car carries out each command 0.15 seconds after it is given. A program that forgets about this delay keeps reacting to a car that is no longer quite where it thinks it is. Each run lasts up to 150 decisions, or 30 seconds.
Solvers can drive in two ways. Some pick from a menu of seven simple commands, which combine fast or slow with left, straight or right, plus a brake. Others set speed and steering directly.
Under the hood, Duckietown.jl rebuilds the physics of the original Duckietown simulator in Julia and checks it against the original, number by number. It can also draw the scene, using the Duckietown textures from the user’s own installation.
Figure 2. The same moment drawn by the original Duckietown renderer (left) and by Duckietown.jl (right). The two views match closely, and the physics behind them was checked against the original simulator. Rendered Duckietown environment courtesy of the Duckietown Project.
A record of what happened on the road
Every run in Duckietown.jl produces a log of events next to the score. It notes how many stop signs the car reached, how often it stopped properly, how many signs it ran, whether it gave way to the duck and how the run ended. A low score then comes with an explanation attached.
The scenarios are also frozen. Once a scenario is released, its map, signs and scoring rules never change. Any revision is published under a new name, and automated tests check this every time the code changes. A result reported today can therefore still be compared with one reported next year.
The package also measures computing effort in a way that is fair across very different methods. A small counter sits between each solver and the simulator and records every time the solver asks, in effect, “what happens if I do this?” Planners that look ahead ask that question thousands of times before each decision. Programs that have already learned a driving habit simply act on it. Counting the questions gives a single unit of effort that means the same thing for both.
All of this runs through one shared evaluation routine that treats every solver the same way, including the ones that ship with the package.
Seven programs and one stop sign
We tested seven solvers from four families. Two learn from experience by filling in a table of good moves (Q-learning and SARSA), and two learn with neural networks (SAC and TD3). Two more plan ahead by searching through possible futures before each decision (MCTS, and DPW, a version built for smooth control of speed and steering). Each planner was given about 1,000 simulator calls per decision. The seventh, DORA, is a route-finding planner that one of us developed in earlier work. Each solver drove 20 test runs.
| Solver | How it decides | Average score | Progress along the road | Left the road / other crashes | Proper stops / signs reached |
|---|---|---|---|---|---|
| SARSA | Learned table | 11.92 | 20.07 | 1 / 0 | 8 of 8 |
| Q-learning | Learned table | 11.63 | 20.08 | 1 / 0 | 8 of 8 |
| MCTS | Searches ahead | 6.78 | 15.96 | 0 / 0 | 4 of 4 |
| SAC | Neural network | −1.32 | 14.52 | 0 / 0 | 18 of 20 |
| DORA | Route-finding | −1.59 | 27.47 | 1 / 0 | 3 of 3 |
| TD3 | Neural network | −213.78 | 4.48 | 0 / 0 | never reached a sign |
| DPW | Searches ahead | −214.46 | 7.13 | 13 / 6 | 0 of 7 |
The duck stayed safe in every run. Beyond that, the event log tells very different stories. TD3 never broke a rule, mainly because it never got anywhere. DPW left the road in 13 of its 20 runs, hit something in six and drove through all seven stop signs it reached. Two very different failures ended up with the same number.
The replay in Figure 3 puts one run of each car side by side. TD3’s speed falls close to zero soon after the first corner. DPW drifts a little wider on each bend until it leaves the road at decision 116.
Figure 3. One test run replayed from the recorded logs, with TD3 on the left and DPW on the right. The charts under each map show distance from the lane center, heading error, speed and turning rate.
The rest of the field did better. SAC made a proper stop at 18 of its 20 stop signs and stayed on the road in every run. The two table-based learners stopped at every sign they reached, though each ran off the road once. DORA covered the most ground and stopped at all three signs it reached. One bad run pulled its average score down to −1.59, while its typical run scored 12.07.
Counting the work behind each decision
Planners like MCTS and DPW are usually compared by giving each one the same number of search rounds, often 1,000. That sounds fair until you measure what a round costs. In our runs, one round took about 27 to 29 simulator calls for both planners. A budget of “1,000 rounds” therefore means about 28,000 calls for every decision the car makes. Counting rounds is a bit like comparing two delivery vans by the number of trips they made, without asking how long each trip was.
Duckietown.jl sets budgets in simulator calls instead. It turns a target, such as 1,000 calls per decision, into a number of search rounds using the measured cost of a round. Then it checks what was actually used. This way, both planners landed within 0.7 percent of their targets. Picking the nearest value from a coarse list of round counts could miss the target by as much as 42 percent.
How much a planner spends also depends on how it drives. With the same target of 1,000 calls per decision, MCTS used 999.4 on average and DPW used only 721.7, about 28 percent less. DPW’s runs often end early when the car leaves the road, after 85.6 of 150 decisions on average. Figure 4 shows its spending falling as a run goes on, while MCTS stays close to its target. For this reason the benchmark reports the calls actually used next to every result.
Figure 4. Simulator calls per decision from the start of a run (left of each chart) to its end (right), across 20 runs each. Points are medians and bars cover the middle half of the values. Q-learning has already learned its driving habit, so it makes no calls.
The size of the budget can also change which planner looks better. In a smaller check over five extra runs used during development, with shorter drives, MCTS scored about the same at every budget, between 5.9 and 7.0. DPW went from −165.3 at 500 calls per decision to −30.2 at 1,000 calls and −26.9 at 2,000. Testing across several budgets gives a fuller picture of how each planner behaves.
Where this fits in our work
Duckietown.jl grew alongside two other projects in our lab. One is vnvspec, our way of writing verification and validation requirements so that a computer can check them. The benchmark’s own promises, such as “released scenarios never change” and “no special code for any solver”, are written in that form and checked automatically whenever the code changes. In other words, we hold the test to the same standard we would ask of the AI systems it measures.
The other is DORA, the route-finding planner in the results above. It comes from a separate codebase, yet it ran on the benchmark with only a small amount of connecting code on the user’s side. That kind of reuse is what the shared Julia toolkit was meant to make possible.
Figure 5. DORA driving laps of the test loop, seen from above. Rendered Duckietown environment courtesy of the Duckietown Project.
Both projects rest on the same idea. A score means more when the events behind it, the computing behind it and the files needed to reproduce it are all open to checking.
How to try it and how to help
Duckietown.jl is listed in Julia’s official package registry and installs like any other Julia package. The repository holds the code, the frozen scenarios and the evaluation results, along with a worked example that is tested automatically. The animations in this post, and many more, are kept in a companion repository so the package itself stays small to download.
We welcome other groups to rerun our evaluation and to bring their own solvers, which only need to follow the POMDPs.jl conventions to run through the shared evaluation routine. Contributions of new maps and scenarios, including ones where the car works from imperfect sensing, are welcome too. New scenarios are released under new names so that every earlier result stays comparable. The GitHub issues page is the best place to start.
Duckietown.jl is an independent community project built on the Duckietown simulation environment. Parts of it are derived from gym-duckietown with the written permission of the Duckietown project.
References
- A. A. A. Irfan and M. M. Arief, “Duckietown.jl: A Closed-Loop Driving Benchmark for Sequential Decision-Making in Julia,” 2026. Code at github.com/ai-vnv/Duckietown.jl.
- M. Egorov, Z. N. Sunberg, E. Balaban, T. A. Wheeler, J. K. Gupta and M. J. Kochenderfer, “POMDPs.jl: A Framework for Sequential Decision Making under Uncertainty,” Journal of Machine Learning Research 18(26), 2017. Code at github.com/JuliaPOMDP/POMDPs.jl. The shared Julia toolkit that Duckietown.jl plugs into.
- L. Paull et al., “Duckietown: An Open, Inexpensive and Flexible Platform for Autonomy Education and Research,” IEEE International Conference on Robotics and Automation (ICRA), 2017. The original Duckietown platform.
- DORASolvers.jl, github.com/ai-vnv/DORASolvers.jl. The route-finding planner used as the seventh solver.
- Duckietown-artifacts, github.com/PannnTastic/Duckietown-artifacts. Recorded laps, animations and figures from the study.
- AI V&V Lab, “vnvspec, Machine-Readable Specifications for AI Systems Tooling,” 2026. How the benchmark’s own properties are written down and checked.