At 7:42 a.m. on a wet Tuesday in Ann Arbor, a self-driving car approached a temporary lane shift, saw the orange barrels, and moved left. Then it moved right again before the merge point. No collision occurred. That is not the same as a pass. I logged the event as severity 3: an uncomfortable planning decision that required a human to monitor closely.
This is how I evaluate autonomy. I do not count press-release miles, watch a single impressive demo, or treat a smooth highway drive as proof that the hard problems are solved. I use repeatable routes, fixed scenarios, synchronized video, and the same severity scale after every software update. Your car talks. I check his homework.
What the benchmark measures
A self-driving car is not one model making one decision. It is a chain. Sensors produce observations, perception assigns labels and tracks objects, prediction estimates what those objects might do, planning selects a path, and controls turns that path into steering, throttle, and braking. A failure anywhere in that chain can look like bad driving from the curb.
My test harness separates those layers where the available data allows it. The basic run includes urban turns, unprotected left turns, construction zones, parked vehicles near intersections, pedestrians approaching the curb, emergency vehicles, and lane markings that disappear under rain or glare. I record the planned trajectory, speed changes, intervention timing, and whether the system recovered without an abrupt correction.
The score is not simply “worked” or “failed.” Severity 1 is cosmetic or inefficient. Severity 2 is clearly wrong but leaves comfortable recovery room. Severity 3 requires immediate human attention. Severity 4 creates a credible collision path or dangerous traffic interaction. Severity 5 is a crash, near miss, or loss of control. A self-driving car that completes a route while generating three severity-3 events has not delivered a clean run.
The corner cases that expose weak planning
The easiest way to make a self-driving car look capable is to give it a clean road, good weather, visible lane lines, and predictable traffic. The useful test does the opposite. I want the delivery van blocking the view, the pedestrian standing just outside the crosswalk, and the construction worker whose reflective vest blends into a sign.
One recurring failure mode is excessive hesitation. The vehicle recognizes a gap but waits so long that another driver fills it. That is safer than forcing the maneuver, but repeated hesitation can create its own risk and confuse surrounding traffic. Another is false confidence: the car begins a turn because its map expects a lane to exist, even though cones have shifted traffic several feet away.
The distinction matters. A perception error says, “I did not see the object.” A planning error says, “I saw it and chose an unreasonable response.” Those need different fixes. More camera data will not repair a planner that treats every uncertain object as irrelevant, and a smoother trajectory will not help if the object tracker loses a cyclist behind a parked SUV.
OTA updates need regression testing
An over-the-air update can improve one scenario and quietly damage another. That is why I rerun the same route after each meaningful release instead of trusting the version notes. “Improved behavior around vulnerable road users” is a hypothesis, not a result.
For each run, I capture the software version, weather, lighting, route, traffic density, and intervention point. I compare the new trajectory with the previous run and tag changes as better, worse, or inconclusive. If the self-driving car brakes earlier for a pedestrian but now stops unnecessarily at an empty curb, both observations belong in the report.
Repeatability is harder than it sounds. Traffic is not deterministic, pedestrians do not follow scripts, and a slightly different sun angle can change camera exposure. I ran one merge scenario three times and got three different answers. That does not prove the system is random; it proves that a single video is weak evidence. The correct response is more controlled trials, not a confident headline.
How I read a demo without being fooled
A polished demonstration hides the denominator. Was the route selected because the system already handles it? How many disengagements happened during setup? Was a safety driver ready to intervene? Did the vehicle encounter an unusual obstacle, or only the exact objects the team expected?
When I review a self-driving car demo, I look for operational boundaries first. Does it require mapped roads? Does it slow or stop when rain blocks sensor confidence? Can the driver understand why the vehicle is waiting? A system that performs well inside a defined operating domain can be valuable. Calling it general autonomy is a separate claim.
Brand names do not answer those questions. Tesla, Waymo, Mercedes-Benz, Ford, and companies using open-source stacks make different system choices, sensor assumptions, and driver-monitoring decisions. I test behavior, not reputation. A familiar badge cannot convert a severity-4 event into a severity-2 event.
A practical test plan for owners and engineers
If you own an advanced driver-assistance system, you can collect useful evidence without pretending your driveway is a laboratory. Start with a short route that includes one repeatable challenge, such as a confusing merge or a protected left turn with a blocked sightline. Drive it at similar times and record the software version. Keep your hands and attention where the manufacturer requires them; these systems are not a substitute for supervision.
Log four things: what the sensors appeared to detect, what the vehicle intended to do, what it actually did, and how much time you had to intervene. Add a severity rating and a short clip. Avoid conclusions like “the AI is stupid.” Write the observable gap: “The car detected the stopped vehicle but maintained speed until the driver braked.” That sentence can be reproduced and investigated.
For engineers, add trajectory error, braking onset, minimum time-to-collision when available, and uncertainty around each measurement. Store raw logs rather than only screenshots. A self-driving car evaluation becomes much more useful when another person can inspect the evidence and disagree with your interpretation.
What a meaningful result looks like
A meaningful result is often less dramatic than a viral clip. It might show that a self-driving car handled ten ordinary intersections cleanly, then produced one repeatable severity-3 hesitation at a specific construction pattern. That is actionable. It identifies a boundary, suggests a scenario for simulation, and gives the next software release something concrete to improve.
I publish failed runs alongside clean ones because the failed run contains more engineering information. If the cause is unknown, I say so. If weather, traffic, or sensor occlusion contaminated the trial, I mark the result inconclusive instead of laundering it into a score.
Autonomy will improve through disciplined evaluation, not optimism. Before trusting a new feature, ask what changed, what was tested, and which corner case still breaks it. Then run the scenario again. The car can make the claim. The log gets the final word.