I live in Ann Arbor, a mile from MCity, and I still commute in a Miata with a manual transmission. But my night job is checking the homework of self-driving systems. Computer vision in autonomous vehicles is the part of the stack I trust least, which is why I built a standardized test harness in my garage. After every OTA, I re-run the same scenarios and log what breaks.
What Computer Vision in Autonomous Vehicles Has to Solve
A self-driving car perceives the world through a stack of sensors: cameras, LiDAR, radar. Computer vision is the software layer that turns pixels into semantics. It has to answer three questions in real time: What is that object? Where is it? What is it going to do next? Those questions map to three core tasks: object detection, semantic segmentation, and trajectory prediction.
Object detection draws boxes around things — cars, pedestrians, bicycles. Segmentation labels every pixel as road, sidewalk, curb, sky. Trajectory prediction takes the detected boxes over time and guesses where they'll move. Modern systems run all three simultaneously, often on a single neural network.
The hard part isn't the happy path. It's the corner cases: a plastic bag tumbling across the highway, a cyclist weaving between parked cars, a child chasing a ball into the street. In my perception days, we called those 'ghost and jelly' scenarios — the ghost is a false positive, the jelly is a missed object. Computer vision in autonomous vehicles lives or dies by how it handles both.
The Severity Scale I Use on the Hallucination Log
When I test a model, I rate every failure on a five-point scale that I've adapted from bug tracking. Severity 1 is cosmetic — the model labels a fire hydrant as a mailbox. Severity 5 is the one that ends with a tow truck and an insurance claim — the model sees a lane-splitting motorcycle as a parked car and merges into it. I've logged more severity 3s than I'd like to admit: the system correctly identified a pedestrian but misjudged their stride, so it braked politely instead of hard.
The scary thing is how often the failures are silent. The model doesn't say "I'm uncertain." It draws a confident box with a 0.94 confidence score and moves on. That's the core risk in computer vision in autonomous vehicles: confident wrongness.
Cameras vs. LiDAR: Where the Wrongness Comes From
Every production system I've tested leans on a different sensor budget. Tesla has bet on cameras alone, and its vision stack has to infer depth from parallax and motion. Waymo runs a full sensor suite with LiDAR giving a point cloud, so its computer vision problem starts with a 3D map already partially solved. LiDAR is not magic — it fails in heavy snow and produces its own ghost points. But when a model relies on cameras alone, a dark unlit road can turn a stopped truck into a gray gradient. That's not a camera limitation; it's a vision model limitation, and it's the kind of thing I test specifically at dusk and in fog.
How I Run the Benchmark
My rig is not the MCity test facility, though I've driven enough loops there to know its quirks. My rig is a set of standardized recorded scenarios plus a simulation loop. I pick 40 scenarios: downtown intersections, highway merges at dusk, construction zones, school pickup lines, dense fog, heavy rain. I feed the exact same sensor data to whichever production system I'm evaluating — via the openpilot interface when I'm testing comma.ai's stack, or through a hardware-in-the-loop setup for others.
Then I score three things: latency (how many milliseconds from frame to action), accuracy (how often the bounding boxes match the labeled ground truth), and trajectory reasonableness (does the predicted path make sense given the road geometry). I've measured detection latency from 15 milliseconds on a beefy GPU down to 90 milliseconds on a phone-class chip. That spread matters more than the marketing says.
Why Every OTA Deserves a Fresh Test
Last spring, a major EV maker pushed an OTA that improved highway lane keeping but degraded pedestrian detection at dusk by a measurable margin. Same camera, same sensor suite, same model architecture — different training distribution. You don't see that in a press release. You see it when you re-run the same scenario and the bounding box flickers on and off for two seconds.
That's why I re-run the full suite after every update. It takes about six hours of compute and one evening of watching logs. I've caught three regressions this year that no automated warning system flagged. Two were harmless. One would have been a severity 4.
The gap between computer vision in autonomous vehicles and human perception is still wide. That's not a reason to give up. It's a reason to test like an engineer, not like a fanboy.
What You Should Do Next
If you own an EV or an ADAS-equipped car, you don't need to build a test harness. You need to pay attention to what your car actually does, not what the marketing says. Try the same commute with and without the system. Keep a log of the weird stuff — the phantom braking, the late merge, the glance at a shadow that wasn't a person.
If you're an engineer, you know the drill: reproducibility, error bars, and a healthy distrust of your own model. Use an open benchmark if you can. openpilot's comma.ai community runs the same kind of standardized scenarios I do, and their release notes show the regressions. Publish your own hallucination log.
Your car talks. I check his homework. The next OTA is already in the queue.