You know that moment when you're staring at two completely different problems and suddenly realize they're the same problem?
Mine happened on a Tuesday night in my basement, around 11 PM, with a pot of coffee that had gone cold three hours earlier.
I was debugging a RAG pipeline that kept hallucinating citations—the retrieval step was pulling the right documents, but the generation step was stitching them together into something that looked correct and felt authoritative but was, in fact, complete fiction. A classic LLM failure mode: fluent, confident, and wrong.
The fix wasn't more data. It wasn't a bigger model. It was a better test harness—something that could catch the hallucination before it reached the user, by measuring not just what the model said, but whether what it said could be grounded in something real.
I closed my laptop, walked upstairs, and spent the next hour staring at the ceiling.
Because I'd seen this exact pattern before. Four years of it. In a completely different domain.
The Perception Problem
From 2019 to 2023, I worked on a perception algorithm team in Detroit's autonomous driving supply chain. We built models that had to look at a LiDAR point cloud, fuse it with camera data, and decide: is that blob a plastic bag or a child?
The industry standard at the time was modular. Perception fed into prediction. Prediction fed into planning. Planning fed into control. Each box was a separate model, a separate team, a separate codebase. And each box introduced its own failure modes.
We spent countless hours chasing false positives—the car sees a shadow and brakes. False negatives—the car doesn't see the pedestrian and doesn't brake. Edge cases no one thought to code for. The stack was brittle, but at least it was traceable. When something failed, you could isolate which box broke.
Then in early 2024, Tesla shipped FSD V12. The rule-based city driving stack—roughly 300,000 lines of C++ control logic—got deleted-. Replaced by a single end-to-end neural network that mapped pixels directly to trajectories-. Perception, prediction, planning, control: collapsed into one model-.
The industry called it a breakthrough. And in many ways, it was. The car drove smoother. More human. Less jerky. The codebase shrunk by two orders of magnitude-.
But something else happened that most people didn't talk about.
The black box got blacker-.
When an end-to-end model makes a mistake—when it invents a merge lane that doesn't exist, or phantom-brakes on an empty highway at 70 MPH, or decides a roundabout is a four-way stop—you can't open the hood and point to the perception module anymore. You can't say "the LiDAR classifier misfired" or "the trajectory planner had a bug." The whole thing is one giant matrix of floating-point numbers.
And matrices don't write postmortems.
The Evaluation Problem
In my day job—the one I took in 2023, after leaving Big Auto for a startup building an open-source LLM inference framework—I spend my time optimizing RAG pipelines and tool-calling accuracy. I evaluate models for a living. Latency. Hallucination rate. Context retention. Tool-call correctness-.
The LLM evaluation community has spent the last few years building sophisticated test harnesses precisely because they don't trust the models. They run them in CI. They score answer relevance, faithfulness, hallucination risk, tool behavior-. They use judges to grade intermediate reasoning-. They treat every model output as guilty until proven innocent-.
The autonomous driving industry, meanwhile, still publishes "disengagement rates" and calls it safety data-.
Here's the thing about disengagement rate: it measures how often a human has to take over. It does not measure how often the model made a bad decision that the human didn't catch. It does not measure how often the model almost caused a collision but recovered at the last second. It does not measure trajectory reasonableness. It does not measure hallucination.
The industry's primary safety metric is, in effect, a measure of how annoying the car is.
That's not good enough.
The Hallucination Problem
In LLM evaluation, "hallucination" means the model generated something that isn't grounded in its context or in reality. It made stuff up. It sounded confident while doing it. And it fooled the user.
In autonomous driving, the same thing happens. The model sees a road, predicts a trajectory, and executes a maneuver that doesn't correspond to anything in the physical world. It hallucinated a path. It hallucinated a vehicle's future position. It hallucinated a traffic rule.
The difference is that when an LLM hallucinates, you get a wrong answer. When a driving model hallucinates, you get a collision-.
And yet, almost no one is testing driving models the way we test LLMs. No standardized benchmark. No repeatable test suite. No systematic hallucination log. No one running the same evaluation after every OTA and publishing the raw data.
I started doing exactly that.
The Airtable
In 2021, while I was still in Detroit, I started keeping a log. Every time a production driving system did something that didn't make sense—every phantom brake, every missed pedestrian, every inexplicable lane change—I wrote it down. Timestamp. Vehicle. Software version. Trigger condition. What the model did. What it should have done.
I didn't know what I was building at the time. I just knew that the industry wasn't keeping this data, and someone needed to.
Four years later, that Airtable has hundreds of entries. Some are minor—a lane-centering wobble on a straight road. Some are severe—a high-speed phantom brake on a clear highway, clear sky, no obstructions-. Some are weird—the same roundabout, same failure mode, across three different systems from three different OEMs.
Every entry has a severity rating, 1 to 5. Every entry has a root-cause hypothesis. And every entry has a follow-up field: Did the next OTA fix it?
Most of them, so far, say "no."
What I'm Building
checkthehomework.com is my attempt to bring LLM evaluation methodology to autonomous driving.
Not drive reviews. Not hype. Not press releases rewritten as blog posts.
Standardized benchmarks. Reproducible test harnesses. A living database of hallucinations and edge-case failures. Updated after every major OTA. Published with full methodology and raw data.
I test production systems—FSD, Super Cruise, BlueCruise, whatever I can get my hands on. I test in simulation. I test on real roads. I run the same scenarios, the same metrics, the same severity scale, every single time.
And I publish everything.
The Scorecard: structured benchmark results, updated after every OTA. Latency, trajectory reasonableness, hallucination rate, long-tail generalization. Comparative tables. Trend lines.
The Hallucination Log: timestamped entries from the Airtable. What happened, what should have happened, what the root cause likely was, and whether the next update fixed it.
The Rig: methodology posts. Test harness design. Metric definitions. Hardware specs. Everything you need to replicate my results—or argue with them.
Track Notes: human driving as the baseline. My Miata ND on track days, my daughter Maya's observations from the back seat, and what 10,000 miles of manual driving teach you about the gap between human and machine reasoning-.
Why This Matters
The autonomous driving industry is in the middle of a massive architectural shift. The old modular pipelines—perception, prediction, planning, control—are being replaced by end-to-end neural networks-. The industry calls it progress-. And in many ways, it is.
But progress without evaluation is just faith.
Faith that the model learned the right patterns. Faith that the long tail won't show up. Faith that the next OTA won't introduce a regression that kills someone-.
I don't have that faith. I've spent too many hours debugging perception failures and too many nights chasing LLM hallucinations to trust a black box on faith alone.
So I'm testing them. Systematically. Repeatedly. Publicly.
Your car talks. I check his homework.