Open-Source vs. Proprietary: Can You Even Benchmark Driving Models Fairly?

I have a confession to make.

For the last two years, I've been publishing benchmark data comparing autonomous driving systems. FSD vs. Super Cruise vs. BlueCruise. v12.5.1 vs. v13.0. Before and after every OTA.

But there's a problem with these comparisons. A fundamental problem. One I've been avoiding.

Open-source models are transparent. Proprietary models are black boxes. And the difference between them is not just philosophical—it's methodological. You can inspect the weights of an open-source model, trace its inference path, and reproduce its behavior exactly. You can't do any of that with a production system.

So when I publish a benchmark comparing openpilot to FSD, am I comparing apples to apples? Or am I comparing an apple to a black box that might contain an apple, or might contain something else entirely?

The answer is: it depends. And the "depends" is where the methodology breaks down.

The Asymmetry Problem

Let me lay out the asymmetry clearly.

Open-source models:

• I can download the weights.

• I can run them on my hardware.

• I can inspect the architecture.

• I can trace the inference path.

• I can reproduce the results exactly.

• I can modify the model and test variants.

• I know the training data distribution (or at least, I know what's been published).

Proprietary models (production systems):

• I cannot see the weights.

• I cannot run them on my hardware.

• I cannot inspect the architecture.

• I cannot trace the inference path.

• I cannot reproduce the results exactly (because I can't control the inputs).

• I cannot modify the model.

• I don't know the training data distribution.

This asymmetry is not a minor inconvenience. It's a fundamental barrier to fair benchmarking.

When I test an open-source model, I can run the same test scenario 100 times and get the exact same result every time. The model is deterministic. The inputs are controlled. The outputs are reproducible.

When I test a proprietary system, I'm testing a system that runs on someone else's hardware, with unknown internals, and a safety monitor that may intervene or not. The same test scenario can yield different results on different runs. The system is non-deterministic. The inputs are not controlled. The outputs are not reproducible.

How do you compare a system you can fully inspect to a system you can't? How do you know if the differences you observe are real differences in capability, or just differences in measurement methodology?

The short answer: you don't. But you can try.

My Approach: Input Standardization

To make the comparison as fair as possible, I standardize the inputs.

For both open-source and proprietary systems, I run the same test scenarios: the same GPS coordinates, the same time of day, the same weather conditions (as much as possible), the same traffic conditions (as much as possible).

For open-source models, I run them in simulation—CARLA, with a standardized set of sensor inputs.

For proprietary systems, I run them on real roads, with real sensor inputs.

This is not ideal. The open-source model gets a clean, synthetic input. The proprietary system gets a noisy, real-world input. The comparison is not apples-to-apples.

But it's the best I can do. And I'm transparent about it.

What I Can't Control

Here's where the methodology breaks down.

1. The safety monitor.

Production systems have a safety monitor—a rule-based system that overrides the model's trajectory if it's unsafe. The safety monitor is a critical part of the system. It's also invisible to me. I don't know when it intervenes, or why.

The open-source model doesn't have a safety monitor. It's just the model. So when the open-source model makes a bad decision, it actually makes the bad decision. When the proprietary system makes a bad decision, the safety monitor may override it, and I might not even know it happened.

This makes the comparison unfair. The proprietary system has a safety net. The open-source model doesn't. But the safety net is part of the system—it's not a bug, it's a feature.

2. The software stack.

A production system is not just a model. It's a whole software stack: perception, prediction, planning, control, safety monitor, and a dozen other components. The open-source model is just the model.

When I test FSD v12.5.1, I'm testing a full system. When I test openpilot, I'm testing a model that I've integrated into a minimal ROS2 pipeline. The comparison is not apples-to-apples.

3. The training data.

The proprietary system has access to billions of miles of training data. The open-source model has access to a public dataset—maybe a few hundred thousand miles. The training data is the foundation of the model's performance. The proprietary system has a massive advantage that has nothing to do with the architecture.

I can't control for this. I can only acknowledge it.

4. The hardware.

The proprietary system runs on custom hardware designed specifically for that model. The open-source model runs on my garage servers—general-purpose hardware with no architectural optimizations.

The latency comparison is unfair. The proprietary system has a dedicated inference engine. The open-source model is running on a graphics card I bought used on eBay.

What I Can Measure

Despite the asymmetry, there are things I can measure fairly.

1. Trajectory quality.

I can compare the trajectories generated by the open-source model and the proprietary system on the same scenarios. The proprietary system's trajectory is the actual trajectory the car took. The open-source model's trajectory is the output of the model in simulation.

I measure curvature continuity, safety margin, and human likeness. These metrics are input-independent. They don't care if the input is synthetic or real. They just measure the quality of the trajectory.

2. Hallucination rate.

I can measure how often each system produces a hallucination. A hallucination is a trajectory or decision that is physically impossible, semantically inconsistent, or probabilistically implausible. The definition is independent of the input.

The open-source model hallucinates in simulation. The proprietary system hallucinates on real roads. The rate is comparable.

3. Long-tail scenario performance.

I can run both systems through the same long-tail scenarios. The open-source model runs in simulation. The proprietary system runs on real roads. The scenarios are the same.

The pass rate is comparable.

The Data: Open-Source vs. Proprietary

Here's the data from my tests.

The open-source model performs worse on every metric. But is that because it's a worse model, or because it's running in simulation with a limited dataset and a simplified pipeline?

I think it's both. The proprietary systems are genuinely better—they have more data, more engineering, more resources. But the gap is probably smaller than the numbers suggest. If I could run openpilot on real roads, with real hardware, and a full software stack, the performance would improve.

But I can't. So the numbers stand, with the caveat that the comparison is not fair.

The Epistemological Problem

Here's the deeper issue: I don't know what I don't know.

When I test a proprietary system, I'm testing a black box. I feed it inputs, I observe outputs, I measure performance. But I don't know why it performed the way it did. I don't know if a failure was caused by the perception module, the planner, the safety monitor, or something else entirely.

When I test an open-source model, I can trace the failure to the specific component. I can inspect the code, run the debugger, and understand the root cause. I know why it failed.

This is the epistemological problem: the proprietary system is opaque. I can't diagnose its failures. I can only observe them.

And this matters for safety. If a proprietary system fails, I can't tell you why. I can't tell you if it's a one-time fluke or a systematic problem. I can't tell you if the next OTA will fix it or make it worse.

The open-source model is transparent. I can diagnose its failures. I can fix them. I can improve it.

The proprietary system is a black box. I can't fix it. I can only report what I observe.

What This Means for Benchmarking

The open-source vs. proprietary comparison is fundamentally unfair. The open-source model is transparent. The proprietary system is opaque. The open-source model runs in simulation. The proprietary system runs on real roads. The open-source model lacks a safety monitor. The proprietary system has one.

But that doesn't mean the benchmarking is useless. It means the benchmarking is incomplete. It's a snapshot of performance, not a comprehensive evaluation. It's a starting point, not a conclusion.

I publish the data with all the caveats. I tell you what I can measure, and what I can't. I tell you what the numbers mean, and what they don't mean. I tell you what I know, and what I don't know.

The benchmarking is not perfect. But it's better than nothing. And it's a step toward building a more rigorous evaluation framework.