Running Open-Source Driving Models on My Garage Servers: A Weekend Experiment

It started as a bad idea. It ended as an education.

I had a free weekend. My wife took Maya to her parents' place. The basement servers were humming, the Miata was clean, and I had three days of uninterrupted time to do something stupid.

So I decided to run open-source driving models on my garage hardware.

Not the production systems I usually test—those are black boxes, proprietary, and locked to specific vehicles. I'm talking about open-source end-to-end driving models: openpilot, UniAD variants, and a few research implementations I found on GitHub.

The goal was simple: could I get these models to drive a virtual car through simulation, and then—maybe—transfer that learning to something resembling real-world behavior?

The answer was: sort of, but not really, and here's why that's the most important thing I learned all year.

The Setup

My garage servers aren't fancy. Three machines:

Total compute: about $8,000 worth of gear, mostly bought used or refurbished. Not a supercomputer. But enough to run a model that takes a camera input and outputs a steering command.

The software stack:

• CARLA (0.9.15) for simulation environment

• openpilot (latest commit from comma.ai's GitHub)

• UniAD (an open-source implementation from a research lab)

• A custom test harness I wrote to run models in a loop and log outputs

• ROS2 (Humble) for message passing and visualization

I also built a small "reality bridge"—a script that could take real-world data from my FLIR cameras and GPS logs, reformat it into the input format expected by these models, and run them on recorded data. This was a crude substitute for real-world testing, but it gave me a way to compare the model's trajectory against what the car actually did.

What Worked: openpilot

openpilot is the most mature open-source driving model. It's been in development for years, has a large community, and is designed to run on real hardware—a Comma Three device that costs about $1,500.

I didn't have a Comma Three. But I had the code, and I had the servers.

Installation was straightforward. The model uses a hybrid approach: a vision model (ResNet-based) processes camera frames, then a planner (RNN-based) generates trajectories. It's not end-to-end in the way Tesla FSD is—it's more of a modular pipeline—but it's close enough for my purposes.

I ran openpilot in CARLA for a few hours. The car drove. Not perfectly, not smoothly, but it stayed on the road. It handled curves, lane changes, and basic intersections.

Then I ran it on recorded data from my test routes. The model output a trajectory for every frame. I could see where the model diverged from reality.

The surprise: openpilot's trajectory was often more conservative than the production system's. It would brake earlier, turn later, and generally avoid any behavior that required confidence. This isn't necessarily a bug—it's a design choice. The openpilot model is tuned to be safe, not to be aggressive.

The catch: the model hallucinated in the same places as the production systems. The drainage culvert on I-94? openpilot swerved right, just like the production FSD did before v12.5.1 fixed it. The roundabout? openpilot hesitated and then attempted to cross the center island, just like FSD, Super Cruise, and BlueCruise.

The hallucinations were reproducible, measurable, and consistent.

This is what I found most interesting: the open-source model, trained on a different dataset, with a different architecture, failed at the same edge cases as the commercial systems. The training data overlap is real. The models are learning the same patterns, and those patterns are wrong.

What Worked (Sort Of): UniAD

UniAD is the most interesting model I tested. It's an end-to-end architecture that unifies perception, prediction, and planning into a single model. It was developed by a research lab and is designed to be a benchmark for autonomous driving research.

The concept is impressive. One model takes in sensor data and outputs trajectories. No separate perception, prediction, or planning modules. Just an end-to-end network that learns to drive.

Getting it to run was a challenge. The codebase assumes you have access to high-end compute, a specific version of PyTorch, and a dataset that has been preprocessed in a specific way. I spent six hours just getting the dependencies to install.

Once it was running, the performance was... underwhelming.

In CARLA, on simple scenarios—straight roads, gentle curves—the model performed well. It produced smooth trajectories, stayed in the lane, and handled basic speed control.

In complex scenarios—intersections, roundabouts, construction zones—the model failed dramatically. It would output trajectories that were physically impossible, or semantically inconsistent, or just plain wrong.

Here's what I learned from UniAD: end-to-end models are brittle. They require massive datasets, careful architecture, and extensive training. The research implementation I tested was trained on a dataset from a specific driving environment (a subset of nuScenes). When I put it in CARLA, the distribution shift was too large. The model didn't know how to generalize.

The model worked in the lab. It failed in the garage.

What Didn't Work: Simulation-to-Real Transfer

This is the hardest problem in autonomous driving, and I felt it in my bones during this weekend.

I ran the models in CARLA. They performed well on the test routes. I was optimistic.

Then I ran them on recorded real-world data from my test routes. The performance dropped by 50-80%.

Here's why:

1. Sensor differences.
CARLA simulates a perfect camera. No motion blur. No lens flare. No dirt on the lens. No low-light noise. The real-world data from my FLIR cameras had all of these artifacts. The models were trained on clean data. They couldn't handle the noise.

2. Lighting and weather.
CARLA's lighting model is simplified. The shadows are sharp. The sun position is deterministic. The real world has soft shadows, changing weather, and dynamic lighting conditions. The models couldn't handle the variation.

3. Dynamic agents.
CARLA's vehicles and pedestrians are scripted. They follow predetermined paths. The real world has drivers who don't signal, pedestrians who jaywalk, and cyclists who swerve. The models couldn't handle the unpredictability.

4. Physics mismatch.
CARLA's vehicle dynamics are a simulation of physics. The real world has tire slip, brake fade, and suspension dynamics that are hard to model. The models' trajectories were physically plausible in simulation but unrealistic in the real world.

5. Distribution shift.
The training data for these models comes from a specific dataset (nuScenes, Argoverse, or similar). The distribution of that dataset is different from the distribution of the real world. The models are optimized for a specific benchmark, not for driving in the real world.

The Real-World Test

At the end of the weekend, I did something reckless.

I took the openpilot model—the one that had performed best in simulation—and tried to run it on real-world data from my test routes. The model was running on my laptop, connected to a CAN interface, processing recorded sensor data from my FSD test runs.

It failed.

The model's trajectory was consistently off. It would turn too early or too late. It would brake too hard or not hard enough. It would make decisions that were physically reasonable but semantically wrong.

The simulation-trained model was fundamentally disconnected from the physical world. It had learned to drive in a simplified environment. It couldn't handle the complexity and noise of real-world driving.

The gap between simulation and reality is not a technology problem. It's a fundamental problem of representation. The simulation is a model of the real world. The model learns from the simulation. But the simulation is not the real world. The gap is irreducible.

What I Learned

The weekend experiment taught me three things.

1. Open-source models are valuable, but they're not production-ready.
They're a great platform for research and experimentation. They're a way to test ideas and understand the state of the art. But they're not ready for deployment. The gap between a research implementation and a production system is enormous.

2. The data is the differentiator.
The commercial systems have access to massive, diverse datasets. Tesla's fleet collected billions of miles of driving data. GM and Ford have their own datasets. The open-source models are trained on smaller, less diverse datasets. The difference in performance is a direct reflection of the data.

3. Simulation-to-real is the hardest problem.
It's not just about training a model. It's about training a model that can generalize to the real world. The distribution shift between simulation and reality is huge. Closing that gap requires innovation, resources, and time.

The Data

Here's what I logged from the weekend:

The drop between simulation and real data is stark. The models perform at 70-80% in simulation, but only at 20-40% on real data.

The openpilot model performed best in simulation and on real data. The UniAD model was more brittle. The custom implementation was a disaster.

What This Means for the Industry

The autonomous driving industry is in the middle of a massive shift to end-to-end architectures. The modular pipelines are being replaced by single neural networks. This is progress.

But the shift to end-to-end brings its own challenges. The models are more powerful, but they're also more opaque. The failure modes are harder to diagnose. The simulation-to-real gap is larger.

The industry needs to focus on bridging the simulation-to-real gap. That means developing better simulation environments, collecting more diverse real-world data, and building evaluation frameworks that can catch the simulation-to-real mismatch.

The open-source community is a critical part of this effort. But the open-source models are not the answer. They're a testbed. They're a way to explore ideas. They're not a replacement for the commercial systems.