I spend my days optimizing RAG pipelines. My team builds an open-source inference framework that helps developers deploy retrieval-augmented generation systems with sub-second latency. We obsess over the same things every LLM engineer does: chunk size, embedding dimension, retrieval speed, generation latency, tool-calling accuracy.
I spend my nights testing driving models. I run the same test harness, the same metrics, the same data collection. I measure latency, hallucination rate, trajectory reasonableness. I log every failure.
For the first six months of this project, I kept these two worlds separate. Day job: LLMs. Night job: driving models. Two different problems, two different domains, two different mental models.
Then one night, while staring at a latency trace from a FSD run, I had a realization: these are the same problem.
The RAG pipeline and the driving model are both doing the same thing. They take a query (a question, or a sensor frame), retrieve relevant context (documents, or map data), and generate a response (an answer, or a trajectory). They both have to do it fast, accurately, and within the constraints of the hardware.
The lessons I learned optimizing RAG pipelines at work directly informed how I measure latency in driving models. And those lessons changed what I look for in the data.
The RAG Pipeline: A Quick Primer
For the non-LLM engineers in the audience, here's what a RAG pipeline does.
A user asks a question: "What's the capital of France?"
The system doesn't just generate an answer from memory. It retrieves relevant documents from a knowledge base, then generates an answer based on those documents. The retrieval step ensures the answer is grounded in something real. The generation step produces a fluent, natural response.
The pipeline looks like this:
Query embedding: The user's question is converted into a vector embedding.
Retrieval: The embedding is used to search a vector database for similar documents.
Re-ranking: The retrieved documents are scored and ranked by relevance.
Generation: The most relevant documents are fed into the LLM, along with the original question, to produce the final answer.
Each step adds latency. Each step is an opportunity for optimization. And each step has a knobs-and-dials trade-off: you can go faster, but you might sacrifice accuracy. You can be more accurate, but you might add latency.
At my startup, we spend our days tuning these trade-offs. We measure retrieval latency (the time to get the documents), generation latency (the time to produce the answer), and end-to-end latency (the total time). We optimize for the 95th percentile, not the mean, because the 95th percentile is what users actually experience.
The same logic applies to driving models.
The Driving Model Pipeline: The Same Structure
A driving model does the same thing as a RAG pipeline.
Perception: Sensor data (cameras, LiDAR, radar) is processed into a representation of the world. This is the "query embedding" step.
Map retrieval: The relevant map data—lane geometry, traffic signs, speed limits—is retrieved from the HD map. This is the "retrieval" step.
Fusion: The perception data and map data are fused into a unified representation. This is the "re-ranking" step.
Planning: The fused representation is fed into a planner to generate a trajectory. This is the "generation" step.
Control: The trajectory is translated into steering, braking, and throttle commands.
The pipeline is structurally identical. Both are processing an input, retrieving relevant context, generating an output, and doing it fast enough to be useful.
The latency constraints are different—a RAG pipeline can take 500 ms and still be acceptable; a driving model needs to produce a trajectory every 50 ms—but the problem is the same: how do you optimize latency without sacrificing quality?
Retrieve Early, Retrieve Often
In a RAG pipeline, the biggest latency bottleneck is almost always the retrieval step. Vector search over millions of documents takes time. The naive approach is to retrieve a large set of documents (say, 100) and then re-rank them to the top 5. But that means you're paying the cost of retrieving 100 documents and the cost of re-ranking them.
The optimized approach is to retrieve a smaller set, re-rank, and then use the re-ranking results to guide a second retrieval pass. This reduces the total number of documents you have to process.
The same lesson applies to driving models. The map retrieval step is a latency bottleneck. The model needs the map data to understand the road, but retrieving the map data takes time.
In the FSD v12.5.1 update, Tesla reduced the latency by 35 ms. My logs show that the reduction came from a change in the map retrieval pipeline. The model is now retrieving a smaller, more focused set of map data—only the relevant section of the road, not the entire surrounding area. The loss of some context is acceptable because the relevant context is what the model needs for the immediate trajectory.
The optimization is the same as a RAG pipeline: you don't need all the documents. You need the relevant ones. And you need them fast.
Parallelize What You Can
In a RAG pipeline, there are steps that can be parallelized and steps that can't. The retrieval and re-ranking steps can be parallelized across multiple GPUs. The generation step, however, is sequential—the LLM generates tokens one at a time, and you can't speed it up without sacrificing quality.
The optimization is to parallelize the retrieval and re-ranking, and then pipeline the result into the generation step. The generation step waits for the retrieval result, but the retrieval result is already ready because it was processing in parallel.
The driving model has the same constraint. The perception and planning steps are parallelizable—you can process sensor data from multiple cameras, LiDAR, and radar simultaneously. But the planning step is sequential—you can't generate the trajectory until you have the fused perception and map data.
The FSD v12.5.1 latency improvement also appears to come from better parallelization. The logs show that the perception and map retrieval steps are now overlapping more than before. The model is processing sensor data while simultaneously retrieving the map data. The total latency is the sum of the longest path, not the sum of all paths.
This is a fundamental lesson from systems engineering: parallelize what you can, serialize what you must. The driving model is doing this more effectively in v12.5.1.
The 95th Percentile Matters More Than the Mean
In the RAG world, we measure the 95th percentile latency, not the mean. The mean is misleading because it's dominated by the common case. The 95th percentile is where the system actually performs in real-world conditions.
The same applies to driving models. The mean latency might be 90 ms, but the 95th percentile might be 200 ms. The 200 ms cases are the ones that matter. They're the moments when the model is struggling to generate a trajectory, the moments when the car hesitates, the moments when the driver might need to intervene.
In my driving model testing, I always report the 95th percentile. It's the metric that correlates with real-world experience. A system with a 95th percentile latency of 112 ms feels faster than a system with a mean latency of 80 ms but a 95th percentile of 200 ms.
The industry tends to report the average. The average makes the system look good. The 95th percentile tells you what the system is actually doing in the hard cases. I don't care about the average. I care about the 95th percentile—the one that might kill you.
Hallucination Is a Latency Problem, Not Just an Accuracy Problem
In RAG, hallucination is often a latency problem. When the system doesn't have time to retrieve enough context, it hallucinates more. The generation step fills in the gaps with plausible-sounding falsehoods. The solution is to either reduce the latency of the retrieval step (so it can retrieve more context) or accept a higher latency for the generation step.
The same is true in driving models. Hallucination is often a symptom of latency constraints. The model doesn't have time to process all the sensor data and retrieve all the map data. It produces a trajectory based on incomplete information. The trajectory is plausible but wrong.
The FSD v12.5.1 hallucination rate dropped by 33% across the board. This is not just an accuracy improvement. It's a latency improvement. The model now has more time to process the sensor data and retrieve the map data, which reduces the hallucination rate.
If you want to reduce hallucinations, you have two levers: improve the model's understanding of the world, or give it more time to process the world. The v12.5.1 improvements are a combination of both.
The System Is the System
In RAG pipelines, the performance is a function of the entire system. You can't optimize one part in isolation. The retrieval step is coupled to the generation step. The re-ranking step is coupled to the retrieval step. You have to optimize the whole pipeline, not just the individual components.
The driving model is the same. The perception, map retrieval, fusion, planning, and control steps are coupled. You can't optimize the perception step without considering the planning step. The entire system has to work together.
The v12.5.1 improvements show that Tesla is thinking systemically. The latency reduction came from a combination of improvements across the pipeline—better parallelization, faster map retrieval, optimized inference. The hallucination reduction came from a combination of improved perception and better latency.
The system is the system. You can't optimize one part without considering the rest.
What I Learned from the Day Job
The day job taught me to think in terms of systems, not components. To measure the 95th percentile, not the average. To parallelize what you can and serialize what you must. To treat hallucination as a latency problem, not just an accuracy problem. To understand that the system is the system, and you can't optimize one part in isolation.
These lessons are not domain-specific. They apply to any system that takes an input, processes it, retrieves relevant context, and generates a response. They apply to RAG pipelines. They apply to driving models. They apply to any real-time system that has to be both fast and accurate.
When I test a driving model, I'm not just testing the model. I'm testing the entire pipeline—the hardware, the software, the inference engine, the map retrieval system. The model is just the generating step. The pipeline is what makes it work in the real world.
One More Thing: The Hidden Cost of End-to-End
The end-to-end architecture in driving models is analogous to the "end-to-end" approach in RAG. In both cases, you're replacing a modular pipeline with a single model that processes the input and generates the output.
The advantage of end-to-end is simplicity and performance. The advantage of end-to-end is that you can optimize the entire pipeline as a single system, and you can potentially learn better representations.
The disadvantage is that you have to test the system as a whole, not the individual components. You can't isolate the perception step from the planning step. If the model hallucinates, you don't know if it's a perception problem or a planning problem.
In the modular pipeline, you can test each component separately. In the end-to-end pipeline, you have to test the entire system. It's the same challenge as testing a RAG system end-to-end.
My day job teaches me to be comfortable with the black box, but also to be rigorous about the methodology. I can't see inside the model, but I can measure what comes out. I can control the input, run the same test repeatedly, and observe the output.
It's the same methodology. The same discipline. The same respect for data.
The autonomous driving industry doesn't think about latency the way the LLM industry does. It thinks about disengagement rates, collision statistics, and safety outcomes. It doesn't think about the 95th percentile of end-to-end latency, the effect of map retrieval on planning quality, or the systems-level trade-offs between accuracy and speed.
It should.
The industry is building real-time systems that have to process high-dimensional sensor data, retrieve relevant map data, and generate a trajectory in the time it takes a human to perceive an event and begin a reaction. The systems are fast, but they're not fast enough. The systems are accurate, but they're not accurate enough.
We need to treat these systems as the real-time systems they are. We need to measure the 95th percentile latency, not the average. We need to parallelize the retrievable steps and serialize the sequential ones. We need to treat hallucination as a latency problem as much as an accuracy problem. And we need to understand that the system is the system, and you can't optimize one part without considering the rest.
The day job taught me that. The night job is helping me build the tools to measure it.