Pedestrian Detection Failure at Dusk – Severity 4

A pedestrian at 200 meters is a dot. A pedestrian at 80 meters is a person.

This is a story about how the model handled both, and what happened in between.

I was heading west on Washtenaw Avenue, just past the EMU campus, running a routine test loop. The sun was setting. Not dramatically—no brilliant colors or dramatic shadows. Just a gradual transition from daylight to dusk, the kind that happens every evening, the kind that any driver handles without thinking.

At 19:41:22, with light levels still decent, the model saw a pedestrian at 200 meters. It classified them correctly. The visualization showed a little figure on the sidewalk. The car slowed slightly, giving them a wide berth.

Twenty-two seconds later, at 19:41:44, at a distance of 80 meters, the model saw nothing. The pedestrian—the same person, wearing the same clothes, walking at the same speed, on the same sidewalk—had disappeared from the model's perception.

The car continued at speed. No warning. No braking. No trajectory adjustment. The car drove past the pedestrian as if they didn't exist.

I logged it as Hallucination #189. Severity 4. A significant safety violation. Not critical because the pedestrian stayed on the sidewalk. But a clear indicator that the model's perception degrades faster and earlier than any human driver's would.

The Root-Cause Hypothesis

This is not a single failure. It's a cascade.

1. Sensor sensitivity.

The camera's sensor has a limited dynamic range. At 200 meters, in daylight, the pedestrian is well-lit against a bright background. At 80 meters, in dusk, the pedestrian is in shadow against a dark background. The contrast has dropped from 85% to 30%. The camera is struggling to resolve the detail.

The camera's exposure is optimized for the overall scene, not the pedestrian. At 180 lux, the camera is adjusting to the darker sky, which makes the shadowed pedestrian even harder to detect.

2. Training data distribution.

The model was trained on a dataset dominated by daylight images. The nuScenes dataset—one of the most widely used benchmarks—contains about 15% low-light images. The Argoverse dataset has a similar distribution.

The model is an expert in daylight. It's passable in low light. It's a novice at dusk—the transitional state between the two.

The pedestrian at 200 meters was in the daylight distribution. The pedestrian at 80 meters was in the transitional state. The model didn't have enough examples of transitional lighting to maintain the classification.

3. Inference at the perceptual edge.

The model's confidence at 200 meters was 0.94. At 80 meters, it had dropped to 0.42. The model is treating the pedestrian as a signal that is fading into noise.

But here's the problem: the model doesn't know how to handle the uncertainty. It doesn't have a state for "I see a pedestrian, but the light is poor, so I should be careful." It either sees a pedestrian and acts, or it doesn't.

The model at 200 meters: "I am confident. I will brake."

The model at 80 meters: "I am uncertain. I will do nothing."

This is the dangerous version of the trade-off. The model's uncertainty at the perceptual edge translates to inaction, not caution. It doesn't tell the planner to be careful. It tells the planner that nothing is there.

4. Temporal inconsistency.

The model had tracked the pedestrian for 2.2 seconds before losing it. That's 132 frames of visual evidence—a person walking along a sidewalk, a sequence of images that any human would recognize as continuous.

The model dropped the track because it relies on frame-by-frame classification, not temporal reasoning. If a frame fails to classify the object, the track is broken. There's no temporal memory that says: "I've seen a pedestrian here for two seconds; even if I can't see it clearly now, it's likely still there."

This is the deep problem. The model doesn't understand persistence.

What This Feels Like From the Driver's Seat

I was driving the car. I saw the pedestrian. The car's visualization showed them for about 20 seconds.

Then the visualization disappeared.

I didn't brake. I didn't intervene. I had logged the event, and I knew what was happening. I let the car continue—it was a test run, and I needed to see how long the model would take to recover.

The car didn't recover. It drove past the pedestrian as if they were invisible. The pedestrian had their face in a phone, didn't look up, didn't step off the sidewalk. There was no near-miss.

But the failure was real. At 80 meters, the model had lost a pedestrian it had seen at 200 meters. That's not a regression. That's a fundamental failure of the perception system to handle a common condition.

A human driver sees the pedestrian at 200 meters and continues to see them at 80 meters. The cognitive model doesn't drop the track. The brain maintains the representation because the scene is continuous.

The model doesn't have that continuity. It's a sequence of independent frames. The pedestrian disappears, and the model simply forgets.

The Fixes: What OEMs Can Do

1. Improve low-light training data.

The distribution of training data needs to be rebalanced. Not just more low-light data, but more transitional data—dusk, dawn, overcast, rainy. The cases where the light is changing are where the model performs worst.

OEMs need to collect more diverse data. They need to actively oversample the edge cases. The data distribution should match the driving distribution, not the training convenience.

2. Add temporal reasoning.

The model needs a memory. It needs to know that a pedestrian it saw at 200 meters is likely still there at 80 meters, even if the lighting conditions are poor. A temporal consistency module—or even a simple Kalman filter—would catch this failure.

The end-to-end models are powerful, but they're also frame-wise. The industry needs to incorporate temporal reasoning into the architecture. The frame-by-frame approach is insufficient.

3. Exposure optimization.

The camera's exposure is optimized for the overall scene. At dusk, this means the camera is exposing for the bright sky, leaving the pedestrian in shadow. OEMs need to design exposure strategies that prioritize the detection of pedestrians, cyclists, and other vulnerable road users, not just the overall scene.

Dynamic exposure—adjusting based on the scene's critical regions—would help. The camera could expose for the pedestrian, not the sky.

4. Uncertainty-aware planning.

The planner needs to know when the perception system is uncertain. The model shouldn't treat "uncertain" as "nothing there." It should treat uncertainty as a red flag—a signal to slow down, widen the trajectory, or ask for driver attention.

The industry's safety philosophy is currently binary: if the system sees something, it reacts. If it doesn't, it doesn't. The uncertainty-aware approach would add a third state: if the system is uncertain, it should behave more cautiously.

5. Real-world validation at the perceptual edge.

The industry tests the models on standard benchmarks—nuScenes, Argoverse, Waymo Open Dataset. These benchmarks are dominated by daylight scenarios. They don't test the perceptual edge—the dusk, the dawn, the transitional states.

OEMs need to validate their systems at the perceptual edge. They need to run test loops at dusk, at dawn, in overcast conditions. They need to measure the performance at the boundary, not just in the center of the distribution.

The OTA Trail

The improvements have been incremental, not transformative. Each OTA has improved the low-light performance, but none have eliminated the failure mode.

The model can now detect a pedestrian in dusk to 30 meters. That's better than 80 meters, but it's still not good enough. At 30 meters, at 32 MPH, the stopping distance is 40 meters. The model would not be able to stop in time.

The Industry-Wide Problem

This is not just a Tesla problem. I've seen similar failures in Super Cruise, BlueCruise, and the open-source models.

The problem is systemic. The industry's training data is dominated by daylight scenarios. The models are optimized for the head, not the tail. The perception systems are frame-wise, not temporally consistent.

The industry's safety philosophy is binary: see something, react; see nothing, do nothing. It doesn't handle uncertainty, which is the most common state in the real world.

The solution is not incremental improvements. It's a shift in how we design, train, and validate the systems.

The models need to handle the full distribution of driving conditions. Not just the sunny California days, but the Michigan dusks, the Seattle rains, the Minnesota winters. The conditions that make driving challenging—the conditions where the most vulnerable road users are hardest to see.

The Data

This log entry includes complete camera data and model output logs. The temporal sequence shows the confidence degradation frame by frame, and the root-cause analysis walks through the perceptual edge.

• Entry ID: HALL-189

• Severity: 4

• Status: Improved but not resolved

• Linked Scenario: Adverse Weather & Lighting / Low-Light and Dusk