All-Weather Pedestrian Detection Benchmarking

Motivation

Automated Emergency Braking (AEB) for pedestrians is a cornerstone of vehicle safety, but its real-world reliability especially in edge cases depends on perception that continues to work outside the dry, well-lit conditions that today's consumer and regulatory test protocols (e.g. Euro NCAP AEB/VRU, NHTSA FMVSS No. 127) are built around. Rain, fog, darkness, and window contamination are explicitly excluded from most standardized tests – yet these are the conditions in which sensing systems are most likely to fail. Manufacturers of cameras, LiDAR, radar, and optical components therefore face a persistent research gap: there is no standardized way to compare how different sensing modalities behave, and degrade, once weather is added to the equation.

What Fraunhofer IOSB offers

Fraunhofer IOSB's Human-AI Interaction group develops and operates end-to-end test methodologies that make sensor and perception performance measurable, comparable, and reproducible under adverse environmental conditions. In a recent industrial R&D project, we designed and executed a systematic, multi-modal benchmark of four ADAS-relevant sensing technologies – RGB camera, long-wave infrared (LWIR/thermal) camera, LiDAR, and imaging radar – for pedestrian detection in rain, fog, and night conditions. The project combined controlled-environment data acquisition, systematic ground-truth annotation, and a novel evaluation framework that separates physical sensing limits from algorithmic detector performance. The results were published in a peer-reviewed conference paper (“All-Weather Pedestrian Perception for AEB across RGB, LWIR, LiDAR, and Radar”) and illustrate the kind of engagement Fraunhofer IOSB can deliver for automotive OEMs, Tier-1 suppliers, and component manufacturers.

This type of project is a representative example of what clients can commission from us: a fully instrumented, independent test campaign covering data acquisition, annotation, and quantitative benchmarking of one or several sensing modalities against a defined operational domain – without requiring the client to build up in-house test infrastructure or annotation capacity.

Our methodology: separating physics from algorithms

A central contribution of our approach is a three-level evaluation framework that disentangles why a sensing system fails – physical signal loss versus algorithmic weakness – so that improvement effort can be targeted precisely:

• Sensor availability – a physics-level recall metric that measures whether a target is present in the raw signal at all, independent of any detection algorithm.
• Detector quality – standard object-detection Average Precision (AP), computed only on the subset of targets that are physically visible to a given sensor, isolating pure algorithmic performance.
• System-level performance – an end-to-end AP metric that scales detector quality by sensor availability, yielding a single, comparable score across fundamentally different sensing modalities.

This framework allows a client to see immediately whether a performance gap in adverse weather is caused by the sensor's physics (e.g. attenuation, backscatter, window contamination) or by the detection software – and therefore where investment should be directed.

Test infrastructure and data acquisition

Test campaigns are carried out at controlled environmental simulation facilities capable of generating reproducible rain (variable rate, up to and beyond 60 mm/h) and fog (multiple density / visibility levels), combined with day and night illumination, including standardized low-beam headlamp conditions. A geometric target grid derived from Euro NCAP / NHTSA pedestrian scenarios is used, with multiple lateral positions and longitudinal distances sampled systematically. Target sets can include certified pedestrian dummies (standard and actively heated, to probe thermal contrast), as well as human subjects in light and dark clothing (to probe optical contrast) – all permuted using a Latin-square design to remove positional and temporal bias.

Where required, sensor housings can incorporate exchangeable optical windows and active wiper/cleaning systems, allowing the impact of window wetting, contamination, and cleaning strategy to be quantified separately from the sensor's underlying detection capability – a critical factor for camera- and thermal-based systems integrated behind vehicle glazing.

Data annotation and ground truth

All recorded sensor streams – images, thermal frames, and point clouds – are synchronized by timestamp and manually annotated per modality (2D bounding boxes for camera/thermal imagery, 3D bounding boxes for LiDAR and radar point clouds). Targets are labeled whenever discernible in the raw signal, even where detection is expected to fail, so that sensing limits and detector limits can be evaluated independently. Annotation and data handling follow data-protection requirements (GDPR/DSGVO) throughout.

Illustrative results

The figures below illustrate the type of output such a benchmark produces: distance-resolved, end-to-end system performance curves per modality and condition, and qualitative detection examples that make sensor-specific failure modes tangible for engineering and management audiences alike.

Across the tested modalities, we found that no single sensing technology provides all-weather reliability on its own: each modality has characteristic, quantifiable strengths and blind spots depending on weather, illumination, target contrast, and optical window condition. Making these trade-offs explicit and measurable is precisely the value such a benchmark project delivers to a client's sensor selection, fusion strategy, or homologation planning.

What you can contract from Fraunhofer IOSB

Building on this project, we offer the following services to automotive OEMs, Tier-1/Tier-2 suppliers, and sensor or component manufacturers, either as a full package or as individual work packages:

• Custom dataset creation – design and execution of controlled-environment (rain, fog, day/night, target-contrast) or naturalistic data-collection campaigns for camera, thermal, LiDAR, and/or radar sensors.
• Multi-modal ground-truth annotation – 2D/3D bounding-box and other labeling services with defined quality assurance, tailored to your object classes and use case, carried out in compliance with data-protection regulations.
• Independent sensor and detector benchmarking – application of our sensor availability / detector quality / system-level AP framework to your own sensors, detectors, or fused perception stack.
• Adverse-weather and night test campaigns – access to controlled rain and fog test facilities and standardized target sets for reproducible, repeatable adverse-condition testing beyond current regulatory envelopes.
• Window / glazing and housing evaluation – quantification of how optical windows, coatings, and cleaning (wiper) strategies affect camera and thermal sensor performance in precipitation.
• Test protocol and methodology consulting – support in defining internal test specifications, functional-safety arguments, or inputs for future homologation and assessment-program discussions.
• Multi-sensor fusion evaluation – assessment of complementary sensor combinations against a shared, standardized adverse-weather protocol.

All engagements are scoped individually and can range from a short benchmarking study using an existing dataset to a full campaign covering test design, data acquisition, annotation, evaluation, and reporting.

Outlook

• Extending controlled adverse-weather testing with explicit accounting for window condition and target contrast.
• Incorporating ego- and target-motion to assess radar performance under dynamic driving conditions.
• Evaluating multi-sensor fusion within the same standardized adverse-weather protocol.
• Expanding the test framework to further obscurants such as road spray and snow.