SensAI develops methods and driver-monitoring systems for measuring cognitive distraction

The challenge of context-sensitive driver monitoring

Cognitive distraction can impair a driver’s response to traffic situations, even when they continue to look at the road. Conventional driver-monitoring systems primarily detect individual visible characteristics or momentary states and are insufficient for this complex, multidimensional condition. In SensAI, Fraunhofer researchers are therefore developing a multimodal system that combines camera, audio, vital signs and vehicle data using AI. The aim is to detect cognitive distraction in a driving context – in real time and in compliance with data protection regulations – and, in the long term, to avoid unnecessary or inappropriately triggered warnings.

IOSB contribution: Computer vision, activity detection

Fraunhofer IOSB coordinates SensAI and contributes its competence in computer vision, occupant monitoring and driver state modelling. Using in-cabin cameras, we investigate gaze behaviour, body posture, activities and interactions with objects. Together with the partners’ contributions on contactless vital sign monitoring and acoustic scene analysis, specialised encoders are being developed for the respective data sources. Their outputs are consolidated to form a digital twin of the driver’s cognitive state. A domain-specific AI model will then use this to assess the driver’s state and present the relevant influencing factors in an easily understandable format.

Project objective: Demonstrator and validation dataset

SensAI is currently in the development phase. The participating institutes are setting up the sensor systems and data processing and preparing for integration into test vehicles. The key project deliverables are intended to be an integrated demonstrator, a multimodal validation dataset and an empirically based measurement definition for cognitive distraction. This should enable driver monitoring systems to be systematically developed, compared and tested. For vehicle manufacturers, suppliers and testing organisations, this offers opportunities in the areas of sensor integration, AI models, data collection, benchmarks and the development of context-sensitive assistance systems.