Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB 
The challenge of context-sensitive driver monitoring
Cognitive distraction can impair a driver’s response to traffic situations, even when they continue to look at the road. Conventional driver-monitoring systems primarily detect individual visible characteristics or momentary states and are insufficient for this complex, multidimensional condition. In SensAI, Fraunhofer researchers are therefore developing a multimodal system that combines camera, audio, vital-sign (i.e., physiological) and vehicle data using AI. The aim is to detect cognitive distraction in a driving context – in real time while complying with data protection requirements – and, in the long term, to avoid unnecessary or inappropriately triggered warnings.
IOSB contribution: Computer vision, activity detection
Fraunhofer IOSB coordinates SensAI and contributes its competence in computer vision, occupant monitoring and driver state modelling. Using in-cabin cameras, we investigate gaze behaviour, body posture, activities and interactions with objects. Together with the partners’ contributions on contactless vital sign monitoring and acoustic scene analysis, specialized encoders are being developed for the respective data sources. Their outputs are combined to form a digital twin of the driver’s cognitive state. A domain-specific AI model will then use this to assess the driver’s state and present the relevant factors in an easy-to-understand form.
Project objective: Demonstrator and validation dataset
SensAI is currently in the development phase. The participating institutes are setting up the sensor systems and data processing and preparing for integration into test vehicles. The project is expected to deliver an integrated demonstrator, a multimodal validation dataset and an empirically based measurement definition for cognitive distraction. This should enable driver monitoring systems to be systematically developed, compared and tested. For vehicle manufacturers, suppliers and testing organisations, this offers opportunities in the areas of sensor integration, AI models, data collection, benchmarks and the development of context-sensitive assistance systems.
