Personalized, Adaptive, Cooperative Systems for Automated Vehicles

Brief description of the project

In the future, automated vehicles will not only relieve the driver, but will also take over complete vehicle control for a certain time. This creates the need to transfer vehicle control between the driver and automation. For a safe handover, the automated vehicle must know the driver's current capabilities and adapt itself to them. Only efficient and comprehensive driver observation allows safe and comfortable transfer of control.

Project goals

The problem: Today's driver observation ignores secondary activities.

The current state of the art in driver observation is mainly limited to fatigue detection and gaze measurement for attention detection. A prognosis about the driver's distraction, e.g. due to the secondary activity he is engaged in, is sometimes not possible. With increasing automation, it is to be expected that secondary activities already performed by the driver today will occur more frequently. And with the reduction of human driving responsibility, a liberalization of the distraction bans in the passenger compartment is obvious, provided that it can be ensured that the driver can take back control of the vehicle in time.

Secondary activities can be detected by camera. Such systems show their strengths in the above-mentioned application because the possibilities in the vehicle interior are limited and a predefined number of activities to be detected can be narrowed down.

Project result - Driver observation for recording secondary activities.

At IOSB, camera-based driver observation is being expanded to include the body skeleton in addition to the direction of vision and head rotation. In this way, the interior detection system gains knowledge about where the driver's hands, head, and finally the driver's attention are located. By capturing the entire interior space of all objects and occupants, physical activities can be put into context and secondary activities can be classified in a robust way.

Based on the detected secondary activities, the project will investigate takeover periods in which the driver is basically ready to resume driving responsibility. Adaptive human-machine interfaces to alert the driver to the need for a take-over will be adapted accordingly.

Project partners

Fraunhofer IOSB with

  • b.i.g. security service GmbH
  • BMW AG
  • Fraunhofer-Institut für Optronik, Systemtechnik und Bildauswertung
  • FZI Forschungszentrum Informatik
  • Karlsruher Institut für Technologie
  • mVISE AG
  • Robert Bosch GmbH
  • Spiegel Institut Holding GmbH & Co. KG
  • Technische Universität München
  • Videmo Intelligente Videoanalyse GmbH und Co. KG

Further information about the project partners and their responsibilities can be found here.


Interactive Analysis and Diagnosis department

Would you like to learn more about our competence and service spectrum in the field of Interactive Analysis and Diagnosis? Then visit the IAD department page.


Other projects of the IAD department

Here you can find more projects with participation of the Interactive Analysis and Diagnosis department.


More information

on the project PAKoS - Personalisierte, Adaptive, Kooperative Systeme für Automatisierte Fahrzeuge:


The project is funded by the German Federal Ministry of Education and Research (BMBF).