INITIATIVE – Intelligent human-technology communication in mixed traffic

Artificial intelligence for the vehicle of the future

Project goals

The project INITIATIVE, funded by the BMWi, aims at developing an AI-based adaptive communication for the integration of automated vehicles in mixed traffic scenarios. For this purpose, the automated vehicle must provide appropriate communication interfaces for external road users (external human-machine interfaces, in short HMI) and for occupants of the vehicle (internal HMI). These interfaces are to be developed and validated in the project, taking into account angle- and time-of-day-dependent recognizability.

In addition, camera-based methods will be used to capture the intention of communication participants in road traffic and take this into account in AI-based communication with each other. This will ensure that messages are transmitted in a way that is adapted to the situation. To avoid misunderstandings between the participants, the messages of the external and internal interaction interfaces must be synchronized accordingly.

The function of the systems and sensor technology is also evaluated across projects. For the identification of relevant participants of a communication, metadata from a networked infrastructure (external sensor technology) is used, which is transmitted via C2X communication. In the overall context of a mixed traffic scenario, AI methods are suitable for synchronizing messages, detecting the intentions of communication participants, and preselecting relevant traffic participants.

Contributions and methodology

The scientific contribution of IOSB lies in the camera-based detection of vehicle occupants and weaker road users, such as pedestrians, in the immediate vicinity of the vehicle. The detection is performed withof machine learning methods, e.g. Deep Learning.

The planned procedure consits of two steps: In the first step, suitable camera systems for the interior and exterior are identified. Each camera and each person involved in the scenario (drivers, pedestrians, etc.) will be detected and individually analyzed using real-time AI methods. For this purpose, basic features are first captured (e.g. body posture) and the relevant gestures for communication in the scenario at hand are evaluated using machine learning techniques.

In the second step, the features are fused in a scene model. Based on this model, further analyses of the overall situation take place. For example, it is necessary to know which persons would like to enter into a dialog with the automated vehicle and which goals these persons pursue. Machine learning methods are going to be used for this. The data processed in this way will then be used by project partners for decision-making by the automated vehicle.

Project partners

Fraunhofer IOSB with

  • Electric-Special Photronicsysteme (ESPS) GmbH: Ambient detection with so-called road side units (e.g. intelligent street lights)
  • Fraunhofer IOSB: AI-based detection of the driver and passers-by in the immediate vicinity of the vehicle. Determination of intention and communication
  • HELLA GmbH & Co. KGaA: Consortium leader, human-machine interaction in the vehicle interior. Operation of the test vehicle
  • Karlsruhe Institute of Technology (KIT) IRS: Agreement and decision making (how should the human-machine interfaces or the vehicle respond to sensor inputs)
  • Karlsruhe Institute of Technology (KIT) LTI: Light-based design of human-machine interaction in outdoor spaces
  • Universität des Saarlandes – Institut für Rechtsinformatik (IFR): Data protection concept, legal framework
  • Version1 GmbH: Assistance in the design of human-machine interaction
  • Würzburger Institut für Verkehrswissenschaften (WIVW) GmbH: Driving simulator studies for data collection and system verification

Interactive Analysis and Diagnosis department

Would you like to learn more about our competence and service spectrum in the field of Interactive Analysis and Diagnosis? Then visit the IAD department page.

More information

about the project “INITIATIVE - Intelligent human-technology communication in mixed traffic“:

  • Project duration: 4/2021 - 3/2024
  • Sponsor: German Federal Ministry for Economic Affairs and Energy (BMWi).
  • Program: New vehicle and system technologies
  • Budget: 4.09 million euros