Intelligent video surveillance enhances safety and privacy

Pilot project in Mannheim: Algorithm-based video surveillance in public spaces to combat street crime

© Fraunhofer IOSB
The algorithm first extracts the position of the bodies and limbs (“digital skeleton“) from the video recordings. On this basis, the AI can then classify certain behaviours.
System architecture of the project: Conventional video evaluation and the intelligent video evaluation under development operate in parallel on different computers.

Computers recognise certain behavioural patterns

The pilot project for intelligent, algorithm-based video surveillance is about fighting street crime in public spaces. A person's face or identity does not play a role, only their behavioural patterns. The aim is for computers to automatically recognise certain behavioural patterns that indicate crimes - such as hitting or kicking - and alert police officers at the command and situation centre. The police officers can then look specifically at these situations and decide whether or not intervention is necessary.

Project goals: Improved safety AND better data protection

On the one hand, we strive to make the work of police officers in the command and situation center easier, which ultimately results in improving the safety of everybody. No one can follow the images from dozens of video cameras in parallel and with constant attention for hours on end. An assistance system that pre-filters relevant scenes and directs the officers' attention specifically to where it is worth taking a closer look is a great help.

On the other hand, this approach offers new possibilities to bring video surveillance and privacy together: Once precarious situations are reliably detected, all images may be pixelated in normal operation; they will only be visible in full resolution when the system comes to the conclusion that a human should take a closer look. 

Implementation steps

In the project, the intelligent system runs in trial mode in parallel to the human evaluation of the recordings (which are stored for 72 hours and then overwritten). Experimental software developed in earlier research projects under laboratory conditions is used. It is now being adapted step by step for use in real life. First, the software is able to recognise people and objects. In a second step, it can detect the postures and movements of people. On this basis, the built-in artificial intelligence (AI) will then learn to recognise police-relevant situations. For this, the AI needs training data, i.e. recordings of corresponding situations, in order to be able to filter out similar behavioural patterns from the variety of images. However, such training data from public spaces practically do not exist for data protection reasons. That is why the project in Mannheim has a real pilot character. How well, with what reliability and what rate of misjudgement the recognition of police-relevant events succeeds, will only be proven in the course of the project.

Project partners

  • Polizeipräsidium (Police Headquarters) Mannheim
  • City of Mannheim
  • Ministerium für Inneres, Digitalisierung und Migration (Ministry of the Interior, Digitalization and Migration) Baden-Württemberg

Mannheim is taking a pioneering role with this project. A specific amendment to the Baden-Württemberg police law at the end of 2017 created the legal basis for such a use of technology in the first place. At that time, the city and the Mannheim police headquarters as well as the Fraunhofer IOSB had already found themselves as project partners. In the course of 2018, video surveillance hardware was procured in Mannheim and installed at locations with significantly increased street crime - a total of 76 cameras are planned, which transmit their images to the situation centre at Mannheim police headquarters via their own, sealed-off fibre optic network. There is no connection to the internet at any point.

Mannheim already made much noted experiences with video surveillance in the years 2001 to 2007: the "Mannheim Way" became synonymous at that time with video surveillance that aims less at recording and subsequently solving crimes than at immediate and quick intervention - and thus ultimately at prevention. At that time, the Mannheim police achieved an average response time of two and a half minutes. As a result, crime at the observed hotspots decreased so much that the legal basis for video surveillance ceased to exist and the then (analogue) system had to be switched off. In recent years, the security situation deteriorated again.

Department IAD of Fraunhofer IOSB

You want to learn more about our projects in the field of “Intelligent Video Surveillance“? Then visit the page of our VID department and find out about other projects.

Project profile

Intelligent Video Surveillance in Mannheim

Project start: December 3, 2018

Project duration: 5 years

Project volume: City and state invest a total of approx. 1.6 million euros, of which 0.5 million euros go to Fraunhofer IOSB