AI-based detection, classification and tracking of vehicles in large-scale video data (WAMI)

Large-scale detection of events

For an up-to-date picture of the situation when protecting large-scale infrastructures, in disaster response, border protection, counter-terrorism or traffic analysis, it is essential to capture events in real time.

Conventional video sensor technology only captures a small scene area, i.e. events can only ever be captured selectively in small sub-areas. This makes it virtually impossible to monitor larger areas of the scene and detect and track several vehicles, for example.

Innovative Wide Area Motion Imagery (WAMI) sensor technology enables large-scale surveillance, as it has a high area coverage (in the square kilometer range) with a high level of detail at the same time. By using unmanned aerial systems, continuous position detection can be achieved.

Fraunhofer IOSB is developing methods for the automatic evaluation of WAMI data that enable the detection and tracking of people and vehicles or the recognition of conspicuous behavior patterns.

Wide Area Motion Imagery sensor technology

Wide Area Motion Imagery (WAMI) is a technology for video data acquisition that enables the wide-area monitoring of square kilometres of ground surface with just one airborne optical sensor system. One or more cameras record images simultaneously, which overlap slightly and can therefore be merged to create a larger image with greater ground coverage. At a suitable flight altitude of one to three kilometres, the ground resolution is sufficiently detailed to detect vehicles or even people. To reduce the data rate, the frame rate is reduced to 2 frames/second, for example. Wide Area Motion Imagery (WAMI) sensor technology thus offers

• High scene coverage
• High resolution
• Low frame rate

An example of a WAMI sensor is an experimental system from Hensoldt:

• 2 Hz frame rate
• Resolution: ~150 megapixels
• Scene coverage. 10 km²

WAMI video analysis

Due to the very large amount of data, automatic video evaluation procedures are necessary to extract the relevant information, as an operator is overwhelmed by the large amount of data and the large number of objects. The evaluation therefore requires

Image analysis methods for

• detection and classification of vehicles and for
• multi-object tracking of the vehicles in order to be able to record the course of the journey.
• As well as methods for trajectory analysis in order to filter out conspicuous trajectories.

The Fraunhofer IOSB has developed corresponding procedures that enable the simultaneous detection and automatic tracking of more than 1000 vehicles. A particular challenge here is the high data rates at low frame rates, which cannot be managed with conventional video processing methods.

Our approach combines AI-based view-based detection and motion detection. A fusion of view-based object detections and motion detections for Persistent Tracking (PT) is performed by mapping object detections to tracks by object descriptor for visual similarity