GRIZZLY - Event-Based Object Tracking

Brief Description of the Project

Sensor-based sorting systems often eject objects using compressed air. This requires very precise knowledge of when each object reaches which nozzle. However, traditional line scan cameras typically capture an object only once and provide little information about its subsequent movement. GRIZZLY solves this challenge by tracking sorted objects after they are detected: Their position is updated right up until just before ejection, rather than being predicted based on a single snapshot. To achieve this, the project uses event cameras that do not capture entire images but only register changes in the scene. This allows moving objects to be tracked particularly quickly and with very low latency.

Project Objectives

The goal of the project is to combine the high-speed motion detection capabilities of event-driven cameras with the classification capabilities of a line scan camera. The line scan camera detects and evaluates objects in the material flow, while the event camera tracks their trajectories with very low latency. IOSB’s contribution to the project includes the mechanical demonstrator, the integration of the line scan camera, sensor fusion, calibration to the valve array, and sorting trials. The goal is to ensure that compressed air ejection is no longer based on fixed assumptions about motion, but is triggered precisely and individually for each object—even for heterogeneous, lightweight, or difficult-to-sort materials.

Contribution by Fraunhofer IOSB

The project resulted in a fully integrated laboratory demonstrator that fuses line-scan and event cameras in real time and precisely controls the valve array. In tests with six material classes, the event-based tracking achieved an effective latency of 4 ms and, for the narrowest activation window tested, an average ejection accuracy of 98.2%. GRIZZLY thus significantly outperformed line-scan camera- and frame-based reference methods. The results show that if object positions are reliably known up until just before the nozzle, compressed air pulses can be applied more briefly and with greater precision—a foundation for more precise and resource-efficient sorting systems.

In the Fraunhofer SME HAWK project, the system is being further developed to evaluate in real time, based on the trajectory of the objects after ejection, whether they were deflected correctly. The goal is to use self-learning methods to develop an object-specific control strategy that takes even more object properties into account.

Read our publication on this topic; you can find the download at the bottom left of the website.

Tracking with an Event Camera

The objects move from left to right, are captured by the line scan camera, tracked by the event camera, and precisely controlled at the nozzle array.

Setup of the Experimental System

Layout of the sorting system. The arrows indicate the movement of the bulk material.

Problems with Traditional Sorting Systems and a Proposed Solution

Traditional sorting systems rely solely on a line scan camera, assuming that objects move at exactly the same speed as the conveyor belt between detection and ejection. However, this is not the case for many objects (those that are very light or roll on the belt). In the Grizzly project, all objects are tracked continuously instead, enabling more accurate sorting. In addition, a conveyor belt is not required, which reduces system costs.

Items to be sorted

The objects selected for the test could all, for example, enter the material stream in food production. They have very different dynamic properties.

Results of the Sorting Tests

By activating the compressed air nozzles for only a very short time, you can save on expensive compressed air and eject objects with precision. Thanks to our event-based tracking with minimal latency, we still achieve a very high level of accuracy compared to frame-based tracking and, in particular, purely line-based systems.