Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB

Automatic support in situational awareness with mini drones

Initial Situation

Right decisions are results of an accurate, precise knowledge of the situation and a correct assessment of possibilities for action. This becomes all the more critical, the more urgent the situation requiring action is and the less likely a second chance is.
Whether the deployment of emergency response teams in disaster situations, the
protection of critical infrastructure or as a part of a military assignment in the face of adversarial forces, situational awareness and orientation primarily comprise the tasks of allocating spatial information and performing exact navigation in the immediate, relevant environment. Visual information, in particular, as provided by our eyes or reconnaissance with imaging sensors, is readily integrated in one’s thought process and consequently in the decision-making process. Particularly for assignments in confined areas as is often the case in security applications, rescue efforts or military operations in urban areas, the mini-drone has recently shown itself to be an agile and versatile reconnaissance tool. Affordable, easy to transport and operate and virtually maintenance-free, modern mini-drone systems are ideal platforms for quickly and flexibly filling any informational gaps in the immediate environment. Despite all the benefits of such reconnaissance systems, one should not forget the disadvantages: the miniaturized navigation systems are prone to drift and in the event of GPS, they may be disrupted easily or fail completely (GPS drop-out in the event of shadowing or multipath propagation), the light  mini-drones shift in the wind, which directly impacts the quality of images or  videos. Though easy operation makes it possible for less specialized personnel to use  mini-drones, they may have difficulty handling the task of spatial orientation in image data.

Task

In order to allow the user to concentrate completely on decision-making, data should be automatically pre-processed to the greatest extent possible. Two tasks are considered decisive in this respect: First of all, the creation of the spatial reference between the sensor images must be handled automatically and not left up to the operator. Situation determination by the operator should therefore take place directly in the image data prepared in the manner described. Secondly, the third dimension should be mapped by means of a 3D reconstruction using image sequences for reconnaissance missions that require more than two-dimensional information. The results can also be used to locate and navigate the mini-drone. Experiences and methods of image processing and 3D object analysis were referred to for the investigation. In particular, the issue associated with reconnaissance using mini-drones must be resolved. This type of reconnaissance is characterized by the use of miniaturized sensor systems, the presence of geometrically adverse flight paths and the analysis of generally unlimited image series.

 



Situation report with super-imposed, geo-referenced video

 

Conclusion

Spatial orientation of sensor images is achieved by means of real-time implicit
geo-referencing by e.g. registering the images on an available orthophoto.
As a result, a global coordinate is generally assigned to each pixel with an
acceptable level of accuracy. Video images can consequently be displayed on a
situation map as can spatial context information from other sources or from a
database and automatically annotated on the image. With 3D reconstruction from mini-drone videos, special attention was paid to the self-calibration of sensors in order to reduce the need for knowledge of internal sensor geometry. Furthermore, sensor self-locating on the basis of unlimited image sequences has been made possible so that concepts for real-time solutions are present. The environment is first reconstructed in the form of dense 3D point clouds. These interim results can be superimposed with the color information from the image data and already provide a

Other than with conventional approaches, the use of explicit model assumptions
like building and roof shapes were refrained from in the modeling of 3D surfaces
that follows in post-processing; instead, triangular meshing of the point cloud
and subsequent surface approximation using L1 splines occurs. This method makes it possible to describe virtually any surface shape and is therefore, in contrast to classic methods, also suitable for reconstructing damaged infrastructure, which is e.g. probable following an earth-quake or another disaster.