Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB Military vocational training traditionally takes place in two separate worlds: field exercises on the one hand, and computer simulations with abstract models on the other. This creates a gap: how can we convey whether a position is tactically good or bad – not through experience alone, but through comprehensible evaluation criteria? Fraunhofer IOSB has developed a demonstrator called “MilitAR-Sandbox” that closes this gap. Users interact directly with sand, while algorithms project the tactical quality of the position back in real time. The exchange between humans and models can be experienced directly.
© Fraunhofer IOSB, Linda Christ
MilitAR Sandbox: Tactical position assessment made tangible
Vocational training at the interface between sandbox and algorithm
Gap between experience and model logic
Projected contour lines, grids, and SWK on modeled sand
The integration of terrain evaluation algorithms into training or decision-making systems poses a fundamental challenge: the evaluation criteria remain hidden. Trainees and decision-makers find it difficult to understand why an algorithm evaluates a position in a certain way. Computer-assisted scenarios do not solve this problem – they are too abstract. Traditional training tools such as terrain models or tabletop exercises, on the other hand, are rigid and do not respond to user input. The MilitAR Sandbox addresses this gap with a modular, open architecture: it supports rapid iteration and direct validation with real users.
The innovation lies not in the augmented reality sandbox technology itself – which has been used for years in geography and spatial planning education – but in the integration of military analysis algorithms with a documented interface and systematic validation with real users.
Representation of the system architecture with three levels: sensors (3D camera) blue, analysis pipeline green (modular plug-in system), visualization pink (projector + GUI)
The MilitAR Sandbox functions as a tangible user interface – an interface that makes digital data tangible through physical interaction. It combines three components in a standardized architecture:
Sensors and data acquisition: A 3D camera continuously captures the sand surface and generates a digital elevation model. Spatial calibration is performed separately – users can flexibly adjust the position and elevation scale depending on the training or validation scenario.
Analysis pipeline: Modular analysis plugins communicate via standardized programming interfaces (API) and a bus system. This enables the exchange and expansion of individual modules without hardware adaptation. The currently implemented “position selection assistant” evaluates positions according to effectiveness, cover, and accessibility.
Visualization and operation: A projector projects the results directly back onto the sand – the color-coded evaluation, contour lines, and coordinate grid. A graphical user interface allows all parameters to be adjusted in real time. All operations are documented in log files – essential for validation and traceability.
The system has a modular design and can be transported in a small delivery van. This allows flexible deployment at various locations for vocational training and testing.
Tests with soldiers
As part of a test and trial campaign lasting several days, soldiers modeled terrain scenarios and evaluated the algorithmic assessments against their tactical experience. The results were revealing: the position selection assistant achieved a high degree of consistency with expert judgments. Where discrepancies occurred, they could be analyzed immediately – not abstractly on a screen, but directly on the sand with the users. This direct feedback enabled calibration and refinement of the weighting parameters much faster than pure computer simulation.
Training participants reported an increased understanding of the factors that make a position tactically valuable. One soldier summed it up succinctly: “The tool is excellent for rapid prototyping.” The system transformed abstract criteria into tangible contexts.
Vocational training, algorithm validation, co-design
Change in results due to sand modeling
The MilitAR Sandbox addresses three primary application scenarios:
- Vocational training and training: Teaching tactical evaluation criteria through interactive, hands-on learning instead of classroom instruction
- Algorithm validation: Systematic testing of evaluation formulas with real users – reveals blind spots that pure simulation overlooks
- Co-design and rapid prototyping: Thanks to modular, open architecture, development teams can quickly vary evaluation logic and discuss it directly with end users on the system – without having to adapt the basic structure or hardware.
From development to operational use
The MilitAR Sandbox focuses on the early phase of algorithm development: it enables rapid iteration, direct validation with specialists, and user training before evaluation logic is transferred to larger systems. The Fraunhofer IOSB's digital map table, on the other hand, addresses the operational phase – live situation visualization, real-time data integration, and distributed decision support in the field. While the sandbox provides the algorithmic foundation, the situation table integrates it into a broader operational context.
Standardization and scaling
The roadmap aims for interoperability and expandability:
- Object recognition: Control and annotate physical objects in the sand (model vehicles, position markers) Algorithm inputs
- Geographic information system integration: Import of real geodata from geographic information systems (GIS) for replication of operational areas
- Standard interfaces: Adaptation to OGC standards (Open Geospatial Consortium) and military data formats for connection to existing staff information systems
Looking beyond the military mission
While the MilitAR Sandbox is currently designed for military training and algorithm validation, the concept also opens up potential applications in other domains. Scenario-based rapid prototyping with tangible user interfaces could be used in infrastructure planning, disaster control, or urban analysis – anywhere where spatial decisions need to be supported by algorithms and validated with stakeholders. Such civilian or industrial extensions are not currently planned, but they demonstrate the potential of the modular architecture.
The MilitAR Sandbox is being showcased at trade shows and events. Consulting on integration projects and customized adaptations is available.
Military departments, training institutions, and research partners who would like to experience the demonstrator or participate in its further development can contact the “Geo-Intelligence” group at Fraunhofer IOSB.