By Steven D. Fleming, Ph.D., and Ryan McAlinden, the University of Southern California; Matt S. O’Banion, Ph.D., Christopher Oxendine, Ph.D., and William Wright, Ph.D., the United States Military Academy at West Point; and Ian Irmischer, Ph.D., the United States Air Force Academy
Geospecific 3D terrain representation (aka reality modeling) is revolutionizing geovisualization, simulation, and engineering practices around the world. In tandem with the rapid growth in unmanned aerial systems (UAS) and small satellites, reality modeling advancements now allow geospatial intelligence (GEOINT) practitioners to generate three-dimensional models from a decentralized collection of digital images to meet mission needs in both urban and rural environments. Scalable mesh models deliver enhanced, real-world visualization for engineers, geospatial teams, combatant, and combat support organizations. In this, reality modeling provides a detailed understanding of the physical environment, and models allow installation engineers and GEOINT practitioners to quickly generate updated, high-precision 3D reality meshes to provide real-world digital context for the decision-making process.
On the facilities engineering front, projects can be planned, designed, and completed more quickly and easily with lower costs. These models integrate with existing CAD tools to save time and money in facility design. Facility operations and maintenance, construction site inspection, asset management, environmental management, and management of military training ranges all benefit from this technology. In deployed environments, ground commanders, military planners, engineers, and practitioners can use 3D models for mission planning and rehearsal, terrain generation, route mapping and clearance, base layout and design, infrastructure planning, IED-modeling and post-blast assessment, cover/concealment, and more. For post-attack recovery efforts, practitioners can quickly send drones to capture existing conditions, then model the damage and map unexploded ordnance to assess the situation and develop a recovery plan—while minimizing exposure to deployed troops. Operational units such as infantry and special operators can produce models to map the battlespace and to enhance defensive preparation efforts or model assault objectives. Units can now quickly determine mission conditions and answer questions such as: Can our vehicles fit in that alleyway? Can we land a helicopter on that roof? What is my line of site at this location?
- This article is part of USGIF’s 2019 State & Future of GEOINT Report. Download the PDF to view the report in its entirety.
Modeling and Simulation Possibilities
The use of non-traditional, decentralized data collection sources supports next-generation digital Earth representation and the desires to achieve unique 3D visualization and terrain development for many U.S. government modeling and simulation (M&S) communities, including the Department of Defense (DoD) and the Intelligence Community (IC). Work is being done for U.S. Joint Staff-funded projects designed to assist the DoD in creating the realistic, temporarily accurate, precise, and informative representations of the physical and non-physical landscape. In addition, and as a part of the Army Synthetic Training Environment (STE) Cross Functional Team (CFT), reality modeling is helping to establish a next-generation government/industry terrain standard for M&S hardware and software for use in training and operational applications. To this end, the following goals are being advanced:
- Construction of a single, authoritative, updated 3D geospatial database for use in next-generation simulations and virtual environments.
- Utilization of commercial cloud-front solutions for storing and serving geospatial data.
- Protocols for procedurally recreating 3D terrain using drones and other collection equipment/sensors.
- Reduction of cost and time for creating geospecific datasets for M&S.
- Utilization of non-traditional, open, and market sources of geospatial data.
There are anticipated impacts of potential applications enabled by this work. The One World Terrain (OWT) effort is principally centered around understanding and planning for the next-generation of M&S technology. More specifically, OWT relates to the feasibility of turning collected terrain data into simulation-usable terrain features that can be employed in real-time by simulation platforms. This work hopes to demonstrate how rapid terrain generation and user-driven social media data may be incorporated in real- or near-real-time into a virtual or constructive environment for geovisualization and simulation applications.
As more and more data saturates the digital landscape, we have become increasingly reliant on technologies to help sift, sort, analyze, and visualize. One example is the way one collects, processes, and uses geospatial data. The field has evolved rapidly from paper maps with acetate overlays, to the digital 2D maps of the 1990s and 2000s, to the 3D immersive representations we see today. This data continues to grow in abundance and requires a new breed of cross-disciplinary collaboration and research to ensure its utility is maximized.
Identifying and developing ways for users to exploit and better understand the 3D world through automation is becoming increasingly popular and relevant. Virtual and augmented reality continue to proliferate and are now mainstays in society. Map-based data are used in many of the most popular applications on common devices, from consumer review apps (like Yelp), to ride sharing, to games. However, the ability to produce and visualize 3D geospatial content for these devices remains elusive. The process for generating such content is existentially a human-intensive process, and, as a result, time-consuming, inefficient, and inconsistent. Spatial scientists are attempting to ease the burden of creating and using 3D terrain content in electronic devices as quickly and cost-effectively as possible. Ultimately, the research goal is to achieve complete automation of how one creates the digital world around us, removing the human from the loop.
In order to understand the challenges with 3D geospatial terrain, the problem is best decomposed into its constituent parts: collection; creation (processing); storage and distribution; and application. More precisely, the questions often asked when assessing 3D terrain include: How is source terrain data collected? How is that data processed into a form digestible by an application? Where is it stored and how is it distributed? And how is it used by consumers?
Research has been conducted on the challenges presented by 3D terrain data for several decades, harkening back to the days of the Topographic Engineering Center (TEC). In the DoD, tremendous efforts have focused on building the Army’s suite of next-generation interactive simulation and training platforms. Years ago, terrain was often considered the “Achilles’ Heel” of simulators. Its generation is time-consuming, expensive, manpower-intensive, and fraught with vagaries that result in unrealistic, unsatisfying, and often uncompelling synthetic experiences. Simulation environments are often created with entities floating above the terrain because of correlation issues, or virtual characters passing through walls because the models were not attributed correctly. And until recently, creating the virtual terrain in applications was purely a manual activity, with artists, modelers, and programmers spending significant time and money to create one-off terrain datasets that were rarely able to be repurposed in other rendering environments. Limitations in processing and artificial intelligence (AI) and poor-quality source data compounded the problem for decades, stalling attempts to fundamentally change the way terrain is created for virtual applications.
However, over the past 5 to 7 years, the introduction of cloud computing, better and cheaper processors and graphics processing units, and new sources of high-resolution terrain data (unmanned systems, airborne and terrestrial LiDAR, small satellites, crowdsourcing, photogrammetry, and commercial industry mapping resources such as Bing or Google Maps) have provided new procedures for terrain generation. The opportunity has arisen to reduce the time and cost for creating “digital dirt” by automating what were previously manual efforts. Automated functions include procedurally-generated textures and polygons, the correlation and linking of datasets, and adding application-specific attribution to models that allows the simulation to reason with colliders, navigation meshes, and other entities. Leveraging these advancements and combining them with specific research areas has allowed the M&S community to exponentially grow its capabilities and output.
Unlike traditional geospatial research (which often falls to academic geography departments), this line of research incorporates the disciplines of geomatics (e.g., remote sensing, surveying, navigation, and positioning) and computer science (e.g., AI, computer vision, image processing, and computer graphics). The ability to automate from “source to runtime” requires algorithmic approaches that can add, manipulate, and preserve data attributes and qualities that allow the data to be rendered and simulated in 3D. This collaboration is crucial as disciplines seek to exploit data, computational resources, and knowledge. Collaboratively, much of the current work is focused on automating the workflow from collection to application. Specifically, the steps to this end include:
- Collection: How can one organically collect and fuse their own 3D geospatial data, use existing open and market-based sources, and leverage previously-collected data? To accomplish this, one relies on automatic geo-referencing and correlation of the data using traditional GIS techniques such as ground-sampling distance as well as newer techniques for 3D terrain data collection such as automated photogrammetric reconstruction.
- Creation (Processing): This dynamic work with the data entails manipulating source GIS data into a form that a runtime application can not only display but reason against. Techniques rely heavily on machine learning as well as more traditional AI techniques to analyze and segment the data into its constituent parts (elevation, vegetation, roads, buildings, etc.).
- Storage & Distribution: 3D terrain data can be very large, on the order of several petabytes to cover the Earth’s surface at one-meter resolution. Storing all of the data in the cloud is cost-prohibitive, and storing it locally is impractical for bandwidth and throughput reasons. Researching strategies and techniques for storing and serving the data is central to addressing these challenges. Basic research centers on identifying intelligent storage means (dynamic load balancing and cloud instancing; hot versus cold storage) that allow for a cost-effective, yet efficient 3D storage and distribution mechanism.
- Application (Rendering & Simulating): Displaying terrain data is where the most fundamental research challenges remain. Adding semantic labels and metadata to the underlying data is critical so the engine can differentiate how the data is to be used at runtime (e.g., whether something will drive on it, shoot through it, move through it, hide behind it, etc.). This is where some of the most manually intensive activities continue to be centered, such as adding colliders around buildings, navigation meshes, lighting properties, and higher-order metadata for AI agent reasoning. Moreover, much of the investment for automating the terrain workflow has been in processing, with rendering and simulation often relegated to the sidelines because they are viewed as production activities.
Advanced Applications and Future Use
This research need stretches across the workflow from collection to application. Early efforts have led to many outcomes, including the purchase of tactical decision kits for the U.S. Marine Corps that allow small units to organically manage their own geospatial holdings. Unit operators now regularly collect image data and provide it to others in the force, as well as researchers for additional classification and segmentation experiments. Agriculture, architecture, and law enforcement professionals have also applied these techniques. Work is also being done in mapping the commercial infrastructure (sports venues, college campuses, and many other urban locations) to assist these and other communities with specific challenges such as infrastructure protection, flood analysis, site surveys, structural integrity, and historical/anthropologic research activities. Ultimately, researchers hope to revolutionize the way the world collects, processes, and serves 3D geospatial data with long-term goals being to obviate the need for human intervention, and to use automation to more quickly and cost-effectively deliver terrain data to the point-of-need. In aggregate, focused research hopes to continue and evolve with outcomes including:
- Machine learning for additional classification and segmentation of meshes and point clouds.
- Alternative sources of data collection and fusion.
- Algorithmically adding attribution to 3D data for use in runtime applications.
Conclusively, 3D geospatial data will continue to increase in abundance and quality. Therefore, its use and the research to ensure its utility, integrity, and production are critical so the GEOINT Community can produce more accurate and reflective digital representations of the physical world. These representations service the reality modeling community as a conduit to revolutionizing geovisualization, simulation, and engineering practices around the globe.
Headline Image: A 3D terrain model and line-of-sight tool created with data collected via autonomous drones. Image by One World Terrain.