With urban and underground warfare looming, GEOINT takes center stage
ONE WORLD TERRAIN
Redefining military readiness through virtual simulation training for urban engagements
Cities are living, breathing things. Like organisms, they’re constantly changing. In the 19th century—before indoor plumbing, electricity, or automobiles—the typical city was a cesspool of decay, disorder, and disease. Without municipal police departments, traffic laws, or indoor plumbing, streets were congested with crime, horses, and waste. Streets were narrow, buildings short, and apartments overcrowded. Skyscrapers? Ridesharing? Condominiums? Urbanites of the past could not have even imagined the cities of today.
Of course, cities are still changing. In a few short decades, futurists predict, cars will drive themselves, buildings will generate their own energy, drones will police streets, augmented reality will make streetscapes searchable, and high-speed hyperloops will transport passengers really far, really fast. Urban planners hope for cities of the future to be smart, sustainable, and safe.
Unfortunately, some of them might also be warzones, according to the U.S. Army, whose Army Future Studies Group (AFSG) has identified megacities—cities with populations of 10 million people or more—as a growing concern for the U.S. military.
“Megacities are rapidly becoming the epicenters of human activity on the planet and, as such, they will generate most of the friction which compels future military intervention,” AFSG, formerly known as the Army Chief of Staff’s Strategic Studies Group, said in its 2014 report, “Megacities and the United States Army: Preparing for a Complex and Uncertain Future.” To ignore megacities, the report said, is to ignore the future.
The U.S. Army Training and Doctrine Command echoed AFSG’s sentiment in its 2018 pamphlet, The U.S. Army in Multi-Domain Operations 2028: “Dramatically increasing rates of urbanization and the strategic importance of cities … ensure that [future] operations will take place within dense urban terrain.”
But urban terrain isn’t just dense. It’s also diverse. Along with threats aboveground—on roads and rooftops or inside buildings—warfighters in cities might also face threats below ground in basements, sewers, subways, and tunnels.
The United States has already faced and defeated enemies both in and under cities. Across the Department of Defense (DoD), however, there’s an acknowledgment that what was once the exception might one day be the rule. And so, military leaders have begun laying a foundation on which to build a future fighting force that’s as ready to engage in urban and underground environments as it is in conventional domains. There’s just one thing they need to complete their mission: more and better geospatial intelligence (GEOINT), which is being developed thanks to forward-looking programs such as the Army’s One World Terrain (OWT) and the Defense Advanced Research Project Agency’s (DARPA) Subterranean (SubT) Challenge.
“The military has recognized that no domain is an unreasonable place to assume we might have to engage,” explained Barry Tilton, chief technical officer for U.S. federal operations and vice president of engineering at 3D GEOINT provider Vricon. “Because the world is nothing but a series of constantly changing political and environmental circumstances, they’re saying, ‘Let’s prepare to engage anywhere engagement might happen.’ … Whether that’s a cityscape or a tunnel: The more knowledge you have about the landscape and where you are in it, the more effective you’re going to be.”
Armies have typically waged land wars in remote or rural areas, away from major population centers. During the Civil War, for example, the Battle of Gettysburg raged mostly in the open fields surrounding the Pennsylvania town. World War II’s Battle of the Bulge unfolded in the dense forests of Belgium, and the Vietnam War’s Battle of Hamburger Hill in the jungle-cloaked mountains. Even Operation Enduring Freedom took place predominantly in the rugged mountains of eastern Afghanistan.
There have been urban battles, too—Aachen, Germany, in 1944; Hue, Vietnam, in 1968; Mogadishu, Somalia, in 1993—but cities are generally a theater of last resort.
There are good reasons why, according to Patrick Cozzi, CEO of Cesium, a geospatial software company whose platform enables 3D applications based on real-world geospatial data.
“Urban environments are infinitely more complex than open land where you can see everything,” explained Cozzi. “Ingress and egress—getting in and out of multi-story buildings—and being able to understand the movement of people to precisely target bad actors and avoid civilians ahead of time and in real time are key challenges.”
Both the challenges and the potential fallout—including widespread destruction and high numbers of casualties—can be even greater in megacities, of which there will be 43 by 2030, according to the United Nations. Most of these cities will be in developing regions, and 37 of them will be 200 to 400% larger than Baghdad, where U.S. forces spent nearly a decade engaged in urban conflict during the Iraq War. While some 6.5 million people live in Baghdad, New Delhi has a population of 29 million and Shanghai a population of 26 million. Mexico City and São Paulo, meanwhile, each have approximately 22 million inhabitants, while Cairo, Mumbai, Beijing, and Dhaka each have nearly 20 million.
According to AFSG, current U.S. Army doctrine calls on troops to “isolate and shape the urban environment and to utilize ground approaches from the periphery into the city.” But in cities with so many people, that doesn’t work.
“Physically controlling an urban population consisting of tens of millions of people spread over hundreds of square miles with military forces numbering in the tens of thousands not only ignores the force ratios recommended in doctrine but actually inverts them,” AFSG explains in its “Megacities and the United States Army” white paper. “Virtual isolation is even more improbable given cellphone saturation in urban environments worldwide and global interconnectedness through the World Wide Web. Ground maneuver from the periphery is also unrealistic. The congestion of ground avenues of approach, combined with the massive size of the megacity environments, makes even getting to an objective from the periphery questionable, let alone achieving an operational effect.”
But it’s not just doctrine that falls short in megacities, it’s also maps.
“If a company commander or squad leader wants to see what’s up over the hill for route planning or threat analysis, they need a level of resolution, fidelity, and currency that you can’t get currently,” said Ryan McAlinden, director of modeling, simulation, and training at the University of Southern California’s Institute for Creative Technologies (ICT), a DoD-sponsored University Affiliated Research Center that works collaboratively with the U.S. Army Research Laboratory.
The deficit is especially apparent in urban environments, which necessitate a three- instead of a two-dimensional vantage point.
“Our world is inherently 3D,” Cozzi said. “In an urban environment where there are buildings that can have overhangs and multiple floors, the third dimension is key to having situational awareness—where I am, what I can see, if there are other actors, where they are, what they can see, and what the potential is for mobility across all three dimensions: X, Y, and Z.”
Unfortunately, 3D data so far have been limited to crude sources like NASA’s Shuttle Radar Topography Mission, which provides global elevation data at 30-meter resolution.
“That’s pretty much useless. You can’t do anything in an urban area with 30-meter data, except perhaps some basic flight planning to make sure you don’t hit buildings,” McAlinden said. “Even 1-meter or sub-meter data is of marginal use in urban areas. Because of the density of structures and the slope contrast—you have very sharp edges that because they’re manmade are less organic and very perpendicular to the ground—you really need high-resolution content.”
TRAINING FOR TOMORROW
Generating, federating, and utilizing high-resolution 3D content is the focus of the Army’s OWT program, the outcome of which will eventually be a realistic and accurate geo-specific 3D map of the world that the Army can use to train warfighters for future conflicts—including those in megacities.
OWT—the 3D geospatial database that will underpin a larger virtual training platform known as the Synthetic Training Environment (STE)—is in some respects a direct response to AFSG, which stated plainly in Megacities and the United States Army that “the Army is not prepared for operations” in megacities, and that “entirely new concepts are needed” to bridge the gap.
OWT is, indeed, an entirely new concept. Led by Maj. Gen. Maria Gervais, director of the Army’s STE Cross Functional Team, the OWT effort will pursue American strategic advantage by addressing a geospatial gap in the Army’s current training regime.
“We have to be able to replicate in a training environment whatever operational environment a soldier or commander might face in the future,” Gervais said. “Our current training environment does not allow us to do that.”
The Army’s current training platform, the Integrated Training Environment, was built using gaming technology from the 1980s and ’90s. Replacing it with modern technology—virtual simulations with high-resolution 3D imagery—will redefine U.S. military readiness.
“Although they’ve been really good for a number of years, game simulations are not designed to support real data within their systems. In order to minimize file sizes, they tend to fill in textures of buildings and create generic structures,” said Tilton of Vricon, which is building OWT for the Army using its large-scale 3D mapping solution. “One World Terrain is about moving from that environment to one where if you’re simulating Paris, not only will the Eiffel Tower be in the right place—which happens in simulations now—but so will the buildings around it, the walkways going up to those buildings, and the restaurant you remember eating at down the street.”
Wherever they’re going to deploy, soldiers can rehearse tactical warfare as if they were there already. That’s especially valuable in the context of densely populated urban environments where it’s not feasible to conduct realistic live training.
“The ability to virtualize an environment—to see it on the ground as it is ahead of time—is so powerful,” explained Gervais, adding that hyper-realistic virtual training environments help soldiers strengthen their decision-making, develop cognitive skills, enhance teamwork, reinforce self-confidence, and build muscle memory.
To realize its OWT vision, the Army must solve myriad challenges that are familiar to the GEOINT Community. One is spatial and temporal resolution. To get the best and most current view possible, Vricon is building a foundation layer of 3D GEOINT using half-meter commercial satellite imagery from Maxar, which co-founded Vricon with Saab in 2015. On top of that, the Army can then add a high-resolution tactical overlay layer comprising 2- to 15-centimeter data collected by unmanned aerial systems (UAS), manned aircraft, ground vehicles, etc.
“OWT will have a … foundation built from satellite imagery to enable consistent and accurate registration of datasets acquired by other sources,” said Cozzi of Cesium, which is partnering with Vricon to execute OWT’s 3D requirements. “These datasets may be more recent or higher resolution or both, depending on the source. The system must be flexible enough to accurately make use of the best available source data—wherever it might come from—to produce the most detailed result possible.”
In that way, soldiers themselves become sensors.
“We will take source data from national technical means and commercial satellites, but we also will produce our own source data,” said McAlinden of ICT, which is conducting research and development in support of OWT and the STE. “Units themselves are going out and collecting imagery and then feeding that imagery into the One World Terrain pipeline to produce highly resolved, highly accurate geo-specific content.”
Several hundred of these UAS 3D mapping kits have been deployed to Army, Marine Corps, and Special Operations units, members of which can use mobile devices to define areas they wish to simulate for training purposes. UAS then automatically survey the defined areas to generate terrain models from which individual features (e.g., roads, vegetation, buildings, doors, windows) can be extracted and classified. GEOINT is subsequently processed, stored, and distributed to soldiers in the form of simulations they can access anywhere and on any device, including virtual reality goggles that allow them to interact with training environments as if they were actually in them.
“We want to be able to take all the terrain data that’s out there … and make it simulator- or game engine-agnostic so we can quickly bring it into applications for soldiers to train on,” explained Gervais, who said depicting roads, structures, and trees in the right location is just the first step in providing accurate terrain representations. The 3D model also requires terrain features with appropriate attributes to support realistic training. For example, a muddy street or dirt trail must constrain vehicles more than an asphalt highway, and concrete walls must react differently to munitions than wooden ones. Machine learning and artificial intelligence are accelerating the assignment of appropriate attributes during the collection and processing phases with the help of cloud computing.
“It’s not just about realism in the pixels; it’s about having semantic data,” Cozzi said. “The fusion of highly accurate 3D geospatial data with the semantics that describe that data … opens up a whole new level of uses cases beyond visualization.”
The key to the entire workflow, according to Gervais, is open architecture that allows terrain data to flow freely among media.
“Different training systems traditionally have required different data formats at different levels of resolution,” said McAlinden, adding that terrain data currently exists in 57 different formats suited to disparate Army training systems. “If you were doing a tank simulator, an aviation simulator, and a ground simulator, you had to build terrain for every single one of them because there wasn’t a mechanism to move them in a concise, coordinated, accredited way. … The STE will consolidate all these different training systems into one, with one foundational terrain dataset.”
With one system instead of 57, training programs will advance from conception to implementation more quickly, more affordably, and with fewer stovepipes.
“The vision obviously is quite grand,” said McAlinden, acknowledging that grand visions face grand challenges. Because of the type and size of data involved, those challenges include data storage, distribution, conflation, and accreditation.
Solutions are underway, according to Gervais, who said OWT is on track to deliver its initial operational capability at the end of 2021 and its full operational capability in 2023. Based on early feedback from soldiers—who might also be able to use OWT data beyond training, to include mission planning, mission rehearsal, and even operations—success is imminent.
“We still have a lot of things to work out,” Gervais said, “but it’s already proven its value.”
While it’s designed to simulate virtually any environment on Earth, OWT will be especially useful in helping warfighters prepare for urban combat.
“To practice warfare, usually you have to break things,” Tilton said. “Obviously, you don’t want to break real cities, and building a pretend city is not easy.”
Virtual cities are, therefore, an ideal solution.
The same logic applies to another, aforementioned Achilles heel in U.S. military readiness: underground environments. Though the Army promises underground terrain will figure just as prominently in OWT as aboveground terrain, it demands a different approach.
“Cities are relatively easy to model because we can take enough pictures to build maps, and because there’s a lot of supporting data from everyone on the planet taking selfies with their cellphones. But because the whole point of underground facilities is to protect certain resources and assets, there are very few models available of what they actually look like,” continued Tilton, who said the Army is hard at work on solutions that will give it the access and technical means it requires to capture geo-specific subterranean data.
The Intelligence Community recognized the paucity in subterranean GEOINT more than 20 years ago. In 1997, the Defense Intelligence Agency (DIA) stood up the Underground Facility Analysis Center (UFAC). A DIA-managed consortium of co-located intelligence professionals from DIA, the National Geospatial-Intelligence Agency, the Defense Threat Reduction Agency, and others, UFAC’s mission is to unearth—literally—information about adversarial underground facilities.
“The underground ‘problem’ has been growing for many years now,” said UFAC Senior Engineer and Analyst Gunnar Radel. “It’s growing in the quantity [of underground facilities] that our adversaries are building and in the number that they have, the quality of the sites that they’re using, and the scope of application.”
Adversaries increasingly are burying not only top-secret facilities, but also weapons and other sensitive assets. Simultaneously, there is an increase in underground civilian infrastructure and in underground activity by non-state actors.
“Our adversaries feel threatened … so they’re protecting themselves from observation and attack by putting distance and rock between us and by keeping their precious assets concealed,” continued Radel, who said subterranean construction simultaneously has become more affordable and feasible for both military and civilian developers. “The technology has improved to the point where there’s not too many places anymore that you can’t build underground. At the same time, we’re running out of surface space, so countries are looking to build vertically down in major cities instead of vertically up like they have in the past.”
Because underground environments are hidden from surveillance, all of this adds up to a significant GEOINT opportunity. “UFAC has an incredible, insatiable appetite for GEOINT,” said Radel, whose team is trained to detect and characterize underground facilities by analyzing above-ground terrain. As the subterranean domain grows in size and strategic importance, however, more advanced tools and techniques are needed.
Radel is bullish about machine learning.
“Our priority is to optimize our most precious analytic resource, which is the analyst’s time,” he said. “We’re looking at how much can be automated upstream before our analysts even have to engage … [which will help them] mine the gems and nuggets out of the mass quantities of data coming in.”
THE SUBTERRANEAN CHALLENGE
Without GPS, underground mapping requires automation, perception, mobility, and networking
While UFAC is investing in better analysis of subterranean GEOINT, its partners in the research community are developing means with which to better collect it. Namely, DARPA, which in September 2018 launched its SubT Challenge, an underground robotics competition with the objective to incubate new technologies to assist with navigating, mapping, and otherwise surveying complex underground environments.
“In places where we don’t have access to GPS and other ways to geo-rectify the data that we have, it’s a severe limiter. One of those places is the underground,” said SubT Program Manager Timothy Chung. “The SubT Challenge allows for a very wide aperture of approaches to address the problem at hand.”
DARPA has a history of using competition to seed innovation. In 2004 and 2005, for instance, it hosted its Grand Challenge to accelerate the development of autonomous vehicles. Thanks in large part to that competition, automakers are preparing to launch their first driverless cars as early as next year. DARPA hopes the SubT Challenge will be a similar catalyst.
“A challenge is a really great way to pose an audacious problem and offer it up to the world to bring together a diverse pool of innovators and see what they bring to the table,” continued Chung, who said SubT technology could one day assist not only soldiers in combat, but also civilian first responders and perhaps even commercial entities in industries such as mining.
The SubT Challenge comprises three “circuits” during which competitors will test and demonstrate novel robotic solutions. The first, the Tunnel Circuit, took place in August at a mine outside of Pittsburgh, Pa. The second, the Urban Circuit, will take place in an urban underground environment—a subway, perhaps, or a sewer—in February. The third, the Cave Circuit, will take place in a natural cave environment in August 2020. A fourth and final event will integrate all three subdomains in August 2021.
To achieve the competition’s GEOINT objectives and win up to $2 million, competing teams must successfully execute on four pieces of the subterranean puzzle: autonomy, perception, mobility, and networking.
Because underground environments can be dangerous for humans, autonomy is especially important, according to Kostas Alexis, director of the Autonomous Robots Lab at the University of Nevada, Reno (UNR).
“The idea is that human supervisors will be aboveground somewhere, away from the underground mission. So, we must have absolute autonomy,” said Alexis, whose team—Team CERBERUS, comprising UNR; Swiss university ETH Zurich; the University of California, Berkeley; Sierra Nevada Corporation; and Swiss company Flyability—was one of 11 that competed in SubT’s Tunnel Circuit. “Because an underground network of tunnels can be kilometers long, the systems must be able to integrate and operate on their own in very large-scale environments.”
Team CERBERUS, which placed sixth in the Tunnel Circuit, has designed a quadruped robot about the size of a dog. When it enters an underground environment, the robot will carry on its back a drone it can deploy if it encounters unnavigable vertical or multi-level environments. Thanks to their autonomy, the robots will be able to determine the appropriate time to deploy their drone companions. Eventually, the robots will be able to exchange and achieve what’s known as collaborative autonomy.
“Robot A needs to be able to share maps with Robot B so as to be able to make intelligent decisions as a team,” Alexis explained.
A prerequisite for autonomy, of course, is perception.
“[Present-day autonomous systems] operate well in environments that are well-lit, well-structured, and with good features. That’s not what an underground environment is,” Alexis continued. “An underground environment can be simultaneously texture-less and dark. And if there’s a potential threat, it might even be full of smoke. That challenges the ability for the robot to know where it is and to map the environment accurately, which in turn challenges its ability to operate autonomously.”
To solve perception challenges, Team CERBERUS uses a multi-modal sensor fusion package that includes a camera, LiDAR, thermal vision, and a gyroscope for inertial navigation. Later, it also plans to add radar.
The area in which Team CERBERUS has swung for the fence, however, is mobility.
“The question is: How can you make a unified robotic solution that can be deployed in a variety of situations?” asked Alexis, whose robot features jointed legs that are designed to function in a variety of underground environments, from mines littered with rocks to subway stations full of stairs. “With legged systems, you have dexterity over different types of terrain.”
Another team, Team Explorer, took a similar yet different approach. Comprised of roboticists from Carnegie Mellon University and Oregon State University, it built two autonomous ground vehicles designed to work in tandem with two drones. Instead of legs, however, its modular vehicles feature large, all-terrain wheels that roll instead of step over obstacles.
“We used big, knobby tires that are actually motocross racing tires for dirt bikes, and that gave us the traction and mobility we needed,” said Steven Willits, lead test engineer for Team Explorer, which placed first in the Tunnel Circuit thanks not only to its superior mobility, but also its approach to the challenge’s final requirement: networking.
“Each of our ground robots is a communication node in and of itself, but then we also have them drop Wi-Fi nodes at strategic locations,” continued Willits, who said one ground robot can be sent deep into a mine to map it autonomously while another robot follows behind to build a local communication network. “That way, when the first robot comes back into range, we’re able to receive all the information it stored while it was roaming around.”
Whether you’re talking about automation, perception, mobility, or networking, the goal is to facilitate mapping in the absence of GPS. For DARPA, however, the underground is as much about relative as absolute location.
“The SubT Challenge focuses on what we like to call actionable situational awareness,” Chung said. “What that means is: We want to move beyond just having knowledge of the layout or the environment by itself and toward understanding how it relates to the mission at hand. In the context of the SubT Challenge, it’s about not only having a map—which in and of itself is hard to obtain—but also being able to use that map to provide insight into, for example, where artifacts are located so that emergency responders or incident commanders can make decisions based on the location of survivors, hazardous areas, and things of that nature.”
Despite their myriad differences, urban and underground environments in that respect are exactly the
same: Whether the Army of the future faces adversaries in the concrete crevices of Dhaka, the cosmopolitan streets of São Paulo, the metros beneath Europe, or the mountainous caves that lurk below Afghanistan, GEOINT is about more than coordinates. It’s also about context. And thanks to efforts like OWT and the SubT Challenge, context will soon be possible in even the most complex places.
Concluded Tilton, “Having knowledge about what the world actually looks like makes our warfighters more effective wherever they engage.”