Activity-Based Intelligence in Mixed Reality

By Robie Mitchell; Patrick Kenney, Whitespace Solutions; Jacqueline Barbieri, Whitespace Solutions; John Bridgwood, Vricon; and Stephen Hodgson, Valorem

Hidden among more than 82 billion data points gathered in the past five years—the last of which were ingested seconds ago at 6:32:12 GMT on May 3, 2023—a Rohingya fishing village in Myanmar is likely in the crosshairs of Buddhist paramilitaries, intent on setting it alight. Supported by the nationalist government, they operate with impunity.

8,260 miles away in a nondescript office a few metro stops from the Pentagon, the symbol of the U.S. national security community, an analyst is working for a non-governmental organization (NGO) that aims not just to record war crimes of the past, but to warn of war crimes in the future. She wears a sleek visor and fingertip sensors that let her see and manipulate a broad, virtual landscape of three-dimensional imagery, icons, relationships, and timescales. Her simple cubicle would blend into any office, but the “view” in her cubicle is out of this world. To the uninitiated, her virtual view is as colorful and overwhelming as the Las Vegas Strip. However, for this analyst, properly trained in the right mind-set and methods, this 3D analytic environment is a wealth of knowledge—a book waiting to be read. She focuses her eyes on a particular bit of data and blinks twice, summoning relevant details and attributes into view. The analyst has been tracking a paramilitary leader via four proxies across 19 datasets during the past 13 months, and she has identified recent aberrations in his “normal” pattern-of-life, indicating his intent to attack the Rohingya fishing village within days.

With her sensor-tipped fingers, she seamlessly integrates additional contextual data and generates a 3D augmented reality experience to convey her facts and findings. This presentation will be dispatched to contacts at the U.S. Department of Defense, the United Nations, and partnered NGOs in Southeast Asia. Just as the adoption of video and digital slideshows surpassed static photos and flip charts as the means to convey the outcome of complex intelligence and research work to great effect, so will this 3D immersive tool allow analysts to better make their case and explain their conclusions.

• This article is part of USGIF’s 2019 State & Future of GEOINT Report. Download the PDF to view the report in its entirety. 

VR and ABI: A Path to the Future

Virtual reality (VR) and activity-based intelligence (ABI) are both buzz terms used imprecisely and hyperbolically. But those familiar with the disparate technologies, methodologies, and mind-sets that underwrite their contributions to traditional analysis have experienced the substance behind the hype. Together, they are particularly powerful and make the aforementioned hypothetical scenario feel tantalizingly real. Synthesizing the described technological and methodological innovations is not a simple task, but it is happening successfully in industry and government alike and will be a foundational element of analyst workflows in the not-too-distant future.

Given the current pace of change in technology, the synthesis is only a matter of time, but to what ends is it taking place? Can the resultant analytic efficiencies and exponentially increased information advantage be brought to bear outside of traditional intelligence problems, and also applied to problems NGOs are tackling? If so, what does this mean for their outcomes? Before diving into these important questions, we will first establish a brief foundation in virtual, mixed, and augmented reality (AR) as well as ABI.

Virtual, Mixed, and Augmented Realities

Paul Milgram and Fumio Kishino introduced the concept of a “virtual reality continuum” in the 1990s, and the idea has held sway since.^[1] The continuum refers to the relative mix of object types—real and artificial—visible in any display environment. At one extreme, everything visible is artificial (i.e., virtual reality). And at the other, only few virtual objects are laid over the real world (i.e., augmented reality). Whereas Milgram and Kishino used “mixed reality” to refer to the entirety of the continuum, nowadays, it often refers to the spectrum’s middle ground.^[2] Today, other than for gaming and recreation, mixed reality technology has been used mainly as a presentation tool. But such capability can do much more than enable high-tech conference calls and heightened consumer engagement.^[3] Properly leveraged, mixed reality experiences afford users a level of engagement and understanding that is impossible with traditional displays.^[4]

For analytic purposes, mixed reality experiences offer enhancements that consider user experience, comfort, and ease without sacrificing cutting-edge visualization. Moreover, the ability to toggle between an immersive VR experience enabled by a set of lightweight glasses and an augmented one projected via a tablet potentially drives more efficient, accurate, and timely analysis more so than a set of capabilities predicated on a particular device. Mixed reality solutions must be hardware agnostic to avoid challenges related to hardware upkeep, lock-in, and interoperability, which can cause as many headaches as their capabilities will be worth. Imagine being able to visualize 3D problems in 3D space—seeing chemical plumes and wildfires spread in real- or near real-time, tracking the spread of cyber threats through complex data infrastructure schematics, or training to breach a room for law enforcement, counterterrorism, or counter-narcotics purposes. The analytic dividends gained by going from 2D geographic information system (GIS) tools to 3D mixed reality capabilities will be like the leap to 2D GIS from static maps and charts.

Activity-Based Intelligence

ABI emerged from the counterterrorism fight in Afghanistan and Iraq and, as such, was organically defined and refined by operator-analyst teams in the field. Its bottom-up origins have led to contentious debates over ABI’s definition. It is important to understand that at its core, ABI is not just a methodology, but a mind-set. While the proper mix of method to mind-set required to maintain rigor varies across the three ABI sub-disciplines, its four constituent pillars—geo-reference to discover, integrate before exploitation, data neutrality, and temporal neutrality—are foundational to all variants of the ABI approach.^[5]

First, all data, to the extent possible, should be geo-tagged so it can be viewed spatially and temporally. Second, all data should be folded together and only exploited after it has been integrated. This means not assessing the various intelligence disciplines, or INTs, in silos. Third, all data can yield significant information no matter how old or new it may be (i.e., one must resist the urge to privilege that which is newly-collected or obtained over that which resides in older stores). Fourth, and relatedly, data from flashier “exquisite” sources should not be privileged over data with humbler origins.^[6] A tweet or a piece of battlefield pocket litter can yield an answer as readily as satellite imagery with creative analysis.

These foundational pillars not only provide answers for pre-existing questions, but also enable the discovery of entirely new questions based on the interrelations deeply embedded within data. This includes seemingly irrelevant patterns of life and anomalies contained therein that are invisible without ABI’s unique temporal, spatial, and network analysis. This significant analytic innovation hinged on the adoption of 2D GIS environments in which all your data can be brought together. Transitioning analytic work into 3D environments will bring the next wave of analytic innovation, unlocking a richer capability to model and understand complex issues.

Geospatial Analysis in Humanitarian Relief

The literature on applying mixed reality technologies to humanitarian workflows is neither practical nor up-to-date. This is understandable because much of the innovation that has increased analytic rigor and allowed for creative problem-solving has come from within government, and much of the technological progress in mixed reality capabilities has come from within the entertainment industry. As a result, academics lack the insider perspective needed to accurately assess the impact of combining the two phenomena. We compare the ability of NGOs to use geospatial data to the Intelligence Community’s (IC) still-dominant tasking, collection, processing, exploitation, and dissemination (TCPED) paradigm.^[7] There are widely acknowledged issues with this process, including a lag in the delivery of final analytic products and a lack of integration among human, geospatial, signals, and open-source intelligence.^[8] Despite TCPED’s shortcomings, the framework provides a useful metric for evaluating NGO performance and mapping a path forward.

NGO responses to natural disasters, along with academic after-action analyses, demonstrate the level at which geospatial understanding and technologies are currently leveraged. These responses provided a baseline with which to evaluate the NGO community’s views and uses of geospatial analysis.^[9] During an exceptionally severe drought in the Iberian Peninsula, wildfires broke out in 2005 near Coimbra, Portugal’s third-largest city. Not until the activation of the “International Charter”—an informal cooperative agreement among the world’s premier governmental space agencies—did Portugal request satellite imagery to aid in the disaster response.^[10] This delay, coupled with limited data integration, showcased shortcomings in NGO tasking and exploitation processes.^[11]

In another example, a rain-induced landslide on the Southern Leyte Island of the Philippines on February 17, 2006, killed 1,126 people and displaced approximately 19,000.^[12] Continual rainfall impeded rescue operations, and Rosette Lerias, the governor of Southern Leyte, ended them on February 24.^[13] A widely cited assessment by Voigt et al. confirms the disaster response was inadequate but frames the problem as a lack of information sharing with government entities that have access to exquisite data sources and ineffective information fusion among those entities.^[14] The assessment states that greater awareness of the landslide extent—derived from timely production and dissemination of analysis—could have informed evacuation routes and emergency response and notes similarly deficient responses to earthquakes in Pakistan and a tsunami in the Indian Ocean region. Yet, this understanding of how geospatial information could inform action against humanity’s most daunting challenges lacks imagination—so much more can be done to improve outcomes than faster analysis and better information sharing.

What Can Be Done?

The capacity of NGOs to leverage geospatial information represents fertile ground for organizational leapfrogging—with proper adoption of ABI-infused VR, these sometimes nimbler and less bureaucratically constrained entities can surpass what the government has accomplished with TCPED.^[15] The intersection of mixed reality technologies, analytic innovation, and humanitarian outcomes remains largely speculative and theoretical. Despite the nature of our research question, we found it imperative to derive experimental data using mixed reality capabilities and industry expertise to better assess the utility of cutting-edge tradecraft and tools within NGOs. Partnering with Amnesty International, our research team selected relevant crisis-affected geographies. Demonstrating the potential of ABI and other elements of tradecraft, we collected a wide range of open-source data that provided insights into political violence, social media, infrastructure, natural resources, population, geographic features, and epidemiology. Layering and geo-referencing this data allowed us to demonstrate how analytic innovations could enhance humanitarian outcomes.

Using ABI, we discovered links among the data that allowed us to infer culpability in Syrian violence, track aid worker movements in Myanmar, and map the spread of election-related civil unrest in Zimbabwe. Incidentally, while we were collecting our live data, the database we used to track politically motivated violence, the Armed Conflict Location & Event Data Project (ACLED), stopped refreshing data for August. So, some of the events that were referenced via Tweets and news articles were not represented in arguably the best open-source database on conflict. Nonetheless, we were able to determine that a neighborhood in Aleppo was currently controlled by rebel forces, meaning that recent violence was more likely perpetrated by non-government actors. We were able to observe differences in the relationship between natural resources and violence in Syria and Myanmar. Namely, that oil fuels violence in Syria but has little effect in Myanmar. Finally, we discovered that unrest from Zimbabwe’s disputed election on July 30 was not centered in the capital of Harare as many open-source datasets suggested, but rather was steadily spreading southward.

These findings are speculative to some extent because of the limitations of open-source data collection, but with enough data points they can have real-world implications and in VR they could be leveraged to lifesaving effect. Although we investigated geospatial capabilities primarily at a specific NGO—Amnesty International—and showcased how tradecraft can improve situational awareness and humanitarian outcomes, our findings are applicable to use cases beyond refugee flows in Myanmar, war crimes in Syria, or election monitoring in Zimbabwe. Given the severity of the events NGOs often respond to, there is a good chance that no matter the use case there will be a wide variety of means across available sources to collect data, track bad actors, and act upon information in new and meaningful ways.

Ethics

A growing body of evidence suggests that people’s exposure to technology is positively correlated with overconfidence in their analysis.^[16] When the capabilities we investigated make their way to routine use by governmental and non-governmental actors, it will be important to seriously consider the role of observers in the events they are omnisciently “seeing.” Whether they are making decisions that affect resource allocation or post-conflict justice, NGOs will have to reconcile with issues the IC has long contended with. The life-and-death nature of the issues at hand leave them inextricably linked to questions of ethics, morality, and values. In addition, performing ABI requires collecting and integrating massive volumes of data, which raises critical privacy questions when done outside of the purview of the strict laws that govern U.S. government activities. Integrating visualization capabilities such as VR into routine analytic workflows should be coupled with processes that compensate for these issues.

Conclusion

There are two interrelated takeaways from our research and discussions. First, geospatial data and advanced analytic approaches are underused among NGOs—which is unsurprising given that their adoption within government is still developing. Institutional inertia, tight budgets, and insufficient technical expertise explain why this gap exists and persists. Second, mixed reality capabilities could allow NGOs to leverage their geospatial data in new ways as disparate as fundraising (by showing potential donors scenes from the ground) and imagery exploitation (through faster and more efficient analysis).

At Amnesty International, for example, volunteers have sorted through imagery and marshalled evidence showing where and when war crimes were committed. In 2016, Amnesty used this model to credibly assert that the Sudanese government had targeted its own citizens in Darfur.^[17] Professional and volunteer analysts can be trained in ABI analytic techniques even at an introductory level quickly and economically to get to these meaningful conclusions and outcomes faster and more efficiently. Amnesty and other NGOs can leapfrog their analytic skills and technology through dialogue with government actors who have suffered the long, expensive trial-and-error phase of moving analysis into the 21^st century. Likewise, NGOs will be able to stand on the global security community’s shoulders as VR technologies advance, are tested, and fall in price through government initiatives.

By sharing their respective problems, potential solutions, and methodological and technical innovations around ABI and VR technologies, these groups could create a collaborative nexus that lowers the cost of doing good for both NGOs and the national security community. This new equation would have tremendous impact for both humanitarian and security outcomes.

1. Paul Milgram and Fumio Kishino. “A Taxonomy of Mixed Reality Visual Displays.” The Institute of Electronic, Information, and Communication Engineers (IEICE) Transactions on Information Systems. 1994:E77-D(12):3.
2. Lucas Matney. “Magic Leap Details What Its Mixed Reality OS Will Look Like.” Tech Crunch. July 27, 2018. https://techcrunch.com/2018/07/27/magic-leap-unveils-what-its-mixed-reality-operating-system-will-look-like/, par. 7.
3. Jeff Miller, Christian Dieckmann, Dario Raciti, Shauna Heller, and Craig Dalton. “Building Virtual Reality Experiences to Maximize Brand Awareness and Prestige” (presentation). VRX Immersive Enterprise Conference & Expo. San Francisco, CA. November 2015.
4. Alejandro G. Iñárritu. “Carne y Arena: Physically Present, Virtually Invisible.” https://carneyarenadc.com.
5. “NGA GEOINT CONOPS 2022.” National Geospatial-Intelligence Agency. p 7.
6. Patrick Biltgen and Stephen Ryan. Activity-Based Intelligence: Principles and Applications. Norwood, MA: Artech House; 2016. p 16.
7. The authors acknowledge that several IC agencies have made strides toward replacing TCPED with alternatives, such as object-based production (OBP). However, given that TCPED is still the doctrinal paradigm, we elected to use it as our comparative lens for this paper.
8. NGA GEOINT CONOPS 2022. National Geospatial-Intelligence Agency. p 3.
9. Stefan Voigt, Thomas Kemper, Ralph Kiefl, Klass Scholte, and Harald Mehl. “Satellite Image Analysis for Disaster and Crisis-Management Support.” Transactions on Geoscience and Remote Sensing. 2007:45(6):1525.
10. The International Charter Space and Major Disasters. The International Charter. https://disasterscharter.org/web/guest/home;jsessionid=DF479CC8AB623E78C289173FA37A1B95.jvm1.
11. Stefan Voigt, Thomas Kemper, Ralph Kiefl, Klass Scholte, and Harald Mehl. “Satellite Image Analysis for Disaster and Crisis-Management Support.” Transactions on Geoscience and Remote Sensing. 2007:45(6):1524.2.
12. Gisela D.A. Luna, Jesusa Grace J. Molina, and Fatima Gay J. Molina. “The Southern Leyte Landslide 2006: Recovery Status Report.” International Recovery Platform. December 2011. p 1. https://www.preventionweb.net/files/26098_26098recoverystatusreportleytemarch.pdf.
13. ibid. p 4.
14. Stefan Voigt, Thomas Kemper, Ralph Kiefl, Klass Scholte, and Harald Mehl. “Satellite Image Analysis for Disaster and Crisis-Management Support.” Transactions on Geoscience and Remote Sensing. 2007:45(6):1520.
15. José Goldemberg. “Technological Leapfrogging in the Developing World.” Georgetown Journal of International Affairs. 2011:12(1):135. https://www-jstor-org/stable/43133873.
16. Brent B. Clark, Christopher Robert, and Stephen A. Hampton. “The Technology Effect: How Perceptions of Technology Drive Excessive Optimism.” Journal of Business and Psychology. 2016:31(1):87. https://doi.org/10.1007/s10869-015-9399-4.
17. “Decode Darfur.” Amnesty International. https://decoders.amnesty.org/projects/decode-darfur.