The history and future of the software that made GEOINT mainstream and changed the way we view the world
In August 2005, Hurricane Katrina ravaged the Gulf Coast of the United States, bursting levees throughout Louisiana and Mississippi and submerging the streets of south Florida. According to the National Hurricane Center, it was the deadliest hurricane since 1928, claiming at least 1,800 lives and causing more than $108 billion in damages.
The U.S. Navy, Coast Guard, and other federal relief groups deployed helicopter teams to rescue people stranded in New Orleans without the resources to escape or survive in their homes. Hurricane victims dialed 911 for urgent help at specific street addresses, but it was impossible for first responders to find them without precise GPS coordinates—street signs and house numbers were invisible beneath the deluge. In the absence of traditional situational awareness, responders were operating blind.
In California, a team from the recently minted Google Earth program launched into action, creating real-time imagery overlays of heavily affected areas on top of its existing 3D globe platform. Fly-by aerial photos from the National Oceanic and Atmospheric Administration (NOAA) and satellite imagery from DigitalGlobe—one of Google Earth’s primary providers—revealed the scope of the hurricane’s destruction. Google Earth made this data publicly available and responders had eyes again.
Now, they could input a caller’s location into Google Earth paired with case-specific details—for example, a target trapped in a two-story house with a clay roof next to an oak tree. Equipped with up-to-date imagery from Google Earth, relief teams saved thousands of people from Katrina’s aftermath.
Years later, the Louisiana Governor’s Office of Homeland Security and Emergency Preparedness would pair internal data with Google Earth Enterprise (GEE)—the desktop software suite for private or offline use of Google Earth—to create 3D globes for emergency response and infrastructural planning.
Today, Google Earth is among the most popular geospatial software in the world, boasting upward of one billion downloads. With it, students take virtual tours of the world’s wonders from their classrooms, house hunters evaluate prospective properties without leaving home, and much more. The U.S. military employs GEE for secure mission planning and intelligence professionals use it to visualize points of interest and detect change. Google’s spinning globe truly represents the democratization of geospatial intelligence.
In the case of GEE, government and military organizations became so dependent on the software’s private storage and visualization capabilities that not even a depreciation announcement from Google two years ago stopped them from using the platform.
As a result of the community’s reliance on GEE, earlier this year Google decided to make the software’s code open source and available for public download on GitHub.
With its future in the hands of its users, GEE is poised to remain at the center of mission planning and situational awareness efforts for the defense and intelligence communities—at least until a supported platform of equal utility arises.
A Giant’s Infancy
At the time Hurricane Katrina made landfall, Google Earth software had been available to the public for only three months. But the story of Google Earth began to take shape 10 years earlier at a computer hardware company called Silicon Graphics (SGI).
Michael T. Jones, then a member of SGI’s computer engineering team, had developed an invention that would revolutionize the firm’s 3D graphics offering, which at the time was used primarily for flight simulation.
“It was called clip mapping. That’s the fundamental hardware feature SGI had that let it do this amazing, smooth flight around the world,” said Jones, now a managing partner at Seraphim Capital.
Jones’ technique displayed a small region of graphics—the region under examination—in high resolution while the peripheral regions were displayed in low resolution. Jones, along with SGI engineers Chris Tanner, Chris Migdal, and James Foran, patented the method in 1998. Clip mapping required powerful supercomputers to run, but enabled a high-fidelity texture map that became the centerpiece of SGI’s final graphics system, Infinite Reality, which at the time boasted the fastest 3D graphics in the world.
Federal agencies such as the National Geospatial-Intelligence Agency (NGA) and the National Reconnaissance Office (NRO) would later follow suit, Jones said, using clip mapping to build data visualization platforms of their own.
To demonstrate the vastness of Infinite Reality’s capabilities, SGI created a demo called “Space to Your Face.” It began with a wide view of Earth from space, slowly zooming into Europe. When Lake Geneva became visible, the program would focus on the Matterhorn in the Swiss Alps. It would continue to zoom until reaching a 3D model of a Nintendo 64 console on the mountainside. Then it would zoom in even more, settling on the Nintendo’s MIPS r4000 graphics chip—a microprocessor created by SGI—before snapping smoothly back to space.
The demo was well received. Educators were excited to see an interactive, classroom-friendly global map tool, and video game developers had never seen such fluid graphics.
Seeking a new home for their brainchild, Jones, Tanner, and former SGI engineers Remi Arnaud and Brian McClendon founded a company of their own. Called Intrinsic Graphics, it focused on developing high-quality 3D graphics for personal computers and video games.
In October 1999, Tanner took the concept further when he designed a software version of the clip mapping feature that allowed a user to “fly” within a 3D visualization of Earth.
“People were blown away,” Jones said. “They were looking at Google Earth.”
Though the software platform wasn’t Intrinsic’s primary product—the graphics themselves were—Jones was intrigued and continued refining the spinning globe.
Yet running the software required expensive and highly specialized computing hardware not available to most of the private tech industry, let alone the commercial user.
“That machine cost $250,000. We wanted to be able to offer this without the specialized hardware,” said McClendon, now a research professor at the University of Kansas. “To be able to get that performance out of a PC meant we could share it with the world. The moment you realize you can transmit this data over the internet, you begin to realize the impact. A group of us at Intrinsic thought, ‘We need to build a company around this.’”
And before long, yet another company was founded. In 2000, Jones, McClendon, and a few others spun out the software from Intrinsic Graphics to launch Keyhole. In early 2001, Keyhole raised first round funding from NVIDIA and Sony Digital Media Ventures, making official its existence as a standalone company. Keyhole’s first product, EarthViewer 1.0, was the true precursor to Google Earth.
Using public data gathered from NASA’s Landsat constellation, IKONOS imagery, and aerial photos of major U.S. cities, Keyhole built a complete digital Earth. Though pixels were beginning to proliferate, high-resolution imagery was mostly limited to U.S. metropolitan areas.
Under the direction of newly appointed Keyhole CEO John Hanke, the company marketed EarthViewer to the commercial real estate and travel industries. Civil engineers also purchased it for the ability to sketch out location information when planning construction projects.
“Intelligence agencies wanted this capability as well, but they wanted to use their own data,” McClendon said.
The Intelligence Community (IC) was intrigued, but wanted to use classified geospatial data gathered through National Technical Means rather than the data on Google’s public server. To accommodate such buyers, Keyhole began offering an enterprise version of its software, allowing large-scale users to stand up private network servers and host their own data on a replica of EarthViewer’s 3D globe.
The National Imagery and Mapping Agency (NIMA) was the first agency to take note of this unprecedented capability. Under the leadership of then director James Clapper and deputy director Joanne Isham in 2001, NIMA launched a research and development directorate known as InnoVision. The new directorate sought to leverage state-of-the-art technologies from industry to help the IC adapt to the changing face of conflict in the aftermath of 9/11.
Isham, a former CIA employee, was well versed in In-Q-Tel, the CIA’s nonprofit venture capital initiative. She approached Robert Zitz, InnoVision’s first director, about collaborating with In-Q-Tel to find partner companies.
“We sat down together with In-Q-Tel and went over what our most urgent requirements were,” said Zitz, now senior vice president and chief strategy officer of SSL MDA Government Systems. “In-Q-Tel started trying to locate companies and [in 2002] discovered Keyhole.”
In-Q-Tel was impressed by the low barrier to entry and EarthViewer’s ease of use.
[Users] will create data files … rapidly and not to spec, put them in Google Earth, and they’ll run somehow. That’s really the reason why no other applications have been able to enter this space as dominantly as Google Earth.
— Air Force Lt. Col. Mike Russell, NGA
“With [EarthViewer], you just click on the icon and all of a sudden you’re flying around the globe,” said Chris Tucker, In-Q-Tel’s founding chief strategic officer and now the principal of Yale House Ventures. “There had been some way earlier-era, very expensive defense contract iterations [of a 3D digital Earth], but none at a consumer level that a regular analyst could make sense of without being a missile defense expert or some other technical user.”
In 2003, In-Q-Tel invested in Keyhole using NIMA funding. It was the first time an intelligence agency other than the CIA had employed In-Q-Tel. NIMA experienced an immediate return on its investment. Within two weeks, the U.S. military launched Operation Iraqi Freedom, which Keyhole supported in its first mission as a government contractor.
“We wanted a capability that would help military planners visualize and seamlessly move through datasets pertaining to particular target areas,” Zitz said. “We also wanted the ability to rapidly conduct battle damage assessments. NIMA was supporting joint staff in the Pentagon, and to sense how effective a strike was after-the-fact was very labor and imagery intensive. With Keyhole, we were able to streamline that process.”
EarthViewer quickly gained public exposure through TV news coverage using its battlefield imagery.
One of McClendon’s junior high school classmates, Gordon Castle, was CNN’s vice president of technologies. McClendon approached Castle with his EarthViewer demos. Castle was wowed, and CNN became one of Keyhole’s first media customers. The network routinely used EarthViewer to preview story locations during broadcasts. When the U.S. invaded Iraq, CNN used the software heavily—sometimes several times an hour—to show troop movement or combat locations.
The Big Break
Realizing its technology could improve people’s understanding of the planet, widespread commercialization became Keyhole’s mission. But Keyhole was a small company, and scaling up its computing infrastructure to handle more traffic was expensive. An annual EarthViewer Pro subscription still cost $599—a price justified by the company’s high operating costs. Keyhole’s bottom line stood in the way of its goal.
“[We wanted] everybody that opened the app to be able to find their house,” McClendon said. “It’s the first thing everybody searches for. If that experience isn’t good, the user thinks the product isn’t good.”
That first step required high-quality coverage of the entire land surface of Earth—a seemingly unattainable achievement for Keyhole’s 29 employees, even with In-Q-Tel backing. And the startup’s network bandwidth wasn’t strong enough to offer a high-resolution 3D globe to millions of consumers worldwide. McClendon recalled making regular trips to Fry’s electronics store to purchase hard drives, struggling to keep up with demand.
“To provide high-resolution data for the whole world was an epic undertaking … that would’ve taken us probably a decade to build up on our own,” he said.
For its vision to materialize, Keyhole needed more capital to scale up imagery procurement and to build powerful data infrastructure to store high volumes of imagery. In 2004, as if on cue, along came Google—one of the few companies powerful enough to manifest Keyhole’s mission. And they wanted to buy.
“It seemed like a tough road. Everybody was impressed with what we had done, but there was going to be competition and we needed to move quickly,” Jones said. “So we sold to Google because our dream would happen.”
As part of the acquisition, the Keyhole team maintained control of the program as it evolved. Most personnel, including McClendon and Jones (Tanner had since departed Keyhole), became executives at Google, developing their software unrestricted by the need to keep a startup afloat.
Once at Google, the program began to operate on an entirely different scale. Instead of acquiring licensing deals for small portions of a vendor’s imagery at a time, Google bought out all the imagery a vendor had available at once. Google also provided access to a rapidly growing user base already hooked on its web search platform.
Before debuting a Google-branded product, the former Keyhole team had to rewrite EarthViewer’s service code to run within Google’s infrastructure. Additionally, pre-release engineering refinements focused on adding data around the globe, making the program accessible to non-English speaking users, and simplifying features. Finally, Google Earth launched in June 2005.
The software exploded in the commercial marketplace. Where Keyhole’s consumer version of EarthViewer was too expensive for most casual civilian users, Google Earth was downloadable for free.
“We had millions of users in the first few days and tens of millions in the first year,” McClendon said.
Keyhole brought to Google a new form of interactive information that mimicked the real world and helped people understand their place in it. A GEOINT tool had finally made it to the mainstream.
Whether to disseminate that data after creating proprietary globes in GEE was, and still is, up to the user. This was the final evolution of the EarthViewer enterprise suite used by the Pentagon at the outset of the Iraq war.
GEE in Action
In the years following its launch, government agencies, businesses, and state municipalities began to deploy GEE at internal data centers to produce 3D globes using sensitive or classified data.
The city of Washington, D.C., for example, has used GEE to model and visualize public safety data including crime, vehicle and fire hydrant locations, and evacuation routes.
Arguably the largest user of GEE is the U.S. Department of Defense (DoD). When Google Earth was first released, military customers had an explicit need for this capability to function in a highly secure private network.
For example, the Army Test and Evaluation Command (ATEC) uses private data on enterprise servers such as Google’s to evaluate a wide range of weapon systems as well as ground and air operations.
At ATEC’s Yuma Proving Ground (YPG) in Arizona, proprietary terrain data, imagery, and operations maps are overlaid on Google Earth and used to plan and schedule launches.
“Knowing where everyone is and moving in a secure range and air space is important to our operations,” said Ruben Hernandez, an Army civilian in the YPG’s engineering support branch. “Much of this data is also armed for range awareness display.”
For example, prior to an indirect fire artillery test, personnel use YPG data within GEE to assess the safest positions on base to conduct the test—when to fire, where to fire from, and what to fire at. That information is disseminated throughout YPG for awareness.
“Many of these munitions have extensive footprints. We want to find out how much air and land space [the blast] is going to consume. Safety is a big component of how these overlays are planned,” Hernandez said.
NGA is another major GEE stakeholder. In 2008, the agency’s new GEOINT Visualization Services (GVS) program invested in the enterprise server. GVS has since produced a proprietary version of Google Earth for warfighters featuring classified NGA data.
According to GVS program manager Air Force Lt. Col. Mike Russell, “GVS was built around providing a version of Google Earth in the secret and top secret domains so users could visualize classified information geospatially and temporally in a common operating picture.”
Now, NGA’s private Google Earth globes are mission critical for more than 30,000 customers daily, including DoD Combatant Commands, the FBI, CIA, NRO, National Security Agency, and Federal Emergency Management Agency. NGA’s current release is the second largest Google Earth globe in the world and is used across the DoD and IC for common situational awareness, tracking vehicles and personnel, delivering intelligence briefings, and more.
Russell praised Google’s efficient rendering of data files in the Keyhole Markup Language (KML) format. KML was created for file building in Keyhole’s EarthViewer platform and has since become an industry standard for visualizing geospatial data.
“[Users] will create data files like the location of an IED or a live dynamic track of an aircraft. They can build these files rapidly and not to spec, put them in Google Earth, and they’ll run somehow. [Competitors] can only render smaller KMLs or those built to spec. That’s really the reason why no other applications have been able to enter this space as dominantly as Google Earth,” Russell said.
GEE served a far more specific client and purpose than the commercial Google Earth services, but its rate of adoption was noticeably low compared to most Google products.
According to McClendon, “Continuing to innovate on a hosted service exclusively for the enterprise community was not financially viable.”
In March 2015, Google announced the depreciation of GEE. After a two-year transitional maintenance period, the company stopped supporting GEE software in March 2017. Though it was being phased out of Google’s product line, GEE remained in use by invested customers relying on it to meet mission demands and house their data.
Hernandez recalled pushback from teams at Yuma who were not keen to change their data storage and visualization system. According to Russell, GVS feared losing its primary product and stranding customers without an application to replace it.
To accommodate the ongoing need, Google announced in January it would publish all 470,000 lines of GEE’s code on GitHub, allowing customers to continue using the software they’d grown loyal to and to improve the product independently.
For customers who prefer transitioning to a supported enterprise software, Google has coordinated with Esri to offer free software and training for GEE customers who migrate to Esri’s ArcGIS platform.
The open-source GEE (GEE-OS) suite includes the Earth server, Fusion, and a portable server allowing users to run GEE on a mobile device or desktop computer not connected to a centralized server. The GEE Client software, which is required to connect to the Earth server and view 3D globes, was not carried forward into the open-source environment. Instead, it will continue to be maintained and provided by commercial Google Earth.
Thermopylae Sciences and Technology (TST), NT Concepts, and Navagis—three longtime Google partners—supported GEE’s transition to open source. In the spring, each of the three companies sent a developer to Google in Mountain View, Calif., to spend several weeks learning the code from Google developers who had been maintaining the software baseline.
TST began a partnership with Google in 2007 through a series of federal government customer engagements supporting Thermopylae’s own Google Earth-based tracking console. When the open-source announcement was made, TST’s Earth Engineering team was reassigned to the company’s Open Source Development Office to create the GEE GitHub site and migrate the source code.
On Sept. 14, TST’s open source team released GEE-OS version 5.2.0, which matches the last proprietary release as well as fixes bugs that emerged during the two-year depreciation period.
“When we pulled the code out from [Google’s] proprietary side, there were a lot of things that needed to be built back up or replaced with open-source components,” said Thermopylae CEO AJ Clark. “Really these first few months are just about providing feature parity with where the code was at its last state inside Google.”
TST’s team aims to release GEE-OS 5.2.1 by the end of 2017.
Now that parity is achieved and the program’s performance is stabilized, developers will begin submitting expanded code contributions. According to Clark, the first value-add propositions will most likely begin to flow in early 2018. Meanwhile, DoD and IC users are eager to discover how they can further adapt the software for their specific missions.
Chris Powell, CTO of NT Concepts, said the company is working with its defense and intelligence community customers to support GEE and their transition to the GEE-OS baseline.
“We’re also actively looking for opportunities to contribute back to the open source baseline for feature improvements and capabilities,” Powell said, adding some possibilities are scaling the GEE processing power to a larger compute platform and examining how the software can be optimized for the cloud.
Hernandez said the planning crew at Yuma is looking forward to new software capabilities that could be built out at the request of the test community. Among these features, he said, is the ability to “grab geospatial objects and collaborate on them between multiple users; to grab, extend, and change the shape of a [weapon] footprint in 2D or 3D; and to provide a simulation of an object’s line trajectory.”
According to Jon Estridge, director of NGA’s Expeditionary GEOINT Office, the agency has committed to provide enhancements and ongoing sustainment to open-source GEE on Github through at least 2022.
“A few specific examples would be multi-threading the fusion process to support massive terrain and imagery updates, enhanced 3D mesh management, and inclusion of ground-based GEOINT content like Street View,” Estridge said.
Open source means more customizability for users with niche wants and needs. No two proprietary Google Earth globes look the same, and teams will have more command over the unique data they store, visualize, and analyze within the program.
“It’s very positive,” Russell said. “[Open source is] an opportunity for NGA to partner with Thermopylae to tie the proprietary and non-proprietary pieces together, and it allows us to sustain Google Earth for our user community for a longer period of time.”
The decision to make GEE code open source only improves the program’s accessibility and potential use cases, and will bolster the software’s longevity. Code sharing is a growing trend in the IC, and Google has provided government, military, and industry unlimited access and control of one of the most useful enterprise GEOINT tools on the market.