28 May 2019

NEWS FROM SOFIC: New Software Can Sift Through Decades of Stovepiped Intelligence Data

By Stew Magnuson

TAMPA, Fla. — Defense contractor Leidos revealed a new big data analytics tool that an intelligence community customer has been using for the past six months to sift through terabytes of data previously stored on different devices.

The company publicly spoke on May 21 for the first time about the Advanced Analytics and Machine Learning Microservices Platform on the sidelines of the Special Operations Force Industry Conference, which is sponsored by the National Defense Industrial Association.

The problem for intelligence analysts has been that data that can be useful for them has been stored on a variety of stovepiped software systems, or in different digital mediums, said Christine J. Harney, data systems engineer in the surveillance and reconnaissance group at Leidos. They are too massive and there isn’t enough manpower to look through it all, she added.

“These data collections are very complex and there are multiple types of devices and the number of those devices grows over time,” she said. An analyst could never hope to see all the data available with manual searches, she added.


Artificial intelligence and machine learning is helping the intelligence community sort through all the data coming from different media and find the content relevant to them, she said.

“We have enabled our customers to discover the previously undiscoverable data and get directly to that mission-relevant content,” Harney said. The technology is gathering current data from open sources such as social media, and going back into data sets on proprietary software and systems from about the time of the 9/11 attacks and ingesting them into the platform as well.

She demonstrated the system using two open source data sets: one was the digital media collected at the raid on Osama Bin Laden’s compound in 2011 at Abbottabad, Pakistan. The other was a collection of open-source intelligence about ISIS, including claims-of-responsibility that the terrorist organization published publicly. That also included hand-written notes, which traditionally have been hard to extract from analytical tools for analysis. 

Harney embedded the image of a Game of Thrones character Tyrion Lannister among all the data to see if it would pop up in a search based on facial recognition. It did.

“The value is that we can build up services and workflows that mean something to the customer,” she said. That may include in the future law enforcement applications for criminal investigations, she added.

“We’re not completely through that 20-year backlog of collection that had been stovepiped to very specific forensic analysts and investigators,” said Matt Reardon, Leidos’ director of operational analytics at the company’s innovation center. It will take more time to ingest it all into the platform, he said. How long would it take? He didn’t know.

Data stovepipes were created for both technology reasons and policy reasons, Reardon said. “We’re now pushing beyond that from a technology perspective. We’re catching up with a lot of the backlog.”

The next step will be fusing all the sensitive-site exploitation data, such as what was found at the Bid Laden compound, with open source data, signals intelligence data and human intelligence data, he said.

No comments: