CIA Looking for Cutting Edge Big Data Technology in Silicon Valley

Jonathan Vanian

June 17, 2015

What the CIA and Silicon Valley have in common

The CIA’s top techie explained why the intelligence agency is interested in the same big data tech that businesses love.

Doug Wolfe, the CIA’s chief information officer, made an unusual sales pitch to Silicon Valley on Tuesday by arguing that the spy agency and the tech industry have a lot in common.

“Remember, a lot of the solutions we need are similar to the private sector,” he told the crowd at a tech conference in San Francisco, using some tech industry jargon in the process.

Wolfe, a 30-year veteran of the CIA, was trying to explain the intelligence agency’s interest in a hot technology for data-processing called Spark that’s the current rage for big data nerds. It lets businesses sift and analyze data much quicker than they could just a decade ago.

Of course, Wolfe stayed silent about why the CIA is so interested in processing huge amounts of data. But the reason was hardly a mystery. As part of its spy mission, the CIA invariably wants to quickly glean insights from huge troves of information it and fellow spooks at the National Security Agency collect on a daily basis. Records of international money transfers, cellphone calls, and biometric data about possible terrorists are just some of the inevitable areas of interest.

It’s the same kind of technology that many big businesses use everyday. It turns out that covert CIA operations and routine online marketing campaigns by retailers are much the same.

Both the CIA and advertisers want results fast — very fast.

Regardless, the type of technology the CIA uses is apparently similar to what theonline restaurant reservation startup OpenTable relies on to manage its customer database. Wolfe said he happened to have had dinner with some employees of OpenTable the night before the conference and was fascinated to learn about the tech the company uses.

In a separate presentation at the conference, OpenTable OPEN 0.00% said it used Spark to improve the personalized restaurant recommendations it gives to customers. While the CIA is probably not interested in learning about new fine-dining options, it could use similar predictive technology to help pinpoint terrorists faster.

Throughout the talk, Wolfe emphasized the need for the tech community to work with the government, echoing similar statements by President Obama and the Department of Homeland Security. In April, that agency opened up a new satellite office in Silicon Valley.

But for the past few years, the technology industry has been critical of the federal government’s data-collection and surveillance tactics. At the same time, government agencies have been trying to convince tech companies to share more information with it as part of a data-sharing hub for national security called theNational Cybersecurity and Communications Integration Center.

Wolfe’s talk emphasized this theme of the government working hand-in-hand with the tech industry by telling the crowd that it’s “critically important that we partner with you all” in the name of national security.

But the crowd, which remained silent during the talk, was left in the dark about how partnering up with the CIA will help protect the nation because Wolfe, in classic CIA fashion, didn’t give any specifics.