Defense and Security: Data Explosion (& Accompanying Challenges)

business| machine learning| defense | | Remi Meunier

The Internet of Things (IoT) is infiltrating the world of defense and security at an astounding pace, so the amount of data available to intelligence and security agencies is, unsurprisingly, staggering. But indeed, collecting data is easy - using it can be hard. And that's exactly the challenge this industry faces today.

landingpage-security-WP.jpg

Here at Dataiku, we've been a part of several projects in this space - looking at how data science and machine learning can help keep us safe, from predicting connections to ISIS based on tweet content to predicting crime in Portland, Oregon. To be sure, there are a lot of opportunities for these technologies to help make our world a better place and keep people safe. However, the defense and security industries face a few main challenges when it comes to harnessing the power of data for the greater good, namely:

Amounts and Sources of data are massive and varied

It is no longer feasible for teams of humans alone to handle the volumes or types of data that we're talking about in this space. It's because of this reality that data is growing exponentially, but insights from that data is growing linearly: people alone simply can’t keep up, and it it's financially impossible to hire teams that would be big enough to analyze the data from IoT outputs. 

Security is Extra Important

Of course, good data governance and security practices are always important no matter the industry (ahem, Equifax). But it becomes truly critical when handing sensitive data for security and defense. And this can really be a roadblock to rolling out large-scale, automated data projects that are working in near real-time. Without the piece of mind that data is safe and there are clear processes and protections in place, projects cannot move forward.

Timing is Everything

This challenge is a double-edged sword. Machine learning and data science are the perfect opportunities for analysis where time is of the essence, because they can work much more quickly than a person (as detailed above). But on the other hand, if the work is not truly real time, it can quite literally cost lives. So this makes investing in data infrastructure more difficult in defense and security, where one must select and stay on top of the most cutting-edge technologies.  

Facing The Challenges

Want to learn more about how to start using data science and machine learning to tackle big data challenges in the defense and security space? Download the white paper for details and case studies.

 Dataiku Production Survey Report

Other Content You May Like