The Internet of Things (IoT) is infiltrating the world of defense and security at an astounding pace, so the amount of data available to intelligence and security agencies is, unsurprisingly, staggering. But indeed, while collecting data is easy, using it can be hard. And that's exactly the challenge this industry faces today.
Here at Dataiku, we've been a part of several projects in this space detailing how data science and machine learning can help keep us safe, from predicting connections to ISIS based on Tweet content to predicting crime in Portland, Ore. To be sure, there are a lot of opportunities for these technologies to help make our world a better place and keep people safe. However, the defense and security industries face a few main challenges when it comes to harnessing the power of data for the greater good, namely:
Amounts and Sources of Data Are Massive and Varied
It is no longer feasible for teams of humans alone to handle the volumes or types of data that we're talking about in this space. It's because of this reality that data is growing exponentially, but insights from that data is growing linearly: people alone simply can’t keep up, and it's financially impossible to hire teams that would be big enough to analyze the data from IoT outputs.
Security Is Extra Important
Of course, good data governance and security practices are always important no matter the industry (ahem, Equifax). But it becomes truly critical when handing sensitive data for security and defense. And this can really be a roadblock to rolling out large-scale, automated data projects that are working in near real-time. Without the peace of mind that data is safe and there are clear processes and protections in place, projects cannot move forward.
Timing Is Everything
This challenge is a double-edged sword. Machine learning and data science are the perfect opportunities for analysis where time is of the essence, because they can work much more quickly than a person (as detailed above). But on the other hand, if the work is not truly real time, it can quite literally cost lives. So this makes investing in data infrastructure more difficult in defense and security, where one must select and stay on top of the most cutting-edge technologies.