Commercial threat intelligence

This post was featured in the magazine of the 2020 One Conference.

Threat intelligence (TI) is information on adversary behavior that can be used to adapt security measures to the evolving threat landscape. It often takes the form of a continually updated list of domains, IP addresses or file hashes that are observed in attacks. Such atomic pieces of intelligence are known as ‘indicators of compromise’, and can be used for network- or host-based detection. Threat intelligence also consists of reports written in natural language that describe the tactics and procedures of specific threat actors.

For organizations, it is hard to generate threat intelligence by themselves. For this reason, they share TI with peers, for example via sectoral Information Sharing and Analysis Centers (ISACs) or informal trusted communities. And, in many cases, they buy threat intelligence from specialized vendors. Leading vendors charge around €300,000-500,000 per year for a subscription to their threat intelligence data. This is a costly security investment. What are customers getting for their money? How good is the view that you get from the threat landscape? No one knows. The high price and restricted access to this data makes it hard to investigate it independently. But we found a way. What did we uncover?

Limited coverage

We carried out a comparison of the indicators of two leading commercial vendors of threat intelligence. If two vendors are tracking the same threat actor groups, you might expect them to be providing similar indicators to their customers. We find that this is not the case.

In our peer-reviewed paper, the vendors have only a tiny 4% overlap between their indicators for the same threat actors. Across all data in their TI feeds, we find at most 13% overlap. Our findings suggest that TI vendors have very limited coverage of the threat landscape they claim to track.

We spoke to 14 professionals that use paid TI, and found that most of them were using a combination of free and commercial sources of threat intelligence. The selection of source was based on heuristics and implicit ideas about the value of each source for their organization, not on metrics on the properties of the sources.

When our respondents did mention metrics, this was often in the context of preventing false positives. In fact, rather than maximizing the coverage of potential attacks, customer organizations seemed to be optimizing for reducing false positive. This is because of the scarce resource they are managing: time of their analysts who investigate the alerts. Customers preferred smaller, more curated datasets that led to a small number of false positives over alternatives that might potentially detect more incidents (low false negatives).

Market characteristics

From earlier surveys, we know that customers in the market for commercial TI are struggling to compare services. Our respondents confirmed that it was still “mostly guesswork” to understand the methods that TI vendors use to collect their data.

The market for commercial TI is characterized by asymmetric information, meaning that the sellers know what they are selling, but the buyers don’t know what they are buying. In time, this might lead to a ‘market for lemons’ in which the willingness to pay of consumers decreases because they cannot distinguish the good from the bad. Vendors could prevent this by being more transparent, for example about how their data is collected and what it can reasonably cover.

Selecting sources

For a more structured approach to selecting sources of threat intelligence, paid or otherwise, organizations could start by drawing up the current threat landscape of their organization, asking questions such as:

• ‘What adversaries might target us?’
• ‘What assets could they be after?
• ‘What tactics would they employ?’

Identified gaps in the organization’s understanding of its threat landscape can then be made explicit. Intelligence requirements are the questions that threat intelligence could possibly help resolve, such as:

• ‘Is actor X targeting our region?’
• ‘What are X’s tactics?’
• ‘What are actor Y’s current indicators of compromise?’

With intelligence requirements in hand, candidate sources of threat intelligence may be evaluated for their ability to provide new information. Vendors often allow for a ‘proof of concept’ trial period of access to their data, letting customer organizations evaluate and compare alternatives. We refer to publicly available resources for more information about planning and direction of a threat intelligence capacity.