AI Threat Modeling

What is Threat Modelling?

In cybersecurity, threat modelling is identifying and understanding potential threats and mitigating threats to protect something of value.

At Matillion, we create threat models for new feature and architecture changes in order to discover potential security concerns relating to the change, which we then look at mitigating. 

What challenges does it bring?

  1. Can be time-consuming
  2. Often requires someone knowledgeable in security to run the session  
  3. Those with less security knowledge may not feel like they can contribute. This is a particular problem given that the most effective threat models require multiple perspectives as…
  4. There is always the possibility that some serious potential threat could be missed

 

How can AI help?

By automating the process, AI-assisted threat modelling could potentially help with all of the above problems:

Speed: LLMs can produce responses extremely quickly, potentially threat modelling a whole feature in seconds.

Engineering-led: By reducing the need for technical security knowledge needed to generate a threat model, threat modelling sessions could be run by the software engineers who know the feature architecture best.

Inclusivity: An AI threat modelling tool could be used by all members of the team, meaning that it won’t be just those with a security background who are comfortable contributing.

Exhaustiveness: Using an additional tool to brainstorm threats could allow us to consider more possible threats which may have otherwise been missed.

Solutions we looked at

ChatGPT
  • Allows for highly customisable prompts + back and forth conversation
  • Familiar interface for most users
  • Free tier which is very capable.
StrideGPT

This is an open-source web application that uses the OpenAI API to generate threat models using the STRIDE framework. The STRIDE model is something we use at Matillion for our own threat modelling sessions. It categorises threats into the following categories:

Spoofing - Pretending to be someone/something you are not

Tampering - Changing data which isn’t supposed to be modified

Repudiation - Denying responsibility for an action (which may be possible with insufficient logkeeping)

Information disclosure - Accessing information you shouldn’t be able to

Denial of service - Preventing others from using a service

Elevation of privilege - Doing something you shouldn’t be able to

This framework is incredibly useful for threat modelling, and at Matillion we use it in order to help with the brainstorming of threats.

  • Custom-made for threat modelling. Built around the STRIDE framework for threat modelling which we use in our own threat modelling sessions
  • Uses the OpenAI API so it can be powered by OpenAI’s powerful GPT-4 model
  • Asks a series of questions about the general security of the product, such as what method is being used for authentication
OpenAI Playground

OpenAI’s playground allows for the creation of custom “assistants” which are essentially versions of ChatGPT: 

  • Allows for highly customisable prompts + back and forth conversation 
  • Slightly different interface to ChatGPT, but included as part of Matillion’s OpenAI organisation.
  • Can be reconfigured to remove the need for users to enter long prompts  
  • Access to GPT 4, which generally gives better results than ChatGPT’s 3.5
Bespoke Solution

This solution would combine the ease of use of a tool like StrideGPT with the ability of OpenAI’s assistants to customise to the specific use case. By creating a basic custom application, we could have some more sophisticated logic than just a short prompt before the user interacts with the program.

  • This would require some programming, so would take more time & effort to implement
  • Could allow for more relevant threat models - the tool can have a better knowledge of Matillion-specific services/features and it can have multiple functions depending on the use case
  • It could be made to be very easy to use, with a high degree of control over inputs and outputs

Why we chose what we chose

StrideGPT does generate some good suggestions. However, from my testing, it repeatedly tries to threat model the whole application when the aim is only to look at a specific feature. This holistic approach means it will often consider aspects of the application that are irrelevant to the feature in question at the cost of going into more depth.

On the other hand, ChatGPT is able to identify threats that are much more focused on a specific feature within an application and, thus, much more useful. It is also free to use and has a familiar interface to most people.

However, from my comparison, the assistant tool in OpenAI Playground is favourable to ChatGPT because:

  • The prompt can be pre-configured beforehand (and also can be changed in the future if necessary). There is no reliance on the user to give a prompt, resulting in the best threat model.
  • The assistant tool has support for GPT 4, which is still getting frequent updates and generally gives better results than ChatGPT’s GPT 3.5
  • Interaction with the assistants tool is not used for training whereas ChatGPT is by default.

The other solution I considered was a custom-coded solution. This would have been similar to the assistant tool in that the prompt would have been pre-written, and all that would be required would be a feature description from the user. However, given that this would be using the same API as the assistant's tool without much of a clear benefit in terms of functionality, I was able to rule out creating a bespoke solution.

The only disadvantage is that the assistant tool is not completely free. However, each interaction with the tool costs a very small amount (several cents). 

In conclusion, the selected tool was the OpenAI assistants tool because:

  • It is easy to use
  • It is highly customisable
  • It is cheap
  • It has a familiar interface
  • The quality of responses was the highest of the tools assessed.

Impact

We tested the AI assistant with three separate engineering teams and got some good feedback.

Positives:
  • It was very simple to set up an assistant—it can be programmed with just a text prompt describing the threat modelling task.
  • Many of the threats were specific and useful.
  • Some threats were ones which the engineers may have otherwise missed.
  • Several engineers with little security experience said they felt like they could better contribute to the session using this tool.
Negatives:
  • The tool requires a text description, which can be time-consuming to generate this feature description beforehand.
  • It initially took some trial and error to get useful responses, which could be difficult for those not familiar with using LLMs for this sort of thing.
  • The tool sometimes generates threats too vague to meaningfully do anything about (though this could be helped by including more specifics in the feature description) 
  • The tool knew little about Matillion services. This meant that each team would spend much time describing services/features other than threat-modelled ones.

Many of these negatives could potentially be alleviated in the future. For example, we plan on creating pre-written technical descriptions on Matillion services to allow the process of describing the feature to the tool much faster.

The engineering teams' feedback was generally positive, and many said it would be a useful tool for making threat modelling sessions more accessible and useful in the future.

One engineer said, “Overall, I think the assistant could be a useful aid in threat modelling, but I don't think it can replace a threat modelling session with a development team just yet.”

Conclusion

Integrating AI into threat modelling at Matillion has shown promising results, enhancing both the efficiency and inclusivity of our security processes. The OpenAI assistant tool stood out due to its ease of use, customizability, and superior response quality. Although there are some challenges, such as the time required to generate feature descriptions and occasional vague threats, the overall feedback from our engineering teams has been positive.

Looking ahead, we aim to refine the process further by preparing pre-written technical descriptions of Matillion services. This will streamline the setup phase and improve the specificity of the threats identified. While AI tools may not yet fully replace traditional threat modelling sessions, they certainly serve as valuable aids, making the process more efficient and accessible.

Michael Lowe
Michael Lowe

Application Security Engineer - Placement Student