Finding Security Bugs in Code With AI Chatbots and Agents
Although you can't trust code written by an AI chatbot or model you can use one to help you better secure your code
Although you cannot trust code written by AI you can use AI to help you write more secure code. It can help you find security problems in your code and fix them. I wrote about how AI is non-deterministic in another post and why you should not blindly trust it here:
Even though AI might not be perfect and may make mistakes, can it find some things faster than a human? In a lot of cases, yes. It might also get stuck or make things up. I’ve experienced both of those. The worst thing it can do is miss something that is a serious problem. You think you’re good because AI pentest. But AI is not a deterministic process and it makes a lot of mistakes. That’s why I wouldn’t rely solely on AI. But at the same time it can give you a fast run through of the code and find things you might have missed or haven’t considered from a security perspective.
Ask if there are security problems in the code
That’s it. Just ask. You may have to refine your query after that point to drill down into the specific problems. You’ll need to figure out if the model is hallucinating or if there are any false positives. But that is a really simple starting point that will generally be enlightening.
Glossing over the details
The problem is that when you have a large project the AI model is going to gloss over things. If you can ask your questions about smaller blocks of code at a time that may help it be more thorough. For example, are there any security problems in file xyz.
Missing the big picture
The next problem is that when you ask on a file by file basis the model may not read related files it need to understand the full picture. It may make assumptions and guess what it things that the other files are doing rather than reading them. You’ll need to include the proper directives in your prompt to get the full picture.
A second set of eyes
Typically what I do when I’m writing code and I want to know if there are any security problems in it is first write the code. Then I use a NEW session that doesn’t have any context about that code to ask it if there are any security problems in a block of code or program I have written. I find that asking an AI agent that just wrote code to review its own code is less subjective. You can also ask different models to review the code.
False positives
I get false positives a lot. Whether I’m using a traditional scanner or some AI magic there are always false positives, so you have to run an exploit to prove it. And even with a so-called exploit I’m getting false positives with the AI tools and methods I’ve used. So you need a human to check whether it’s a true issue or not. Hence why the smart companies are still using a human-in-the-loop model and not calling their tools “AI security agents” rather than “AI pentests.”
Gibberish (AI slop)
I had an emergency pentest for a client and was trying to get them any potential findings quickly so they could launch a product. I tried to find the actual issue in all the noise in a particular finding. I weeded out as much of the noise as I could and quickly sent it over with the warning the prose was AI-generated and it still had a bunch of gibberish in it. It wasn’t all wrong but there was a bunch of garbage tacked onto the issue. These types of reports are submitted to bug bounties are why many programs are complaining about “AI slop.” They waste everyone’s time. Don’t do it. I only sent that because the client was in a hurry and won’t be doing it again because it’s annoying.
Ask again. And again.
I find that asking over and over again is a good way to get more information or better results. The models generally don’t do the same time each time you ask a question so ask more than once.
Ask “Are you sure?”
You may start spinning in circles when you do this but ask are you sure over and over again to see what the model says. It will check and recheck and hopefully come to a correct conclusion eventually.
Ask a different model
Sometimes you can ask a different model. When a model gets stuck I’ll go back and forth between two different models until I get them to agree.
Check other sources
When you get a result, check other sources. This sounds obvious but some people aren’t doing it. They assume that was is coming out of the model is intelligence. That depends on your definition of intelligence I guess. It’s a statistical amalgamation of what the model has seen most and has been tuned to produce. I suppose that is how are brains work too. I remember reading in college that a person has to see an ad at least seven times before they will remember it. How many times do you have to see something before an idea because a fact in your mind? Do you check alternate sources?
Write tests
When you find security problems, ask the tools to write tests that check for security problems. I use this approach when security is paramount. You can’t trust AI as I already mattered. If I want to make sure something is secure, I will evaluate it with deterministic tools. If I see a security problem I write a test that will fail if that same security problem appears again.
That is an important step when vibe coding because often AI bots will reintroduce problems they have fixed in the past. It’s also great if you can set up tests that will check for and prevent all similar problems in your code moving forward. You can also integrate other tools into your tests or overall process to deterministically test for security flaws.
Do you use solid logical proofs that cannot be disputed to prove things are true or false? You can do that with computer bugs. With other aspects of our lives sometimes the appropriate answer is, “I don’t know” because you can’t know. You might trust someone else’s research or believe something is true. When we are talking about computer bugs and code, it is deterministic. We can prove that it works or it does not work using deterministic (not AI) tests to validate the answer form the AI tool is correct or not.
Reverse engineering
You can use AI to help with reverse engineering. When you see some tricky code you can ask for an explanation of the code. Perhaps you get a broad overview of the code. Then you can drill down line by line and ask what each individual line of code is doing. This can help you understand much more quickly whether there is a potential security bug that might be exploitable.
Pretty print the code
If you are reviewing code, you can prompt to “pretty-print” the code or make it more human readable. This can help you view obfuscated or minified code. Add a “and do not change anything” to the end of the prompt to try to minimize the chance that the AI model “summarizes” the code instead of giving you the exact code. There’s still a chance of hallucination but it can help.
Try the AWS Security Agent
Personally I don’t like giving tools credentials to my source code accounts but if you are ok with that you might want to try the AWS Security Agent:
https://aws.amazon.com/security-agent/
It can run tests on your repositories to find security problems. I wanted to use it on my last penetration test but I ran into some problems. First, I don’t upload my customer’s code when I find it to my own GitHub account for analysis. I do that on a locked down encrypted host in a private network.
In general, when I’m performing a penetration test I don’t push customer data into third-party tools hosted in the cloud because the customer may not have authorized those tools for use within their organization. They may not want their data, code, credentials, and security vulnerabilities in those external systems.
I also don’t ask customers to put records in public DNS. When you create a DNS entry to use a tool, that DNS entry is visible to the entire world. It gives people clues about the tools and services you are using. I would prefer to keep those things private on my penetration tests.
Where is the network traffic coming from? That was the other question I had when I was trying to use the AWS Security Agent. I haven’t had a chance to try it out. What if I am testing a customer’s account and application and someone else is using the security agent to test their accounts and applications at the same time. Can you tell which is the valid penetration tester and which is the imposter? I tend to give customers the IP addresses from which I am testing. Can it be set up in a private network? For external pentesters can they specify to a customer which IPs they will be using for the test by setting up a set of elastic IP addresses specifically for that test?
Portswigger’s AI tools in Burp
Some tools embed AI into their tools and you can access models from there. I have been trying out Portswigger’s AI tools embedded into Burp. This is an interesting way to use AI as a pentester as I can keep the data and actions by the tool segregated from the vendor for the most part.
But I do have this question. When you submit an AI prompt where that prompt and the response end up? If my request has authentication tokens in it are those making their way into an LLM? Are they traversing a third-party network or staying within my host and being sent straight to the website I’m testing? Burp is generating and performing exploits. At some point a model is involved. How much of the data is going to the model or coming back from the model?
The other issue is that when you are submitting information to the model is it getting your credentials in order to run or formulate tests inside the model? How does that work exactly? Beyond the security and sensitive data exposure questions, the AI in Burp seems to have a lot of limitations right now, similar to the issues I had using for troubleshooting with some tools.
You are looking at a request. You submit it with a prompt and ask a question about it. It may evaluate the request and run a bunch of tests. Then you see the results and you want to expand on that - but you can’t. You have to start over. The problem is that when you start over, you lose all the context from the last run and sometimes the next scan repeats all the same thing as the last even when you try to create a prompt to tell it not to do that.
In addition any time you ask a question it either gives you a super minimal and not that helpful response that you can’t ask follow up questions on, or it simply wants to run more tests. I had a lot more luck just using chatbots so far but it did help me understand some things about a complex architecture more quickly than I would have figured it out manually.
Security problems in a Lambda function
Here’s a concrete example where I used AI to check to see if some code had any security vulnerabilities in it. I wrote about how I was using AI to help me write a Lambda function here:
I was using Google AI since the code wasn’t proprietary and I didn’t care if it ended up in their model. It was for a blog post.
I followed that up with a prompt asking if there were any security problems with that Lambda code.
Here’s what I got back:
Sensitive data could be leaked from logs into the AI model from the CloudFormation logs.
Possible indirect prompt injection if someone inserts malicious code into the logs.
Lambda role may be over-provisioned leading to account take over.
Over collection of events with no filters other than time could cost a lot or exceed rate limits.
Lack of input validation on startDate and endDate could lead to code injection.
Those are some very valid points and it would behoove me to fix those problems in a production environment. If you are going to use the sample code I or anyone else or even an AI model provides you will want to do your own evaluation of the code and fix things like that before running it.
Threat modeling
You can also use this method of asking questions about your code to help you with threat modeling before you ever write the code or build a system. Ask it to help you think of all the different ways someone might attack the system. Don’t stop at the first answer. Ask the question in a number of different ways and use different models as needed to try to think about how can you architect a more secure system.
Consider the data flows
As you are using AI for all of the above, consider where all the data is flowing. Access to this sensitive data itself might be a serious security problem. When you ask a question about a system what data are you sharing? Where is it going? If you find a security problem who has access to that information? Using secure AI options from AWS will help. I wrote about the secure chatbot I wish I had here and thought through some of the data flows when using AI models.
The danger when allowing AI to run exploits
Some AI systems claim to run automated penetration tests. True penetration tests are not fully automated. They involve humans. I would consider those tools penetration test agents, not “penetration tests.” When you are allowing a tool to exploit your systems, where is the data flowing? What are they attacking? Can they end up attacking something they shouldn’t? Are they going to delete something or exfiltrate data? You’ll need to think through the guardrails of agents allowed to exploit things in your environment.
Security professionals warn about running honeypots in environments when you don’t know what you are doing. You could be running a honeypot that gets compromised and ends up compromising your other systems. A tool that performs exploits has the same risk or more. You need to be extremely careful where you run that tool. Make sure you can stop it if it has gone down the wrong path or somehow escaped whatever sandbox you have it in.
You do have your autonomous security agents running in a sandbox, right?
Hopefully that gives you some ideas about how you can use AI to find security problems in your code. Be careful where that data ends up if it is sensitive and contains vulnerabilities for critical applications. These are some of the basics of using AI to find security bugs.
Subscribe for more stories like this and follow Good Vibes
— Teri Radichel