When to Use An AI Agent
Considering when AI agents are useful and when they burn tokens unnecessarily or may produce a harmful result
I started writing about batch jobs about three years ago on my old blog. A batch job is just a way to run some automated process to complete some task on some data. I started out with this idea that I was “just” going to run some batch jobs to produce cybersecurity metrics. But the whole adventure turned into this: Setting up the infrastructure to run batch jobs securely on AWS and writing all the code to automate it was insanely time-consuming and error prone.
So along comes AI and it’s all the rage now and people are talking about AI agents. What are AI agents? Well really they are just like batch jobs but they are kicked off by a user natural language prompt instead of some logically defined parameters. They are using AI models to complete the tasks which means they are at least at some point non-deterministic.
And that’s why I had to stop and write this post before writing more about AI and AI agents. This is the most important concept you need to understand when choosing to use AI in your application architecture or for data processing.
Check out my post on generative AI on AWS and AI terminology if you see any terms you don’t recognize below.
AI Agent and Batch Job Security
I spoke about how AI agents are basically batch jobs with some new functionality and you need to take all the same steps to secure them at AWS re:Inforce (which will be part of AWS re:Invent in 2026 and AWS Community Day at the Community History Museum in Mountain View last year.
I’ll be heading back to speak at that event again this year. Check it out!
When I started creating custom Kiro agents after that presentation, the first thing I noticed was that it was trying to use sudo and attempting to access my AWS credentials when I had none on the machine and the code I was writing had absolutely no reason to use those credentials. This has even greater risks than someone inserting malware into a batch job or gaining access to it, and especially if you give the user that runs the agent free reign to access your entire system. I immediately stopped and spent weeks locking that down before proceeding.
I started with Q CLI agents using specific Linux users that had very sandboxed permissions. I opted NOT to give these AI agents credentials of any kind - github, AWS, or otherwise.
Later AWS renamed that to the Kiro CLI but it’s essentially the same. I was able to transition that code pretty easily to Kiro once I figured out a work around.
https://medium.com/cloud-security/upgrade-amazon-q-cli-to-kiro-cli-57c2c25d4fa5
Amazon Bedrock Agents
If we take a look at the AI services on AWS, we have different options for creating AI agents beyond the Q and Kiro CLI agents I was using in those posts. I started looking at Amazon Bedrock agents and how I might use those. First I had to find the documentation for Bedrock Agents. It is embedded within the Amazon Bedrock documentation here:
https://docs.aws.amazon.com/bedrock/latest/userguide/agents.html
One of the benefits of using Amazon Bedrock agents should be that some of the heavy lifting of securing your environment should be offloaded and baked into the service. Of course, you still have to understand the difference between deterministic and non-deterministic outputs and how they are used and secure the portion of the architecture for which you are responsible.
Amazon Bedrock Agents is the service to use if you don’t want to manage any of your infrastructure and use Amazon’s predefined logic for creating agents. Amazon Bedrock AgentCore aims to create a secure framework for running AI agents regardless of which tool you used to create them and provides more customization options.
https://docs.aws.amazon.com/bedrock-agentcore/
Either option will be more secure than letting agents run rampant in your environment with no guardrails or constraints. The non-deterministic nature of inputs and outputs means you cannot ever really trust AI agents. Design your architecture accordingly.
Bedrock AI Agent Basic Steps
Now let’s take a look at how an Amazon Bedrock agent works at a high level as compared to the Kiro agent I’ve been using for generating code.
To configure a Bedrock AI Agent do at least one of the following:
Configure and use a knowledge base referenced by the agent
Configure an action group that defines what actions the agent takes
That translates to configuring at least one of the following:
A dataset it can query to complete its task
Which Lambda functions or actions it can execute
You can optionally do the following:
Create prompt templates for pre and post processing that augment your prompts to increase accuracy.
Test your agent in the console.
When you are ready to deploy:
Create an alias for your agent.
Set up your application to call the agent using the alias.
Amazon Bedrock API calls
Here’s a list of the AWS service API calls associated with Amazon Bedrock
https://docs.aws.amazon.com/bedrock/latest/APIReference/API_Operations_Amazon_Bedrock.html
Build Time (Configuration) APIs
Some of the APIs are used at build time to configure an agent. Configuration may include the following:
Model
Instructions (What the agent is supposed to do)
Action Groups
Knowledge base
Prompt templates
Runtime APIs
Some of the APIs are called while an agent is executing. Here are the basic runtime steps taken by an agent.
Preprocessing the prompt in advance of execution
Processing (non-deterministic)
Interpret the inputs with a model
Generate a plan
Predict which action group or knowledge base it should use
Query the knowledge base or call the action group Lambda function
Produce output and use it to add to the original prompt
Loop until complete or it needs to prompt the user for more information
Post processing of the response if enabled
Notice: The runtime processing is predicting which actions it should take. This is not a concrete or deterministic process. It is using a map of relationships to guess what queries to make or Lambda functions to call rather than using pure logic based on defined keywords. It is using a model to predict what it should do next and what the output should be. It is making an educated and informed guess as to whether it should continue to loop or ask the user for additional input. In all my testing while using AI, although the results can sometimes be exactly what you want, this is never, ever consistent.
When should you use an AI agent?
Now that we understand how an agent works, what types of tasks would be appropriate to give an agent? When you use an agent it burns your tokens. And tokens are essentially money. They are not cheap.
I burn way more tokens than compute in terms of cost for the things I have been testing. I have been working on ways to optimize that. You pay for tokens as you use them. However, if you were to create a deterministic process you only pay for the compute to process your action and that seems to be significantly cheaper. So why use AI at all? In return I get some things done faster — but not all.
So if you are trying to determine whether to use a deterministic process or an AI agent the first question you should ask yourself is: Which is more cost-effective for your use case and will produce the level of accuracy needed for your particular task?
If I look at how an Amazon Bedrock agent works above, it is prompting a user for information and taking actions based on their prompt. It can get a natural language prompt and based on that prompt, it chooses which Lambda function to execute.
If I have a list of Lambda functions and all I need a user to do is pick one to execute, do I really need an AI agent for that? Not really. It is likely more cost-effective to simply create a web page with a list of Lambda functions and have the user select from a defined list and execute the function they selected.
But wait, I want to use natural language! How much is that worth to you? You could create a slack chat bot that takes specific key words and executes a Lambda function using a deterministic process (though I don’t think Slack is really secure enough for this use case if doing anything very sensitive like modifying your AWS account.) But you could create or use your own chatbot in your own secure environment for that purpose if it is something very specific and repeatable and you need a specific outcome.
If I am performing mathematical calculations for a bank statement, is that an appropriate use case for an AI agent? No. We need those calculations to be precise and accurate every time with no variation. I explained why you can’t count on the out in my post on the non-deterministic nature of AI.
If I am performing a database query should I send my request to an AI agent? That depends. Do you want to pay for the tokens on top of the database query? Is there some reason you cannot just directly run the database query? Perhaps you are making it easier for a non-technical person to run the query. You have to weigh the cost of the token usage against the functionality the AI agent provides over and above a direct query of the database or a web page with a search box to find the data.
So when would it make sense to use an AI agent?
Let’s say you have a data set and you don’t know what’s in it. You want to read the data set and recommend what the user should do next. The recommendation is not a concrete pre-defined answer. It’s based on some set of data that you can’t know in advance and could have a lot of different types of information in it. You can’t concretely define every possible input because there are too many variations and options that could occur. You don’t know every possible answer either since you don’t know exactly what the user is going to ask.
That could be a good use case for an AI agent.
Here are some examples I can think of that might be a good use case for an AI agent:
Read Google news and parse market data for a stock portfolio and try to analyze it for buy, sell and hold recommendations.
Deterministically query all AWS CloudTrail logs for all errors and then send that output as a prompt to an AI agent to review them and make recommendations to fix the errors in the logs.
Review network logs (packet captures) with AI, identify threats, and make recommendations based on a customer’s specific environment if the threats apply.
Analyzing anything with mountains of data that is too great to classify in advance or review in detail and provide a summarized result. The output may not be perfect or the same every time but it’s better than what we could do manually or deterministically.
Generate something that we want to be creative where the results should not be the same every time. For example, I have been using Nano Banana to try to generate images for blog posts. There’s no concrete definitive answer for what I want that image to look like based on the text of the blog. I also might not like certain styles or colors and ask for different variations so I can pick the one I like best. I did that for this blog post but in the end I had to ask for three different images and put them together manually to get the layout I wanted.
There’s one use case I am especially interested in using AI agents for and that is medical research. The human body is incredibly non-deterministic. Every body is unique. All the cells in our bodies have relationships to each other and external inputs and those produce random outputs. In my mind, I imagine this map of relationships of cells that is similar to the map of relationships of tokens in an AI model. I imagine finding the answers to medical problems more quickly or providing pinpointed individualized treatments. But I’m not a doctor or a scientist. Perhaps the relationships are different and the alignment is not exactly as I imagine it. We’ll see what the future holds for generative AI and medical research.
The caveat
When applying AI to any problem it is important to understand the non-deterministic nature of the response. AI can vastly speed up our understanding, the learning process and production of an initial first draft summary of output we might want to use related to some problem. That initial output production must be followed by steps to validate the output and to make sure nothing was missed in the summarization process.
In my own research and writing, I use AI a lot to get a grasp on new topics. I wrote how I learned the basics of the Rust programming language in one day. Note the word basic. Also note that I knew what to ask because I have 30 years of programming experience so I knew what to ask and how to craft the prompts to get what I wanted out of the CLI agent I was using.
Although AI got me started in one day - something I had been putting off for years - I spent many hours after that diving into the constructs and getting to know what each keyword meant, how it was used, best practices and directing the AI CLI tools to actually produce quality code and a maintainable architecture. As it turns out, I had to build a lot of framework code and guardrails around the AI to keep things on track. And that is essentially what the Kiro IDE is doing for developers.
When I write, I ask AI a lot of questions and generate answers. Then I refine that information. It is almost never usable in its original form. It is overly wordy, unclear, inconsistent, not answering the right question, and sometimes just plain wrong. It takes some effort to ask and re-ask and drill down and get what I actually want out of a model. Then I need to reformulate that into something more palatable, organized, and useful.
If you are going to simply throw prompts at AI models and spit it back out to customers as a customer service agent, I don’t think that’s going to work too well for you in many cases. You might end up making your customers angry when they have very specific questions and you are regurgitating general answers that gloss over the problem.
However, you could use a combination of AI and access to links and humans to provide options. In some cases the AI answers might be fine. In other cases, like the answer I just got from a particular company, the answers might feel like a colossal waste of time. They didn’t actually address the question I asked. The AI bot skimmed some keywords and gave me a canned response that was absolutely no help at all.
Hopefully that helps give you some ideas about how and when to use an AI agent. As always, things are changing constantly and I expect some of these tools we are using will continue to improve and produce better results than they do today. As a system architect pondering any new technology, you have to choose the optimal tool for the job based on your specific requirements such as system performance, data sources, resources, cost constraints, and your desired outcome.
For more posts like this subscribe and follow Good Vibes.
— Teri Radichel