Your agent reads email and files tickets. So what does it do with an email written to trick it?
Sooner or later an assistant that reads customer emails gets one written to manipulate it. We test LLM applications and agents for prompt injection, over-broad tool permissions and data leakage, then change the design so a manipulated model can't do much damage.
AI security is the work of protecting applications built on large language models from prompt injection, leaks of sensitive data, and agents taking actions they shouldn't.
We review AI applications against the OWASP Top 10 for LLM Applications and red-team prompts and tool use. We design least-privilege permissions for agents and MCP servers. We also add controls that keep confidential data out of prompts, logs and outputs.
you probably need this if
An agent uses one API key with admin rights to every system it touches.
Your assistant reads uploaded files and web pages. Nobody's tested what happens if one of them has instructions in it.
Full prompts and responses, customer data included, sit in a logging tool that lots of people can open.
terms on this pagePrompt injectiondefinitionInstructions tucked into content an AI reads, like a document or a ticket comment, meant to make it do something its user never asked for.explained in a field note →
From the first call to a setup your own team can run.
Here's what usually prompts a call about your AI Security, what we do about it and in which order, and where you end up.
01 · where it hurts02 · what we change03 · how it runs04 · where you end up
incident board · ai security3 open
openAn agent uses one API key with admin rights to every system it touches.
recurringYour assistant reads uploaded files and web pages. Nobody's tested what happens if one of them has instructions in it.
blockedFull prompts and responses, customer data included, sit in a logging tool that lots of people can open.
exposure
Prompt injection testingTool and agent permission designData leakage controlsOWASP Top 10 for LLM…Output handling and downstream…AI supply chain and model…
startrehearsehandover
Inventory and threat model
Test
Fix the design
Retest and hand over
we are here
ready to hand over
Someone can still write an email that confuses the classifier, but it won't get the…
We fixed it so the model only sees documents the asking user may read, and we added the…
AI application inventory with data, tools and identities
Threat model per application
chapter 01 · where it hurts
Why teams reach out
An agent uses one API key with admin rights to every system it touches.
Your assistant reads uploaded files and web pages. Nobody's tested what happens if one of them has instructions in it.
Full prompts and responses, customer data included, sit in a logging tool that lots of people can open.
chapter 02 · what we change
How we solve it
Prompt injection testing
Tool and agent permission design
Data leakage controls
OWASP Top 10 for LLM Applications review
Output handling and downstream safety
AI supply chain and model inventory
chapter 03 · how it runs
Our approach
Inventory and threat model
Test
Fix the design
Retest and hand over
chapter 04 · where you end up
What changes
Someone can still write an email that confuses the classifier, but it won't get the agent to send mail or open other mailboxes.
We fixed it so the model only sees documents the asking user may read, and we added the leakage tests to release testing.
AI application inventory with data, tools and identities
Threat model per application
what we do
Where we spend the hours
6 areas of work, each one done by an engineer who has run it in production.
We try direct and indirect injection through every input the model reads, whether chat, uploaded files, retrieved documents, web pages, email or tool results. The test is what an attacker can make the application do, and also what it can make the model say.
We gave each agent and MCP server its own identity with the narrowest scopes that work, used user-delegated tokens where the action belonged to a person, and put human approval in front of anything irreversible such as payments, deletes or external email.
Retrieval that enforces the asking user's document permissions, redaction of secrets and regulated data before prompts leave your boundary, and logging policies that don't copy sensitive content into trace stores.
We go through the 2025 list from LLM01 Prompt Injection to LLM10 Unbounded Consumption. Findings point at the specific component, explain its risk and give a fix your team can actually build.
Model output treated as untrusted input before it reaches a browser, shell, SQL query or API. Schema validation for tool arguments, and encoding for anything rendered back to users.
We keep an inventory of models, providers, plugins, MCP servers and datasets, with owner, data classification and version for each. If something turns out to be vulnerable, or a model is retired, it's quick to find where it's used.
capability 01 / 06in production
Prompt injection testing
We try direct and indirect injection through every input the model reads, whether chat, uploaded files, retrieved documents, web pages, email or tool results. The test is what an attacker can make the application do, and also what it can make the model say.
We gave each agent and MCP server its own identity with the narrowest scopes that work, used user-delegated tokens where the action belonged to a person, and put human approval in front of anything irreversible such as payments, deletes or external email.
Retrieval that enforces the asking user's document permissions, redaction of secrets and regulated data before prompts leave your boundary, and logging policies that don't copy sensitive content into trace stores.
We go through the 2025 list from LLM01 Prompt Injection to LLM10 Unbounded Consumption. Findings point at the specific component, explain its risk and give a fix your team can actually build.
Model output treated as untrusted input before it reaches a browser, shell, SQL query or API. Schema validation for tool arguments, and encoding for anything rendered back to users.
We keep an inventory of models, providers, plugins, MCP servers and datasets, with owner, data classification and version for each. If something turns out to be vulnerable, or a model is retired, it's quick to find where it's used.
We pieced these together from AI Security situations that come up a lot. They're patterns. You won't find a specific client in them.
case file 01
An email triage agent that can also send email
composite · not a client story
Beforethe situation
There's a shared-mailbox agent that classifies inbound customer emails, drafts replies and is allowed to send them. Its mail connector has full mailbox permissions. Nobody has tested an email that tells it to forward the inbox.
What we dothe work
Write indirect injection tests in real email formats and run them against the agent. Split read and send into separate tools, require a person to approve any outbound message to a new address, and reduce the connector to the one shared mailbox.
Afterwhere you end up
Someone can still write an email that confuses the classifier, but it won't get the agent to send mail or open other mailboxes. Those injection cases now live in the regression suite.
case file 02
RAG search over HR files with permissions applied after retrieval
composite · not a client story
Beforethe situation
HR policy and case files are indexed by an internal search assistant. Filtering for the user happens after the model has already read the top documents. Sometimes the summaries reflect content that user can't open.
What we dothe work
Move permission filtering into the retrieval query using the user's group claims, re-index with document-level access metadata, and test with accounts from different departments for leakage through summaries.
Afterwhere you end up
We fixed it so the model only sees documents the asking user may read, and we added the leakage tests to release testing.
The model gets untrusted content from user input, retrieved documents and tool results. Controls at the gateway filter and redact. The agent acts only through scoped identities and needs approval for risky actions. Each step goes into the logs used for detection and testing.
01 · Untrusted inputs
User promptschat · uploads → AI gateway · input
Retrieved contentdocuments · web · email → AI gateway · retrieved text
02 · Guard layer
AI gatewayredaction · filters · limits → Model · filtered prompt→ Security logs · blocked attempts
We don't start a step until the last one has produced something you can check.
step 01
Inventory and threat model
We list every AI application, its data sources, tools and identities, then draw what an attacker could reach through each input.
step 02
Test
We red-team injection, leakage and tool misuse by hand and with automation, and map what we find to the OWASP LLM Top 10 and MITRE ATLAS techniques.
step 03
Fix the design
Fixes come next. Tighter permissions, approval steps, output validation and enforced retrieval permissions. Every attack case goes into the regression suite.
step 04
Retest and hand over
Rerun the attacks after fixes, write up residual risk for the application owner, and leave the test suite running in CI.
you keep
What you're left with
We write these for your AI Security setup, and they're working documents. Your team should be able to keep things running from them long after we've gone.
inventory01
AI application inventory with data, tools and identities
yours to keep
design02
Threat model per application
yours to keep
report03
Red-team findings mapped to OWASP LLM Top 10 entries
Some clients want a project team, some want a couple of engineers on theirs. Either works. Onsite, nearshore or offshore, with handovers across time zones that keep things moving.
Put these to anyone bidding on your AI Security work. The ones who've done it answer differently from the ones who've read about it.
Q1
Do you test indirect prompt injection through our documents and tool results, or just the chat input?
The serious attacks mostly come in through content the model reads. What the user types is a smaller part. Testing only chat input misses the most damaging forms of LLM01.
Q2
What can our agent do if it's fully manipulated, and how will you shrink that?
LLM06, Excessive Agency, is about how much damage the agent can do. Expect concrete permission and approval changes in the answer, not a better system prompt.
Q3
How will you check that retrieval respects each user's document permissions?
If access is filtered after the model has already read the content, RAG leaks sensitive information. It happens a lot.
Q4
Which of your findings will become automated tests we keep?
Models and prompts change often. One-off test results go stale, while a regression suite in CI keeps catching the same attacks.
Q5
How do you review MCP servers and plugins before we install them?
Supply Chain (LLM03) risk now covers third-party tools that receive tokens and data. Ask for a review process that looks at scopes, code source and update path.
01Can prompt injection be fully prevented?
Not with current models. Filters and system prompt instructions reduce it, but a determined attacker can often get through. What works reliably is limiting what a manipulated model can do, with narrow permissions, approval steps and treating its output as untrusted.
02What's on the OWASP Top 10 for LLM Applications?
In 2025 the list runs from Prompt Injection and Sensitive Information Disclosure through Supply Chain, Data and Model Poisoning, Improper Output Handling and Excessive Agency, to System Prompt Leakage, Vector and Embedding Weaknesses, Misinformation and Unbounded Consumption. There's now a separate OWASP Top 10 for Agentic Applications for 2026 as well.
03How should an MCP server authenticate?
It's based on OAuth 2.1. The current MCP authorization specification requires authorization servers to implement it. Practically, that's a real identity provider, scoped tokens for each client and no shared static API keys. We check the scopes each tool really needs.
04How long does an AI security assessment take?
It depends on how many applications, tools and data sources are in scope. One assistant with a few tools is a short engagement. An agent platform that many teams use needs an inventory first, and we size that before quoting the testing.
05Do you also handle AI governance and policy?
Yes. Policy, risk classification and regulation sit with our AI governance practice. What's on this page is the technical testing and design work.
06Is this different from a normal penetration test?
It overlaps. The web app, APIs and cloud configuration still need standard testing. What's new is attacks that go through the model, like instructions hidden in a document that make an agent call a tool. Traditional scanners don't try those.
related
Often paired with
Projects on AI Security usually end up touching one of these, so they're worth a look.
