Security testing an LLM application, beyond "try some prompt injection"

“Can you pentest our AI feature?” is now one of the more common requests I get. It’s a good instinct, but the question usually hides an assumption worth unpacking: that testing an LLM application means testing the model. In most engagements, the interesting findings are in everything wrapped around the model — the retrieval layer, the tool integrations, the trust boundaries the LLM sits on. Here’s how I scope one.

The model is one component, not the system

A typical LLM application looks like: user input → your application → a prompt assembled from (system instructions + retrieved context + conversation history + user input) → the model → output that your application then acts on. Every arrow is an attack surface, and most of them are ordinary application security, not AI magic. The OWASP Top 10 for LLM Applications is a useful map here; I’ll reference its 2025 categories where they apply.

Prompt injection is real, but it’s a means, not the finding

Direct prompt injection — the user telling the model to ignore its instructions — is worth testing, but on its own “I made the chatbot say something rude” is a low-severity finding. What raises severity is what the injection can reach:

Indirect prompt injection (LLM01) is the one that actually keeps me up at night. If your app retrieves web pages, emails, documents, or support tickets and feeds them to the model, an attacker who controls that content controls part of your prompt. The classic chain: attacker plants instructions in a document → your RAG pipeline retrieves it → the model follows the injected instructions → it calls a tool with attacker-chosen arguments. Test it by planting payloads in every data source the model ingests, not just the chat box.
The question is always “and then what?” Injection into a model that can only produce text is a content problem. Injection into a model that can call functions, query a database, or send email is a privilege problem.

Excessive agency is where injection becomes compromise

OWASP calls this LLM06: Excessive Agency, and it’s the category I spend the most time on. If the LLM can invoke tools — send emails, query APIs, execute code, modify records — then the real test is whether the tool layer enforces authorization independently of the model. Key questions:

Does each tool re-check the user’s permissions, or does it trust that “the LLM decided to call it” implies authorization? (It must be the former. The model is not an authorization boundary.)
Are tool arguments validated? An LLM instructed to “look up order 12345” that instead calls get_order(user_supplied_id) is a textbook IDOR — with a language model as the confused deputy.
Is there a human-in-the-loop for high-impact actions (payments, deletions, external sends), or can a single injected instruction trigger them?
What’s the blast radius of the credentials the tools use? An agent with a broadly-scoped API key is a broadly-scoped incident.

Sensitive information disclosure (LLM02)

Two flavors: the model leaking its system prompt (usually low impact, but revealing if the prompt contains secrets or hints at internal architecture — it shouldn’t), and the model leaking data from its context window across trust boundaries. The dangerous version: a multi-tenant RAG system where user A can craft a query that surfaces documents belonging to user B, because retrieval filters on the vector store aren’t scoped to the requesting user. That’s not a model flaw — it’s a missing authorization check in the retrieval layer, and it’s a serious one.

The classic web vulnerabilities don’t go away

The output of an LLM is untrusted data. Treat it accordingly:

Output handling (LLM05): if model output is rendered as HTML, you have a stored/reflected XSS surface where the payload can be induced via prompt injection. If it’s passed to a shell, eval, or a SQL query, you have injection of the classic kind. The model laundering the payload doesn’t make it safe.
Rate limiting and cost (LLM10, “Unbounded Consumption”): LLM calls cost money and compute. An endpoint that proxies to a paid model without rate limiting is a denial-of-wallet vulnerability. Test for it.
Ordinary API security: authentication, authorization, and injection on the endpoints that front the model. These findings are often the most impactful in the whole engagement and have nothing to do with AI.

What a scope document should say

When I scope an LLM application test, I want to enumerate:

Every data source the model ingests (for indirect injection).
Every tool/function the model can invoke, and the privileges behind each (for excessive agency).
Every trust boundary — especially multi-tenant data separation in the retrieval layer.
How model output is consumed downstream (for output-handling flaws).
Rate limiting, quotas, and cost controls.

Notice how little of that is about the model’s weights or its propensity to hallucinate. Model-level robustness (jailbreak resistance, safety alignment) matters, but for a deployed application it’s usually the provider’s problem, not yours. Your problem is the system you built around it — and that’s testable with the same rigor as any other application, plus a few AI-specific abuse cases.

Building or shipping an LLM feature and want it assessed before it goes live? AI/LLM security assessment is one of the services offered — get in touch to scope it.