Access:

  1. Least Privilege / Minimize Scope, Tools & Permissions and enforce that

    • Why:

    This practice is useful for minimizing the damage that can be caused by prompt injection. Even if malicious instructions are sneaked into the LLM's input, they won't be as harmful because the LLM doesn't have enough leverage to cause major damage. It also guards against unbounded consumption, as this practice limits how much damage a runaway loop or recursive call can actually do, even if it fires thousands of times.
    Narrowing access only helps if that boundary is actually enforced: if the LLM itself is trusted to decide whether a specific action is appropriate, a convincing injection or hallucination can still talk it into misusing the access it does have. That check needs to happen in your own application code every time a tool is called, not in the model's judgment.

    • Examples of what this could look like:
    • Replace broad tools (e.g., a generic "run shell command" or "fetch any URL" tool) with narrow, single-purpose ones, i.e., a tool that only writes to one designated file, not one that can run arbitrary commands.
    • Give the model's DB connection a read-only role on only the tables it needs, not full read/write access to the whole schema.
    • Add a deterministic policy check in front of every tool call that re-validates the action's arguments and authorization at execution time - don't let the model's decision to call the tool be the only thing standing between it and the action.

  2. Server-Side, Chunk/Tenant-Level Access Control for Retrieval (RAG/Vector)

    • Why:

    This is actually complementary to point 1. Not only must the LLM have a minimized scope, but the retriever in your system must also be bounded. This practice can help prevent sensitive information disclosure.

    • Examples of what this could look like:
    • In a multi-tenant RAG app, filter by tenant ID as part of the vector-database query itself (a metadata filter or WHERE tenant_id = X clause passed into the search call), not by retrieving results for everyone and discarding the wrong-tenant ones afterward.
    • Apply access control at the chunk level, not just the document level - a mostly-public policy document with one confidential paragraph should have that paragraph excluded from the index an unauthorized user can query, rather than relying on whole-document permissions that assume uniform sensitivity.

  3. Human-in-the-Loop Approval for High-Impact/Irreversible Actions

    • Why:

    Human approval can guard against irreversible damage caused by hallucinations from the LLM or in cases where prompt injection manages to pass malicious instructions to the LLM.

    • Examples of what this could look like:
    • An agent that can send emails or post to social media on a user's behalf drafts the message and shows it to the user, requiring an explicit "send" confirmation rather than posting automatically.
    • A coding/ops agent that can run destructive commands (deleting files, dropping a database table) pauses and asks for developer confirmation before executing anything irreversible, instead of auto-running it.

  4. Sandboxing & Isolation of Untrusted Data/Tools/Execution

    • Why:

    This practice is very important, especially in cases where code execution is possible. It limits the damage caused by hallucinations (non-existent packages/bad code) and the damage that can be done by prompt injection. Even if something bad gets generated or injected, it runs in an isolated environment with limited access to secrets, the filesystem, or the network, rather than directly on your production system.

    • Examples of what this could look like:
    • Restrict generated code to a fixed, pre-approved list of allowed packages/functions (validated via an AST check before execution, not just a runtime try/catch), so even if the LLM hallucinates or suggests a non-existent package, it gets rejected before anything runs.
    • Isolate LLM-Generated Code Execution: Run the generated code in an isolated subprocess with execution timeouts. For production-grade isolation, use a VM or microVM (or a user-space-kernel sandbox like gVisor) with enforced CPU and memory limits. Subprocesses and standard containers share the host's kernel, so they can be escaped or exhaust shared resources.
    • Restrict Sandbox Network Access: Allow access only to approved external APIs, and explicitly block requests to internal services, private IP addresses, and cloud metadata endpoints to prevent unauthorized access (SSRF).

Input:

  1. No Secrets or Sensitive Data in Prompts/Hidden Context

    • Why:

    Including any secrets, credentials, or API keys in the system prompts introduces two issues:

    • If you are using an LLM via an API, your sensitive information is exposed to the API provider.
    • If the system prompt leaks to the user - which can happen because of LLM hallucination or prompt injection - your sensitive data will be in the hands of the wrong people.
    • Examples of what this could look like:
    • Instead of hardcoding an API key or database credential directly into the system prompt, store it in a secrets manager (environment variables, Vault, AWS Secrets Manager) and keep it entirely out of any text the model sees.
    • If the model needs a tool that requires a credential (e.g., a database password), handle the authentication in your application code / tool-calling layer so the LLM should only ever see "call tool X with argument Y," never the actual credential value.

  2. Input Validation & Sanitization

    • Why:

    This practice is important for preventing dangers like prompt injection and unintended uses of your application, since it lets you classify the intent of a user's input and check whether it matches an intended use case for your app.

    • Examples of what this could look like:
    • A user asks a medical-assistant app for a diagnosis of their symptoms, something the app shouldn't provide, since a wrong diagnosis could expose the company to liability. An intent classifier can detect this kind of request and block or redirect it before the model responds.
    • Pair OCR with a text/intent classifier on image inputs, so if a user uploads an image containing hidden malicious instructions, the extracted text still gets scanned for injection attempts before it reaches the model, the same way a normal text input would.
    • Regex/pattern matching can be paired with an intent classifier as a cheap first-pass filter for obvious attack signatures, for example:
    • Catching known jailbreak phrases like "ignore previous instructions".
    • Suspicious long base64-looking blobs, or unexpected character encodings.
    • Flagging suspicious invisible characters, with normalization tailored to the languages your app supports.

  3. Trust-Boundary Separation & Provenance Labeling

    • Why:

    This is different from point 1: point 1 limits what the LLM is allowed to do even if it gets manipulated, while this point is about preventing the manipulation itself.
    The problem is that an LLM can't naturally tell an instruction apart from data, since both are just tokens to it, so a prompt injection hidden in a retrieved document or web page looks exactly like a real command.
    By structurally separating external content from the system prompt and labeling where that content came from, you make it harder for injected text to be mistaken for a legitimate instruction.

    • Examples of what this could look like:
    • When a RAG system retrieves documents to answer a question, wrap the retrieved content in clearly delimited tags (e.g., <retrieveddocument source="...">...</retrieveddocument>) and instruct the model to treat anything inside as data to reference, never as commands to follow.
    • Track where every piece of retrieved content came from (source, ingestion time, trust tier), so if a document later turns out to be malicious, you can trace and invalidate everything pulled from that source, instead of only knowing that something, somewhere, was bad.

Output

  1. Output Schema Validation, Encoding & Safe Rendering

    • Why:

    This practice is extremely important for many reasons:

    • If malicious instructions get past the input defenses, we can still catch the damage at the output stage by checking whether the response leaks secrets or does something outside the intended use of the app.
    • If the LLM hallucinates an incomplete, incorrect, or nonsensical output, validating it lets us catch that and retry instead of passing garbage downstream.
    • Even when nothing malicious happened upstream, treating the model's output as automatically safe to render or execute is risky on its own, as it can still contain something that looks like a script tag or a SQL fragment, and if that's passed straight into a browser, database, or shell without encoding, it's enough to trigger XSS, SQL injection, or command execution, regardless of what caused it.
    • Examples of what this could look like:
    • Validate the LLM's output against a strict schema (JSON structure, expected fields) in application code before using it, and if it doesn't match, reject and retry instead of passing a bad response downstream.
    • Before rendering LLM output in a web page, apply context-aware encoding:
    • HTML-encode for an HTML context.
    • JS-encode if it's inserted into a script block, so a stray <script> tag or quote character in the model's text can't execute as code in the browser.
    • The app uses predefined, parameterized queries, with model-provided values passed separately.

  2. Grounding / Verify-Before-Act ("Claim-Check-Act")

    • Why:

    This practice can help guard against misinformation, since it forces any claim to be checked against a real, current source before the system treats it as true or acts on it.

    • Examples of what this could look like:
    • An agent about to approve a refund based on a claim like "the customer's order was never delivered" should check the actual shipment/tracking record before acting, not act on the model's restated claim alone.
    • A support bot answering "What's our return policy?" should pull the current policy document via retrieval and ground its answer in that actual text, rather than relying on what it remembers from training, which could be outdated or simply wrong.

Ops

  1. Rate Limiting, Quotas & Spending/Resource Caps

    • Why:

    This practice is really important in case the input and output defenses against hallucinations and prompt injections fail. Having strict, reasonable budgets can limit the damage from potentially millions of dollars down to just a couple hundred.

    • Examples of what this could look like:
    • If a request triggers a loop that keeps calling the model over and over, a hard, non-overridable spending cap per API key/user can stop it in its tracks. A simple alert might not get seen until the damage is already done.
    • A single agent stuck in a recursive tool-calling loop can burn through a budget in seconds, even with a per-user spending limit already in place. A step/recursion limit on the agent's own run catches that specific failure mode.

  2. Know & Verify What You're Building On

    • Why:

    This practice can help guard against attackers who tamper with packages/models/datasets/websites. You have to check every element used in your application and verify that it's coming from a trusted resource, as everything can be manipulated.

    • Examples of what this could look like:
    • Random Hugging Face models can include hidden biases, backdoors, or other malicious features. You have to verify that the publisher of that model is trustworthy before using it in your application.
    • Public datasets on Kaggle and other websites can be modified or poisoned between versions, and a model fine-tuned on them can pick up hidden biases or backdoors without anyone noticing, so be careful when choosing your data.
    • Before trusting an external website as a data source (e.g., a RAG pipeline pulling from the web), vet it the same way you'd vet a supplier: reputation, terms of service, and track record, rather than treating every site as equally trustworthy by default.

  3. Test Your Defenses Like an Attacker Would

    • Why:

    You might need help from a friend on this one! A defense that only gets tested against attacks it already knows about will look solid on paper but fail against a real attacker who studies your system first. Testing your own app the way an attacker actually would is the only way to know if those defenses actually hold up, instead of just assuming they do.

    • Examples of what this could look like:
    • Test your prompt-injection filters against attacks that already know your rules, not just ones that don't. If they still hold up even when the attacker knows exactly how they work, that's a good sign.
    • Before using a third-party model, test it yourself with prompts related to your specific use case, not just generic benchmarks. A model can pass a popular benchmark and still fail at your actual task.
    • Regularly test a deployed model with unusual or suspicious inputs to check for hidden backdoors. A backdoor stays silent until it sees its specific trigger, so normal use won't reveal it.

  4. Monitoring, Logging, Anomaly Detection & Incident Response

    • Why:

    This practice is really important for knowing what has happened and what is happening within your application. Documenting every detail helps you figure out how to improve and how to catch issues within your app, including - but not limited to - model performance degradation. It's also what lets you catch an attack early, like an unusual spike in tool calls or resource use, and investigate exactly what happened afterward, instead of only finding out once the damage is already done.

    • Examples of what this could look like:
    • A sudden, unexplained spike in tool calls or token usage from one session can be the first sign of a recursive loop or an attack in progress. Catching that pattern in real time lets you intervene before the costs, or the damage, pile up.
    • An unusually high volume of near-identical queries against a retrieval/embedding endpoint can be a sign that someone is trying to extract your underlying data. Flagging that pattern catches an extraction attempt that a single request wouldn't reveal on its own.

  5. Protect Your Data Wherever It Goes

    • Why:

    Encrypt sensitive data wherever it's stored or sent, and use confidential computing when it also needs protection during processing. These protections work alongside access controls and input/output defenses, not instead of them.

    • Examples of what this could look like:
    • Chat histories, uploaded documents, and backups stored with encryption enabled.
    • HTTPS connections between your app, model API, and tools.
    • Encryption keys kept in a key manager rather than in code.

If you got to the end of the article, then I want to congratulate you and say thank you so much. I know that those are a lot of points to go through :) I hope you enjoyed reading them, and I hope that this list can be helpful for you. If you have any points you want to add or any comments or feedback, please let me know.
My LinkedIn: Aml Hassan Esmail | LinkedIn
My email: aml.hassan.esmil@gmail.com