There is a comfortable lie going around in the sale of AI projects: the claim that the system is secure because "it has guardrails." Guardrail is the nice name for the rules that try to keep the AI from saying or doing what it should not. It is necessary, but treating it as definitive security is like saying your store is protected because it has a "no trespassing" sign on the door. The sign helps. It does not stop anyone who has decided to walk in.
This text explains, in business language, what real AI security is: what the three largest makers in the world (Anthropic with Claude, Google with Gemini, and OpenAI with Codex) do to protect their own products, why none of them relies on guardrails alone, and the list of measures that separates a genuinely protected project from an ordinary one that only followed the standard checklist. If your company uses or is going to use AI connected to real systems, with real customer data, this is probably the most important text you will read before signing any contract.
1. What can go wrong, in business terms
First, it is worth naming the risk without jargon. A chatbot that only answers questions can, in the worst case, say something silly. Embarrassing, but recoverable. An AI agent is different: it acts. It accesses the order system, looks up customer records, issues a duplicate invoice, schedules a visit, sends an email. It does things on behalf of your company, with the permissions your company gave it.
That changes the nature of the risk. The questions stop being "what if the AI answers wrong?" and become: what if someone convinces the AI to show one customer's data to another? What if the AI grants a discount that does not exist and the customer demands it be honored? What if it deletes or alters a record it should not have? What if a competitor figures out how to extract your internal price list by chatting with your own virtual agent? None of these scenarios is fiction. All of them have already happened in real companies, and most happened in projects that had guardrails.
2. Why the guardrail does not hold on its own
The guardrail works like a new employee who is very well trained: he received clear instructions about what he can and cannot do. The problem is that he is still someone who talks to the public all day long. And there is a whole category of scam, called prompt injection, that consists precisely of talking to the AI until it is convinced that the rule does not apply in that case. "Ignore your previous instructions." "I am the system administrator." "This is an authorized test." It sounds silly written out like this, but the sophisticated versions of this scam fool the best models in the world on a regular basis, including when hidden inside an email, a document, or a page the AI read in order to help you.
The detail almost no one tells you
A traditional system is deterministic: the same input always produces the same output, and a system lock blocks it 100% of the time. A generative AI is probabilistic: it gets it right almost always, but "almost always" is a dangerous phrase in security. If the guardrail holds 99% of manipulation attempts and your agent handles ten thousand conversations a month, that leaves a hundred conversations a month where it can slip. An attacker does not need to win every time. He needs to win once. Your company needs to defend every single time. That asymmetry is the reason a guardrail, on its own, will never be definitive security: it is a layer of risk reduction, not a wall.
3. What Claude, Gemini, and Codex do, and the lesson it teaches
The strongest argument against relying on guardrails alone does not come from a security consultant. It comes from the model makers themselves. The three largest in the world surround their products with controls external to the model, because they know better than anyone that the model can be fooled.
Claude (Anthropic)
Anthropic trains Claude with a proprietary method that embeds principles of behavior inside the model, and it maintains a public safety escalation policy: the more capable the model, the more controls it requires before being released. But the most revealing point is in the product. Claude Code, Anthropic's own tool that uses AI to work on systems, asks for human confirmation before executing sensitive actions and runs commands inside an isolated environment (a kind of closed room where, if something goes wrong, the damage does not spread). In other words: the company that makes the model, which knows every detail of it, does not let its own model act alone without an external fence.
Gemini (Google)
Google publishes an AI security framework (called SAIF) whose central idea is defense in layers: filters before the conversation reaches the model, classifiers that watch the conversation looking for manipulation attempts, limits on what the model can access, and review of the responses before they go out. Google openly states that prompt injection has no known definitive solution, and that this is why the strategy is to stack barriers: the scam that gets past the first layer has to get past the second, the third, and the fourth.
Codex (OpenAI)
Codex, OpenAI's coding agent, is perhaps the most instructive example. By default, it works in a closed box: no internet access, seeing nothing beyond the folder it is working in, and asking for human approval for anything outside of that. OpenAI could have said "our guardrail is great, go ahead and open everything up." It did not. It chose to assume the agent can be fooled and to design the product so that, even when fooled, the possible damage is small.
The lesson from all three is the same and fits in one sentence: the company that makes the model does not rely on the model's behavior alone. It relies on structure around it. If the maker does this with its own product, the project your company hired needs to do the same with yours.
4. The measures that separate a protected project from an ordinary one
The ordinary project installs the model, writes the behavior rules, tests half a dozen conversations, and delivers. The protected project assumes the AI will make mistakes and will be attacked, and builds for that scenario. In practice, the difference comes down to eight measures.
Least privilege. The AI gets the intern's key, not the master key. If the agent only needs to look up orders, it cannot have access that allows it to alter or delete. Test question: "if this AI is completely fooled today, what is the worst it can do with the permissions it has?" If the answer is scary, the permissions are wrong.
Separation between who gives orders and who converses. Everything that comes from outside (customer message, email, attached document, web page) must be treated as content to be read, never as an order to be obeyed. It is the difference between the agent reading a letter from the customer and the agent obeying the customer's letter. Well-built projects mark that boundary technically; ordinary projects mix everything in the same bucket.
Human confirmation on anything irreversible. A reversible action (look up, calculate, draft) can be automatic. An irreversible or costly action (send, pay, delete, sign, grant a discount) goes through a human or a system lock. It is not a lack of trust in the AI, it is the same approval-authority principle your company already uses with people.
Isolated environment. The AI works in a closed room, with access only to what it needs, and what it produces passes through a controlled door before it touches the real system. If something goes wrong, the problem stays contained in the room.
A record of everything. Every conversation, every decision, every action the AI takes is recorded, like a security camera. It is not bureaucracy: when something strange happens (and it will), the difference between resolving it in two hours and resolving it in two weeks is having the record of what the AI saw, decided, and did.
Volume limits and an off switch. The AI has a cap on actions per hour, a cap on spending, a cap on messages. And there is a button, accessible to a human on duty, that shuts the function off instantly, without depending on a programmer, without waiting for a vendor. An AI incident happens in minutes; the response cannot take days.
A rehearsed attack. Before it goes live, someone is paid to try to break the system: extract data it should not, pull an nonexistent discount, make the AI disregard its own rules. In the security world this is called red team, and it is the equivalent of hiring a former burglar to test your lock before the real thief shows up. An ordinary project tests whether the AI works. A protected project tests whether it holds up.
Continuous monitoring. Security is not a phase of the project, it is a routine of the operation. Someone looks at the records, tracks strange patterns (the AI suddenly accessing ten times more records than normal is an alarm, not a curiosity) and repeats the attack tests periodically, because the scams evolve every month.
5. The right question is not "how to prevent the mistake"
Ordinary projects ask: how do we make the AI never make a mistake? That question has no answer, and anyone who promises it does is selling a "no trespassing" sign as if it were a vault. Protected projects ask something else: when the AI makes a mistake or gets fooled, what is the maximum size of the damage, how fast do we find out, and how fast do we shut it down? If your vendor answers that question with concrete numbers and mechanisms, you are looking at a serious project. If it answers "that will not happen because we have guardrails," you are looking at the problem.
The guardrail is the seatbelt: indispensable, and insufficient. A safe car has a seatbelt, brakes, an airbag, a speed limit, and a trained driver. Your company's AI deserves the same standard. The checklist below sums up the questions that are worth an entire meeting with whoever is building or selling your AI project.
