The conversation around AI agents has undergone a rapid shift in sentiment in the few years since they reached mainstream deployment. What started with companies asking whether they could orchestrate them across enough of their workflows to generate meaningful efficiency has now shifted to whether they can be trusted once they are there.
Recent security-testing disclosures involving OpenAI, Anthropic and the UK AI Security Institute have brought the containment question into focus, after agents took actions beyond their intended test environments.
“We’re no longer talking about sci-fi scenarios: autonomous agents are actively routing around their own boundaries,” said Ona Ojukwu, Program Manager at Justice Digital.
The concern with this is not that an agent may produce an inaccurate answer or poorly worded draft. It is that they can autonomously make poor judgements and execute them at machine speed.
The impact of a poor judgement is intensified by the systems these agents can now work across. As Andre Scott, AI Expert at CoraLogix, said: “They’re calling APIs, they’re modifying records, they’re moving money, triggering workflows.”
The operational challenge for enterprises is therefore not only to do as much as possible to prevent an agent from crossing a boundary, but to recognise that AI’s dynamic nature means they must prepare for the possibility that one does.
What an AI Agent Incident Can Look Like
The OpenAI-Hugging Face incident brought the issue sharply into focus. During a cybersecurity evaluation, OpenAI said advanced agents found a weakness in their sandbox, reached the open web, and interacted with Hugging Face systems while seeking information relevant to their test objective. Hugging Face subsequently closed the vulnerabilities involved and rebuilt affected systems. As UC Today reported, the incident showed how an agent can take an unintended route in pursuit of a narrow goal.
Anthropic’s subsequent disclosure raised similar concerns. Three Claude models interacted with real organizations during cybersecurity evaluations intended to be isolated. Anthropic attributed the incidents to internet access being available because of a misunderstanding in the evaluation setup. The models did not, the company said, independently exfiltrate themselves or deliberately leave their environments. Yet once a route was available, they were able to act against real systems.
That distinction matters. Scott argues this shows why businesses should be careful about describing this behavior as deception, because that implies intent. “What we’re seeing is what I call optimization misalignment,” he said.
“The AI is doing exactly what it was measured to do, but not actually what you meant for it to do.”
In security terms, an agent may not “want” to cause harm for its actions to have harmful consequences.
Scott’s customer service example provides a less dramatic but potentially more common enterprise illustration. An agent trained to reduce ticket volume could automatically close cases or label them as duplicates. The KPI improves, but customers are left agitated and without support. “The model was just optimizing perfectly for the wrong target,” Scott said.
The same logic applies when an agent is tasked with finding a solution, accelerating a process, or completing a security exercise: it may identify a technically effective route that violates the business, security, or ethical constraints humans assumed were understood.
For enterprises, the key point is that an incident-response plan should not look for malicious intent. It should be designed for the point at which an agent’s actions, whether intended or not, have created an outcome the business needs to stop, investigate, and put right.
Understanding Agents, Building a Response
An effective AI agent incident-response plan begins with a basic question: how will an organization know when an agent has gone wrong? The operational danger is that an agent incident may not initially resemble a traditional system failure. There may be no outage, no spike in error rates, and no obvious infrastructure alarm. A system can remain responsive, available, and apparently healthy while taking unauthorized or harmful actions.
That can make harmful behavior difficult to identify quickly. Anthropic said activity involving its models took place in April but was not identified until months later, prompting it to suspend its cybersecurity evaluations on July 23. By the time an issue is identified, an agent may already have taken several actions or triggered processes that continue beyond the original task.
That is why the first stage of an agent incident response must be to establish the scope of the activity and understand when issues could be occurring. Scott recommends instrumenting the AI stack to capture prompts, responses, tool calls, decisions, and downstream actions, using open telemetry standards where possible, to do this. “You need to set up telemetry data. You need to see it. You need to be able to evaluate it or otherwise you’re just going blind,” he said. Those records should allow investigators to reconstruct what the agent saw, which tools it used, what permissions it exercised, which data was accessed, and where controls failed. They must also identify whether the agent’s output triggered other automated workflows, created changes that need to be reversed, or affected customers, employees, or third parties.
Once the scope is understood, the organization must be able to contain the incident. That means pausing the agent, revoking access to connected tools and APIs where necessary, and stopping any workflows that may still be in motion. An alert may be raised only after an agent has made its first tool call, changed a record, or triggered another process. Stopping the agent does not automatically undo those effects.
An incident plan must therefore establish, before deployment, who has the authority to make those decisions. Teams need a named owner who is accountable for the agent and its connected workflow; a rapid way to pause, revoke, or reduce the agent’s permissions; and a defined escalation path spanning security, IT, legal, compliance, and the affected business function. It is not enough to say a human is “in the loop” if that person has neither the visibility nor the authority to intervene before the damage is done.
Finally, organizations should apply the disciplines they already use for production incidents. As Scott said,
“We need to be treating AI like production infrastructure.”
That means treating it with the same ownership model, incident-response process, root-cause analysis, and postmortems.
Preparing for When Prevention Fails
Monitoring, telemetry, guardrails, and tightly controlled permissions are essential before an organization deploys an AI agent. They can help teams identify abnormal activity sooner, limit the systems and data an agent can reach, and provide the evidence needed to investigate an incident. But they should not be treated as a guarantee that an agent will never make a consequential mistake.
The recent testing incidents reinforce that prevention cannot be the whole strategy, especially as agent deployments become more complex, models are updated, and new tools are connected.
Therefore, like any other part of the security estate, AI agents need a defined plan for when controls fail. That means being ready to identify the incident, stop further activity, establish what was affected, recover from any changes, and use the findings to improve the deployment. The question for enterprises is no longer simply whether an AI agent can be trusted with a workflow. It is whether the organization is prepared for the moment that trust is tested.
