An AI service may be online and meeting technical performance targets, yet still fail the customer. That's precisely the operational challenge facing organisations as they move AI agents and conversational systems from controlled pilots into production.
For traditional IT operations, an incident is often defined by a clear technical signal: an application is unavailable, latency breaches a threshold, a server fails or a network connection drops. The ticket is logged, categorised and routed to the team responsible for that service.
AI changes the picture.
A customer-facing chatbot or voice agent can remain technically available, continue answering requests and pass conventional uptime checks. But it may provide incomplete answers, misunderstand intent, fail to retrieve the right data or hand an increasing number of customers over to human agents.
In that situation, the system is not necessarily down. But the service is still failing.
Speaking with UC Today, Lokesh Vijay Kumar, AI Engineering Lead at GEICO, said organisations need to distinguish between a system being operational and it actually delivering the intended outcome.
“The system is up” and “the system is really doing the right thing” are separate questions, he said.
Technical Health Is Not Customer Success
The earliest warning signs of an AI issue may not appear in conventional infrastructure monitoring. Instead, teams may need to watch for changes in customer behaviour.
Kumar pointed to signals including rising frustration, more abandoned journeys, increased fallbacks and a higher volume of transfers from an AI service to human support agents.
For example, a conversational AI system may be responding within its normal latency range. However, if a growing proportion of customers cannot complete their task and require escalation, the organisation has a service problem even though its core availability dashboard remains green.
“That’s a very typical thing you want to capture, but it’s not so easy,” Kumar said.
This means AI observability cannot be confined to infrastructure metrics. Teams need visibility into the quality of interactions and whether a customer actually completed the task they arrived to perform.
That requires a broader operational view. Organisations need to understand the intent behind an interaction, whether the AI interpreted it correctly, what information was retrieved, which tools and downstream services were involved, and whether the final outcome met the customer’s need.
For AI systems, a technically successful response is not always a successful customer interaction.
The Root Cause May Sit Elsewhere
The visible issue will often emerge in the application itself. A customer sees a chatbot give an incomplete response, take too long to answer or fail to complete a request. As a result, the application team is likely to receive the first report and begin the initial triage.
But the application may not be where the fault originated.
Kumar described an environment in which an AI model can respond correctly, while an upstream legacy system returns incomplete data, a cloud dependency introduces latency, or another service fails to respond in time. To the customer, it is simply an AI experience that has not worked. Behind the scenes, however, that experience may depend on a complex web of models, APIs, data sources, cloud platforms and enterprise systems.
“The customer sees one AI experience, but behind that experience, there may be hundreds of different dependencies.”
This is where distributed tracing becomes central. IT teams need to be able to follow an interaction across the full chain of dependencies, rather than treating AI as an isolated application.
Without that end-to-end view, teams can lose time investigating the most obvious layer rather than the actual failing dependency. A customer-facing issue may initially look like a model-quality problem, for instance, when the underlying cause is a timeout, an incomplete data return or an unavailable downstream service.
The network, cloud, application, data and security teams may all hold part of the evidence. But none of them will have the full picture if telemetry remains fragmented.
Tickets Need More Context
This also exposes a limitation in conventional IT service-management processes. A traditional incident ticket might state that an “AI response failed” or that a customer received a poor answer. That may be enough to open a case, but it is not enough to diagnose a distributed AI workflow.
Kumar said the objective should be to prevent multiple teams from having to reconstruct the same incident independently.
“Don’t make five teams rediscover the same incident at different times,” he said.
The ticket needs to carry meaningful context from the moment of failure. That includes the specific interaction affected, the workflow step involved, the systems and downstream services called, latency at each stage, any fallback that occurred, and whether the customer completed the intended task.
That changes ownership, too. The application team may appropriately take the first ticket because it owns the visible customer experience. But it should not necessarily own the incident through to resolution if the underlying cause lies with a cloud dependency, a data platform, a legacy system or network connectivity.
The operational model needs clear accountability for coordinating the incident, while allowing the right specialist teams to investigate the relevant layer. Otherwise, ownership becomes a hand-off exercise rather than a route to resolution.
Observability Must Be Designed In
For organisations deploying customer-facing AI, observability cannot be an afterthought added after launch.
Kumar said monitoring and guardrails should be considered during the proof-of-concept and design stages, alongside familiar non-functional requirements such as availability, latency, errors and dependency resilience.
But AI also introduces additional operational requirements. Teams need to assess whether responses are grounded in appropriate information, whether the system is correctly understanding customer intent, when it fails over to another route, and whether users can complete the journeys that matter.
Fallback strategies, sensitive-action controls, traffic reduction and the ability to roll back behaviour changes all need to be available before an AI system reaches production.
“If you can’t observe it and you can’t evaluate it safely, then you better stop it right there – it’s probably not ready for production.”
As enterprises scale AI beyond pilots, the incident-management question is becoming more urgent. AI does not remove the need for established ITSM discipline. It ultimately expands it – requiring richer signals, cross-team visibility and a definition of service health that begins with the customer outcome, not simply whether the system answered.
