AWS details a two-layer monitoring setup for production multi-agent systems
AWS published an architecture that scores live agent quality separately from infrastructure health, demonstrated on a four-agent airline booking system.
AWS has published a technical write-up arguing that multi-agent systems in production fail in ways traditional monitoring misses. In one example it gives, an agent cannot invoke its foundation model and returns an empty response because of a missing IAM permission on its execution role, without throwing a 500 error. In another, a supervisor agent with a poorly scoped prompt routes 20 percent of requests to the wrong specialist while error rates stay flat and infrastructure metrics stay green.
Two questions, two layers
The architecture answers two separate questions: is the agent working well, and is the infrastructure healthy? Amazon Bedrock AgentCore Evaluations scores live interactions on helpfulness, correctness and goal completion. It samples a configurable percentage of production requests and evaluates them in the background, attaching reasoning to each score based on conversation context, tools used and task requirements. When quality drops, it runs pattern analysis across recent low-scoring sessions and produces concrete recommendations: prompt changes, tool selection adjustments or orchestration logic fixes.
The second layer, AWS DevOps Agent, watches metrics, logs and error patterns. When something breaks it pulls the relevant Amazon CloudWatch logs, builds a topology graph of affected resources, correlates errors across IAM, Amazon Bedrock and the agent runtime, traces the failure path and returns fix recommendations. AWS describes this as connecting a blank agent response to a missing IAM permission, or a spike in timeouts to Bedrock throttling in a specific Region.
The airline demo
To show the setup end to end, AWS built an airline reservation system with four specialized agents using the Swarm pattern. A sample request covers two routes in a single conversational turn, pulls loyalty status and a companion certificate from a separate data source, and sequences bookings, since the certificate cannot be applied before the flight is selected and the fare class is known.
Full details are in the AWS Machine Learning blog post.
