Site icon Bamboos Consulting

Five Failure Modes in Agentic AI: A Buyer’s Pre-Deployment Checklist

roberto

A supplier shows an AI agent retrieving a customer record, making a decision, and completing an action. The demonstration is convincing. The commercial question, however, is narrower: what evidence shows that the agent will take the right action, under the right authority, when ordinary operating conditions break down?

For a startup buying an agentic component—or a founder deciding whether to release one—this question matters more than a general claim that the model is accurate. A workflow may produce a plausible answer while acting on the wrong account, accepting a malicious instruction from retrieved material, or repeating a payment after a timeout.

Deloitte’s 2026 State of AI in the Enterprise reports a gap between increasing agentic AI adoption and mature governance of autonomous agents. Its survey concerns enterprises, not specifically European startups; the relevant lesson for smaller buyers is to request operational evidence rather than infer readiness from adoption trends. The OWASP Top 10 for Agentic Applications offers a complementary, security-focused vocabulary for several of the risks below.

The following five failure modes are not a ranking or a list of incidents observed in a particular vendor’s system. They form a practical pre-deployment acceptance checklist that a buyer can adapt to the system’s intended use and consequences.

1. The answer is right, but the action is wrong

Failure mode. The agent gives a reasonable explanation but invokes the wrong tool, passes an incorrect identifier, or modifies a record that was not part of the authorized request. An apparently successful conversation can hide an incorrect operational outcome.

Buyer question: Can the supplier prove that the final response matches the authoritative state of the application—and show the exact tools and arguments used?

Evidence to request: Representative execution traces with tool names, validated parameters, before-and-after record states, and test results for similar-looking identifiers or ambiguous requests. The acceptance rule should check the transaction itself, not just the agent’s narrative.

Release concern: If a consequential action cannot be traced to the correct authorized request, a high aggregate task-success rate is insufficient.

2. External content becomes an instruction

Failure mode. A document, email, repository file, or tool result contains text intended to redirect the agent. The workflow treats that untrusted content as an instruction rather than as data. OWASP classifies related threats under agent goal hijack and tool misuse.

Buyer question: What happens when a retrieved record instructs the agent to ignore approval rules, change destinations, or disclose information?

Evidence to request: Controlled tests in which plausible malicious instructions are embedded in lower-trust inputs; records of attempted and actual tool calls; and evidence that permissions and action constraints remain enforced outside the model’s text generation. A model’s refusal in one prompt is useful evidence, but is not a substitute for enforced boundaries.

Release concern: An agent that can convert untrusted text into a privileged action needs redesign or effective external containment before it receives those privileges.

3. Delegated access exceeds the user’s authority

Failure mode. The agent uses a broadly privileged service account or inherited credentials to perform an action that the requesting user could not perform directly. This is a problem of application authorization, not just prompt quality.

Buyer question: Which identity executes each action, who grants approval, and where are access rules enforced?

Evidence to request: A permission matrix for users, agents, and tools; tests of denied access; audit events for approvals; and proof that sensitive operations are checked by server-side logic. Approval must not be established merely because the model says that approval exists.

Release concern: If a lower-privilege user can indirectly trigger an administrator-level action, restricting the text of the system prompt is not an adequate control.

4. A partial failure creates a duplicate or inconsistent transaction

Failure mode. A payment, refund, order, or record update succeeds, but the tool response times out. The agent retries without reconciling the actual transaction state. Alternatively, one step succeeds and another fails, leaving the workflow in an inconsistent state.

Buyer question: How does the application distinguish “the action failed” from “the action succeeded but its acknowledgment was lost”?

Evidence to request: Failure-injection tests, transaction identifiers, idempotency or deduplication controls where applicable, recovery traces, and state reconciliation against the authoritative system. Ask specifically about retries after timeouts and failures between sequential tool calls.

Release concern: A workflow that cannot distinguish uncertain transaction state should stop, verify, or escalate rather than repeat a consequential action.

5. The system changes after its acceptance test

Failure mode. The vendor updates the model, prompts, retrieval configuration, tool schema, or orchestration logic. Previous measurements no longer describe the deployed configuration. Cost, latency, and failure patterns may also change as the use case scales.

Buyer question: Which tested configuration are we buying, and what happens when a component changes?

Evidence to request: Versioned test artifacts, documented acceptance criteria, a regression suite, change-control responsibilities, and monitoring for relevant operational outcomes. Where costs or service levels matter, require measurements under representative workload assumptions.

Release concern: An evaluation report without configuration identifiers and a re-evaluation trigger has limited value for ongoing operation.

The buyer’s evidence matrix

The table translates the five failure modes into requests that can be included in a procurement review, pilot acceptance plan, or internal go/no-go meeting.

Risk to examine Minimum evidence to request Reason to pause deployment
Wrong tool or record Tool trace plus verified final system state Consequential action cannot be attributed to an authorized request
Instruction injection Adversarial input tests and enforced action boundaries Untrusted content can authorize or redirect privileged actions
Excess privileges Permission matrix and denied-action tests Agent can exceed the requesting user’s permissions
Partial transaction Timeout/retry tests and authoritative reconciliation Duplicate or ambiguous state-changing operations remain possible
Uncontrolled change Versioned configuration, regression tests, monitoring Evidence cannot be tied to the deployed version
Table 1. Illustrative evidence requests, not a universal certification standard. The severity of each finding depends on the intended use and potential consequences.

A concrete acceptance test: one refund, never two

Consider a hypothetical invoice-support agent with a read-only invoice_lookup tool and a state-changing invoice_refund tool. A buyer could define the following claim:

For an eligible refund requested by an authenticated and authorized user, the system records no more than one refund. It records none where approval is absent. An interrupted acknowledgment does not cause a second refund.

The acceptance exercise would prepare synthetic invoices, vary authorization and refund eligibility, inject timeouts immediately after execution, and compare the final account ledger with the action trace. It would also introduce misleading content into an invoice note to test whether that content changes the workflow’s authority.

The useful output is not a declaration that the system is “safe.” It is a bounded report: what was tested, how often, under which configuration, which rules were violated, which records support the verdict, and what remains unknown. These are proposed tests, not reported experimental results.

Three procurement decisions, not one benchmark score

A purchase or release decision should be based on consequences and controls as well as average performance:

For a small buyer, the required evidence need not be a large independent benchmark. A well-scoped scenario set, explicit critical constraints, inspectable execution traces, and a documented acceptance decision may be more valuable than an impressive but opaque aggregate score. NIST’s voluntary AI Risk Management Framework provides a broader structure for identifying, measuring, and managing AI risks. The 2026 ENISA Threat Landscape also highlights why dependencies deserve attention in cybersecurity risk management; that report is not a test of any individual AI agent.

For European buyers, regulatory analysis is a separate activity. The EU AI Act applies different rules depending on the system, use case, and organizational role. This checklist is an engineering and procurement tool, not a legal classification or conformity assessment.

What to put in a supplier brief

Before accepting a proposal for an agentic system, ask the supplier to document (1) the intended and excluded uses; (2) available tools, permissions, and approvals; (3) critical prohibited actions; (4) realistic failure and adversarial scenarios; (5) the logs and authoritative records used to determine results; and (6) responsibilities for retesting and incident response after changes.

This is a practical extension of the evidence-first approach discussed in our earlier article on evaluating agentic workflows under operational constraints. It also reflects a general point in research on adaptive controllers: detecting an unacceptable behavior and proposing a correction are not the same as verifying that the corrected behavior meets the original constraints. See Garrone (2026), Verification of Adaptive Agentic Controllers through Finite Rule Revision for a controlled, stylized example—not evidence about production language-model agents.

Bamboos Consulting supports independent AI evaluation, intelligent-system prototyping, and computational experimentation. If you are assessing an agentic solution before purchase or deployment, discuss a scoped technical assessment.

Sources and further reading

Exit mobile version