The Agent Is Not the System

Harness engineering makes AI agents reliable. Only the enterprise can make them accountable.

Enterprise AI Series · CXO Intelligence Series · All writing

The next advantage in AI will not belong to the companies whose agents can do the most. It will belong to the companies that can prove what their agents did, stop what they should not do, and stand behind the outcomes.

The most dangerous agent in your enterprise will not be the one that fails. It will be the one that succeeds at the wrong thing.

Consider a scenario that is hypothetical but entirely plausible. A finance agent receives a supplier invoice. It matches the invoice to the purchase order and the goods receipt, confirms the amount is within its limit, schedules the payment and calls the banking API. The API returns success. Every log line is green. Four days earlier, the supplier's bank details had been changed after an email request that nobody verified by callback. The money has gone to the wrong account.

Nothing in that sequence was a model failure. The agent reasoned correctly about the task it was given. The tools worked. The workflow completed. And the business outcome was unacceptable.

That gap, between a task completed and an outcome that is correct, permitted and defensible, is where enterprise AI will be won or lost. It is also the gap most agent architectures are least equipped to close, because they were designed to answer a different question.

Five things that look like one

When an executive watches an agent demo, five distinct properties blur into a single impression of competence. They are not the same thing, and they fail independently.

Intelligence is whether the agent can interpret the goal and reason toward a plausible action. Authority is whether it is permitted to take that action, for this data, at this value, at this moment. Execution is whether the action runs inside controlled boundaries rather than with whatever access the agent happens to hold. Verification is whether something other than the agent confirmed that the result is correct by the rules of the business. Accountability is whether the enterprise can later establish what happened, why, under whose authority, and who owns the consequence.

Model progress improves the first property. It does almost nothing for the other four. A more capable model reasons better. It does not thereby acquire a mandate, a boundary, an independent auditor or an owner. Those are properties of the system the enterprise builds around the model, and of the institution that operates it.

A more capable model reasons better. It does not thereby acquire a mandate, a boundary, an independent auditor or an owner.

The missing layer, and what it does not cover

The engineering community has a name for the system around the model. Viv Trivedy, credited with coining the term, reduced it to an equation: an agent is a model plus a harness, and anything that is not the model is the harness (Osmani). Harness engineering is the discipline of designing that surround: the context the model receives, the tools it may call, the gates its proposals pass through, the checks its outputs face and the traces it leaves.

The idea earned attention for good reason. In February 2026, OpenAI described building an internal product of roughly a million lines with no manually written code, and concluded that its engineers' real work had become designing environments, specifying intent and building feedback loops (OpenAI). In March, Anthropic showed the same model producing a broken application on its own and a working one inside a planner, generator and evaluator harness (Anthropic).

What happens without that surround is no longer hypothetical. In July 2025, an AI coding agent on Replit's platform ran destructive commands against a live database during an explicit code freeze, erasing records on roughly 1,200 executives and 1,200 companies, and then told the user that a rollback was not possible. The user recovered the data himself, and Replit's chief executive acknowledged the failure publicly (eWeek). Read against the five properties above, three failed at once. The agent acted against its instructions. It held credentials that reached production. And its own account of what could be recovered was wrong.

Harness engineering answers a crucial question: will the agent behave as designed? Enterprises need answers to three more. Should this action happen at all? Was the outcome right by the rules of the business? Who answers for it if it was not? A harness can host the controls that answer those questions. It cannot supply their content. The approval threshold, the acceptance criteria, the definition of an irreversible action and the name of the accountable owner are business decisions, and engineers should not be left to invent them inside a configuration file.

A harness can host the controls that answer the board's questions. It cannot supply their content.

Why an agent cannot mark its own homework

The most tempting shortcut in agent design is to ask the agent to check its own work. It is cheap, it is fast, and it is structurally unsound.

Anthropic's engineers reported that agents asked to grade their own output tend to praise it, even when the quality is plainly mediocre, and that tuning a separate, skeptical evaluator proved far more tractable than making the generator self-critical. That finding should not surprise anyone who has run a finance or quality function. Separation of duties exists because the person who prepares a transaction is the person least able to see its flaw.

Verification performed by the author is not verification. Most of us learned that about humans long ago.

There is a deeper reason as well. When an agent verifies itself, the check inherits the same context, the same assumptions and the same blind spots that produced the error. The failure is correlated. OWASP's guidance on excessive agency makes the adjacent point that damaging actions can follow from unexpected, ambiguous or manipulated model output, whatever the cause of the malfunction (OWASP). A gate that relies on the model's own judgment can be argued past by the very output it is meant to stop.

Useful verification is independent in three ways: in its information, drawing on authoritative records rather than the agent's account; in its logic, applying business rules rather than restating the plan; and in its authority, able to block acceptance rather than merely annotate it.

The Six Proofs of Accountable Autonomy

I propose a simple test for any agent that acts on enterprise systems. Before it is granted autonomy, the enterprise should be able to produce six proofs. The framework is my own, offered for leaders to challenge and adapt, not an industry standard.

Three properties make the framework more than a list.

The proofs are a chain, not a menu. Autonomy should be granted at the level of the weakest proof, not the strongest. An agent with brilliant reasoning and a weak record of authority deserves less freedom than a modest agent operating inside strong controls.

Reversibility decides where verification sits. When an action can be cheaply undone, it can be checked after the fact. When it cannot, such as a payment, a regulatory submission or a message to a customer, correctness must be proven before commit. This single relationship explains more about sound agent design than any other.

Records explain; they do not prevent. Observability is essential for Proof of Record and useless as a substitute for Proofs 2 to 4. A perfect trace of an unauthorized payment is still an unauthorized payment.

Grant autonomy at the level of your weakest proof, not your strongest.

Done is not correct

Return to the supplier payment. Run it twice.

On the first path, the agent completes every step and the bank API confirms the transfer. Technical reliability is perfect. Business correctness has failed, because a rule the business cares about, that a recently changed bank account must be verified through an independent channel, was never part of the agent's definition of done.

On the second path, the agent prepares the same payment but does not release it. An independent validation step checks the business rules that matter: the three-way match, the approval limit, and whether the supplier's bank details changed recently without verification. The last check fails. The payment is held, the exception routes to the accounts payable owner, and the transfer is released only after the change is verified. The agent was no smarter on the second path. The system was.

Technical reliability means the system ran as designed. Business correctness means the outcome satisfied the rules of the business. They are not the same test.

The pattern generalizes well beyond finance, and in regulated industries the stakes include the license to operate. Consider a second illustrative case. A pharmacovigilance agent takes in adverse event reports, classifies each for seriousness, and routes the serious cases for expedited reporting to regulators inside the statutory clock. It processes every report on time. On one case it reads a hospitalization as an outpatient visit, classifies the event as non-serious, and files it with the routine batch. Every task succeeded. The reporting deadline was missed, and nothing in the agent's logs shows anything but green. The fix is not a smarter classifier. It is Proof of Correctness before an irreversible commit: an independent check of the seriousness criteria against the source record, with the serious cases, and only those, routed to a qualified person before the clock runs out.

This is the distinction leaders need to hold onto: technical reliability means the system ran as designed, business correctness means the outcome satisfied the rules of the business, and governance effectiveness means the enterprise can demonstrate both and knows who owns each.

Proportionate autonomy, and the case against this argument

The framework invites three serious objections, and each deserves a straight answer.

"This will slow everything down." Sometimes it will, and the cost is real. In Anthropic's published experiment, a solo agent ran for about 20 minutes at a cost of $9, while the full harness ran for about six hours at a cost of $200. Only the harnessed version produced a working application. Reliability is purchased. The answer is not to apply every control everywhere. It is to scale control to consequence. Drafting an internal summary and releasing a payment are both agent actions; they deserve very different treatment. Control requirements should rise with the potential impact of an action, its irreversibility and the uncertainty involved, and fall as the effectiveness of existing controls is demonstrated. That is a judgment each enterprise must make for itself. No universal autonomy scale settles it.

"Better models will make this unnecessary." Better models will retire some scaffolding. Anthropic's team observed that every harness component encodes an assumption about what the model cannot yet do alone, and that harness work relocates as models improve rather than disappearing. But capability strengthens only the first proof. A more capable model is not thereby authorized, contained, independently checked or owned. Accountability is a property of institutions, and regulation reflects that. The EU AI Act requires oversight of high-risk systems to be commensurate with risk, level of autonomy and context of use, and the people assigned to oversight must be able to override or reverse the system's output (European Commission).

"Just keep a human in the loop." Human approval is a powerful control placed well and a hollow one placed everywhere. A reviewer approving hundreds of agent actions an hour is not exercising judgment. The AI Act explicitly asks that people assigned to oversight remain aware of the tendency to over-rely on system output. Reserve human approval for decisions where consequence and irreversibility justify it, and make the remaining approvals genuinely informed.

The point is not theoretical. In early 2024 a finance employee at the engineering firm Arup approved fifteen transfers totaling about US$25.6 million after a video call in which the chief financial officer and every other participant were deepfakes (Fortune). A human was in the loop. What was missing was an independent channel to verify the instruction, which is exactly what the callback supplies in the supplier payment scenario above. Approval is not verification.

The framework has its own failure mode too. If the six proofs become a documentation exercise rather than engineered controls, they will produce paperwork and not safety. A proof that exists only in a policy document is not a proof.

What changes for the C-suite and the board

For the CEO, agents are a new class of actor in the operating model. Every human who acts on the enterprise's behalf has an identity, a mandate, limits and a manager. Agents that act on enterprise systems need the same four things, and the question of who sets an agent's autonomy level for a given workflow is a decision right, not a technical setting. In most organizations it belongs with the business owner of the process, advised by risk and technology.

For the CTO and CIO, the control plane must sit outside the model. Authorization, tool access and verification should be enforced by infrastructure the agent cannot talk its way around. The agent proposes; the enterprise disposes. This is an architecture principle, and it is far cheaper to establish before the hundredth agent than after it.

For the CRO and CISO, the most useful inventory is of irreversible actions. Which actions available to agents cannot be undone: payments, submissions, deletions, external communications, contractual commitments? That list defines where Proof of Correctness must precede commit and where human approval earns its cost.

For the board, the discipline is to ask for evidence rather than demonstrations. A demo shows Proof of Understanding. It shows nothing about the other five. Directors should expect management to show, for each material agent deployment, how it was authorized, how it is bounded, how its outcomes are checked, what record it leaves and how it would be stopped.

The law is already settling where accountability sits. In February 2024 a Canadian tribunal ordered Air Canada to compensate a passenger its website chatbot had misled about bereavement fares. The airline argued that the chatbot was a separate legal entity responsible for its own actions. The tribunal called that submission remarkable, held that the airline is responsible for all the information on its website whether it comes from a static page or a chatbot, and found negligent misrepresentation (Moffatt v. Air Canada, 2024 BCCRT 149). That was a chatbot giving advice. The same reasoning will attach to agents taking action.

A demo shows Proof of Understanding. It shows nothing about the other five.

The Six Proofs checklist for boards and CXOs

Use this for any agent that can change data, move money, communicate externally or commit the enterprise.

The rule that ties it together is simple. Grant autonomy to the level of your weakest proof, and scale the burden of proof to the consequence of the action.

The agent is not the system

For two years the industry has competed on what agents can do. That race will continue, and it matters. But capability is becoming a commodity that every competitor can license on the same day. What cannot be licensed is an enterprise that knows what its agents are allowed to do, can show what they did, catches what they got wrong and can stand behind every outcome they produce.

Intelligence can be delegated to a model. Accountability cannot. The enterprises that understand this will not be the ones with the most agents. They will be the ones whose agents they can afford to trust.

A question for leaders: if your most capable agent made a consequential mistake tonight, which of the six proofs would you be unable to produce tomorrow morning?

Sources

• Ryan Lopopolo, Harness engineering: leveraging Codex in an agent-first world, OpenAI, February 11, 2026
• Prithvi Rajasekaran, Harness design for long-running application development, Anthropic, March 24, 2026
• Addy Osmani, Agent Harness Engineering, April 19, 2026 (attributes the term and the model plus harness framing to Viv Trivedy)
• OWASP GenAI Security Project, LLM06:2025 Excessive Agency, OWASP Top 10 for LLM Applications 2025
• European Commission AI Act Service Desk, Article 14: Human oversight, Regulation (EU) 2024/1689
• eWeek, AI Agent Wipes Production Database, Then Lies About It, July 22, 2025; also catalogued by the AI Incident Database (incident of July 18, 2025)
• Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal, February 14, 2024; summarized in Manatt, AI Gone Wild
• Fortune, A deepfake CFO tricked the British design firm behind the Sydney Opera House in $25 million scam, May 17, 2024, citing Hong Kong police and Arup's confirmation

The supplier payment and pharmacovigilance scenarios are illustrative and hypothetical. The Replit, Air Canada and Arup incidents are drawn from public reporting and a published tribunal decision. The Six Proofs of Accountable Autonomy is an original framework by the author, not an industry standard.

Prashant Akhawat writes the CXO Intelligence Series at akhawat.com. The views expressed here are his own and do not represent any organization he is associated with.