Calling AI Agents Autonomous Will Not Shift the Blame

72% of a 50-plus expert panel said responsible AI governance that treats agents as autonomous decision makers will fail. The vote, published September 8, 2026 by Elizabeth M. Renieris, David Kiron, Steven Mills, and Anne Kleppe with Boston Consulting Group, lands after courts and regulators already refused to let software carry the blame.

Companies still sell agents that plan, call tools, transact, and run across workflows without waiting for a click. That operational freedom is the product. It is also the word that will not move liability off a named human.

A Chatbot Could Not Carry Air Canada’s Blame

In November 2022, Jake Moffatt used a chatbot on Air Canada’s website after a grandmother’s death. The bot said bereavement fares could be claimed after travel and pointed to another page. The cheaper fare was not available that way. Moffatt paid full price, then spent months trying to get the difference back.

Air Canada told the British Columbia Civil Resolution Tribunal that the chatbot was, in effect, a separate legal entity responsible for its own words. Tribunal member Christopher C. Rivers rejected that in the February 2024 tribunal decision on the chatbot, cited as 2024 BCCRT 149.

In effect, Air Canada suggests the chatbot is a separate legal entity that is responsible for its own actions. This is a remarkable submission. While a chatbot has an interactive component, it is still just a part of Air Canada’s website. It should be obvious to Air Canada that it is responsible for all the information on its website.

Christopher C. Rivers, tribunal member, 2024 BCCRT 149

Rivers found a duty of care between the airline and the customer. He found that Air Canada did not take reasonable care to keep the bot accurate, and that Moffatt was reasonable to rely on it. A customer, he wrote, has no reason to know that one corner of a site is trustworthy and another is not.

The award was small and specific. Moffatt had paid $1,630.36 for two flights and should have paid $979.48 at the bereavement rate. Damages were $650.88, plus $36.14 in pre-judgment interest and $125 in tribunal fees, for $812.02 in total. The legal point was larger than the check. A tool that talks like staff still speaks for the company that put it on the page.

Seventy-Two Percent of the Panel Drew the Same Line

Renieris and her co-authors asked the panel, now in its fifth year, to react to a single claim: responsible governance that treats agents as autonomous decision makers will fail. Most agreed. The split underneath that number is about vocabulary, not about whether the software can act.

EnBW chief data officer Rainer Hoffmann said agentic autonomy is real and growing. Renato Leite Monteiro, vice president of privacy, data protection, AI, and intellectual property at e&, said self-improving agents are moving faster than failure modes can be mapped. Ben Dias, chief AI scientist at IAG, said agentic systems are already taking action on a user’s behalf rather than only answering questions. Simon Chesterman, vice provost at the National University of Singapore, listed the new range: plan, call tools, transact, and operate across workflows.

THE PANEL AND THE SURVEY

• Coworker framing: A 2025 global executive survey from the same research program found 76% of respondents view agentic AI as more like a coworker than a tool.
• Management gap: On an earlier prompt, 69% of the expert panel agreed that holding agentic AI accountable requires new management approaches.
• Adoption: The same research stream found 35% of organizations had adopted AI agents, with another 44% planning to deploy.

Those figures explain the marketing copy. Executives already talk about agents as teammates. Linda Leopold, an AI speaker and consultant, said agents are autonomous decision makers in a technical sense because they act without constant human approval. Bruno Bioni, founder of Data Privacy Brasil, cut that down: what looks like autonomy is delegated execution, selecting steps and using tools inside limits someone else set.

WHERE EXPERTS DISAGREE

• The majority line: Technical independence does not create a party that can be sued, fined, or morally blamed, so governance aimed at the agent leaves an empty chair.
• The dissent: Tshilidzi Marwala of United Nations University argued that governance can recognize rising autonomy and still keep human oversight, legal duty, and transparency, rather than deny that the systems act on their own.

Marwala’s view is the one product teams prefer, because it lets them keep the word autonomous in the pitch. The majority is closer to what a tribunal can actually order.

Why Operational Autonomy Still Leaves a Named Owner

Chesterman’s distinction is the one that survives contact with a docket. Autonomy in the engineering sense is not autonomy in the moral or legal sense. Amit Shah, chief executive of Instalily.ai, called “autonomous decision maker” a governance fiction that enables blame laundering with better vocabulary. Jai Ganesh, formerly vice president of technology at Wipro, said agents can make choices and still cannot be held to account for what follows. Carolina Aguerre, a professor at Universidad Católica del Uruguay, put the same point in one line: responsibility is a human faculty.

The more we speak as if agents decide, the easier it becomes for firms and governments to launder responsibility through the machine: The model recommended, the agent acted, the human shrugged.

Simon Chesterman, vice provost, National University of Singapore

Riyanka Roy Choudhury, a Stanford CodeX fellow, said treating an agent as the decision maker severs liability from capacity. The software holds no assets to attach, no license to suspend, and no interest that a fine can deter. Öykü Işik added that these systems are stochastic and context-dependent, not coherent actors with stable intent. Yan Chow of Automation Anywhere pressed the persistence problem: each inference is stateless, so the agent at the next step is not the same subject as the one before, and blame is assigned to something that never existed as a continuous party.

That is why a limited-liability wrapper around an agent is such a tempting idea, and such a bad one. It would turn Air Canada’s failed argument into a corporate form. The human who set the goal and took the upside would cash out, while a shell with no assets sat in the dock. Practitioners arguing over agent identity keep returning to the same objection: if the software can transact, someone still has to stand behind the transaction when it is wrong.

Europe Will Not Treat Agents as Their Own Category

The European Commission’s AI Act Service Desk is blunt. The term AI agent is used loosely in public debate, and it is not a defined class in the statute. The definitions of an AI system and of a general-purpose AI model are wide enough to cover agents, which means the existing duties apply. In the Commission’s words, agents are not a separate category of AI.

The Act entered into force on August 1, 2024 and became applicable on August 2, 2026, the same date the AI Office and national authorities began enforcement. Chatbots and other interactive systems must tell people they are dealing with a machine. Deepfakes need labels. AI-generated or altered content needs machine-readable marks. The Commission said more than 180 organizations had signed a code of practice on transparency of AI-generated content, and it pointed to transparency rules that started in August.

High-risk duties are the slower shoe. After the AI Omnibus entered into force on July 27, 2026, rules for many high-risk uses, including biometrics, critical infrastructure, education, employment, and border control, apply from December 2, 2027. Systems tied to sector product law have until August 2, 2028. Those files still demand risk controls, logging, documentation, and human oversight. The Commission’s risk-based rules for developers and deployers never create a robot that can take the stand.

Autonomy or tool use in an underlying general-purpose model can even push that model toward a systemic-risk designation, which brings extra risk-management duties. The AI Office says its thinking on agents is still preliminary and that it is watching the category, including through safety-evaluation work. Preliminary does not mean a carve-out. It means the old categories still bite.

WHO STILL HOLDS THE BAG

Katia Walsh, AI lead at Apollo Global Management, drew a stakes line the panel kept repeating: high-stakes choices should not treat agents as autonomous decision makers, while lower-stakes ones might. Richard Benjamins, co-chief executive of RAIght.ai, said impactful decisions should not be taken fully by agents, and trivial ones can be. Pierre-Yves Calloc’h, a consultant, boiled the job down to knowing exactly where autonomy must stop. Europe’s calendar now writes that line in dates rather than slogans.

Estonia’s AI ID Codes Still Point at a Person

On June 17, 2026, at Stenbock House, Estonia’s Eesti.ai advisory board backed a plan to issue AI ID codes. Prime Minister Kristen Michal endorsed official digital identities for AI agents so that software can act for a person, a company, or an institution inside limits that can be checked and audited.

“In the future, AI will increasingly carry out digital tasks on our behalf, compiling reports, preparing declarations or interacting with information systems,” Michal said. “To that end, it must be clear who is acting on whose behalf with what rights, and who is ultimately responsible.”

The design is borrowed from a country that already lets an accountant file a client’s taxes and lets an adult manage a parent’s health portal with defined rights. An AI ID is meant to stop the lazy pattern of handing an assistant the owner’s full keys. The government listed the kinds of mandate the code should be able to express.

POWERS AN ESTONIAN AI ID IS MEANT TO SCOPE

• View only: The agent may look at a record and nothing more.
• Draft: It may prepare a document without sending it as a final act.
• Payment file: It may draw up a payment that still sits inside a named mandate.
• Money cap: It may act only within a set financial limit.

Liina Vahtras, managing director of e-Residency, has described the failure mode the ID is built to block: actions that cannot be traced to a responsible party, unclear permissions, and misuse that no one sees. Under the proposal discussed around the plan, operations by an agent are treated as operations by a machine, and the natural person used as the identity anchor remains liable. That is the opposite of personhood. It is a nameplate that still points at a human.

The rest of Eesti.ai is still in pilot form. A “Most AI-Savvy Nation” series ran 35 workshops in six cities from April to June and drew about 1,200 people, with a later target of 10,000 workshop participants by year end. The ID itself is a build, not a finished registry. Even as a build, it answers the panel’s fear in infrastructure rather than in a policy PDF.

Insurers Are Rewriting the Word Autonomous

Cyber underwriters spent years arguing over what counts as a hack. Agents that take a broad instruction and then choose their own steps do not fit that script. Ryan Kratz, head of cyber for North America at MSIG USA, said carriers will need to keep reviewing policy language as AI becomes able to find weaknesses and carry out attacks without a person at the keyboard. MSIG, QBE, and Beazley are among the firms going back through forms.

Two questions sit on the desk. Does an autonomous system count as a cyber attacker under the old wording. And if an agent acting as designed makes a costly choice, is that even a cyber event, or a business decision the policy was never meant to cover. Munich Re put the global cyber insurance market at nearly $15 billion in 2025 and roughly $28 billion by 2030. Aon has forecast that generative AI will be involved in nearly 20% of cyberattacks by 2027. Those are market-size figures, not a verdict on any one claim. They are why the wording fight is happening now.

A parallel D&O conversation is already treating weak AI governance as a claims risk. Boards are being told to put AI use inside the same oversight they already apply to other enterprise risk, including written policies and a named owner. Poor controls are expected to show up later as disclosure failures and compliance breakdowns, which is how directors and officers end up in the file.

A new product class is trying to sit in the gap. Ines Boutemadja, co-founder of Klaimee, said the firm raised $5.5 million to write insurance-backed performance warranties for agent actions, on the claim that ordinary tech errors-and-omissions and cyber policies were not built for autonomous AI.

We just raised $5.5M to answer one question:

Who pays when your AI agent **** up ?

Until now, nobody had a good answer.

That's why we built @klaimee_ai : insurance-backed performance warranty for AI agents.

AI agents are taking real actions inside enterprises. They can… pic.twitter.com/V9NUCDFhBb

— Ines Boutemadja (@inesboutem) July 21, 2026

The practical brake on agent rollouts is not whether the model can finish the task. It is what happens on the first wrong payment, the first bad promise to a customer, or the first file deleted in production, and whether anyone’s policy will pay. If the exclusion turns on “autonomous operation,” the sales deck that bragged about hands-free action becomes evidence.

Limits Belong in the Architecture, Not the Prompt

Mark Surman, president of Mozilla, said agents do not come from nowhere: people build them, companies deploy them, and someone profits from the decisions they make. His instruction was to treat agents as extensions of human and institutional choices. Chesterman went further. The unit of governance is not the agent as a little corporate citizen. It is the developer, the enterprise, the data and tools, the permissions, and the humans who benefit.

The organizers’ own list matches that unit. Calibrate autonomy by stakes and reversibility, not by how capable the model looks in a demo. Encode the stop in permissions, approval gates, and hard technical limits, rather than in a prompt the agent can talk itself around. Name a person for every consequential outcome before go-live, including when the workflow crosses department lines. Aim the audit at the system around the agent, not at the model card. And keep a culture in which staff can challenge an agent and are rewarded for raising the alarm, which is harder to demand of outside customers if the agent is a vendor product.

Belona Sonna of the Australian National University noted that some domains, including driving, are designed for real-time autonomous action because a person cannot sit in every loop. That does not dissolve the owner. It makes the alignment, the logging, and the override more important, not less. Calloc’h’s warning sits on top of those cases: high-stakes trade-offs between conflicting goals are governance choices, and they do not encode cleanly.

THE ACCOUNTABILITY CALENDAR

1. February 14, 2024: The British Columbia tribunal holds Air Canada liable for chatbot misstatements and rejects the separate-entity defense.
2. August 1, 2024: The EU AI Act enters into force.
3. June 17, 2026: Estonia backs AI ID codes that still tie agent acts to a human identity anchor.
4. August 2, 2026: The Act becomes applicable, transparency duties begin, and the AI Office starts enforcement.
5. September 8, 2026: The 50-plus expert panel publishes the 72% vote against treating agents as autonomous decision makers.
6. December 2, 2027: High-risk duties for many Annex III uses are scheduled to apply.

Shah’s line still fits the file: a machine can make the call, and it cannot own the outcome. Moffatt’s $812.02 already showed where that ownership lands. Estonia is trying to print the owner’s name on an ID before the agent spends a euro. Europe is forcing the machine to admit it is a machine. The companies that keep calling the software autonomous in the brochure will meet that word again in discovery, in an exclusion clause, and in a hearing, and it will not help them.

Disclaimer: This article is news reporting and analysis of public research, tribunal records, and official rules. It is informational only and does not constitute legal advice, insurance advice, or a recommendation to deploy or withhold any AI system. Readers facing a dispute, a policy renewal, a board resolution, or a regulatory filing should consult a qualified attorney and, where coverage is at issue, a licensed insurance broker before acting. Figures, enforcement dates, and product statuses reflect the cited sources as of the dates on those documents and may change as agencies, courts, and carriers issue new guidance.