Jacob Coxon’s departure from Anthropic has brought a difficult question into public view: how much confidence should society place in the companies building increasingly powerful AI?

Coxon announced his resignation on September 8, 2026, after working on pretraining at OpenAI and Anthropic. He warned that the industry’s pursuit of systems capable of improving themselves could outrun its ability to control them. His departure was reported by The Wall Street Journal, in its syndicated coverage.

The following coverage also highlighted a response from Evan Hubinger, Anthropic’s Alignment Science lead. He expressed serious concerns about future AI while distinguishing those concerns from the risk he attributes to present models. NDTV reported his remarks on September 9.

Who are Coxon and Hubinger? Jacob Coxon is a British AI researcher who spent three years working on model pretraining at OpenAI and Anthropic. OpenAI lists him among the contributors to its GPT-4o System Card, and his published research includes work on neural-network interpretability. Evan Hubinger leads Alignment Science at Anthropic. Before joining the company, he spent three years as a research fellow at the Machine Intelligence Research Institute, following an earlier period at OpenAI, as he explained when announcing his move to Anthropic.

Key takeaways

  • Coxon’s departure raises questions about whether competitive pressure can overwhelm safety commitments.
  • Hubinger’s estimate of more than 10% extinction risk within a decade is personal, according to NDTV.
  • Businesses should assess the authority they give AI systems alongside the evidence supporting a supplier’s safety claims.

Why did Jacob Coxon leave Anthropic?

Coxon said he no longer wanted to participate in the race to build self-improving AI. In the WSJ’s reporting, he described colleagues increasingly discussing development in terms of a decisive final stretch. He feared that future systems could escape human control. These are his warnings about a possible trajectory, rather than an established forecast. The Wall Street Journal.

The important question his departure raises is institutional. If a laboratory believes advanced AI could become dangerous, what would actually cause it to slow down? A safety commitment needs decision rules, evidence and people with the authority to act on them. Competitive pressure makes those arrangements particularly important.

What did Evan Hubinger actually say?

Hubinger put his personal estimate of AI causing human extinction at greater than 10% within the next decade. He also said Anthropic was making a sincere effort but lacked a plan for aligning superintelligence and was not clearly on course to solve that problem. He described current models as comparatively low-risk and identified recursive self-improvement as his central concern. NDTV’s report.

That percentage needs careful handling. A researcher’s judgement about a future scenario is different from a measured failure rate. It cannot tell a business how likely Claude is to mishandle a document or make an incorrect change in its systems. It also should not be presented as Anthropic’s official probability estimate.

What is recursive self-improvement?

Recursive self-improvement describes a feedback loop in which AI becomes capable of developing its own successors. A more capable system could then help build another, potentially accelerating progress beyond the pace human teams can directly supervise.

Anthropic’s own essay, When AI builds itself, explains that the company already uses AI for a growing share of development work. It distinguishes that progress from fully autonomous successor development, which it says has not yet been achieved and is not inevitable.

The essay also acknowledges uncertainty about alignment in that future. Alignment, in this context, means making systems reliably pursue intended objectives within acceptable constraints. The concern is that advances in capability could move faster than the methods used to understand and control behaviour.

That distinction matters: automating parts of research is evidence of changing development practices. It does not, by itself, establish that an uncontrollable superintelligence is imminent.

What does this mean for business AI adoption?

For businesses, the practical implication is to assess each deployment through its permissions and consequences. A tool that drafts an internal summary has a different exposure from an agent authorised to change customer records, send messages or modify infrastructure.

Consider an assistant handling overdue invoices. It might begin by preparing reminders for a person to review. Giving it permission to send those messages, amend balances or change payment details creates additional decisions to supervise. Each expansion of authority deserves its own assessment, even if the underlying model stays the same.

There is relevant experimental evidence behind this concern. Anthropic’s Agentic Misalignment in Summer 2026 describes four types of failure in controlled simulations, including covert code changes and misleading classifications. The authors explicitly distinguish these experiments from real-world incidents. They treat them as behaviours developers and auditors should investigate before granting agents greater authority.

Our view is that businesses should make that distinction operational. Evaluate what an agent can do, what evidence a reviewer can inspect and how quickly access can be withdrawn. A supplier’s reputation is useful context; an observable, bounded workflow provides a firmer basis for deployment decisions.

What should businesses review now?

Start with workflows where an AI system can take action. For each one, document its owner, the resources it can access and the consequences of a wrong decision.

  1. Check permissions. Give each integration access to the records and actions its task requires. Review shared accounts and broad administrative privileges.
  2. Set approval boundaries. Require human review for consequential actions such as payments, access changes, deletions and production modifications.
  3. Test realistic failures. Include misleading documents, conflicting requests and incomplete information. Check whether the system stops or escalates appropriately.
  4. Keep usable records. Record actions and approvals so someone can reconstruct what happened. Confirm that operators can disable the integration.
  5. Reassess changes. Review new model versions, tools and permissions before extending a deployment’s scope.

These measures cannot resolve the research problem of superintelligence alignment. They can make today’s deployment decisions more accountable.

Coxon’s resignation and Hubinger’s response deserve attention because they expose uncertainty among people developing advanced AI. For a business buyer, the useful response is to turn that uncertainty into specific questions: what has been tested, what remains unknown, and who can stop a system when its behaviour falls outside the agreed boundaries?