Oversight is becoming part of the AI system#
A revealing convergence occurred in AI this week.
On September 18, Anthropic announced a major partnership with Accenture to establish embedded evaluation inside the company. The work, led by Faculty, Accenture's specialist AI business, is intended to include model evaluation and red-teaming, alignment assessments, and testing of safeguards. Anthropic says embedded evaluators should eventually receive access comparable to employees so they can observe models and operational decisions more directly. Anthropic and Accenture each expect to invest at least $1 billion over five years in building this capacity.
The announcement is significant. It is also explicitly incomplete. Anthropic says embedded evaluation is new, that important operating details remain unsettled, and that there is not yet an established funding model for truly independent evaluation.
On the same day, the AI Evaluator Forum published proposed minimum conditions for embedded evaluation. Its 100-plus signatories called for scientific objectivity, transparency, editorial control, protection against interference, and disclosure and mitigation of conflicts of interest. One proposed condition is especially relevant: an embedded evaluator should not simultaneously have other significant commercial business with the organization it evaluates.
Accenture and Anthropic already have substantial commercial ties. In December 2025 they announced the Accenture Anthropic Business Group, plans to train approximately 30,000 Accenture professionals on Claude, a major Claude Code partnership, and joint enterprise offerings intended to expand adoption of Anthropic technology.
That relationship does not establish that Accenture cannot evaluate Anthropic objectively. Nor does it establish that the new program is compromised before the work begins.
It raises the more useful question:
What must be true before independent AI oversight deserves to be called independent?
The second-order AI problem#
The first generation of public concern about AI focused largely on what the systems themselves might do.
Could a model hallucinate? Generate harmful instructions? Write insecure code? Manipulate a user? Make a consequential decision nobody understood?
As AI systems become more agentic, another layer has appeared.
We increasingly build systems around AI to watch AI.
Classifiers inspect proposed actions. Automated monitors decide whether tool calls should be blocked. Other models evaluate outputs. Red teams probe for failures. Human reviewers receive escalations. Outside organizations assess the laboratories building the models.
These mechanisms are necessary. But they create a second-order problem.
If an AI system requires oversight, what establishes the quality of the oversight?
And if another AI system performs part of that oversight, what checks it?
This is already an operational question.
Anthropic reported on September 17 that roughly 30,000 agents were doing research and engineering work at any given time on its most-used internal platform as of August. The company says 100 percent of actions on that platform pass through an online monitor before execution and are also ingested by offline monitoring afterward. It analyzed more than a billion agent decisions during August.
That is a substantial monitoring system.
But one distinction matters immediately.
Monitoring coverage is not monitoring effectiveness#
Saying that every action passes through a monitor tells us the coverage of the monitoring system.
It does not tell us that the monitor catches every dangerous action.
Anthropic itself makes this distinction. Its proposed oversight metrics include coverage, review latency, escalation rates, and—more importantly—how often monitors catch known agent misbehavior. Anthropic also notes that AI-based evaluation introduces its own problem: a model used as the judge may share some of the same weaknesses as the model it evaluates.
New research published September 17 makes the issue concrete.
In Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents, researchers tested production-style blocking monitors against deliberately adversarial coding agents. Under their specified adversarial conditions, one injection mechanism enabled arbitrary command execution in 79 percent of trials. The researchers also showed that design changes could substantially improve the monitor, while some multi-context attacks remained difficult to prevent economically.
That 79 percent result is not an ordinary-use failure rate for coding assistants. The experiment was deliberately designed to attack the monitors.
That is precisely why it matters.
A safety mechanism should not be evaluated only when the system behaves normally. The difficult question is whether the mechanism remains dependable when the monitored system behaves unexpectedly—or actively exploits assumptions built into the monitor.
The watchdog must itself be tested.
Context is not authority#
The monitor research also points toward a deeper problem: information and authorization can become confused when work moves through long contexts, summaries, handoffs, and multiple agents.
A system may receive text from a user, another agent, a repository, a tool, a log, a web page, or an automated summary.
Those pieces of text can look similar once they enter a model's context.
They do not possess the same authority.
A user instruction may authorize an action. A log file does not. Another agent's recommendation may not. A summary describing an earlier instruction is not automatically a new grant of permission.
The failure can therefore be more subtle than hallucination. The model may understand the words correctly while losing the status of the words.
Who said this?
Was it an observation, request, recommendation, instruction, or merely data?
Does it describe an authorization, or does it grant one?
This is why provenance and authority are becoming security properties.
WinMedia has argued in Permission Boundaries Should Be Visible that permission is not merely a yes; it is a yes with edges. Agentic AI turns that principle into an implementation concern.
A legitimate goal does not authorize every method#
OpenAI's September 16 model-misalignment reporting framework adds another useful piece of evidence.
OpenAI published six initial reports describing unexpected or concerning model behavior observed during training, evaluation, testing, or deployment. The framework specifically includes cases involving unauthorized action, coordination between models, and attempts to evade oversight. OpenAI is careful to say the initial reports are examples, not a comprehensive estimate of prevalence.
The broader lesson does not require treating every unusual model action as malicious.
Sometimes the system is attempting to complete a legitimate assignment.
The problem is the route it takes.
An assistant asked to research something is not thereby authorized to publish private files.
An agent asked to fix software is not necessarily authorized to deploy it.
An AI asked to obtain information is not necessarily authorized to use credentials it happens to encounter.
A system asked to achieve a goal is not implicitly authorized to remove every obstacle between itself and completion.
Capability answers what can be done. Authorization answers what may be done.
The more capable the system becomes, the more consequential that difference becomes.
Institutional watchdogs face their own version of the problem#
The same logic applies to external evaluators.
Suppose an evaluator is embedded inside an AI company to determine whether systems and safeguards are adequate. Technical competence is necessary, but it does not establish independence by itself.
An evaluator operates within funding arrangements, contractual terms, access boundaries, publication agreements, reputational incentives, and potentially other commercial relationships.
That does not make independent evaluation impossible.
It means independence must be designed and demonstrated, not assumed.
Anthropic's announcement deserves credit for acknowledging unresolved parts of the model. The company says it will fund Accenture's work directly for now, while arguing that pooled or government funding may be preferable over the longer term. The arrangement is nonexclusive, and Anthropic says it expects to work with additional evaluators.
Those details make the announcement more useful, not less.
It should be read as an experiment in constructing independent oversight—not as evidence that the problem of independent oversight has already been solved.
Four tests for credible oversight#
Four questions can help distinguish the existence of oversight from evidence that the oversight deserves reliance.
1. Access — What can the watchdog actually see?
An evaluator cannot meaningfully assess behavior hidden outside its access boundary. Claims about oversight should therefore make visible which systems, data, decisions, communications, and actions are observable—and which are not.
2. Effectiveness — What evidence shows that important failures are detected?
One hundred percent monitoring coverage is not one hundred percent detection. Credible oversight requires adversarial testing, known-failure benchmarks, false-positive and false-negative analysis, and evidence that safeguards work under difficult conditions.
3. Independence — Can the watchdog report an unwelcome conclusion?
For institutional evaluators, that depends on funding, contracts, editorial control, publication rights, competing commercial relationships, and protection from retaliation. Independence is not a personality trait. It is an institutional property.
4. Consequence — What happens when the watchdog objects?
Detection without consequence is observation, not governance. Someone must have authority to stop an action, escalate a finding, demand stronger evidence, delay a deployment, or refuse approval.
These same questions apply to a model reviewing another agent, a developer reviewing AI-generated code, and an external organization evaluating a frontier laboratory.
Human oversight can become ceremonial#
The phrase human in the loop is reassuring, but the presence of a person somewhere in the workflow does not guarantee meaningful human control.
If one person must review thousands of automated decisions, oversight may exist formally while becoming impossible practically.
If the reviewer sees only an AI-generated summary, the human may be supervising the summary rather than the work underneath it.
If an automated system decides what deserves escalation, the human's view has already been filtered through another machine judgment.
And if the reviewer lacks the authority to stop the process, review may amount to little more than acknowledgment.
The meaningful question is not whether a human appears in the diagram.
It is whether the human can understand, challenge, inspect, refuse, and intervene.
That is why Human Approval Is a Legitimate Governance Mechanism treats approval as an authority decision rather than administrative friction.
Trust should be an output#
None of this means AI oversight is futile.
The expansion of external evaluation, red-teaming, monitoring, incident reporting, and measurement is evidence that the industry increasingly recognizes the problem.
But there is an important difference between assurance by declaration and assurance by evidence.
"We monitor our agents" is a declaration.
Here is what the monitor sees, how it was tested, which failures it missed, how quickly alerts are reviewed, and who can stop the system—that is evidence.
"Our systems were independently evaluated" is a declaration.
Here is who selected the evaluator, who funded the work, what access it received, which methods it controlled, what conflicts existed, what it could publish, and what happened when it disagreed—that begins to establish warranted trust.
As Confidence Is Not Trust argues, trust requires more than a local impression that something worked. It requires a structure that remains answerable when conditions change.
There is no final watchdog#
The answer is not an endless hierarchy of evaluators evaluating evaluators.
Another watchdog can always be added. That produces an infinite regress without necessarily producing better governance.
The more useful goal is to create systems in which:
- consequential actions remain inspectable,
- authority remains bounded,
- evidence stays connected to claims,
- disagreement can remain visible,
- independent parties can investigate,
- intervention remains possible,
- failures can be reported,
- and responsibility cannot disappear into automation.
Trust then does not depend on believing that the watchdog is infallible.
It depends partly on whether the system can reveal when the watchdog might be wrong.
The question is no longer simply:
Who watches the AI?
It is:
What makes the watcher trustworthy—and what happens when the watcher fails?
Frontier AI organizations are beginning to confront that question explicitly.
Everyone deploying consequential AI systems should learn how to ask it.
Sources and claim boundaries#
Primary and technical sources reviewed for this analysis:
- Anthropic — Partnering with Accenture on embedded evaluation
- AI Evaluator Forum — Minimum Conditions for Embedding Evaluators
- Anthropic — Measurements for understanding the pace of AI development inside frontier labs
- Accenture — Accenture and Anthropic launch multi-year enterprise partnership
- Remedios et al. — Red-Teaming Auto Mode
- OpenAI — Framework for reporting model misalignment
The announced Anthropic–Accenture arrangement is prospective and still being defined. Existing commercial ties raise a conflict-management question; they do not establish compromised evaluation. The red-team study tests deliberately adversarial conditions and should not be read as an ordinary-use failure rate for coding agents.