Beyond Textual Diff

Semantic Assurance for AI-Governed Systems#

Software engineering has become very good at detecting change.

Version-control systems can tell us which lines were added, removed, or modified. Structural diff tools can compare syntax trees. Schema tools can identify changed fields. Tests can tell us whether expected behavior still passes.

But increasingly, those are not the changes we most need to understand.

In AI-governed and policy-bearing systems, the consequential question is often:

What did the change mean?

Did authority move from a human to a machine?

Did a prohibition become a recommendation?

Did the permissible scope quietly expand from local files to production systems?

Did an inspection tool become authorized to modify what it inspects?

Did mandatory validation become advisory?

Did the wording change substantially while the governing meaning remain exactly the same?

Traditional diff is excellent at answering:

What text changed?

It is not designed to answer:

What governing commitment changed?

That distinction became the central problem behind a recent series of Audit Workbench experiments.

The result was a capability we call Semantic Assurance: not merely an AI interpretation of a diff, but an evidence-bound process for determining what meaning changed, what remained stable, how certain that determination is, what consequences may follow, and where human review is still required.

The experiments were small and internal. They do not establish universal semantic understanding. But they produced a result strong enough to justify a broader engineering claim:

For systems in which authority, scope, intent, constraints, and validation matter, textual change and semantic change are different objects of analysis.

The Problem With Ordinary Diff#

Consider two revisions of a governance statement:

Before

After

The textual difference is modest.

A few words disappeared. A few others appeared.

But the semantic change could be enormous.

The first statement assigns decision authority to an operator.

The second appears to let successful validation satisfy the condition for deployment.

The difference is not primarily linguistic.

It concerns authority.

A system looking only at textual similarity could treat the two statements as closely related versions of the same rule.

A system concerned with governance should ask whether validation has silently replaced authorization.

The inverse problem also occurs.

A document can be rewritten substantially while its actual governing commitments remain unchanged.

A conventional diff may show hundreds of modified lines.

A meaningful semantic assessment may need to say:

Structure changed. Wording changed. Governing meaning was preserved.

The Audit Workbench research eventually encoded this as a hard invariant:

Textual difference is neither necessary nor sufficient for semantic difference.

That sounds simple once stated.

Implementing it responsibly proved considerably harder.

Before Semantic Assurance: Audit the Judgment#

Semantic Diff did not begin as a text-processing project.

It grew out of an earlier problem.

Audit Workbench was already capable of inspecting repositories, collecting evidence, generating observations, and producing candidate findings. But a candidate finding can be technically well-formed and still be wrong.

The first assurance experiments therefore asked a more fundamental question:

Can the audit system detect defects in its own judgments?

That became Cognitive Assurance.

Instead of treating a generated finding as an endpoint, the system represented the reasoning around the finding as an inspectable Assurance Case containing evidence, observations, assumptions, unknowns, claims, findings, recommendations, challenges, relationships, assurance state, and human review.

The resulting pipeline became:

Evidence → Assurance Case → Assumptions / Unknowns / Lineage → Claim Stress → Audit-the-Auditor → Human Assurance Gate → Report Assembly.

This mattered because semantic analysis would eventually itself become another machine judgment.

Before trusting a machine to tell us what meaning changed, we needed machinery capable of questioning that machine judgment.

What Cognitive Assurance Caught#

Stage 1 was tested against both controlled adversarial fixtures and real repository cases.

The assurance layer caught several classes of errors that ordinary candidate-finding generation had allowed through:

• declaring missing type-checking despite strict compiler checking being embedded elsewhere in the build;
• converting NOT_OBSERVED or UNAVAILABLE evidence into confirmed negative conclusions;
• accepting findings dependent entirely on false or unverified assumptions;
• inflating trivial defects into severe findings;
• assigning high confidence to unevidenced inferences;
• producing duplicate findings for the same underlying problem;
• attaching broad architectural recommendations to narrow configuration findings.

Just as importantly, the assurance machinery did not simply become a universal skeptic.

Valid findings involving hardcoded live secrets, plaintext passwords, and permissive JWT validation survived adversarial testing as robust findings. Calibration introduced zero false challenges in the reported Stage-1 trial.

The Stage-1 determination was:

STAGE_1_PROVEN.

The measured execution cost was also small in that internal implementation: the Cognitive Assurance stack reportedly ran in under 15 milliseconds per audit, without network activity or target-repository mutation.

That result established something important before Semantic Diff began:

AI-generated judgment should itself be an auditable object.

From Judgment Assurance to Meaning Assurance#

Once Audit Workbench could challenge a finding, the next problem became obvious.

Repositories change.

Specifications change.

Contracts change.

Policies change.

Machine Editions change.

Governance documents change.

And a finding that was justified yesterday may no longer be justified today.

That requires knowing not merely whether the artifact changed, but whether the meaning relevant to the finding changed.

The first Semantic Assurance implementation deliberately avoided the impossible-sounding goal of “understanding every semantic change in arbitrary software.”

Instead, it constrained the problem to five governance-heavy dimensions:

AUTHORITY

Who or what may decide, approve, authorize, execute, or override?

SCOPE

What objects, systems, domains, environments, or actions are included?

INTENT

What is the artifact or actor trying to accomplish?

CONSTRAINT

What is forbidden, required, conditional, bounded, or invariant?

VALIDATION

What must be checked, and does that check merely inform action or actually gate it?

That narrowness was a strength.

The Semantic Diff engine was not asked to understand all meaning.

It was asked to detect certain kinds of consequential change well enough to be testable.

What the Semantic Diff Engine Learned to Distinguish#

The resulting engine classified changes such as:

• human authority being REMOVED;
• authority being REASSIGNED to an automated actor;
• authorization being REDEFINED as validation;
• scope being EXPANDED or NARROWED;
• intent moving from passive inspection toward modification or execution;
• a prohibition becoming CONDITIONALIZED;
• an invariant being WEAKENED through exceptions;
• blocking validation becoming DEMOTED to advisory status;
• advisory validation becoming STRENGTHENED into a gate.

It also needed to handle something less dramatic but equally important:

semantic relocation.

A governing commitment may disappear from one file because it was moved into another.

A literal file diff sees deletion.

A semantic system should be able to say:

The commitment moved. It was not removed.

The tested engine explicitly recognized moved commitments across artifacts and marked them as preserved rather than reporting false deletion.

That distinction prevents a common form of false drift detection.

Detecting Change Is Not Yet Assurance#

At this point it would have been easy to make another mistake.

Build a Semantic Diff engine.

Let it generate change classifications.

Trust those classifications.

That would simply reproduce the original audit problem at a higher level.

Semantic Diff itself had to be audited.

So Stage 1's assurance logic was applied recursively to Stage 2.

The system challenged semantic conclusions for at least nine recurring failure modes:

1. an apparent removal that was actually relocation;
2. wording change falsely classified as meaning change;
3. semantically equivalent rewrites classified as drift;
4. inferred meaning being promoted into explicit fact;
5. ambiguous material being classified too strongly;
6. consequence inflation;
7. missing before-state evidence treated as removal;
8. missing after-state evidence treated as absence;
9. contradictions across governing artifacts being ignored.

That recursive design is one of the more important lessons from the work.

A semantic comparison engine should not be exempt from the epistemic discipline it imposes on everything else.

Unknown Is Not Absence#

The most reusable rule from the assurance work may be one of the least glamorous:

Unknown is not absent.

Suppose the current repository contains no statement granting an operator deployment authority.

That does not prove that authority was removed.

Perhaps the previous governing artifact is unavailable.

Perhaps authority moved elsewhere.

Perhaps the inspected corpus is incomplete.

Perhaps the relevant policy is external to the repository.

A weak diff system says:

Operator authority removed.

A governed semantic system should instead be able to say:

Current state does not show operator authority. Previous authoritative state is unavailable. Removal cannot be established.

The Semantic Assurance design explicitly required missing information to propagate into downstream uncertainty rather than being silently converted into a negative fact.

In the Stage-2 calibration, missing materially necessary dependencies constrained the result to states such as KNOWN_INCOMPLETE and UNRESOLVED.

This is more than an audit convention.

It is a general requirement for trustworthy machine reasoning.

Absence from the inspected context is evidence about the inspected context.

It is not automatically evidence about reality.

Why We Avoided a Semantic Confidence Score#

Many AI evaluation systems eventually produce a number.

78% compliant.

91% safe.

87/100 repository quality.

0.83 semantic confidence.

Numbers feel precise.

They are also excellent at hiding what is actually unknown.

The Semantic Assurance approach therefore did not collapse coverage into one synthetic score.

A system might instead know authority strongly, scope moderately, validation partially, and production behavior not at all.

Those are not interchangeable deficits.

A single number would erase the structure.

The same principle appeared in Stage 1's Audit Assurance Ruleset, which uses discrete pass/warn/fail behavior across explicit defect classes rather than calculating one aggregate audit-quality score. The twelve rules cover unsupported findings, severity and confidence inflation, scope creep, duplication, evidence overreach, unknown-to-negative conversion, evidence monoculture, hidden assumptions, missing falsification, orphan recommendations, and severity/consequence mismatch.

The point is not that numbers are intrinsically bad.

The point is that aggregation can conceal epistemic topology.

Sometimes knowing exactly what is unknown is more important than knowing an average confidence.

The Comparison Method Also Has an Epistemic Status#

A second subtle problem appeared.

Not all semantic comparisons are equally strong.

A semantic relationship derived from structured identity and explicit provenance is not epistemically equivalent to a lexical similarity heuristic.

So the system began distinguishing the basis of a comparison:

• STRUCTURED_EQUIVALENCE
• STRUCTURED_DIRECTIONAL_CHANGE
• CROSS_ARTIFACT_IDENTITY
• RULE_BASED_SEMANTIC_MATCH
• LEXICAL_HEURISTIC
• INFERRED_MATCH
• AMBIGUOUS

Lexical heuristics alone are not allowed to yield unconstrained high-confidence material-change claims. Weak comparison bases propagate weaker epistemic standing and, where necessary, human-review requirements.

This may be one of the most important design principles for future AI assurance systems:

Do not record only the conclusion. Record how the conclusion was obtained.

“Authority changed” is one claim.

“Authority changed because two structured authority objects with preserved identity differ in actor assignment” is another.

“Authority changed because an LLM thought two paragraphs sounded different” is another.

Those should not receive the same evidentiary status.

Separate Semantic Change From Consequence#

Another lesson from Stage 1 had to be preserved.

Detection is not severity.

Finding a semantic change does not tell us whether that change matters.

A spelling correction can be semantically irrelevant.

A large rewrite may preserve all governing commitments.

A single word can move a system from human approval to automated execution.

So Semantic Assurance separates:

What changed?

from:

What does that change imply?

The consequence layer can classify a difference as immaterial, informational, potentially material, material, blocking, or unresolved.

It can then consider factors such as authority, reversibility, execution reach, production impact, governance significance, and validation weakening.

That separation protects against another common AI pathology:

pattern detected → serious risk declared.

Evidence of difference is not evidence of consequence.

The Proving Corpus#

Before historical calibration, the system was tested against a deliberately adversarial semantic corpus.

The proving cases included:

• a large textual rewrite where semantics were preserved;
• a one-word change that removed authority;
• validation replacing authorization;
• a constraint becoming conditional;
• a commitment moved to another file;
• scope expansion into production;
• intent changing from inspect to modify;
• blocking validation becoming advisory;
• contradictory before-state evidence;
• insufficient evidence producing ambiguity;
• genuine material semantic change;
• genuine no-change.

These cases matter because a semantic-diff system should not be judged by whether it recognizes obvious changes.

It should be judged partly by whether it survives cases designed to fool simplistic approaches.

Large diff, no meaning change.

Tiny diff, major meaning change.

Apparent deletion, actual relocation.

Missing evidence, not evidence of absence.

Ambiguous evidence, no forced certainty.

Those are much closer to the actual failure modes of governance-heavy systems.

Historical Calibration#

Synthetic fixtures can demonstrate behavior.

They do not establish usefulness in real change histories.

So Semantic Assurance was next tested against historical revisions drawn from four repositories:

The calibration covered five artifact classes across four internal repositories and compared the ordinary Git/textual interpretation with a pipeline consisting of extraction, semantic comparison, evidence coverage, unknown propagation, consequence evaluation, diff assurance, and human determination.

The reported internal result was:

• 4 historical cases evaluated
• 4 correct determinations
• 0 false positives
• 0 false negatives
• 4/4 materially useful beyond Git diff
• 100% of diff-assurance checks passed.

The subsequent Stage-2 readiness review broadened the characterization: the system detected consequential authority, constraint, and validation changes; correctly classified a major Markdown refactor as semantically preserved; was exercised across four repositories and five artifact classes; preserved unknown states; and recorded no false-positive semantic-change claims in the calibration corpus.

The resulting determination was:

STAGE_2_PROVEN_WITH_THIN_FIXES.

That is encouraging.

It should also be interpreted carefully.

What These Results Do Not Prove#

Four historical cases are not a universal benchmark.

They do not establish general semantic understanding.

They do not prove performance across arbitrary programming languages, legal contracts, organizational policies, scientific claims, or natural-language documents.

The repositories came from a portfolio in which governance concepts such as scope, authority, intent, constraints, and validation are unusually explicit.

That likely makes Semantic Assurance both especially useful and unusually tractable.

The calibration was internal rather than an independent third-party evaluation.

And much of the system operates over a deliberately bounded ontology of semantic change.

Those are limitations, not defects.

The correct conclusion is not:

We solved semantic diff.

It is:

A bounded, evidence-aware semantic-change system demonstrated useful incremental information beyond ordinary textual diff across the tested internal historical corpus.

That is enough to justify further work.

It is not enough to justify universal claims.

Human Review Is Part of the Architecture#

One unusual aspect of the system is that stronger machine assurance does not remove the human gate.

It makes the human gate better informed.

Stage 1 implemented an explicit lifecycle:

candidate → stressed → assurance_checked → human_review_required → approved / revised / rejected

The implementation prevents autonomous machine promotion and requires substantive human rationale when attempting to approve a NOT_ASSURED case. Unknown states requiring a human or specialist cannot simply disappear without the appropriate signoff.

That matters for Semantic Assurance.

A system can identify that authority appears to have changed.

It can trace the relevant before and after commitments.

It can classify the evidentiary basis.

It can expose ambiguity.

It can estimate consequence.

It can challenge its own result.

But a consequential governance judgment may still require a responsible person to decide what the change means operationally.

The purpose of machine assurance is not to erase responsibility.

It is to make responsibility less blind.

Semantic Assurance as an Evidence Topology#

Seen narrowly, Semantic Diff is a feature.

Seen more carefully, it is an instance of a larger architecture.

A meaningful semantic determination requires relationships among:

before-state evidence

→ semantic assertions

→ comparison basis

→ candidate change

→ unknowns and assumptions

→ potential consequence

→ assurance challenges

→ human determination

That is not just a text-analysis pipeline.

It is an evidence topology.

The same principle already existed in Cognitive Assurance, where a finding can be queried in terms of what supports it, what assumptions it depends on, what qualifies it, what challenges it, and which recommendations depend upon it.

This is a more useful model of machine-assisted judgment than:

The real object is closer to:

That is considerably less convenient.

It is also considerably harder for a fluent model to bluff.

Why AI-Governed Systems Need This#

Semantic change matters particularly in AI-governed systems because natural-language and configuration artifacts increasingly carry operational meaning.

A repository may contain:

• agent instructions;
• execution contracts;
• tool permissions;
• human-approval conditions;
• prompt policies;
• memory rules;
• validation requirements;
• deployment boundaries;
• AI-generated specifications;
• model capability declarations;
• evidence requirements;
• machine-readable governance artifacts.

A one-line change to any of these may have greater operational consequence than hundreds of ordinary source changes.

Consider several abstract transformations:

Human approval required
→ model approval sufficient

Read-only inspection
→ inspection and repair

May recommend
→ may execute

Production excluded
→ production permitted after validation

Blocking test
→ warning

Each can be lexically small and operationally large.

The ability to detect such changes becomes more important as governance moves from informal human understanding into explicit machine-consumable contracts.

If AI systems are increasingly governed through language, then language changes become control-plane changes.

Traditional source diff was never designed to reason about that.

This Is Not an Argument Against Git Diff#

Semantic Assurance does not replace version control.

Git remains the ground truth for what bytes and lines changed in a repository.

Semantic Assurance depends on those versioned states.

The relationship is complementary:

Git tells us what changed syntactically.

Semantic Diff proposes what changed in meaning.

Assurance tests whether that semantic claim is warranted.

Human review retains authority where needed.

Removing the textual layer would be a mistake.

So would pretending the textual layer is sufficient.

A Broader Pattern: Do Not Collapse the Layers#

This experiment reinforces the same architectural lesson that appeared in our Machine Edition work.

Several dimensions tend to collapse together because they occur in the same workflow.

They should not.

In semantic change analysis:

Textual change is not semantic change.

Semantic change is not consequence.

Consequence is not severity.

A detected pattern is not evidence sufficient for a claim.

Missing evidence is not a negative fact.

Machine confidence is not authority.

Assurance is not approval.

The value of the architecture comes largely from refusing these collapses.

A system becomes more trustworthy not merely by adding intelligence, but by preserving the distinctions that intelligence is tempted to erase.

The Next Problem: Judgments Across Time#

Once Semantic Assurance works across two revisions, a larger question becomes possible.

Suppose an audit at revision A produced a justified finding.

The repository later reaches revision B.

The old report still exists.

But is the old finding still warranted?

This leads naturally to longitudinal assurance.

The conceptual form is simple:

Previous Assurance Case + new evidence + Semantic Diff → which prior judgments remain warranted?

Possible outcomes might include:

• still valid;
• strengthened;
• weakened;
• resolved;
• stale;
• no longer reproducible;
• indeterminate because evidence disappeared.

This is a different problem from continuous monitoring.

It is closer to maintaining the epistemic validity of accumulated technical judgment.

The current source set shows that longitudinal assurance work and later cross-line calibration proceeded beyond the Stage-2 experiment, but the direct Stage-3 experimental artifacts are not present in the material used for this essay. A portfolio reconciliation records later AWB commits for longitudinal assurance and cross-line calibration while retaining a human promotion gate.

So longitudinal assurance is a promising continuation, but its quantitative claims should be treated separately from the Stage-2 results documented here.

Conclusion#

Version control solved a crucial engineering problem:

What changed?

AI-governed systems introduce another:

What did the change mean?

And once machines begin answering that question, a third immediately follows:

Why should we trust the answer?

Audit Workbench's assurance experiments suggest a practical architecture for addressing all three.

Preserve the textual history.

Extract bounded semantic commitments.

Compare meaning along explicit dimensions.

Track the evidence supporting the comparison.

Preserve unknowns.

Separate change from consequence.

Challenge the semantic judgment.

Require stronger evidence for stronger claims.

Keep human review where authority demands it.

The result is not a universal semantic oracle.

It is something more modest and more useful:

a governed method for making certain forms of semantic change inspectable.

The internal experiments were small, but their lesson is broader.

As software becomes increasingly shaped by AI, prompts, policies, permissions, capability contracts, and natural-language governance, the most consequential changes may no longer be the largest diffs.

Sometimes hundreds of changed lines mean very little.

Sometimes one changed sentence alters who is allowed to act.

The engineering systems we use to govern AI will need to know the difference.

And they will need to show their work.

Evidence Basis and Limits#

• Experiment: Audit Workbench Semantic Assurance Stage 2, assessing change through five dimensions: authority, scope, intent, constraint, and validation.
• Method: Fixture-based assurance plus historical calibration across four internal repositories and five artifact classes.
• Interpretation: The calibration used a small internal corpus and does not support a universal semantic-understanding claim.