Drift — regression testing with nothing declared¶

Every other gate in Veridor costs you a declaration. Drift costs you nothing: run map once to record what your API does today, commit that matrix, and every run after that reports what moved.

Nobody writes a policy. Veridor records reality on Monday and a human reads a diff on Friday — the bargain snapshot testing makes, and the reason snapshot tests got adopted at a scale hand-written assertions never reached. The declaration still exists. It moved from write it before to accept it after, and accepting it is keeping this run's matrix file.

Drift is not a fourth verb. It is what a map means read against an earlier one, which is why it lives under map in the CLI and inside the map view in the local viewer.

What it needs¶

This is the whole configuration. There are no checks, because map evaluates none.

{
  "version": 1,
  "target": {"base_url": "https://staging.example.com"},
  "identities": {
    "owner":    {"headers": {"Authorization": "Bearer ${OWNER_TOKEN}"}},
    "outsider": {"headers": {"Authorization": "Bearer ${OUTSIDER_TOKEN}"}}
  },
  "matrix": {
    "specimens": {
      "/api/invoices/{id}": [{"label": "inv", "values": {"id": "inv-100"}}]
    }
  }
}

Two credentials, one real object id per templated route, and the OpenAPI file you already have. Two identities is the one floor that does not move — a single principal cannot express a boundary at all.

What each part buys you, measured against the demo target by flipping it from patched to vulnerable and running the same command twice:

Configuration	Drift caught the regression?
`version`, `target`, two identities	No. Every `{id}` route is unmeasured, so there is nothing to compare
…plus one `specimen` per templated route	Yes — `reach_opened`: the outsider went from refused to served
…plus a `marker` on the owner	Yes, with proof — also `marker_appeared`: the owner's data was in the outsider's response

The specimen is what makes drift work at all, and it is one line per route you care about. The marker is what upgrades "the status code changed" into "another tenant's data crossed the boundary". Neither is a policy; both are facts about your test data.

Record, commit, diff¶

veridor map --config veridor.json --openapi openapi.json --matrix veridor/baseline-matrix.json

That run records the matrix. It finds nothing, says nothing, and exits 0. Then commit the file:

git add veridor/baseline-matrix.json

Every build after that is the same command with one flag:

veridor map --config veridor.json --openapi openapi.json --baseline veridor/baseline-matrix.json

Access-matrix drift — 2 change(s), 2 that mean more access
  baseline 20 cells · current 20 cells · 20 comparable
  drift saw 60% of the grid: 12 of 20 comparable cell(s)
     were measured in BOTH runs. A quiet report below is quiet about the rest
     of it, not evidence about it.
  !! 8 of 20 cell(s) WERE NEVER PROBED, by this plan's own rules
     (skipped_mutating x8)
     Drift is blind there and will stay blind: re-running changes nothing,
     because nothing failed. Widen the plan -- a specimen, an
     --allow-mutating-method, a dropped matrix.ignore -- or read that
     half of the matrix as out of scope for drift.

  MORE ACCESS THAN THE BASELINE
    [reach_opened] GET /api/invoices/{id} other_tenant [specimen owner_invoice]
        refused -> reached
        this identity was refused here and is now served. That is more
        access than the baseline recorded; whether it is intended is not
        something Veridor can tell you
    [marker_appeared] GET /api/invoices/{id} other_tenant [specimen owner_invoice]
        absent -> present
        this response now carries 'invoice_owner''s planted marker and did
        not before. That is a measured fact about data flow, and it is the
        strongest signal in this report

  A change is an observation, not a verdict. Reach that opened may be a feature you shipped;
  reach that closed may be a broken deploy rather than a fix. Veridor reports what moved.

  Accept these as the new baseline by keeping this run's matrix: marker_appeared=1, reach_opened=1

--drift PATH writes the report as JSON. Without --baseline there is no report to write, so it is refused with exit 2 before anything is sent.

The change kinds¶

Kind	Means	Significant?
`reach_opened`	this identity was refused here and is now served	yes
`marker_appeared`	this response now carries another identity's planted marker and did not before	yes
`reach_closed`	this identity was served here and is not now. May be a fix, may be a broken deploy	no
`reach_appeared`	this identity is now served here, but the baseline never measured it	no
`marker_cleared`	this response no longer carries another identity's marker	no
`status_changed`	the status code changed without changing whether the identity was served	no
`operation_added`	this operation was not in the baseline inventory	no
`operation_removed`	this operation is no longer in the inventory — removed, renamed, or simply not probed	no
`identity_added`	this identity was not probed for the baseline, so none of its cells are comparable	no
`identity_removed`	this identity was not probed this time	no
`measurement_lost`	this run could not measure this cell — a rate limit, a server error, a timeout	never gates
`measurement_gained`	the baseline could not measure this cell, so there is nothing to compare against	never gates

Significant means more access than the baseline recorded. Those two kinds are what --fail-on-drift opened gates on.

An unmeasured cell is never a difference — under any policy, including any

measurement_lost and measurement_gained say one of the two runs did not measure a cell. That is a fact about the probe, not about the target: the cell was not measured, so there is nothing to differ. Against a target with a fixed boundary and realistic transients, treating them as differences made --fail-on-drift any fire on 100% of runs, which is a gate somebody switches off in a week.

They are surfaced as a comparison-quality number instead — the drift saw N% of the grid line — because a report that saw less than the whole matrix has to say so, or its silence gets read as coverage it does not have.

A cell neither run could measure produces no change at all. A 429 in one run and a timeout in the other is two absences, not a difference.

The gate¶

Drift produces no SARIF and no findings, and fails nothing unless you ask. One exit code does not wait to be asked: map exits 3 when its own grid measured nothing, under every policy (map).

veridor map --config veridor.json --openapi openapi.json --baseline veridor/baseline-matrix.json --fail-on-drift opened

Policy	Fails the build when
`none` (default)	never on a difference — the report is printed and `map` exits `0`, unless this run's own grid measured nothing (`3`)
`opened`	an identity that was refused is now served, or a marker appeared
`any`	anything at all moved, except the two unmeasured kinds

opened is the one worth wiring into CI. any is noisy by design — a status code changing from 401 to 403 trips it — and is there for the case where you want to review every difference before it lands.

opened does not count a cell the baseline answered with a redirect and this run serves: Veridor never followed that 302, so it never saw whether it was a denial, and counting it would fail the gate on every trailing-slash fix. An anonymous 302 to /login that became a 200 may still be an authentication bypass, so every such cell is counted in redirect_now_served_cells and in the report's headline, listed with the changes, and named on stderr under every policy but any, which fails on it instead.

--fail-on-drift without --baseline is refused with exit 2 rather than accepted and silently never gating. A gate that was never comparing anything is worse than no gate.

veridor: --fail-on-drift needs --baseline: there is nothing to compare this matrix
against. Record one with `map --matrix veridor/baseline-matrix.json`, commit it, and
pass that FILE on later runs.

A comparison that measured nothing exits `3`¶

Under --fail-on-drift opened or any, a comparison in which not one cell was measured in both this run and the baseline exits 3, not 0. A difference (1) still wins. --fail-on-drift none does not apply this rule: it exits 0 unless this run's own grid measured nothing, which exits 3 under every policy.

A baseline recorded while the target was down is the usual cause. Every cell in it is unknown, so every later comparison is unknown → something, which neither policy counts, and the gate used to exit 0 against the demo target serving another tenant's invoice (recorded in the Veridor CHANGELOG). Measured for this page on an earlier tree (before 38025ed), with the demo map configuration and specification:

Step	Exit
record a baseline with the target down (`--matrix`)	`3`: `not one of 12 probed cell(s) was measured, so this matrix records nothing`
gate the vulnerable target against it, `--fail-on-drift opened`	`3`
the same, `--fail-on-drift none`	`0`
control: a baseline recorded against the fixed target, then the same gate	`1`

veridor: not one of the 20 cell(s) in both runs was measured in both, so this
drift report compared nothing and --fail-on-drift opened could not have failed.
It is not a pass. The baseline measured 0 of its 20 cell(s); this run measured
12 of 20. A baseline recorded while the target was down, unreachable, or
redirecting every request (a login wall, an SSO proxy, a maintenance page)
records nothing: re-record it against a healthy target with `map --matrix
FILE`, review it, and commit that file.

A redirect is not a measurement. Veridor never follows a redirect, and the matrix records a 3xx as indeterminate, so nothing behind it was observed. A cell that is a redirect in either run does not count toward the cells measured in both, and a map whose grid is nothing but unmeasured cells and redirects exits 3: a login wall, an SSO proxy, a maintenance page, or http:// for an https-only host. Measured for this page: recording against a listener that answered 302 to everything exited 3, naming the 12 redirects. Before this, such a baseline recorded with exit 0 and no warning, and a gate against it could never fire (recorded in the CHANGELOG). A redirect beside measured cells, such as an anonymous 302 to a login page, is still compared, still reported, and does not trip --require-complete; a 400, 409 or 422 still counts as measured.

The drift JSON records measured_in_both_cells, each side's measured_cells and redirect_cells, and the text and Markdown reports say in words when no cell was measured in both runs. In the run above the drift document read measured_in_both_cells: 0, baseline measured_cells: 0, current 12.

Under a gate, a directory baseline is refused¶

--baseline takes a matrix file or a directory, and a directory resolves to the most recent matrix in it. Point it at .veridor/matrix and the comparison is always against your last run, which is useful while you are exploring.

Under --fail-on-drift it is refused with exit 2, and the reason is the whole mechanism: map writes into that same directory. So the directory form ratchets. The run that catches a regression records it as the newest matrix, and the re-run everybody does next compares the regression against itself, reports Nothing moved, and exits 0 with the regression still live.

veridor: --baseline .veridor/matrix is a directory, which resolves to the NEWEST
matrix in it -- and this run is about to write one more. Under --fail-on-drift that
ratchets: the first run catches the regression and exits 1, and a re-run compares the
regression against itself, reports 'Nothing moved' and exits 0 with the regression
still live. A gate compares against a FIXED, reviewed artifact.

The refusal happens before the run writes anything, and before it probes — a mistyped path should fail in a second, not after a full probe of somebody's API.

Anything that did not change. An endpoint that should be admin-only and has been open to everyone since before your first run never moves, so it never appears here. Drift catches the regression you introduced. Finding what was already wrong is what the access matrix, the candidates, and a declared run are for. "Nothing moved" is reported as exactly that, never as a clean bill of health.

Anything map did not probe. On a CRUD specification the mutating half of the inventory is skipped by default, so those cells are unknown in the baseline and in every later run — which means drift can never report more access there, only measurement_gained. On the demo specification, 5 operations across 4 identities, that is 8 of 20 cells: drift speaks about 60% of the grid, and says so above its own counts. This is a scope choice, not noise and not a tuning problem — re-running changes nothing, because nothing failed. Widen it with --allow-mutating-method against an environment you can recreate, or read the writes as out of scope for drift and cover them with a declared run.

Any cell the baseline itself could not measure, a redirect included. This one is not fixable by tuning, and it is the reason detection is not a constant. Every later comparison on such a cell is unknown → something, which is never more access. A baseline recorded through one bad minute is blind there forever.

Veridor cannot repair that, so it refuses to hide it. map warns when the matrix it just wrote would make a poor baseline, and any drift report against a blind baseline says so above its own counts — in the words THE BASELINE NEVER MEASURED N of M cell(s). Drift is BLIND there. The two blindness warnings carry opposite advice: a cell lost to a rate limit is worth re-recording against a quiet target, and a cell this plan's own rules never send will be byte-identical forever, so only widening the plan changes it. That is why the example above says re-running changes nothing, because nothing failed.

A diff whose two matrices name different targets is called out above the changes, because that compares deployments rather than changes.

What the numbers are¶

Two false-alarm rates, measured rather than asserted, with scripts/noise_floor_study.py against scripts/noisy_server.py — an API whose authorization boundary never changes and whose everything else moves like a real one. Every change reported against it is a false positive by construction.

	Runs that fired
`--fail-on-drift any`	0 of 90
`--fail-on-drift opened`	0 of 90

0 of 90 is not 0%. Ninety runs that did not fire bound the false-gate rate below roughly 4% with 95% confidence (Wilson, [0.0%, 4.1%]). They cannot establish that it is zero, and no number of clean runs could. All ninety are one synthetic server at one noise profile; a real API with a different failure shape is not covered by this measurement.

Detection is stated separately and on purpose — it is not a false-alarm rate, and adjacent rows invite a summary neither number supports. Over the same seeds, a planted boundary opening on one fixture was caught on 87 of 90 runs (96.7%, 95% CI [90.7%, 98.9%]). That is a per-run reliability figure for one regression on one target, conditional on a baseline that measured the cell. Where the baseline is blind, detection is 0% and no amount of re-running changes it. It is not a detection rate for authorization bugs in general, and nothing here measures that.

In CI¶

Drift needs an earlier matrix, and a CI runner has no .veridor/ from last week. Commit the baseline to the repository:

veridor map --config veridor.json --openapi openapi.json --baseline veridor/baseline-matrix.json --fail-on-drift opened --matrix current-matrix.json

--matrix writes this run's matrix, so a failing build gives you the file to review and commit. Accepting a change is then a reviewable diff in a pull request: copy the new matrix over veridor/baseline-matrix.json and commit it with the change that caused it. A reviewer sees this PR opens GET /api/invoices/{id} to other_tenant next to the code that did it.

Caching the file between runs works too, but a cache miss silently means no baseline, and a committed file cannot go missing without somebody noticing.

The packaged Action does this as mode: drift, where fail-on-drift defaults to opened rather than none — selecting that mode in CI is itself the request for a gate. A matrix-baseline that shares no measured cell with the run fails the step with exit 3, and its annotation names drift's causes. See the GitHub Action. Any other CI that runs map --fail-on-drift gets the same exit code, so a job that treats only 1 as failure would pass a comparison that measured nothing: fail on every non-zero code.

Next¶

map — the command drift is a mode of, and the matrix it diffs.
run — the declared gate that catches what was already wrong.
exit codes — including map's 1, which only drift can produce.