Practice
Commerce regression testing
Commerce regression testing verifies that a change to a store — a theme update, an app install, a platform upgrade, a remediation — did not break the commercial outcome that previously worked.
Conventional regression testing asks a different question
Conventional regression testing asks whether the code still runs. Commerce regression testing asks whether the buyer still ends up with the right item.
A theme update that renames a variant attribute breaks nothing a test suite would notice, and everything a buyer would.
Comparability comes first
Two inspections are only comparable if they are about the same thing. Before comparing anything, CRAB requires the same seller, the same requested product, the same mission scope and compatible method versions.
If any of those differ, the comparison is refused outright and no dimension is compared. Producing "three improvements" from a comparison whose premise is invalid would be worse than producing nothing, because somebody would read it.
The five outcomes
| Outcome | Meaning |
|---|---|
| Unchanged | Both inspections found the same thing. |
| Improved | The invariant was violated before and is positively established as holding now. |
| Regressed | It held before and is violated now. |
| Newly observed | Visible now only because CRAB gained the ability to look. Not a regression. |
| No longer observable | Established before, could not be checked this time. Neither a pass nor a failure. |
Why "not observed" is not "fixed"
This is the rule everything else rests on. A cart CRAB could not read produces exactly the same silence as a cart that now holds the right item — and only one of those is a remediation.
A fix therefore requires positive evidence that the invariant now holds. A merchant told their defect is no longer observable learns something true. One told it is fixed learns something CRAB did not establish, acts on it, and finds out later.
The mirror rule protects the merchant in the other direction: a finding visible only because CRAB acquired a new capability is newly observed, never a regression. Nobody should acquire a defect because we got better at looking.
One defect is one finding
A single wrong variant shows up in several places at once — as an identity finding, as a cart-stage result, as a verify-item result. All are true; none is a separate thing that went wrong.
Findings are therefore counted as canonical change events, with the technical views kept underneath them. Reporting three regressions to a merchant who has one defect would misdescribe both the problem and the work ahead.
For agencies
Baseline before the work, reinspect after it, and require positive evidence that the specific failure is corrected. The agency implements; the comparison is determined from evidence. An agency judging its own remediation is not assurance.
Next steps
- Check your commerce journey — request an inspection of a store you are authorised to test.
- CRAB Q&A — fifty questions on commerce execution assurance.
- The Commerce Execution Method — the method itself, as a prepublication draft.
- Commerce Execution in the Age of AI Agents — the whitepaper.
CRAB is built by VSN.