ADR-0006: One event stream, and every interaction persisted

Status: Accepted, amended at M1.4, M1.6, M1.7, M11.1, M9.1 and M11.3, and in #325, #327 and #334, and on 10 and 19 October 2026 Date: 2026-09-11 (amended 2026-09-12, 2026-09-13, 2026-09-15, 2026-09-22, 2026-09-23, 2026-09-24, 2026-09-27, 2026-09-30, 2026-10-10, 2026-10-19)

Context

Two requirements point at the same mechanism.

Reporting: RESTest 1.x implements oracles as REST-Assured filters injected into generated code, and exports results by shelling out to a 19 MB Allure distribution committed into the repository. Adding an output format means touching the execution path. Schemathesis, by contrast, ships six output formats without the engine knowing about any of them, because the engine emits one uniform stream of events and every reporter is an independent consumer.

Future semantic oracles: the maintainers intend to add oracles inferred from sets of requests and responses, or from the specification. An oracle that only ever sees one interaction at a time cannot express an invariant or a metamorphic relation. Retrofitting that later would mean rewriting the oracle engine.

Decision

One event stream. The engine publishes typed events — test case planned, request sent, response received, failure detected, phase finished, run finished. Reporters, metrics collectors, feedback listeners and oracles are all independent subscribers. The engine knows none of them.

Two oracle kinds.

Oracle examines a single Interaction.
CorpusOracle examines a queryable set of interactions — the whole run, one operation's slice, or a sliding window.

Each oracle declares what it consumes, so the engine can schedule it correctly and restest explain can describe it.

~~Every interaction is persisted~~ Amended at M1.7: every interaction is persisted when a run is asked to keep them. A run keeps nothing by default. What is kept, when it is kept, is unchanged — exact request and response bytes, timings, the test case that produced it, and the provenance of each parameter value. ~~SQLite by default, NDJSON as an alternative.~~ Amended at M1.4: SQLite, and only SQLite. See the amendments below.

restest recheck <run> re-evaluates a stored run against a different oracle set, offline, with no API calls.

Consequences

Adding an output format is a new class implementing one interface.
A researcher can try a new oracle against a corpus they already have, in seconds, reproducibly, without touching the API under test. This is the workbench the deferred semantic-oracle work needs, and it is built years before that work starts.
Post-hoc analysis becomes honest: the raw evidence is on disk, not summarised away.
Storage cost. A one-hour run against a chatty API produces a large database. ~~Mitigated by configurable response-body truncation and by the store being replaceable.~~ Amended at M1.7: measured. Body truncation is not the mitigation — bodies are 6% of what is stored. Not writing the file unless asked is.
A small runtime cost per interaction for publishing and writing. ~~Measured by the overhead regression test (ADR-0009).~~ Amended at M1.7: measured there instead, because the whole pipeline ran for the first time at M1.7 and M6.2's regression test is milestones away. As first built it was not small — a third of a run — and that turned out to be the write path rather than the cost of persisting.

Alternatives considered

Reporters called directly by the engine. What 1.x does. Every new format touches the engine.
Only single-interaction oracles, with corpus analysis bolted on later. Rejected on the evidence of how expensive "later" was for 1.x's stateful support.
Keeping results only in memory. Cheaper, and forecloses recheck, corpus oracles and any post-hoc analysis.

Amendment (M1.4)

Date: 2026-09-12

One store, not two: SQLite. NDJSON is a report format, not a second container.

Why

The original decision named NDJSON "an alternative", and building it at M1.4 would have meant two implementations of one interface, doing the same job, maintained in parallel for the life of the project. They do not complement each other: whichever one a run uses, the other is idle.

The roadmap already places NDJSON where it earns its keep — M3.5, among the report formats, beside HTML, JUnit XML and HAR. That is what design principle 8 (open formats out) actually asks for: that a finished run can leave RESTest in a format anybody can read, not that RESTest keep two kinds of database.

They are also not equally good at the job the store exists to do. Everything downstream of the store asks it questions repeatedly — restest recheck (M3.3), corpus oracles, reports. SQLite answers them from an index; a plain-text file answers them by parsing itself from the beginning, every time, and holds no more than a run's worth in memory while doing it. A run killed halfway leaves a database consistent and a text file with half a line in it.

This is the same trade ADR-0007 made at M1.2 when it went back to one parser backend, for the same reason stated there: "One well-tested backend is simpler to build, simpler to keep passing against the golden corpus, and simpler for a reader to reason about."

How

restest-store has exactly one InteractionStore, backed by SQLite: one file per run, the facts worth filtering on as indexed columns, and the interaction itself stored beside them as a JSON document.
There is one serialised shape in the project, defined in one class (InteractionDocument), and M3.5's NDJSON report writes that shape rather than inventing a second one. Note what this does not yet settle: restest-report may not depend on restest-store under the current layer rules, so M3.5 must either take that dependency deliberately (a rule change, argued at the time) or move the document shape into restest-core — which can hold it without gaining a JSON library, because building a JsonValue needs none; only turning it into text does. Whichever is chosen, the requirement is that no second mapping from an interaction to JSON is written.
The InteractionStore interface stays, with one implementation behind it, for the reason it was introduced: it is what M3.3 and the corpus oracles consume, and replacing the container later is a change to one module.

Consequences

Half the code and half the tests for the same capability, and one on-disk shape to keep correct.
A stored run is readable without RESTest by anything that reads SQLite — sqlite3, a notebook, a spreadsheet — and the document column is readable by anything that reads JSON. "Open formats out" is satisfied at M1.4, and satisfied again as a file anyone can grep at M3.5.
The tool now depends on a database driver that carries native libraries. The GraalVM native binary at M7.3 will need configuration for it. This is routine, and it is the price of not having a pure-Java container; it is recorded here so M7.3 does not meet it as a surprise.

Amendment (M1.6)

Date: 2026-09-13

One way of turning a value into JSON text, and it lives in restest-core. The M1.4 amendment above left this open in as many words; M1.6 is where it came due, and it came due a milestone earlier than expected.

Why

io.restest.store.Json was a thin adapter over jackson-core, kept at the store's edge because the store was the only thing that needed it. At M1.6 two more parts of the tool need it and neither can see restest-store: the JSON report has to write a document, and the schema oracle has to write the location of a schema it hands to the validator. The layer rules are right and were not going to be relaxed for this.

That leaves the choice the M1.4 amendment already framed — take the dependency deliberately, or move the shape into restest-core — with a third thing now settled by the same move. InteractionDocument goes with it. The amendment's own requirement was that "no second mapping from an interaction to JSON is written", and M1.6 is the first increment with a reason to write one: a fault in a report quotes the request that caused it. It quotes it through the same class the stored run uses, so a fault in a report and the same fault in a stored file are the same text.

How

io.restest.store.Json becomes io.restest.core.json.JsonText, and io.restest.store.InteractionDocument becomes io.restest.core.json.InteractionDocument. Neither changes otherwise; they gained a test of their own, which the second of them had never had.
io.restest.core.json.JsonException replaces the store's exception in both. SqliteInteractionStore catches it and says which stored interaction was damaged, exactly as before.
restest-core gains jackson-core and restest-store loses it. The architecture rule confining the JSON library to one module is not dropped; it names a different module.

Consequences

restest-core is no longer literally dependency-free, which was a stated property of it. What it gains is a streaming reader and writer with no transitive dependencies of its own, and no object mapper; CLAUDE.md's description of the module says so rather than leaving the line to be found false. "No parser" there was always about the OpenAPI parser, and that is still true.
One answer in the project to "what does this value look like written down", instead of one per module that happens to need it. M3.5's NDJSON report writes the same shape rather than a second one, which is what the M1.4 amendment asked for.
Coverage moved with the code: InteractionDocument had been covered only through the store's tests, and counted against restest-core the moment it arrived there. It now has its own test, which is where a shape two separate features depend on should have been tested all along.

Amendment (M1.7)

Date: 2026-09-15

A run keeps nothing by default. --store turns keeping on, and then everything is kept, in full.

This reverses, for the default case, the sentence this ADR is named after. The capability is unchanged and the promise is unchanged when it is asked for; what changes is that it is now asked for. The reasoning is below, because a reader two years from now will find the title and the default disagreeing and deserves to know that was deliberate.

Why

M1.7 assembled the whole pipeline for the first time, which is the first time any of this could be measured rather than assumed. Every number below comes from restest run against the pinned webfuzzing/wfd-swagger-petstore image the smoke test uses, in Docker on the same machine (macOS, Apple silicon), seed fixed:

docker run -d -p 18080:8080 webfuzzing/wfd-swagger-petstore@sha256:\
26951cb671d013c44ea978aa659afe89d438d0fb431411c127ead2633dc876a4
./restest run http://localhost:18080/api/v3/openapi.json --url http://localhost:18080/api/v3 \
  --budget 60s --seed 20260914 --out /tmp/run

What a run costs on disk.

Budget	Requests	Faults	`run.sqlite`	`report.json`
10 s	67,761	3,989	97 MiB	1.8 MiB
60 s — the default budget	462,033	27,181	661 MiB	1.8 MiB

The report does not grow with the budget, because M1.7 already caps it. The store is 98.5% of what a ten-second run writes and 99.7% of what a sixty-second run writes. Whatever the storage problem is, the report is not it.

Where the store's bytes go. Measured rather than assumed, and the assumption would have been wrong. Of the 97 MiB, 89 MiB is the table, 8 MiB the five indexes, and 14.2 MiB of the table's pages is slack — space inside 4 KiB pages too small to hold another 1.4 KiB row. The documents are 60.7 MiB, a mean of 940 bytes per interaction: response headers 22.8%, request headers 19.9%, JSON keys and punctuation 18.3%, parameters with their provenance 9.0%, method and URL 6.8%, response body 6.2%, and the rest the two ids, the times, the status line and the operation.

So the question this amendment was opened to answer — full bodies, truncated bodies, or bodies de-duplicated by hash — was the wrong question. 88.2% of interactions carry a body; there are 4,022 distinct bodies among 59,790; storing each distinct one once saves 3.4 MiB, 3.5% of the file. The repetition is in the headers: 26 MiB of the 60.7 MiB is header lists, of which there are 2,164 distinct in 67,761 interactions — 41,873 of them the same four request headers, byte for byte. Stored once, all of them are 0.5 MiB.

What a run costs in time. Six configurations of the same ten-second run, three repetitions each, medians below. Differences under about 5% are noise at this sample size:

Listeners	Requests	Idle
everything, as built today	75,750	18.5%
everything, with the write path fixed	111,850	0.9%
no store; oracles and reports only	114,060	0.5%
store only, as built today	92,185	13.0%
store only, with the write path fixed	114,897	0.4%
no listeners at all	114,951	0.3%

Three things fall out of that table, and the second was a surprise.

As built, the store costs a third of the run — 75,750 against 114,951 — and is essentially all of the idle time. ADR-0015 says where that idle comes from: "Pausing shows up as idle time, which is the honest place for it: the tool was not testing, and it was our own work that stopped it." This is that work. Every oracle and every report together cost about 4%.
Once the write path is fixed, the store costs nothing measurable. 111,850 with everything including the store, against 114,060 with no store at all: the same number. The third of a run was never the price of persisting; it was the price of committing a transaction per row.
Fixing the time therefore makes the disk problem worse, not better. Per interaction the file shrinks 12%, from 1.46 KiB to 1.29 KiB. But the tool now tests 47% more in the same budget, so a ten-second file grows from 108 MiB to 141 MiB, and the default budget goes from 661 MiB to roughly 850 MiB.

What the report keeps. The cap works; it is on the wrong axis. Those thousand findings, measured, are two distinct summaries across two operations — 99.8% of the file is near-identical copies of the same two faults, because the allowance is spent in arrival order and the first fault to repeat takes all of it. A fault kind first seen in the ninth second is not in the file at all. It is also a bound on count and not on bytes: a Finding holds its whole Interaction, and a thousand of those against an API answering with large bodies is up to a gigabyte held until the run ends.

Why a default of keeping nothing is defensible. Because nothing in an ordinary run consumes the store. ADR-0013's fifth decision already settled that generation never reads it — the dictionary of observed values and the memory mutation works from are listeners with their own bounded in-memory index, and restest-gen's module-info.java cannot even see restest-store. Everything that does consume a stored run — restest recheck, restest replay, corpus oracles over a whole run — is something a person asks for deliberately, after the fact. A default that keeps 850 MiB for the benefit of a command nobody ran is the wrong way round.

How

Keeping is off unless asked for. A flag on restest run turns it on; without it no database is opened and no file is written. The flag's spelling and the file contract belong to ADR-0015, which is amended alongside this.

When it is on, everything is kept, in full — the original decision, unchanged. Exact request and response bytes, timings, the test case, and the provenance of every value. No store-level truncation and no de-duplication:

Bodies are stored whole, with the engine's existing ceiling the only one. The engine already keeps at most 1 MiB of a reply and records how long the whole thing was (Payload.partial, Payload.truncated()), so a stored run already says "we kept 1 MiB of 12 MB" rather than pretending. A store-level cap low enough to matter would cut ordinary bodies, and a truncated JSON body is not a smaller fact but a false one: recheck at M3.3 re-runs the schema oracle over what is stored, and half an object parses as broken JSON. It would invent F101 faults the live run never reported, which is the opposite of what re-checking a run is for.
Nothing is de-duplicated by hash — not bodies, where there is nothing to win, and not the headers, where there is. Interning the header lists would take about 26% off the file at the price of a row that is no longer one interaction: read by sqlite3, a notebook or a spreadsheet it would become a fragment needing a join, and M3.5's NDJSON and HAR reports would each have to do the joining. It is also competing with gzip, which takes the measured file from 101.8 MB to 14.3 MB, 7.1 times, precisely because the redundancy is across rows. Spending the self-contained row to claw back a quarter of something that compresses sevenfold anyway is a bad trade. Per-row compression of the document column was measured too and is worse than it looks: 54% row by row against 10% over the column as a stream, and it would cost the property the M1.4 amendment put in writing, that the column is readable by anything that reads JSON.

The write path is fixed whether or not the default changed, because a run that did ask to keep its evidence should not pay a third of its budget for it:

Page size 16 KiB, issued as a plain statement on a new file before the journal mode. This one fails silently otherwise: a page size cannot be changed once a database is in WAL mode, and SQLiteConfig.setPageSize is applied after the driver has turned WAL on, so it is accepted and discarded with no error. Verified both ways — 16384 before WAL, 4096 through the settings object. Whoever implements this should assert the page size the file actually has rather than trust the setting.
One prepared statement, reused, rather than one prepared and discarded per interaction.
Inserts batched inside a transaction, flushed by count or by elapsed time, whichever comes first, and always before any query and on close. The count bound keeps memory flat. The time bound keeps the store's own promise that a run can be read while it is still going on — and its exact strength is worth stating, because the obvious reading is wrong: the deadline is noticed when the next interaction arrives, so a reader outside the process is always at most one interaction behind, which is a quarter of a second against a busy API and a minute against one answering once a minute. A background thread would tighten that, at the price of a thread per store to start, join and reason about, for the benefit of somebody watching a run that has already gone quiet. What is also given up is that a run killed outright — not closed, kill -9 — loses the interactions since the last flush instead of none.

The report keeps an allowance per operation and per fault kind, not the first thousand of anything, and it trims the bodies it quotes.

Exact counts for every (operation, fault kind) pair, all of them written. This is the part that must be complete, and it can be: the table has at most one row per pair the specification allows, so it does not grow with the budget. Nothing is counted approximately and no fault disappears from the totals.
The first few of each pair written out whole — the request, the reply, the provenance and the curl command.
Only the first few kilobytes of any one body are quoted, with how long the whole body was recorded beside it. This is the decision everything else rests on, and it came from asking why the mechanism had grown so complicated. Measured on one real run, a write-up of the same kind of fault ranged from 1.7 KB to 158 KB — a factor of 96, all of it the reply body. While that is true, counting write-ups bounds nothing, which is why an earlier draft of this amendment needed budgets counted in bytes, a second "naming" layer with two limits of its own, and a guard against re-serialising a fault the budget had already refused. Trim the body and all of that goes away: write-ups become roughly equal in size, so counting them is a real limit and the file needs no other. Three integers replaced two integers, two byte budgets, a size-measuring pass and the bookkeeping that went with it — including the one place a defect had hidden.
What was rejected for the store is right for the report, and the asymmetry is the point. The store may not trim a body because recheck re-judges the stored bytes and half a JSON object is not a smaller fact but a false one. A report is read, never re-judged. --store still keeps every body whole.

Every attempt is accounted for, not only the faulty ones. The report says how many attempts came back in each family of status code, how many had nothing found wrong with them, and which exact codes were commonest. This is not a refinement: a run where nine replies in ten are refusals is a run whose requests were the problem, and a report that only ever mentions faults cannot say so. It is also the number that catches our own generator breaking, which is the failure that put RESTest 1.x mid-field in 2026. Note that it counts attempts where everything else counts faults, and the two never add up — one attempt can be found wrong in several ways, and most in none.

Faults are also counted by the family of status code that carried them, crossed with the fault kind. A status code is what a developer already works in, and the cross-tabulation makes one thing visible that a list of fault kinds cannot: how many faults came back as a perfectly ordinary 200, invisible to anybody watching their own logs for 500s. By family rather than exact code, because as more kinds of fault are looked for an exact-code list becomes a long tail that summarises nothing; the exact codes are counted too, and listed beside it. An attempt that got no reply at all has no code and gets a name of its own rather than being lost. How this is presented in every report format is ADR-0016's; what is counted is here.

This is not deduplication or clustering, and the distinction is the whole point. That work, deferred out of v2.0 in docs/DESIGN.md, means deciding that two failures are the same underlying bug — inference, with a notion of sameness that has to be argued for and evaluated. A quota fills a fixed allowance per pair of fields already recorded on every finding. It asserts nothing about whether two findings share a cause, merges no counts, and changes no total. If that inference is built later it reads the stored run, as the deferred list says; it is not started here.

A run says what it wrote, and how big it is. One line beside the two it already prints.

Consequences

A default run is 51% more productive: 114,951 requests against 75,750 in the same ten seconds, with idle falling from 18.5% to 0.3%. That is the headline number M1.9 and M8 are compared on, and it arrives from deleting work rather than adding any.
A run that asks to keep its evidence now pays nothing measurable for it — 111,850 against 114,060 without a store — and writes 12% less per interaction than before.
The sharp edge, stated rather than buried: a default run cannot be re-examined afterwards. If something surprising turns up, it has to be run again with the flag. Today that works, because the seed reproduces the run exactly. From M2.7 it stops working for any strategy with a memory: ADR-0013's seventh decision says those are reproduced "by replaying the stored run", not by the seed, and RandomTestCaseGenerator's own comment says the same. A surprising result in a default run will then be irreproducible by any means. M2.7 owns that problem and should decide it deliberately — the cheapest answer is for a run using such a strategy to say so when no store was asked for, rather than letting the evidence be discovered missing.
Nothing in generation is affected, now or as planned: ADR-0013's fifth decision, the layering rule confining Store to Cli, and restest-gen's module declaration all say the same thing three ways. The one place this could still leak is M2.7's dictionary disk cache, where what fills it is not decided anywhere; if it is filled by harvesting a stored run rather than by a listener, this default becomes load-bearing for it.
restest recheck at M3.3 re-examines exactly what the live run examined, because no oracle is ever handed a body the live run did not have — the thing truncation would have quietly taken away.
A stored run stays one row per interaction, readable by sqlite3 and by anything that reads JSON, and compresses about sevenfold with tools everybody already has. Design principle 8 is unchanged rather than traded for a quarter.
The title of this ADR is now about a capability, not about every run. "Every interaction is persisted" remains true of a run that asks; it is false of a run that does not. This amendment is the record of that, rather than a later reader finding the code and the title disagreeing.
M1.8's campaign problem largely dissolves: an evaluation loop that does not pass the flag writes nothing and goes faster, and one that wants evidence opts in per invocation and owns the disk.
Interning the repeated headers is now measured rather than guessed: 26% of the file, at the price of a row that no longer stands alone. If a later milestone wants it, the number and the objection are both here.
The report is a run report now, not only a fault report. It says how the API answered across every attempt, which is a question nobody could put to it before. It also grows with the number of operations and kinds that went wrong rather than with the number of faults: measured against a public demonstration server, eight pairs went wrong and the file is hundreds of kilobytes; against a containerised pet shop where one pair went wrong, tens of kilobytes, against 1.8 MiB before.
The report's mechanism is smaller than it was, despite doing more. Trimming bodies removed the byte budgets, the second layer of detail, the size-measuring pass and the guard that went with them. Every limit in the report is now an integer compared with <.

Alternatives considered

Keep persisting by default and make it smaller. What this amendment set out to do. Every mechanism measured — truncation, body de-duplication, column compression — is worth between 3% and a quarter, against a default cost of 850 MiB. None of them is the difference between a tolerable default and an intolerable one; not writing the file is.
Truncate response bodies in the store. Addresses 6.2% of the problem while making recheck report faults that never happened.
De-duplicate identical bodies by hash. 3.5% of the file. The intuition — that thousands of replies are byte-identical — is right; the bodies are simply not where the bytes are.
Keep only failing interactions. 5.9% of the interactions, so about 8 MiB per ten seconds. It survives as a coherent middle option, and it forecloses every corpus oracle on the roadmap: an invariant is a statement about what the API did in general, and cannot be checked against a corpus filtered down to what already looked wrong.
Leave the default on and rely on --out plus housekeeping. Puts the cost on everyone by default to spare the minority who want it a flag. The reverse is cheaper for both.
Write every finding to the report. What M1.6 did. At 27,181 faults in the default run, each carrying its whole attempt, the report becomes the reason the run ends.
Bound the report in bytes instead of trimming bodies. Built, measured and then removed inside this same increment, which is the honest way to record it. It works, and it costs a size-measuring pass on the event-delivery thread, a second layer of detail with two limits of its own, and a guard against re-serialising a fault the budget has already refused — a guard whose absence was a real defect found in review. Trimming the body attacks the cause rather than the symptom: the variance was never in how many faults there were, it was in one field of one of them.
Name every fault beyond the allowance. The second layer of detail, also built and then removed. It showed the spread of what was tried, which is worth something, and it cost two more limits and a tenth of the file for something a kept run answers completely. The allowance and the exact counts carry the rest.

Amendment (M11.1)

Date: 2026-09-22

The YAML reader joins the JSON one in restest-core, for the same reason and by the same rule.

The M1.6 amendment moved the JSON library into restest-core when a third part of the tool came to need it, and argued that one answer to "what does this value look like written down" is worth more than three. The reader of the files a person writes has now reached the same point.

Three files are written by hand for RESTest: a list of values (ADR-0020), a plan (ADR-0023), and — from M11.1 — a file of settings (ADR-0025). All three are YAML, and all three are read by parts of the tool that cannot see one another: the lists and the plan in restest-gen, the settings in restest-cli, because ADR-0025 assembles the four layers there and nowhere else. The small class that reads them, YamlText, therefore moves to io.restest.core.json beside JsonText, and becomes part of this module's published surface.

It is the same job as the one already in that package. JSON is a subset of YAML, the class turns the text of a hand-written file into JsonValue, and the two traps it solves — a key written twice, and YAML's older rules turning NO into false — are traps for every one of the three files equally.

What this changes, and what it does not. restest-core now carries two third-party libraries rather than one, both behind io.restest.core.json, and restest-gen carries none of its own for this purpose. The confinement rule in the architecture tests moves with the library rather than being dropped: it read only restest-gen and restest-spec may see the YAML reader and now reads only restest-core and restest-spec, still two modules, restest-spec because the library arrives there through the specification parser on its own account. Nothing else about this record changes.

The alternative, and why not. restest-cli could have been added to the confinement rule as a third module and read the file itself. That would have meant three modules holding the library and either a third copy of those two traps or a shared reader living somewhere — and the somewhere is this module, which is where the change ends up either way. Exporting the reader from restest-gen instead would have had the command line read its settings using a class from the module that invents request values, which is a dependency nobody could explain from the names.

Amendment (M9.1)

Date: 2026-09-23

The phase event exists. The decision above lists phase finished among the events a run publishes, and nothing published one until the first round of a run gave a stretch of it a purpose of its own. RunEvent gains PhaseStarted and PhaseFinished, the second saying whether the stretch was cut short, and the two listeners that handle every kind of event - the console and the JSON report - handle both. The run announces a stretch; the reports work out what it achieved, matching each request to the stretch it was sent in, so an answer that arrives late still counts where it belongs.

Waiting for the listeners is a mark on the queue, not a count. EventStream.awaitDelivery waits until everything announced so far has been heard, by queueing a mark that is let go when the delivering thread reaches it. The first round needs it - a step must not be built before the answers to the step before have been taken in by whoever keeps what the API returns - and counting what is still waiting cannot answer that safely, because a listener that announces something while it is being told something can make the count come out even with the last reply still queued. The record is ADR-0026.

Amendment (naming what a run skips)

Date: 2026-09-23

What a run cannot test is announced, and both reports name it. RunEvent gains OperationSkipped: an operation the run will not try, with the reason. The command announces one for each operation the generator could not build a request for, before anything is sent, in the order the generator keeps them - path by path as the document declares them, and within a path in the fixed order of methods the parser reads them in. Since the operations the parser drops, those are announced too, and first. ADR-0007 promised from the start that "the run summary states how many operations were skipped and why", and ADR-0015 that a run "reports what it skipped" - but the reasons lived in the generator and reached a person only when nothing at all could be tested, and then only the first of them. A run that could test eleven of twenty operations printed the count and nothing else, and report.json held neither the operations nor the reasons.

The JSON report writes them as skippedOperations, every one, in that order: operation and reason, and an empty list when there are none. It is not bounded, for the reason the tallies per operation are not: it is as long as the document, never as long as the run. The console names the first report.skippedOperationsShownOnTheConsole of them, five unless somebody says otherwise, each with its reason, and counts the rest, pointing at the report.

Why the summary, after its verdict. Because they qualify the verdict: "no faults found" says nothing about an operation that was never tried, and the list says which ones that is. The place a reader might look first, under the count at the top, has a second reason against it. The lines a run begins with, that count among them, are printed by the command itself, and the one line a listener prints there, naming the address, is written from the stream's own thread - so the two already reach the screen in either order, and a line of one can land inside a line of the other. Measured before this change: 90 of 200 runs of one command in one process, and 1 of 25 separate launches, printed the count first. A listener naming the operations there would join that race, and the command naming them itself would give the screen a second way of learning what the stream carries. The race is older than this and is fixed separately. Fixed since: see one writer for the screen, below.

Two kinds of operation are not announced. One the plan left alone was not skipped: somebody asked for it to be left alone, and the count at the top already says how many. And one the parser could not read at all never becomes an operation for the generator to judge: it is in neither the count nor report.json, and the screen mentions it only as one of the parts of the document that could not be read, of which it names the first five. That is a gap of the same kind, left for a change of its own. Closed since, by the amendment to ADR-0007 on the operations the parser drops.

Amendment (one writer for the screen)

Date: 2026-09-23

While a run lasts, only the console report writes to the screen. The command used to print its own account of the run - how many operations can be tested, the seed, the budget, what could not be read - straight after announcing that the run had begun, while the console report printed the line naming the API when that announcement reached it. The two wrote at the same time with nothing to order them, and a line of one could land inside a line of the other, ...at http://localhost:11 of 4 operations can be tested.... How often depends on the document and the machine. Measured on v2 as it stood before this (9612aabc): one command run 200 times inside one program, against a document of two operations, printed the count above the address 90 times; separate launches of that command did it once in 25; separate launches against the four-operation pet shelter with a budget of 50 milliseconds did it 17 times in 25.

Now the command writes its account into a string before the run begins and hands it to the console report when it builds it, and the report prints it straight under its first line. Three other ways were weighed.

Wait for the report before printing. EventStream.awaitDelivery exists for ordering of this kind, but it needs a bound, and the natural one, the deadline, has already passed in a run whose budget is spent before testing starts - exactly where the order would go back to chance. A fixed bound would be one more decided number for the settings, for a wait nobody would want to tune.
Put the account on the stream as an event, the way the operations a run skips are, and let the report render it. What it would carry is the command's own words about itself: it names command-line options, --store and --print-settings, that nothing outside the command-line module knows about, and no other listener has a use for it. So it stays the command's text, handed over whole - which is not a second way of learning something the stream carries, since the stream does not carry it.
Move the line naming the API into the command. That would also leave one writer, but the report would stop printing the start of a run for anybody else who uses it, and the start of a run is something the stream does carry.

The rule, and its one exception. From the moment a run starts until its summary is printed, the screen belongs to the report. The exception predates this and is left as it is: if the reports have still not caught up half a minute after the run ends, the command stops waiting for them and says so itself, and those lines can then land among the summary's.

What the command prints before a run starts and after it ends - a list of values it could not read, where the report was written - is unchanged, because nothing else is writing then.

Amendment (naming the lists a run holds)

Date: 2026-09-27

Which lists of values a run holds, and which it could not read, are announced, and both reports name them. RunEvent gains DictionaryRead — a list's name and the file it came from — and DictionaryRefused — a file or directory that gave the run nothing, and why. The command announces them beside the operations it will skip, before anything is sent: every list held first, RESTest's own among them, then every refusal, each in the order the files were read.

Until now a list that could not be read was one line on standard error as the run began, and nothing else. 8.3 lost every one of its dictionaries that way without anybody noticing, because a run made without a list finishes exactly like a run made with one that did not help — ADR-0020's amendment has the story. So:

The console repeats every refusal at the very end of the summary, after the operations it could not test, and says nothing about lists that were read.
report.json gains a member dictionaries after skippedOperations: read, one {name, from} per list held, and refused, one {from, reason} per refusal — both lists, empty when there is nothing to say.

The same argument as for skipped operations puts the refusal after the verdict rather than at the top: it qualifies what the run found. Held is not drawn on — a plan decides which lists each kind of request asks — and read claims only the first.

Amendment (M11.3)

Date: 2026-09-30

A kept run holds every exchange whole, except the keys the run was handed. The decision says the exact bytes of every request and reply are kept. When a key is handed over (ADR-0029), every exchange first passes through a door, on its way back from the engine and before the loop announces it. The door replaces each appearance of each key with text naming the key, such as REDACTED-AUTH.api_key, so nothing the store, the reports or any listener sees holds a key. Nothing else in an exchange changes, and a run handed no key is kept byte for byte as before.

Three things follow:

A stored request cannot be sent again as it is. A replay (3.5) is handed the keys again and puts each back where its replacement names it. Judging the stored replies again (3.3) gives the verdict the run gave, since the rules judged the same hidden replies.
A body a key was hidden in may not be the length its Content-Length header says. Nothing compares the two today.
The layout of the store does not change.

Amendment (10 October 2026)

The schema fault is F200. Where this record speaks of F101 faults for replies off their schema, it uses the numbering of Faults 0.7.0; since Faults 0.8.0 that fault is F200 (ADR-0014's and ADR-0016's amendments).

Amendment (19 October 2026): replies a check could not judge

RunEvent gains ReplyNotJudged: a reply some checks could not judge, and which, each with the reason it gave. The listener that runs the checks announces it after all of them have judged the reply, once for the reply however many checks could not judge it, and not at all for a reply every check could judge. Both reports count it, in one place for both, and say how much of the run could not be checked (ADR-0014's amendment of the same day). The stored run does not keep it: judging the stored replies again gives it again.

It is announced on the one thread that delivers everything, and on some APIs nearly every reply brings one: 287,676 in a minute against restcountries, whose document gives no shape for any reply. Each is small, and every listener but the two reports passes it over in one comparison. Measured on 19 October 2026 on swagger-petstore, restcountries and rest-scs, three runs of a minute each before and after, the same seed for each pair, on battery power: the run sent on average 2% to 4% fewer requests, the nine pairs differing by between −21% and +16%, four up and five down, and the time with no request in flight stayed between 0.5% and 0.8% of the run. Three runs do not tell that apart from what two runs of the same build differ by.

Amendment (25 October 2026): every operation, and what the document could not give

RunEvent gains DocumentRead and OperationSetAside. DocumentRead carries every operation the document describes that could be read, path by path in the document's order, and every part of the document that could not be used as it is written. OperationSetAside names one operation the plan asked the run to leave alone. The command announces both as the run begins, inside the block that guarantees RunFinished: DocumentRead first, then the lists of values and the operations skipped as before, then one OperationSetAside for each operation the plan sets aside. Neither is stored; the console passes both over, since the command prints the same facts itself before the run begins.

Why. The HTML report planned next shows every operation beside what happened to it, and a report shows only what report.json holds (maintainer, 25 October 2026). Until now report.json named only the operations something went wrong on, by name alone: an operation that answered every request well and one the run never sent anything to were both absent, and there was no method or path to show for any of them. Any report that lists every operation, passing ones included, needs the same list. An event rather than an argument handed to the report when it is made, because what the document describes is part of the run, and every listener can hear it; what the command itself was handed - the plan and the file of settings as they were named, whether the run is kept, the keys nobody gave - is the command's own account, and is handed to the JSON report when it is made, as its settings are, which is what one writer for the screen asked of the command's account.

Set aside is not skipped. An operation the plan leaves alone is never announced as skipped, as naming what a run skips decided, and is announced on its own so that a report can keep the two apart.

What the JSON report writes from them, each always there, an empty list when there is nothing: operations (every operation, its state - tested, notReached, couldNotBeTested or leftAloneByThePlan - and its requests by class of status code, which add up to the whole run's), documentIssues, keysNotGiven and run, beside verdict, worked out by the rule the exit code follows (ADR-0015's amendment of the same day). A report that never hears DocumentRead lists the operations requests went to, by name, so that its counts still add up. docs/report.md describes every key.