ADR-0024: v2.0 is the version submitted to the 2027 competition, and what that leaves for 2.1

Status: Accepted, amended on 3, 4, 5, 11 and 12 October 2026 Date: 2026-09-22

Context

The 2027 edition of the REST League, the tool competition RESTest 1.x came fourth and fifth in a year earlier (ADR-0017), takes tool submissions until 9 October 2026 — seventeen days after this record — announces results on 13 November, and takes a four-page solution paper until 4 December. The maintainer's decision is that v2.0 is the version submitted: not a snapshot of the v2 branch, but a closed release with a command line that has stopped moving, documentation, a manual, a container image, and the standing to replace master.

What is measured

Ten APIs, five known and five that are "fresh APIs never used in previous studies". One hour per API, five runs averaged, eight cores and sixteen gigabytes reserved for the tool. Four measurements per API, each normalised across tools before it is added: unique server failures, told apart by their error message; operations that answered 2XX at least once; the API's own code coverage; and the area under the curve of each of those three over the hour. Three rankings: effectiveness adds the first three, efficiency adds the three areas, fault detection is the first alone.

Two facts about that protocol decide most of what follows.

The benchmark judges the replies itself. It counts the 5XX, it counts the operations that answered 2XX, and it instruments the API for coverage. The tool's own verdict on a reply changes no score. An oracle that says this 404 should have been a 400 is worth nothing to the ranking, however much it is worth to a user.

What counts is which requests were sent, and when. A tool that covers fifteen operations in the first minute beats one that covers fifteen in the last, on three of the six numbers. RESTest 1.x's failure in 2026 was exactly this: seven times less area under the operations curve than the winner, against a ceiling only sixty per cent lower.

Where the tool stands

Twenty-five increments delivered in twelve days. The last campaign against the five known APIs, before the fix that followed it and before the memory of what the API returned, covered sixteen operations on flight-search and one on kafka-rest-proxy, found three operations answering 500 on gestao-hospital and none on flight-search, and tested nothing on pet-clinic because the base path in the document was discarded. It idled 0% of the budget on every API. The tool is fast and does not crash; what it lacks is identifiers that exist, requests that create before they read, and requests built to break things in more than one way.

What the roadmap had in front of that

In its numbered order the roadmap put, before packaging and evaluation: eight increments of oracles and reports, six of stateful testing including its oracles and an Arazzo exchange format, five of IDL and constraint solving, and two of live constraint sources. Of those twenty-one, the ones that move a competition measurement are the mutation operators of 3.1b and the narrow, identifier-shaped core of 4.1, 4.2 and 4.4. The rest are the tool's value to a user — which is real, and which is why none of it is dropped — but not to this ranking.

Decision

1. v2.0 is the version submitted, and is closed

The commit tagged v2.0.0 is the commit submitted. Closed means: the command line frozen and documented, with its exit codes; a README a stranger can install from; the plan, dictionary and settings formats documented; a user manual; a container image and a distribution archive published from the tag; Ctrl-C leaving a report behind; and master replaced by it. ROADMAP.md's M12 is that list as increments. Replanned by the maintainer on 1 October: the competition receives the harness repository pinned to the commit 8.6 rehearsed, which stays on v2, and v2.0.0 is tagged on master, on a commit that differs from it in nothing under src/main and in no dependency, with documents about the tool alone. If that master or the image is not ready on 8 October, the tag goes on v2 and both follow as 2.0.1 (ROADMAP 12.4, 12.5).

If the manual is not merged by the submission date, the submission is tagged v2.0.0-rc.1 and v2.0.0 is tagged when the manual lands, with a check that the two commits differ in documentation only. The alternative — tag v2.0.0 without the manual and ship it in 2.0.1 — was put to the maintainer and declined on 22 September 2026: the first keeps the submitted version is 2.0 and 2.0 has a manual both true.

2. What is in v2.0 is what moves a measurement, plus what closes the release

Four milestones are added to the roadmap and taken before anything else:

Milestone	What it is	Measurement it moves
M9 Reach	A scheduler that owns the clock and sends every operation its best request first; path parameters filled from the identifier of the resource the path names; a producer sent before a consumer that has nothing to consume; budget withdrawn from operations whose recent answers all say this will never work as asked — 9.4 names the codes, and they are a setting. 9.3 was measured and not merged, and 9.4 set aside before it was built. Five rows were added as 9.5–9.9 by the maintainer on 2 October, from the campaigns of 1-2 October: a memory of replies that keeps what operations ask for, formats the name implies, the values of accepted requests sent on, plural and name gaps, and HAL's reserved properties left out of bodies	Operations covered, coverage, and the area under both
M10 Break	Mutations of requests the API accepted; bodies of the wrong shape; sequences over real resources — delete then read, create twice	Unique server failures and the error branches of the API
M11 Settings	One place for every number decided during development, layered from defaults, a file, the environment and the command line; every lever with a switch (ADR-0025). API keys were added as 11.3 by the maintainer on 29 September: sent where the document says, handed over by the person running the tool	None directly. It is what makes the ablation a campaign rather than a branch per variant. 11.3 moves nothing on the five known APIs, which the benchmark's proxy signs in; it is there in case the undisclosed five hand the tool a key
M12 Closing	The list in §1	None. It is what makes the submission a release

Two increments already on the roadmap are taken with them — 2.9 (optional parameters drawn by count) and 7.2a (the container image and release on tag, split from 7.2) — and four campaigns added to M8 — a baseline, a screening after each of M9 and M10, and a dress rehearsal under the competition's exact protocol from the frozen commit.

3. What waits for 2.1, and is not dropped

Every oracle beyond the two that exist (M3, 4.5, 5.5); every report format beyond the console and JSON (3.5); offline re-checking (3.3); per-operation oracle configuration (3.4); the dependency graph proper with its synonym table and its measurement against word vectors (the rest of 4.1 and 4.2); the lifecycle model and Arazzo (the rest of 4.4, 4.6); IDL, the solver and constraint-based generation, whole (M5); live constraint and flow sources (6.1); the overhead regression test (6.2); authentication inferred from the document (2.6) — all but API keys, which the maintainer added to v2.0 as 11.3 on 29 September; the external value provider (2.8); the dictionary writer and cache (2.7b); Maven Central, Homebrew, SDKMAN, jbang and the native binary (7.1, 7.2b, 7.3). Each keeps its row and its number, and restest-idl ships in v2.0 as the module descriptor it is today.

4. What the competition does not change

• The tool stays general. Nothing is tuned to the five known APIs: the undisclosed five are new APIs, and a lever that helps on five documents and hurts on the fifty of the wider corpus is not merged. Every M9 and M10 row is measured on the priority corpus and checked on the rest. The submission hands RESTest dictionaries for the five known APIs, as the maintainer decided on 3 October; the code stays general (amendment of 4 October).
• The harness stays outside (ADR-0011). What crosses the boundary is the published command line. The container image of 7.2a is the tool's own and knows nothing about being measured; the benchmark-compliant image the competition asks for is the harness repository's, built on top of it, and that repository — source and Dockerfile — is what is submitted and what the dress rehearsal runs. Replanned by the maintainer on 1 October: the harness repository's image compiles the tool from the frozen commit itself, as every campaign's has, so that the dress rehearsal can start before 7.2a lands (ROADMAP 7.2a, 8.6).
• The budget is the whole invocation (ADR-0015, ADR-0017). The opening lap of 9.1 is charged to the clock like everything else. No preparation phase runs off it.
• The deferred list still needs approval. This record asked for exactly one item from it — the narrowest version of the first open question of ADR-0017, weighted sampling over per-operation counters with no reward and no learning rate, as 9.4 — and the maintainer approved it on 22 September 2026. The two neighbouring questions are not taken, and the reward-shaped version stays deferred. The maintainer later narrowed 9.4's list to the answers that do not depend on what the tool sends, and replayed over recorded runs it had next to nothing to act on, so it was set aside before it was built (ADR-0017, M9.4 amendment).
• The rest of the design principles are untouched: zero configuration to start, never crash on a document, test cases as data, one event stream, no global state, nothing blocking the loop.

5. Evaluation is part of the plan, not after it

The baseline is itself an ablation, over where a run's values come from — the document alone, the document and what the API returned, and those plus a hand-written dictionary per API — so that the worth of each source is known before any lever is built on it, and so that the most favourable case, which the competition does not offer, is known as a ceiling. Every M9 and M10 row is then measured before it is merged, on a restarted containerised API from the priority corpus, switched on against switched off, five seeds, and the number goes in the pull request. The five rows added on 2 October, 9.5–9.9, are measured where their evidence is, which for some of them is outside the priority corpus, as the maintainer chose that day; each also counts what its rule reaches over the fifty documents of the wider corpus. 9.5 has no switch, by the maintainer's choice, and is measured before against after (ROADMAP 9.5). Each milestone is then screened as a whole on the five known APIs with short budgets, one switch off at a time, and a lever that moves nothing is switched off in the shipped settings. The dress rehearsal runs the competition's own protocol from the frozen commit for twenty-five hours, and nothing that changes a request is written while it runs. After the submission, the comparison against the published field and the grouped ablation produce the paper's tables, and the harness repository is made public as the replication package.

Consequences

• The roadmap is read in the order of work, not in numerical order. Numbers are names. The file says so at the top, and CLAUDE.md's instruction to "take the next increment in order" now means that order.
• Seventeen days, twenty-one increments, three campaigns, one rehearsal. The pace so far says it fits with nothing added. An increment that grows is split, and the half that moves no measurement goes to 2.1 — never the other way round.
• A row's pull request carries a measurement, not only tests. "Better on pet-clinic, five seeds, by this much" is the acceptance criterion for M9 and M10.
• The tool's own verdicts are weaker in v2.0 than the first roadmap promised. Two oracles rather than forty. The documentation says so plainly, and docs/DESIGN.md's comparison table describes v2.0 rather than the tool it will be.
• FeedbackListener acquires a milestone. ADR-0017 noticed the seam had none; 9.4 is its first implementation. 9.4 was not built, so the seam is without an implementation and without a row again.
• Two decisions ADR-0013 left to M4 are taken at 9.3, because M4 is now after v2.0: the unit of work of a sequence, and what to do about interference between concurrent sequences. 9.3 amends ADR-0013 with both. 9.3 built both and was not merged, because it moved no measured number; the decisions stay open, and ADR-0013's M9.3 amendment records what was built as the starting point.
• This record stands on ADR-0017, which is still Proposed. Five of its ideas are adopted here and four of its refusals are relied on. Accepting it is the maintainer's; until then the plan is built on a proposed record, knowingly.
• The paper is four pages and due eight weeks after the tool. Ablations run at twenty minutes a run, because they rank variants rather than post a number; the field comparison runs at the 2026 edition's own hour on its own eleven APIs, because its published table is the comparison and a different scale would footnote every row.

Alternatives considered

• Keep the order and submit whatever exists on 9 October. Rejected. It would submit a tool with no mutation operators and no sequences — the two levers the shakedown campaign showed missing — and spend the seventeen days on oracles the benchmark does not run.
• Skip 2027 and aim at 2028 with the full roadmap. Rejected by the maintainer. The rewrite exists to be measured, the 2026 result is the reason it was undertaken, and a year's delay buys nothing that a 2.1 release after the competition does not.
• Build the WFC oracles first, because they are "the other half of the problem". Rejected for v2.0, not refuted. They are the other half for a user; the benchmark supplies its own half.
• Adopt more of the winner — the learned tables, the word vectors, the preparation phase. Rejected on the grounds ADR-0017 gives, none of which a deadline changes.
• Take all three open questions while asking for one. Rejected. Questions 2 and 3 are online acceptance prediction and reading error text, each a deferred row of its own with costs ADR-0017 spells out, and neither has the "no constant to calibrate" property that makes 9.4 defensible.
• Tag v2.0.0 on 9 October whatever the state of the manual. Declined by the maintainer in favour of the release candidate, §1.
• Write the manual for one format and decide the format first. Rejected: it makes the format decision expensive and puts it on the critical path. One Markdown source built to both HTML and PDF was approved instead, and only the choice of which to publish is left, at no cost. Taken on 1 October: the Markdown as GitHub shows it, and a PDF on the release (ROADMAP 12.3). The PDF was dropped on 11 October 2026, after 2.1.0: ADR-0031.

Amendment (3 October 2026): 2.0.0 before the image

§1 said that v2.0 closed includes a container image and a distribution archive published from the tag, and that v2.0.0 is tagged on master on the day of the submission. The maintainer moved both on 3 October, once every row that changes a request had been merged and the code was taken as final:

• master is replaced and v2.0.0 tagged and released now, rather than on 8 October, so that the day of the submission carries no release work. A code change the competition still needs after this ships as 2.0.x, and the harness repository is pinned to whichever commit that is.
• The image and the archive follow in 7.2a, on master. Its release workflow is run once by hand against v2.0.0, so 2.0.0 gets its image and archives without a version of its own for them.
• The clean-up master receives (ROADMAP 12.5) covers what a visitor reads; the decision records, this one included, stay as they are, as the record of how 2.0 was decided. The whole record is on the tag history/2.0-development.

Amendment (4 October 2026): what the submission is handed

§4 says that nothing is tuned to the five known APIs. That still holds for the tool: 2.0.0 is the code submitted, and nothing in it knows which APIs the competition uses. It no longer holds for what the submission hands the tool. On 3 October the maintainer decided that the submission runs no local language model, and that the image the harness repository builds hands RESTest, for each of the five known APIs and for those alone, a dictionary of values prepared before the competition. The undisclosed APIs get no dictionary. §5 called those dictionaries the most favourable case, which the competition does not offer; for the five known APIs, the submission now takes that case.

The dictionaries were written offline, before any run and to a written rule (about ten values per place, from the document first and public knowledge last), in sessions with a coding assistant. The five known APIs' files were revised on 2 October so that values that must agree do agree - a flight that arrives after it leaves, a login with the password its e-mail registered - a revision that followed the analysis of earlier campaigns on those APIs, though its values still come from the documents and public knowledge. The other APIs' files were not revised that way, so the files submitted are not quite the kind measured in the last paragraph below.

The same image hands every API a plan of its own rather than the one RESTest carries: the same sources, with the shares of the four strategies (nominal, sequences, mutation, fuzzing) at 45, 15, 30 and 10 instead of 45, 10, 20 and 25. Both cross the boundary as §4 requires, on the published command line: the dictionaries with --dictionary, the plan with --campaign. The plan is about where a run's time goes, not about any API, so it is no exception to §4. It was chosen on 4 October from two overnight campaigns of 2.0.0, sixteen APIs, 20 minutes, two runs each, identical except for the plan (20261003-192803 with RESTest's plan, 20261004-000542 with the new one). On the fifteen APIs other than erc20, whose error messages make almost every failure look unique:

• unique server failures by the reference paper's count (one per distinct stack trace, or per distinct message once the values the request carried are taken out) rose from 792 to 919, higher on three APIs and lower on two; by the benchmark's count, with ncs and scs also left out, from 212 to 220;
• operations covered went from 272.5 to 277.5, lower on one API;
• branch coverage went from 29.3% to 30.0%, lower on two APIs.

No total fell, none of the differences is significant across APIs, and the first ten seconds were slower, mostly because one API started late. ADR-0027 kept the pushing strategy's quarter because a campaign without it lost faults and coverage; a tenth is not none, and here no total fell. RESTest's own plan keeps its shares: evidence this weak is enough to choose between two configurations for one event, not to change what every user gets by default.

A third campaign, 20261003-134551, ran the new plan with a dictionary for every API. Against it, the ten APIs other than the known five and erc20 lost branch coverage without their dictionaries, from 37.2% to 31.1% - lower on nine APIs, higher on none - though one API, scs, accounts for most of it (80.0% to 40.7%; 32.5% to 30.0% without it). Operations fell from 171 to 160.5, and the failures found moved both ways without a significant difference. That is what a model writing dictionaries inside the hour could have been worth if it wrote files as good as these, which, written offline and with time to spare, are probably a ceiling for it.

Amendment (5 October 2026): 2.1.0 is the version submitted

The amendment of 3 October said that a code change the competition still needs ships as 2.0.x. The changes the maintainer asked for on 4 October add behaviour rather than fix it, so they ship as 2.1.0, and on 5 October the maintainer decided that 2.1.0 is the version submitted: the harness repository is to be pinned to v2.1.0, with the plan and the dictionaries of the amendment of 4 October unchanged. 2.0.0 stays released, as the first version of the rewrite. What §1 says of v2.0 being closed holds of 2.1.0, except the container image and the distribution archive, which follow in 7.2a as the amendment of 3 October says, its workflow run by hand against v2.1.0 as well as v2.0.0. What §3 and other records leave "for 2.1" is left for a later 2.x: 2.1.0 holds none of it.

2.1.0 changes three things in what a run sends:

• Five more kinds a name implies (ADR-0022, 2.1 amendment): a password that passes the usual strength rule, a short fresh user name, a person's given or family name, a gender, and, for a property called name and nothing more, a fresh name of letters - the name of a product, a pet or a notebook as often as a person's. That last rule is kept below the bar the corpus sets, nine right in ten (it is 53 right and 7 wrong), by the maintainer's choice. A field marked format: password is now read by its name too.
• Three in ten of the POST bodies RESTest invents are filled afresh (ADR-0020, 2.1 amendment): a value the API returned is passed over for every property that is not an identifier, so that a registration does not carry the name of a user that already exists.
• Optional properties go into a body eight times in ten, and optional bodies are sent nine times in ten, where both were a half (ADR-0025, 2.1 amendments).

The second has a switch of its own, and the third is two settings that can be put back to a half; the new name rules ride the switch of the older ones, generation.impliedFormats, and have none of their own. Nothing in them knows which APIs the competition uses, so §4 holds for the tool as the amendment of 4 October left it: they were drawn from what campaigns showed on APIs no dictionary covers, where sign-ups and creations were refused for the values they carried.

They were measured together, in the configuration submitted, sixteen APIs, 20 minutes, two runs each: campaign 20261004-142720, at ddedf474 (kept on the branch eval/2.1.0-levers), whose code is that of 2.1.0 and whose documents are older, against 20261004-000542, the same configuration at 2.0.0. The criterion fixed before the launch - nothing lower at p < 0.05, the APIs that get no dictionary higher in operations and coverage, the known ones not significantly lower - was met. erc20 is left out for the reason given above. gestao-hospital, one of the five known APIs, is set apart: that afternoon it refused, about 12,000 times in a row, a creation it had accepted byte for byte about 1,000 times a run in the 2.0.0 campaign the night before. 2.1.0 therefore has no measurement on it, and it was not measured again before the release. On the other fourteen:

• operations answered with a 2XX went from 259.5 to 268, higher on six APIs and lower on one;
• unique server failures by the reference paper's count, from 910 to 1,020.5, higher on five and lower on two;
• branch coverage, from 30.4% to 31.1%, higher on five and lower on four.

On the ten that get no dictionary, operations went from 160.5 to 168, failures from 706 to 821.5 and branch coverage from 31.1% to 32.1%. Four operations of user-management and market's registration, never accepted in a run of 2.0.0, were accepted in both runs, and blog's photos went from two accepted creations in two runs to 32 and 42. On the four known APIs measured, operations went from 99 to 100, failures from 204 to 199 (lower on one API) and branch coverage from 28.6% to 28.5% (lower on two, higher on none).

The areas under the curve, by the benchmark's own scoreboard over fifteen APIs with gestao-hospital included, went from 324,176 to 329,914 for operations (higher on eight APIs, lower on seven), from 662,887 to 710,006 for server failures (nine and five) and from 354 to 351 for branch coverage (seven and eight), where gestao-hospital's -71.8 and scs's +71.9 cancel out. Scored the way the 2026 league's leaderboard is, on its APIs other than erc20 at 20 minutes, with the areas extended to an hour as a lower bound, 2.1.0 ranks where 2.0.0 did - third in fault detection, first in efficiency and in effectiveness - with an effectiveness of 60.21 against 2.0.0's 59.78 without gestao-hospital, and of 59.85 against 60.06 with it.

None of the differences across APIs is significant, and the three changes were not measured apart: a campaign with the two chances back at a half, which would have told them apart, was prepared and not run, by the maintainer's decision.

Amendment (11 October 2026): no image for 2.0.0 or 2.1.0

The amendments of 3 and 5 October said that the release workflow would be run by hand against v2.0.0 and v2.1.0, so that both got the image and the archives without a version of their own. The maintainer decided otherwise on 11 October 2026: neither does, and 2.2.0 is the first release that carries them. Neither tag holds a Dockerfile or a launcher, so the image would have been built from old code with files from a later commit, added to releases already published, by a workflow whose first run would have been against a public release. Users of 2.0.0 and 2.1.0 build them from the source, as before. ADR-0032 says how a release is published.

Amendment (12 October 2026): RESTest's own plan takes these shares

The amendment of 4 October kept RESTest's own plan at 45, 10, 20 and 25 while the competition's runs were handed 45, 15, 30 and 10. From 2.2.0 RESTest's own plan has 45, 15, 30 and 10 too, by the maintainer's decision of 12 October 2026 on the same evidence, with no new measurement. ADR-0023's amendment of that date is the record, and says what the evidence does and does not show for short runs, and for runs handed no dictionary.