The settings

A setting is a number somebody decided about how the tool behaves — how hard it pushes, how much it keeps, how deep it goes, how long it waits. You never need to change one: RESTest starts with nothing beside it and works against an unknown API. When you do need to change one, it is one line, and you never recompile.

restest run --print-settings > settings.yaml
# edit a line
restest run api.yaml --url http://localhost:9966 --settings settings.yaml

The decisions behind all of this are in ADR-0025. The settings that are true or false and each turn off one thing the tool does are also listed on a page of their own, the switches, which says what turning each off does, what leaving it on costs, and gives the files that turn off a whole idea at once.

Settings are not a plan

Two files, and it is worth being clear which is which.

A plan says where a run's values come from and which operations it may touch. It travels with an API: somebody writes one per API, and it means the same thing on every machine. See the campaign file.

Settings say how the tool behaves. They travel with a machine or with an experiment: a concurrency that suits your laptop is wrong on a build server, and an experiment that turns a behaviour off should not have to rewrite a file about the API.

The test for which one something belongs in: would it mean the same thing for a different API on the same machine? If yes, it is a setting.

A key the API asks for is neither. It belongs to one deployment of one API and to the person running the tool, so it fails the test above. It would also be printed by --print-settings and recorded in report.json if it were a setting, which is exactly what must not happen to it. It is handed over with --auth, or left in RESTEST_AUTH — a variable named like a setting's and not one of them, never printed or recorded with them. See ADR-0029.

The four places a setting can be given

Each one wins over the one before it.

	Where	Looks like
1	What RESTest does when nobody has said otherwise	—
2	A file named with --settings	maxConcurrency: 8 under engine:
3	An environment variable	RESTEST_ENGINE_MAX_CONCURRENCY=8
4	--set, repeated as often as you like	--set engine.maxConcurrency=8

The same words name a setting everywhere. engine.maxConcurrency in a file and after --set; RESTEST_ENGINE_MAX_CONCURRENCY in the environment, which is the same name in the spelling environments use — capitals, and an underscore wherever the name has a capital of its own.

No file is looked for anywhere unless you name it. Starting the tool with nothing beside it has to behave the same way in every directory, and a file found by accident is a run nobody can explain afterwards.

The template, in full

This is what restest run --print-settings writes. It is generated from the settings themselves rather than kept as a file in this repository, so it cannot drift from what the tool actually does, and so that it can say where each value came from — which a file sitting in a repository could never know. A test keeps the copy below in step with it.

Save it, change the lines you care about, delete the rest or leave them, and hand it back with --settings. Every line is optional: what you leave out keeps the value RESTest uses.

# The settings this run uses, and where each of its values came from.
#
# Save this, change a line, and hand it back with --settings <file>. The same
# values can be given as --set engine.maxConcurrency=8 on the command line, or as
# RESTEST_ENGINE_MAX_CONCURRENCY=8 in the environment. What is typed on the command
# line wins over the environment, which wins over a file, which wins over what
# RESTest does when nobody has said otherwise.
#
# These are settings of the tool. Where a run's values come from, and which
# operations it may touch, is a plan instead: --print-campaign writes one out.
#
# A setting written as a comment, such as "# callTimeout", follows others until
# somebody gives it: change one of those and it moves with them. Take away the #
# in front of its name to fix it at that value instead.

engine:
  # how long to wait for the API to accept a connection at all
  connectTimeout: "10s"       # default
  # how long to wait for the API to answer once connected
  readTimeout: "30s"          # default
  # how long to wait while sending a request body
  writeTimeout: "10s"         # default
  # the longest one request may take, its whole reply included, however slowly that arrives
  # callTimeout: "1m"         # default
  # the fewest requests kept in flight, however badly the API behaves
  minConcurrency: 1           # default
  # how many requests are in flight before anything is known about the API
  # initialConcurrency: 4     # default
  # the most requests ever in flight at once. 1 for a fragile API
  maxConcurrency: 16          # default
  # whether the run keeps to mostRequestsPerMinute as well; on by itself once that is named
  limitTheRate: false         # default
  # the most requests a minute while the rate is limited, evenly spaced: 600 is one every tenth of a second
  mostRequestsPerMinute: 600  # default
  # how much slower than its best answer so far counts as the API struggling
  slowdownFactor: 2           # default
  # how much of a reply body is kept in memory
  maxRetainedResponseBytes: 1048576 # default
  # whether a redirection is followed instead of being reported
  followRedirects: false      # default
  # what the tool calls itself in the User-Agent header
  userAgent: "RESTest/2.0"    # default

schedule:
  # how many requests may await an answer, as a multiple of maxConcurrency
  workAheadFactor: 2          # default
  # how many announcements may await the reports before the run pauses
  announcementsAllowedToPileUp: 1000 # default
  # how long past the engine's own timeout, once the time is up, to wait for answers already asked for
  stragglerGrace: "10s"       # default
  # how long a run stopped from outside waits for answers already asked for
  interruptGrace: "2s"        # default
  # whether a run starts by sending every operation once, the request likeliest to work
  openingLap: true            # default
  # how long each step of that opening lap waits for the answers to the one before
  openingLapPatience: "2s"    # default
  # whether a run stops sending once the API has answered 401 to every request of its opening lap
  stopWhenTheOpeningLapIsRefused: true # default
  # whether an operation the API answers 429, or 503 with Retry-After, is left alone for as long as the API asks
  waitWhenToldTo: true        # default
  # how long it is left alone the first time the API does not say, doubled each time it is told again
  firstGuessedWait: "1s"      # default
  # the longest that doubling goes
  longestGuessedWait: "1m"    # default
  # whether a run stops once it has sent a set number of requests, before its budget is spent
  stopAfterRequests: false    # default
  # how many requests that is
  requestsBeforeStopping: 1000 # default
  # how long a run stopped from outside is then given to write what it found
  interruptWritingGrace: "5s" # default
  # how long it then gives its directory to say what was written, when writing did not finish
  interruptLookingGrace: "1s" # default

generation:
  # below this depth, only what the description insists on is built
  optionalNestingDepth: 4     # default
  # where building stops, however insistent the description is
  hardNestingDepth: 8         # default
  # the longest word invented when the description does not demand more
  usualLongestString: 64      # default
  # beyond this, a demanded length is declined rather than built
  longestString: 10000        # default
  # where an invented number starts, when the description states no bottom, unless its top is lower
  lowestNumber: 0             # default
  # how far above that it may go, when the description states no top; and below a top beneath lowestNumber
  roomAboveIt: 1000           # default
  # decimal places for a number allowed to have them
  decimalPlaces: 2            # default
  # the most items put in a list when the description does not demand more
  usualMostItems: 4           # default
  # beyond this, a demanded number of items is declined rather than built
  mostItems: 100              # default
  # how often an optional property is included anyway, between 0 and 1
  optionalPropertyChance: 0.8 # default
  # how often a request that merely accepts a body sends it anyway, between 0 and 1; a GET or a HEAD never does
  optionalBodyChance: 0.9     # default
  # how often one more optional parameter is added on top of the ones already chosen, between 0 and 1
  optionalParameterContinueChance: 0.5 # default
  # whether how many optional parameters go in is drawn first, favouring few; off, each is decided on its own at optionalBodyChance
  optionalParametersBySize: true # default
  # one time in this many, a value allowed to be absent is sent as nothing
  nullInOneIn: 8              # default
  # how many times a fresh element is attempted for a list of distinct items
  uniqueAttempts: 8           # default
  # how many times a value is invented again after an unsendable one
  sendableAttempts: 8         # default
  # how many bodies are drawn while looking for one its media type can carry
  writableBodyAttempts: 8     # default
  # whether an invented word takes the kind its name or description implies, such as an e-mail address for email, where none is declared
  impliedFormats: true        # default
  # how often it does where a kind is implied, between 0 and 1; the rest are ordinary invented words
  impliedFormatChance: 0.5    # default
  # whether _links and _embedded, which HAL keeps for what a server writes, are taken out of every body before it is sent
  omitHalProperties: true     # default
  # whether, in the body of a POST, a property the API has shown values for is sometimes invented afresh, so that what is made is not a copy of something that exists; identifiers never are
  freshWhereMade: true        # default
  # how often it happens, between 0 and 1, decided once for each body
  freshWhereMadeChance: 0.3   # default
  # how many times a fresh value is asked for, to change one value in a thing the API returned
  changedValueAttempts: 4     # default
  # how many times an open-ended repetition runs in a word built to a pattern, unless the length demanded needs more
  usualRepetitions: 8         # default

mutation:
  # whether accepted requests are changed in ways the description forbids
  violations: true            # default
  # leave out a parameter or body property the description says is required
  dropRequired: true          # default
  # send a required parameter in the query, a header or a cookie other than its own
  wrongLocation: true         # default
  # send a value of another kind, such as a word where a number is declared
  wrongType: true             # default
  # send a value one step past a limit the description states
  outsideABound: true         # default
  # send a value that is not on the closed list the description states
  breakAnEnumeration: true    # default
  # send a word the stated pattern refuses
  breakAPattern: true         # default
  # send a word that looks like the stated format but is not, such as 2021-02-30 for a date
  breakAFormat: true          # default
  # send null for a body property that may not be null
  sendNull: true              # default
  # send an empty word, list or object where the description forbids one
  sendEmpty: true             # default
  # send a word or list far longer than the longest the description allows
  oversize: true              # default
  # send the whole body as another kind of thing, such as a list where an object is declared
  wrongRoot: true             # default
  # send a body of no bytes at all where the description requires one
  emptyBody: true             # default
  # send a body that is not JSON: the accepted one cut off or broken, or plain words
  notJson: true               # default
  # send the accepted body under a media type the description does not offer
  wrongContentType: true      # default
  # send a number past what its declared format holds, such as 2147483648 for int32
  beyondItsWidth: true        # default
  # how many of each operation's accepted requests are kept to be changed, newest first
  acceptedKept: 16            # default
  # how long an oversized word is in characters, and a far too wide number in digits
  oversizedLength: 10000      # default
  # how many items an oversized list has
  oversizedItems: 1000        # default

sequences:
  # create a thing, delete it, then read it and what hangs from it: is it gone?
  readAfterDelete: true       # default
  # create a thing and delete it twice: is the second deletion answered calmly?
  deleteTwice: true           # default
  # create a thing, delete it, then add or change something under it
  writeUnderDeleted: true     # default
  # create a thing and send it the same PUT twice, reading it after each
  putTwice: true              # default
  # create a thing, read it, send other reads, then read it again: unchanged?
  safeGet: true               # default
  # send the same creation twice
  createTwice: true           # default

memory:
  # how many values the run remembers under any one name, and things of any one kind. 0 remembers none
  mostValuesUnderOneName: 20  # default
  # how many different names, shapes and kinds of thing are remembered at all; a new one makes room by letting go of the one heard of longest ago
  mostNames: 2000             # default
  # how long one remembered word or number may be, written out
  longestValueKept: 10000     # default
  # the largest reply the run reads looking for values to remember
  longestReplyRead: 524288    # default
  # how far into a reply, or an accepted request's body, that search goes
  asDeepAsAReplyIsRead: 6     # default
  # whether a gap such as {petId} in /pets/{petId} is filled from the ids of the pets the API returned; off, only a value named petId fills it
  identifiersByResource: true # default
  # whether those pets are asked before any value named petId; off, the name is asked first
  identifiersByResourceFirst: true # default
  # whether the values a request carried are remembered too when the API accepted it, such as the password a registration was accepted with
  rememberAcceptedRequests: true # default
  # whether a gap such as {ids} in /persons/{ids} is filled like {id}, with the id of one of the persons the API returned; only while identifiersByResource is on
  pluralIdentifiers: true     # default
  # whether a gap such as {productName} is filled with the name of one of the products the API returned, a plain list of words included; only while identifiersByResource is on
  namesByResource: true       # default

document:
  # how long to wait for a description fetched over the network
  fetchTimeout: "10s"         # default
  # the largest description that is read at all
  mostBytesRead: 67108864     # default

checks:
  # report a reply of 500, which says the API fell over while handling the request (F100)
  internalServerError: true   # default
  # report a reply of a server error other than 500 and 501, such as 503 (F121)
  otherServerError: true      # default
  # report a reply whose body is not the shape the document gives for its status code and media type (F200)
  replyOffSchema: true        # default
  # report a reply of 501, not implemented, to an operation the document declares (F205)
  notImplemented: true        # default
  # report a request broken on purpose, with one thing the document forbids, that the API accepted (F206)
  invalidDataAccepted: true   # default
  # report a request that got no reply, on a connection open to the API, before RESTest stopped waiting (F312)
  noReplyInTime: true         # default
  # report a reply whose status code, below 500, the document does not declare for the operation (F920)
  undeclaredStatusCode: true  # default
  # report a reply with a body whose content type the document does not declare for its status code, below 500 (F921)
  undeclaredContentType: true # default
  # end the run with 1 when it found warnings, as it does for errors
  failOnWarnings: false       # default

report:
  # how many faults of one kind, on one operation, are written out whole
  writeUpsPerOperationAndKind: 5 # default
  # and how many in the whole file
  writeUpsInTotal: 1000       # default
  # how much of any one body the report quotes
  mostBodyBytesKept: 24576    # default
  # how many faults are printed in full before the screen stops being the place
  faultsShownOnTheConsole: 50 # default
  # and how many of those may be warnings, so that errors found later still have room
  warningsShownOnTheConsole: 10 # default
  # how many operations that could not be tested are named on the screen
  skippedOperationsShownOnTheConsole: 5 # default
  # how many of a fault's particulars are printed under it on the screen
  detailsShownPerFault: 10    # default
  # how many of the status codes answered most the report lists
  commonestStatusCodesListed: 10 # default
  # how many equal intervals, up to 10000, the budget is cut into for the run over time in the report
  timelineIntervals: 60       # default
  # write report.html beside report.json: the same run, as a page to open in a browser
  html: true                  # default

What a settings file looks like

engine:
  maxConcurrency: 1
  readTimeout: 2m

report:
  faultsShownOnTheConsole: 10

That is all of it. Groups, and the settings inside them. Anything you leave out keeps the value RESTest uses by default, so a file naming one line is a perfectly good file.

engine.maxConcurrency: 1 on a line of its own works too, which is what somebody who has only ever seen the command line will write.

What --print-settings writes is a file of exactly this shape, carrying every setting there is with a line saying what each one does and a note saying where its value came from. The notes are comments, so handing the file straight back changes nothing.

What is refused

A run that could not be configured the way you asked does not happen. Running with different numbers would answer a question nobody put, so RESTest stops before it sends anything and answers 2, the command line was wrong.

• A setting that does not exist is refused, naming the one it most looks like. --set engine.maxConcurrancy=8 answers did you mean engine.maxConcurrency?
• A value the setting cannot take is refused, saying what it wanted: engine.maxConcurrency takes a whole number, and 'lots' is not one.
• A value outside its range is refused with the range, and with the other setting it disagrees with when there is one.
• A file that cannot be read is refused, naming the file.
• A wait longer than the HTTP engine can count to, about 24 days, is refused, naming the setting.
• A value the tool could not write back out is refused: a length of time finer than a millisecond or longer than milliseconds can count, a number too large for the machine to hold, a number that would take more than a thousand characters to write out. Every setting has to survive being printed and recorded, or --print-settings and report.json would break on it later.

To check a file of settings without testing anything, hand it over with --print-settings: restest run --settings <file> --print-settings needs no document and no address. It answers 0 and prints every setting, each with where its value came from, or 2 and what is wrong in the file, as a run would. A file with a setting misspelt:

restest: there is no setting called 'checks.invalidDataAcepted'. Did you mean 'checks.invalidDataAccepted'?

Where a run says how it was configured

Two places, and you need neither to be watching.

report.json carries a settings block with every setting, its value, and which of the four places decided it — so a directory of results carries the configuration that produced it, and two runs can be compared line by line.

{ "key": "engine.maxConcurrency", "value": "1", "source": "command line" }

And a run whose settings are not the ones RESTest ships says so on the screen, once, because an environment variable is invisible in the command you typed and in the transcript you paste into a bug report.

One source is worth knowing about: worked out. A few settings follow others, and moving those moves them. --set engine.maxConcurrency=1 also moves where the engine starts, and --set engine.readTimeout=2m how long a whole request may take, without you mentioning either. Such a value is recorded as worked out rather than as a default — because it is not what the tool does by default, and two results directories would otherwise carry different values under the same word.

The lengths of time

Anywhere a setting is a length of time, it is written the way people write one: 500ms, 30s, 5m, 2h. A bare number means seconds. The formal spelling PT1M30S works too. It is the same spelling --budget takes.

Every setting

engine.*

Setting	Default	What it does
connectTimeout	10s	how long to wait for the API to accept a connection at all
readTimeout	30s	how long to wait for the API to answer once connected
writeTimeout	10s	how long to wait while sending a request body
callTimeout	1m	the longest one request may take, its whole reply included, however slowly that arrives
minConcurrency	1	the fewest requests kept in flight, however badly the API behaves
initialConcurrency	4	how many requests are in flight before anything is known about the API
maxConcurrency	16	the most requests ever in flight at once. 1 for a fragile API
limitTheRate	false	whether the run keeps to mostRequestsPerMinute as well; on by itself once that is named
mostRequestsPerMinute	600	the most requests a minute while the rate is limited, evenly spaced: 600 is one every tenth of a second
slowdownFactor	2	how much slower than its best answer so far counts as the API struggling
maxRetainedResponseBytes	1048576	how much of a reply body is kept in memory
followRedirects	false	whether a redirection is followed instead of being reported
userAgent	RESTest/2.0	what the tool calls itself in the User-Agent header

schedule.*

Setting	Default	What it does
workAheadFactor	2	how many requests may await an answer, as a multiple of maxConcurrency
announcementsAllowedToPileUp	1000	how many announcements may await the reports before the run pauses
stragglerGrace	10s	how long past the engine's own timeout, once the time is up, to wait for answers already asked for
interruptGrace	2s	how long a run stopped from outside waits for answers already asked for
openingLap	true	whether a run starts by sending every operation once, the request likeliest to work
openingLapPatience	2s	how long each step of that opening lap waits for the answers to the one before
stopWhenTheOpeningLapIsRefused	true	whether a run stops sending once the API has answered 401 to every request of its opening lap
waitWhenToldTo	true	whether an operation the API answers 429, or 503 with Retry-After, is left alone for as long as the API asks
firstGuessedWait	1s	how long it is left alone the first time the API does not say, doubled each time it is told again
longestGuessedWait	1m	the longest that doubling goes
stopAfterRequests	false	whether a run stops once it has sent a set number of requests, before its budget is spent
requestsBeforeStopping	1000	how many requests that is
interruptWritingGrace	5s	how long a run stopped from outside is then given to write what it found
interruptLookingGrace	1s	how long it then gives its directory to say what was written, when writing did not finish

generation.*

Setting	Default	What it does
optionalNestingDepth	4	below this depth, only what the description insists on is built
hardNestingDepth	8	where building stops, however insistent the description is
usualLongestString	64	the longest word invented when the description does not demand more
longestString	10000	beyond this, a demanded length is declined rather than built
lowestNumber	0	where an invented number starts, when the description states no bottom, unless its top is lower
roomAboveIt	1000	how far above that it may go, when the description states no top; and below a top beneath lowestNumber
decimalPlaces	2	decimal places for a number allowed to have them
usualMostItems	4	the most items put in a list when the description does not demand more
mostItems	100	beyond this, a demanded number of items is declined rather than built
optionalPropertyChance	0.8	how often an optional property is included anyway, between 0 and 1
optionalBodyChance	0.9	how often a request that merely accepts a body sends it anyway, between 0 and 1; a GET or a HEAD never does
optionalParameterContinueChance	0.5	how often one more optional parameter is added on top of the ones already chosen, between 0 and 1
optionalParametersBySize	true	whether how many optional parameters go in is drawn first, favouring few; off, each is decided on its own at optionalBodyChance
nullInOneIn	8	one time in this many, a value allowed to be absent is sent as nothing
uniqueAttempts	8	how many times a fresh element is attempted for a list of distinct items
sendableAttempts	8	how many times a value is invented again after an unsendable one
writableBodyAttempts	8	how many bodies are drawn while looking for one its media type can carry
impliedFormats	true	whether an invented word takes the kind its name or description implies, such as an e-mail address for email, where none is declared
impliedFormatChance	0.5	how often it does where a kind is implied, between 0 and 1; the rest are ordinary invented words
omitHalProperties	true	whether _links and _embedded, which HAL keeps for what a server writes, are taken out of every body before it is sent
freshWhereMade	true	whether, in the body of a POST, a property the API has shown values for is sometimes invented afresh, so that what is made is not a copy of something that exists; identifiers never are
freshWhereMadeChance	0.3	how often it happens, between 0 and 1, decided once for each body
changedValueAttempts	4	how many times a fresh value is asked for, to change one value in a thing the API returned
usualRepetitions	8	how many times an open-ended repetition runs in a word built to a pattern, unless the length demanded needs more

mutation.*

What a strategy of kind mutation may change in a request the API accepted — see the campaign file. Every kind of change breaks something the description states, so a refusal is the right answer. A kind is made only when its own switch and violations are both on; with violations off, such a strategy builds its requests the ordinary way.

Five of the kinds, from wrongRoot to beyondItsWidth, break the body as a whole rather than one value in it — its kind, its bytes, its media type — or push a number past what its declared format can hold. They aim at the code that reads a body before the API's own code runs. The switches has the files that turn off, at once, the kinds that change one value or these last five.

Setting	Default	What it does
violations	true	whether accepted requests are changed in ways the description forbids
dropRequired	true	leave out a parameter or body property the description says is required
wrongLocation	true	send a required parameter in the query, a header or a cookie other than its own
wrongType	true	send a value of another kind, such as a word where a number is declared
outsideABound	true	send a value one step past a limit the description states
breakAnEnumeration	true	send a value that is not on the closed list the description states
breakAPattern	true	send a word the stated pattern refuses
breakAFormat	true	send a word that looks like the stated format but is not, such as 2021-02-30 for a date
sendNull	true	send null for a body property that may not be null
sendEmpty	true	send an empty word, list or object where the description forbids one
oversize	true	send a word or list far longer than the longest the description allows
wrongRoot	true	send the whole body as another kind of thing, such as a list where an object is declared
emptyBody	true	send a body of no bytes at all where the description requires one
notJson	true	send a body that is not JSON: the accepted one cut off or broken, or plain words
wrongContentType	true	send the accepted body under a media type the description does not offer
beyondItsWidth	true	send a number past what its declared format holds, such as 2147483648 for int32
acceptedKept	16	how many of each operation's accepted requests are kept to be changed, newest first
oversizedLength	10000	how long an oversized word is in characters, and a far too wide number in digits
oversizedItems	1000	how many items an oversized list has

sequences.*

Which series of requests a strategy of kind sequences may send around a thing it created itself — see the campaign file. Each series starts with that strategy's own creation, asks one question, and sends each of its steps once the answer to the one before has arrived, with the identifier the API gave the thing. With every series off, such a strategy builds its requests the ordinary way; the switches has that file.

Setting	Default	What it does
readAfterDelete	true	create a thing, delete it, then read it and what hangs from it: is it gone?
deleteTwice	true	create a thing and delete it twice: is the second deletion answered calmly?
writeUnderDeleted	true	create a thing, delete it, then add or change something under it
putTwice	true	create a thing and send it the same PUT twice, reading it after each
safeGet	true	create a thing, read it, send other reads, then read it again: unchanged?
createTwice	true	send the same creation twice

memory.*

Setting	Default	What it does
mostValuesUnderOneName	20	how many values the run remembers under any one name, and things of any one kind. 0 remembers none
mostNames	2000	how many different names, shapes and kinds of thing are remembered at all; a new one makes room by letting go of the one heard of longest ago
longestValueKept	10000	how long one remembered word or number may be, written out
longestReplyRead	524288	the largest reply the run reads looking for values to remember
asDeepAsAReplyIsRead	6	how far into a reply, or an accepted request's body, that search goes
identifiersByResource	true	whether a gap such as {petId} in /pets/{petId} is filled from the ids of the pets the API returned; off, only a value named petId fills it
identifiersByResourceFirst	true	whether those pets are asked before any value named petId; off, the name is asked first
rememberAcceptedRequests	true	whether the values a request carried are remembered too when the API accepted it, such as the password a registration was accepted with
pluralIdentifiers	true	whether a gap such as {ids} in /persons/{ids} is filled like {id}, with the id of one of the persons the API returned; only while identifiersByResource is on
namesByResource	true	whether a gap such as {productName} is filled with the name of one of the products the API returned, a plain list of words included; only while identifiersByResource is on

document.*

Setting	Default	What it does
fetchTimeout	10s	how long to wait for a description fetched over the network
mostBytesRead	67108864	the largest description that is read at all

checks.*

The checks a run makes on what came back, or did not, each of which reports a kind of fault, and, last, what the faults they find do to how a run ends: an error ends it with 1, a warning only when failOnWarnings asks (errors and warnings). The faults says what switching a check off changes, and what it does not.

Setting	Default	What it does
internalServerError	true	report a reply of 500, which says the API fell over while handling the request (F100)
otherServerError	true	report a reply of a server error other than 500 and 501, such as 503 (F121)
replyOffSchema	true	report a reply whose body is not the shape the document gives for its status code and media type (F200)
notImplemented	true	report a reply of 501, not implemented, to an operation the document declares (F205)
invalidDataAccepted	true	report a request broken on purpose, with one thing the document forbids, that the API accepted (F206)
noReplyInTime	true	report a request that got no reply, on a connection open to the API, before RESTest stopped waiting (F312)
undeclaredStatusCode	true	report a reply whose status code, below 500, the document does not declare for the operation (F920)
undeclaredContentType	true	report a reply with a body whose content type the document does not declare for its status code, below 500 (F921)
failOnWarnings	false	end the run with 1 when it found warnings, as it does for errors

report.*

Setting	Default	What it does
writeUpsPerOperationAndKind	5	how many faults of one kind, on one operation, are written out whole
writeUpsInTotal	1000	and how many in the whole file
mostBodyBytesKept	24576	how much of any one body the report quotes
faultsShownOnTheConsole	50	how many faults are printed in full before the screen stops being the place
warningsShownOnTheConsole	10	and how many of those may be warnings, so that errors found later still have room
skippedOperationsShownOnTheConsole	5	how many operations that could not be tested are named on the screen
detailsShownPerFault	10	how many of a fault's particulars are printed under it on the screen
commonestStatusCodesListed	10	how many of the status codes answered most the report lists
timelineIntervals	60	how many equal intervals, up to 10000, the budget is cut into for the run over time in the report
html	true	write report.html beside report.json: the same run, as a page to open in a browser

Five things worth knowing

A whole request is given twice the longest wait inside it. callTimeout is the longest one request may take, from sending it to the last byte of its reply. It is what ends a reply that never stops arriving - an endpoint that streams, or sends a byte every few seconds, and so never lets readTimeout run out. When it is not given, it is twice the longest of connectTimeout, readTimeout and writeTimeout, so --set engine.readTimeout=2m for a slow API also gives each request four minutes, recorded as worked out. When it is given and the other three are not, they are shortened to fit inside it: --set engine.callTimeout=5s is one line. Naming both a wait and a whole request it cannot fit inside is refused. --print-settings writes callTimeout as a comment until somebody gives it, and initialConcurrency too, so a printed file with readTimeout or maxConcurrency changed moves them the same way; take away the # to fix one of them instead.

Moving the concurrency range moves where the engine starts. initialConcurrency is where to begin inside the range rather than a number of its own, so --set engine.maxConcurrency=1 works as typed and gives you one request at a time. Naming a starting number outright that lies outside the range is still refused.

Naming a rate is how a rate is promised. limitTheRate is off, and a run sends as fast as its requests in flight and the API's answers allow. --set engine.mostRequestsPerMinute=300 turns it on by itself, recorded as worked out, so that one line is enough to promise an API's owner five requests a second. The requests go evenly, never two closer than a minute divided by that number — here a fifth of a second — and never a burst to make up for a quiet spell: not three hundred in the first seconds and then nothing. limitTheRate: true alone keeps to ten a second; limitTheRate: false named beside a rate leaves the rate alone. A printed file writes limitTheRate: false out, so changing only mostRequestsPerMinute in it keeps no rate, and the run says so under its first lines:

  engine.mostRequestsPerMinute is 300, but engine.limitTheRate is false, so no rate is kept to

The rate is kept where RESTest sends; the network on the way may bring two requests a little closer. The quiet it makes is not the tool being slow, and the summary says how much of the idle share it was. Against pet-clinic for 30 seconds at 300 a minute:

145 requests to 43 operations in 30.0s, 85% of it idle (81% keeping to the rate it was given)

roomAboveIt is a width, not a ceiling. When a description states no bounds for a number, RESTest invents one between lowestNumber and lowestNumber + roomAboveIt. When the description states a bottom of its own, that bottom is used and the same room is allowed above it, so the room to move in is the same wherever the numbers begin. When the only limit it states is a top lower than lowestNumber, the same room is allowed below that top instead.

optionalBodyChance also decides optional parameters, when optionalParametersBySize is off. With optionalParametersBySize on, which is the default, how many optional parameters a request carries is drawn first and optionalParameterContinueChance shapes that count. Switched off, each optional parameter is decided on a coin of its own, weighted by optionalBodyChance — the same number that decides whether a request sends a body it may leave out. Bodies and optional parameters cannot be given different chances, so an experiment that turns optionalParametersBySize off and changes optionalBodyChance changes how often optional parameters are sent as well as bodies. A setting of its own for the coin would correct it, and is left for a later version (known limitations).

What is not a setting

A number that is a fact rather than a decision stays in the code: 500 is a server error, a file format has a version, a length of a day is a length of a day. The rule for moving one is whether a reasonable user or a reasonable experiment might want a different value.

Some numbers stay in the code on purpose. The builder of words that must match a pattern keeps the limits on how much work it hands the library that builds them — how many candidates it draws, how many characters it builds before giving up — and a value changed to break a pattern is checked against that pattern only up to sixteen characters long. These are safeguards against a pattern that would otherwise run for ever, not choices about what is sent. Others are decisions that have not moved yet: the size of the batches the run's own file is written in, how long the end of a run waits for its reports, how deep the check of a reply's shape follows references. They move when the code around them is next touched, rather than in a sweep.