Known limitations

What RESTest does not do yet, or does in a way that may surprise you. Each entry says what happens today and, where there is something you can do about it, what that is. The list is kept to the current release: an entry leaves it in the release that lifts it, and that release's notes say so.

Pointed at somebody else's API

• A run waits when the API asks it to, but never stops for it. An operation answered 429, or 503 with Retry-After, is sent nothing for as long as the API asks, and the whole API when a second operation is told to wait while another waits (toldToWait). An API that asks for a wait longer than the run has left is obeyed until the run's time is up, and the run then ends with the number its findings give, not with one that says the API kept it waiting. What is not done yet:

  • the RateLimit and RateLimit-Policy fields an API may send to say how much of its quota is left are not read, nor the X-RateLimit-* headers some APIs use: only Retry-After is;
  • the request the API turned away is not sent again; the operation's next turn builds a new one;
  • the requests for an operation already on their way when it is told to wait still go out, at most schedule.workAheadFactor × engine.maxConcurrency of them; a wait of the whole API holds back every one of them, and those it still holds when the run stops sending are never sent, and are counted with the requests that got no reply;
  • a limit over the whole API is only known as one once a second operation is told to wait, so each of the first two operations draws a refusal of its own.

  Each 429 the document does not declare is still a warning, F920, and each 503 a fault, F121, like any other reply — there are few of them once the run waits. schedule.waitWhenToldTo=false sends to such an API as RESTest did before it waited; against an API whose limit you know, engine.mostRequestsPerMinute (the settings) set below it keeps the run under it rather than finding it by going over.

• A rate is kept to only when it is given, for the whole run, and where RESTest sends. By default nothing bounds a run but its requests in flight, sixteen at most (engine.maxConcurrency), and against a fast API that is thousands of requests a second. engine.mostRequestsPerMinute keeps a run to that many a minute, evenly spaced. What it does not do:

  • it is one rate for the whole API, not one per operation;
  • it counts the requests of the test, not the one that fetches the document when its address is given rather than a file;
  • it is kept to where RESTest sends: the network on the way may bring two requests a little closer together, most of all the first, behind the connection it opens;
  • with engine.followRedirects on, which it is not by default, a redirection is followed within the request that drew it, without waiting for a turn of its own;
  • the requests already built and waiting for their turn when the run's time is up are never sent, and are counted with the requests that got no reply; there is hardly ever more than one;
  • the page shows the idle share without saying how much of it the rate made; the summary and report.json (idleKeepingToTheRate, under engine) do.

• A run stops for a refused key only when the API refuses the whole opening lap. When every request of the opening lap is answered and every answer is 401, the run stops sending there, says which key it held, or how the document's is handed over, and ends with 5 (unauthorized). Every other run the API refuses goes on until its budget ends: one that loses its key part-way — revoked, expired, or taken away by something the run sent, such as signing out or changing its own password; one with no key against an API some of whose operations anybody may call; one answered 403 rather than 401; one answered with redirections to a sign-in page rather than 401; one a request of whose lap got no reply at all; and one with no opening lap, or whose lap never ends because its budget or schedule.requestsBeforeStopping runs out first. A 401 the document declares is not a fault, and one it leaves out is reported, F920, on every request; the summary at the end counts the replies by status. Try a new key with a short --budget first.

• A run does not stop when the API stops answering. The limiter falls to one request in flight as unanswered requests pile up, and the run keeps trying until its budget ends. Keep --budget to what you are prepared to spend on an API that may go down. An API that still takes connections and no longer answers on them has every request from then on reported as a fault, F312, though the run's own pace may be what brought it down; checks.noReplyInTime=false stops that (the faults).

Every request names the tool in its User-Agent (engine.userAgent), so that an API's owner can tell RESTest's traffic apart and refuse it, unless the document declares that header as a parameter of its own. Whether you may test an API at all is not something any tool can check.

Credentials

• Only credentials you already hold. An API key, or a bearer token or cookie given with the place it goes, through --auth (an API that asks for a key). RESTest does not sign in by itself, does not do OAuth2, does not refresh what expires, and holds one set of credentials per run.

What a run sends

• Dependencies between parameters are not read. RESTest 1.x read them in IDL - "if from is given, to must be too" - and built requests that honoured them. This version reads no IDL yet, whether in the document or beside it, so a request may break such a rule and be refused for it; the refusal is counted like any other.
• Optional parameters and optional bodies share one number. With generation.optionalParametersBySize off, each optional parameter is decided on a coin weighted by generation.optionalBodyChance, the same number that decides whether a request sends a body it may leave out, so changing one changes the other (the settings). Leave optionalParametersBySize on, which is the default, if you want to change only bodies.
• A value can be offered back to the operation that first used it. A value an accepted request carried is offered to the other operations and not back to its own, but a value seen again replaces what was known of it. An e-mail address goes from a registration to a login; the login is accepted; the address is then remembered as the login's, and the registration is offered it again and refused as already made. There is nothing to set; it costs the run some refused requests.

What a run says

• A reply nobody could judge does not change the exit code. A run counts the replies each check could not judge, and why — a body the document gives no shape for, a shape the checker cannot use — and says so in its summary and under notChecked in report.json (what a run leaves behind); but those are no faults, not even with checks.failOnWarnings, so a run whose only replies were of that kind still ends with 0.

• With checks.undeclaredStatusCode or checks.undeclaredContentType off, some replies go unchecked without a word. The shape of a reply whose status code or content type the document does not declare is never compared with anything, since the document gives no shape for it, and is not counted under notChecked: those replies are what F920 and F921 report. With their check switched off, nothing says those replies were not checked.

• A reply with a body and no Content-Type, to a status code declared as */*, is neither checked nor counted. */* promises any type, so F921 passes it, and with no type to look a shape up by, its body is not compared with the shape. A Swagger 2.0 document that names no type is read as */*, so this is where it is met.

• A check is switched off for the whole run, never for one operation. Turning checks.replyOffSchema off because one operation's error replies are not the shape its document gives hides every other reply off its schema too, on every operation (the faults).

• A document that leaves out its refusals makes many warnings. Every reply whose status code the document does not declare is a warning, F920, and so is every reply sent as a content type it does not declare, F921; a document that lists only what goes well can make most of a run's replies warnings. They fail no run unless checks.failOnWarnings asks, and each line of the summary says where to look, but RESTest cannot tell, reply by reply, whether the API or the document is the one to change: fix the document, or switch the check off for the run (checks.undeclaredStatusCode, checks.undeclaredContentType; the faults).

• Many F206 are an API reading leniently what its document forbids. A request RESTest broke on purpose that the API accepted is an error, F206, however the API read it. Many frameworks read an empty value or null in an optional place as if it had been left out, trim a space from around a number, turn a number or true into the word the document asks for, or take a date-time without its offset, and each of those is reported. Of 28 F206 read by hand from runs of 52 APIs of the Web Fuzzing Dataset, 22 were of that kind, and 3 the API taking what it should have refused. Read what each one sent before changing the API, and loosen the document where the API is right; or switch the check off for the run (checks.invalidDataAccepted; the faults).

• The warnings written out on the screen are the first ten of any kind. The screen writes out the first 50 faults whole (report.faultsShownOnTheConsole), of which at most ten may be warnings (report.warningsShownOnTheConsole), so that a 500 found after a flood of F920 still reaches it. Those ten are the first warnings found, whatever their kind: an F200 found after ten F920 is only counted there. Read the counts under the faults, and report.json, which writes out the first five of each kind on each operation.

• A type that only resembles a declared one is reported. application/problem+json, where the document declares application/json, is F921, though both are JSON (the faults).

• A reply the document declares in a form RESTest cannot read is reported as undeclared. The library RESTest reads documents with drops a reply written as something other than an object — "404": "not found" — and says so before the run, without naming the operation. That operation's 404s are then F920, though the document meant to declare them. On an operation one of whose replies RESTest knows it could not read, such as one given as a reference to a reply the document never defines, a status code it could not find among what it read is left alone, and counted under notChecked.

• A body the document describes only in words is not checked against a type. A reply with a body whose status code the document declares with no content type at all is left alone, whatever type it comes as, and counted under notChecked. A Swagger 2.0 document that names no type at all is read as promising any, */*, so its replies count as checked.

• Not every failure of RESTest's own ends in 4. Exit code 4 means RESTest itself broke (the exit codes), but three such failures still read as something the API did (ADR-0015):

  • a key that cannot be added to a request: the request is not sent and is recorded as a failure on the way to the API, so a run where every request goes that way ends with 3 and points at the address, not at the key;
  • a failure in RESTest's own code inside the engine, in the code that records what went over the wire, say, which is recorded as a network failure;
  • a failure while the run deals with an answer, such as handing it to a series, which is lost: the request counts as answered and no report hears of it.

  A run that ends with 3 although the API is reachable is worth reading as one of these.

• A request on a connection something in between dropped is reported as one the API sat on. The engine keeps a connection open between requests, for up to five minutes. A load balancer or a firewall that drops one without a word leaves a request sent on it unanswered, and RESTest reports it as F312 once it stops waiting, although it reached nobody (the faults).

• Over HTTPS, an API that answers in HTTP/1.0 loses every other request. An API that answers in HTTP/1.0 without asking to keep the connection closes it after every reply, as Python's http.server does unless its protocol_version is HTTP/1.1. Over plain HTTP, RESTest opens a new connection for the next request. Over HTTPS, it sends the next request on the connection the API has closed, and that request is counted under no reply, although it reached nobody. An API that answers in HTTP/1.1 is not affected.

• With redirections followed, a request the next address sits on is reported on the operation. With engine.followRedirects on, which it is not by default, an operation that sends the request on to another address that takes it and never answers is reported as F312, and the curl printed with it shows only the first answer, the redirection. The same goes for the reply at the end of the redirections, which is judged against what the document declares for the operation that was asked: a login page answered with 200 as text/html is F921 there.

• A reply that comes, but slowly, is not reported. Only a request that got no reply before RESTest stopped waiting is a fault; one answered in twenty-nine seconds, against the half a minute it waits by default, is not. Nor is a reply whose status line and headers arrived and whose body never finished.

• report.json does not say that a run broke, nor does the page drawn from it. When a run ends with 4, the file reads like that of a finished run, its verdict says what the run amounts to otherwise, and the page gives the number that verdict ends with. Read the exit code, not only the file.

• The page shows the first request of each kind of fault on an operation, of the up to five report.json writes out whole; the others are in report.json. It needs JavaScript to draw itself, and holds the whole of report.json, so it is about as large.

The files a run reads

• The settings file has no JSON Schema. A dictionary and a plan have one (the JSON Schemas); a file of settings is checked only by RESTest, as it starts. Start one from what restest run --print-settings writes, which RESTest reads back as it is.
• A file is read up to three million characters and fifty levels deep. A dictionary, a plan or a file of settings longer or deeper than that is refused by name, before the run starts. A dictionary a program writes for a large API is tens of thousands of characters and three levels deep, so neither limit is one a file somebody meant reaches. A list or a map that contains itself, through a YAML alias to something it is inside of, is refused the same way.