How we test, in full

The rig

Same machine, same line, same content

Every client in a comparison runs on one machine, against the same providers, on the same content, within the same round. Nothing is carried over from a previous day.

The machines. An Apple M1 Ultra on a 10 GbE line in Europe, and an M3 Ultra on a slower line in the United States. Comparisons never mix the two: a table is one machine.
The line matters more than the processor. On a 10 GbE line most clients can saturate the connection on an easy job, which is why the interesting scenarios are the ones where something other than raw bandwidth decides the outcome.
Providers. Five commercial providers, the same set for every client, over TLS. We exclude one provider from these rounds because it restricts how many locations may connect at once, which would make a shared test machine unfair to whoever ran second.
Connections. Every client gets the same number per provider in a given round, and the round says which. More is not automatically better; we have measured our own client holding the same finish time across a ninefold change in connection count.

The rule that decides everything

A client is judged on the file, not on its own status message

This is the single most important thing on this page. Every leg ends by finding the largest file the client produced, anywhere in its output, and hashing it against a known-good copy. Size and MD5, on every leg, not on a sample.

We do this because a client's own report of success is not reliable evidence. We have measured a client reporting done on a job whose output directory was empty. We have measured another reporting Completed having delivered raw archive volumes it never unpacked. A repair tool in wide use exits with a failure code after a successful repair. None of those clients was lying; status strings simply mean different things to different authors, and a benchmark that trusts them is measuring vocabulary rather than behaviour.

So the finish line is the same for everyone: the payload exists, at the right size, with the right hash. If a client stops before that, it does not get a time, and the whole column is marked rather than quietly omitted. A slow client that finishes is doing better than a fast one that does not, and the table should say so.

The corollary is that a non-finish is a result, not an error on our part. When we publish one we describe the end state in plain terms, including anything that suggests it is specific to that job rather than general, and we say what the client did instead of finishing.

Order and repetition

Why legs alternate, and why single runs are not published

Provider throughput varies minute to minute, sometimes by a factor of two or three. Any two measurements taken at different times are partly measuring the weather.

So clients run back to back and interleaved, and the order reverses between repetitions, so that no client is always the one going first into a warm or cold line. A result is the pattern across repetitions, not a best run. When a spread is wide, we publish the range rather than an average that would hide it, and if we do not know why a client's numbers are spread out, we say so instead of inventing a cause.

Comparisons across different rounds are avoided wherever possible. If a claim requires two numbers, we prefer to re-race both in one round rather than subtract one day's figure from another's, because rig state, provider conditions and client versions all drift. When we compare two of our own builds, both builds run in the same round, alternating, on the same content.

The unglamorous checks

What runs before and during every leg

Most of these exist because a silent fault once produced a healthy-looking, publishable, wrong number. Each one now runs automatically and stops the round rather than warning.

Every provider answers before the leg starts. A leg run while two providers are down measures an outage, and it looks exactly like a slow client.
The binary is identified by hash before every leg, not by filename or version string. Two builds of ours reported the same version number while differing by 88 commits; only the hash tells them apart, and tables cite it.
Every client runs as native code. On Apple machines an incorrectly built helper can silently run under translation, which costs processor time that belongs to our packaging choice rather than to the client's code. We check the running process, not just the file.
Open-file limits are raised for everyone, identically. macOS starts programs with a limit of 256 open files, which a full set of connections can exceed. A client that hits it tends to vanish without a crash report, which reads exactly like a failure of the software. One competitor's apparent non-finish turned out to be this; raised, the identical job finished in 44 seconds.
Disk traffic is recorded per leg, which doubles as a check that nothing else was using the machine. Those counters cover the whole disk rather than one program, so if a second job were running, its work would appear inside our measurements. Flat readings across a round are how we know a result belongs to the client it names.
Each client writes to its own output tree, and the tree is wiped before each leg, so no run can be satisfied by a previous run's file.

Where a client needs particular settings to perform well, it gets them, and the round says which. Examples: request pipelining enabled where a client ships with it off, duplicate checking disabled when the same job is downloaded repeatedly, and unpack helpers taken from the client's own bundle rather than a system copy. The aim is that each client is measured at its best rather than at its defaults, since defaults are a separate argument from capability.

What we will not do

The limits of these numbers

We do not publish a figure we have not measured. If a number is derived rather than observed, it is labelled as derived, and we would rather leave a gap than fill it with arithmetic that reads like a measurement.
We do not quietly update a losing result. A leg we lose stays published until a re-run replaces it, and when a loss turns into a win the change gets an explanation beside the table rather than a silent edit.
Provider legs are not yet independently reproducible, because the content is not ours to distribute. The component tests, which use generated content, are. We are preparing test files we can publish so the provider legs become checkable too.
These are our benchmarks of other people's software. Every version is named and dated, and when a client ships a new release we re-run the legs it affects rather than leaving a stale comparison up. If you think a result is wrong, the settings and versions are published so it can be argued with specifically.