Explained · two
A download that goes perfectly is the easy case, and every client handles it competently. The interesting differences appear when part of the post is missing, which happens far more often than the clean case would suggest. This is what is actually going on, and why it costs so much time.
The shape of the problem
A Usenet post is not one file. It is thousands of small articles, each stored independently on each provider's servers, and the NZB you download is a list of their names. A 6.5 GB release is roughly 9,000 of them.
Articles go missing for ordinary reasons. Providers keep them for a fixed retention period and then delete them. Some providers never received a particular article in the first place, because propagation between servers is imperfect. Occasionally an upload was incomplete when it was posted, and part of it never existed anywhere. Whatever the cause, the effect is the same: your client asks for an article, and the answer is a refusal rather than data.
This is normal enough that the people who post to Usenet plan for it. Almost every release ships with parity data alongside it, usually PAR2 files, which is extra redundant information calculated from the original. If some of the payload is missing, parity can reconstruct it, provided you have enough of it. A typical post carries around 10% parity, so a post can lose a fair fraction of itself and still be perfectly recoverable.
So a damaged post is not usually a broken download. It is a download that requires some arithmetic before it is complete. The question is only how long a client takes to work that out, and that is where they differ by a factor of two or more.
The expensive part
When a server does not have an article it returns a refusal, and the client then tries the next server on your list. That is the correct thing to do, because the second provider very often does have it. The cost appears when nobody has it.
In that case the client walks your entire provider list, one server at a time, and gets refused by every one of them before it can conclude the article is genuinely gone. The walk is serial, and refusals are slower than people assume. We measured them directly, on an idle line, with no payload moving at all:
| provider | time to answer a normal request | time to refuse a missing article |
|---|---|---|
| Provider A | 77.5 ms | 78.8 ms |
| Provider B | 10.5 ms | 454 ms |
| Provider C | 10.8 ms | 871 ms |
| Provider D | 9.9 ms | 1,227 ms |
| Provider E | 10.6 ms | 2,239 ms |
Two things stand out. First, the spread is enormous: a refusal costs anywhere between 79 ms and 2.2 seconds depending on which provider is answering, a difference of nearly thirtyfold. Second, one provider refuses in essentially the time it takes to ask, while another takes more than two hundred times its own normal response time to say no. That is not network latency, it is work happening on their side, and asking in fewer round trips barely helps.
Add it up and a full walk across five providers, for a single article that no longer exists, costs about 5 seconds. Now consider that a damaged post has many such articles, and that a client which processes them in sequence pays that cost repeatedly while your connection sits idle. The download is not slow because the data is arriving slowly. It has stopped arriving at all, and the client is waiting to be told what it already has enough information to work out.
The fix
The insight is not to make refusals faster, because we do not control the providers. It is to notice when the answer no longer matters.
At any moment during a download, the client knows two things: how many pieces are still unaccounted for, and how much parity data it already holds. If the parity in hand is enough to rebuild everything still outstanding, then the answer to "does anybody have this article?" cannot change the outcome. Whether it is a slow yes or a slow no, the file gets rebuilt either way. Continuing the walk buys nothing except delay.
So the current build stops asking at that point and goes straight to repair. The reconstruction itself is quick and always was: about 2.3 seconds on a 6.5 GB post, and that figure is the same in every version we have measured. The time saved is entirely the waiting that no longer happens.
| 6.5 GB post, 60 missing articles | time to a verified file |
|---|---|
| waiting for every provider to refuse | 23 s |
| stopping once parity covers the gap | 13 s |
| of which, the actual repair arithmetic | 2.3 s |
The same change does nothing at all on an undamaged post, which is the clearest evidence that it is doing what we think: with no missing articles there is no walk to cut short, and the two builds finish a clean 6.5 GB job in an identical 7 seconds.
It is a trade, not a free win. Repairing early means holding the parity data in memory instead of discarding it and fetching more later, so peak memory (RSS) on a heavily damaged post rises from about 0.8 GB to about 1.3 GB, and we pull roughly 400 MB more from the providers. On an undamaged post memory is unchanged at 0.24 GB. We think that is the right trade for most people, since the memory is only spent in the case where it buys you ten seconds, but it is a real cost and there is a setting for it.
Provider names are withheld in the refusal table deliberately. The figures are a property of one measurement on one evening from one location, and a provider that indexes differently, or that we happened to reach on a bad night, should not be labelled slow on the strength of a single round. What matters for the argument is the shape: refusals vary enormously, and the total is large enough to dominate a damaged download. The per-provider figures and the raw probe output are in our internal record, and the end-to-end numbers they explain are on the benchmarks page.