parfast Round Book

What it does and does not show

One machine, one set, one kind of damage: a single member deleted, the common real failure, costing 1,399 blocks. It ranks the field; the ladders after it characterise the two tools that matter across their range. ParPar appears only in the create chart because it creates parity and does not verify or repair. Every repair leg on this page is gated on a SHA-256 of every member afterwards, never on the tool's exit code (par2j returns 16 on a successful repair).

The second phase below rebuilds the same set at 100% parity, so the six repairing tools can be taken deeper. par2cmdline and phpar2 ran once per depth: their 4,096-block legs alone took an hour each, and deeper legs would only repeat what the shallower ones say. The quick tools' deeper depths (5,500 to 13,990) have not been run yet.

Read this section as who is the one to beat. The ladders after it are the characterisation, and they are of parfast and turbo only, because those are the two that can be run that far.

The "verify only" row is not a verify result

It says 20x on one machine while the verify figures everywhere else on this page say 2.1x to 2.6x. Both are measured correctly; the fixture explains the difference, and the number to quote is the small one. On a clean set parfast defers the parity payloads and turbo hashes them. At 15% parity that is 1.5 GiB out of 11.5 and barely shows; on this 100%-parity fixture the parity is as large as the payload, so deferring it halves the work (33.9 CPU-seconds against 62.8). The rest is parallelism: parfast ran that verify across 21.9 cores, turbo across 2.0, and that part holds at any parity level. So 20x is 1.9x of reading less times 10.7x of using the machine, and at a realistic parity level only the second factor remains.

Your drive decides this, not your memory

The obvious explanation for a verify advantage that fades on large sets is memory - that it lasts while the set fits in RAM and collapses when it does not. That is checkable, and it predicts something specific: two machines with the same processor and different memory should break at different sizes.

They do not. An M1 Ultra with 64 GiB and an M5 Max with 128 GiB, at sizes straddling each machine's own memory:

No cliff, on either machine. The M1 keeps 2.3x on a set half again its memory; the M5 keeps 2.5x on a set 56% bigger than its memory, still reading at 13 GB/s.

It is the drive. On the Intel machine at 80 GiB every tool converged on 0.78 GB/s, which is that drive's sustained read. When the disk is the slowest part it sets the pace for everyone and no amount of CPU shows through. On the Apple machines storage never becomes the slowest part: parfast reaches 13-14 GB/s while turbo sits at 5.2-5.5 GB/s, and the fact that they differ at all is the proof. If the drive were the limit they would be equal.

So the rule is: parfast's verify advantage holds until your storage becomes the bottleneck, and then every tool meets at the floor. Memory matters only because cache can hide a slow disk for a while. It is the drive that decides, and a slow drive reaches that floor at a much smaller set.

This cuts both ways. Most parity sets people actually verify sit comfortably inside the machine verifying them, so the advantage is the common case rather than a laboratory one. But a set larger than your memory on a slow drive is a disk benchmark, whoever runs it, and a ratio quoted without that sentence is worth nothing.

The single-file rows behave the same way on Apple as on Intel - 1.1x on both, at under 1 GB/s. One file is one chain, the parallelism has nowhere to go, and whatever is left is waiting on the device.

Two cases where parfast is slower

At 1 GiB in a single file parfast takes 1.37 s against par2cmdline's 1.02 s, and at 20 GiB in a single file 23.5 s against 18.9 s. Both are single-file sets, and both read at disk speed. A one-file set gives parfast nothing to spread across cores - the work is one chain, and the parallelism that wins the other rows has nowhere to go. The 20 GiB set appears twice in the table on purpose: as 20 files it takes 3.67 s, as one file 23.5 s. Same bytes, same parity, six times the wall - the shape of a set matters more than its size.

Yes, and it is a normal thing to want

The protection arithmetic says anyone with a set past about 23 GiB who wants real protection has to buy more parity than the customary 15% or 20%. This says that doing so is nearly free on parfast and expensive on everything else - so the region the format pushes large releases into is exactly the region where the gap is widest.

The direction is consistent across five sizes and two parity levels; at 50 and 80 GiB the ratio is a range, 23x to 32x, because of the spreads printed beside those rows.

Do these numbers understate the version that ships?

Every timing table on this page names its binary in its header line; all but the fast-mode ladders were built on 10 September, before any default flipped. Fast mode is now on by default on every class of machine this page measures, so every repair table here understates the shipping build for repairs past about 1,500 blocks, by the amounts the ladders below give. Nothing has been adjusted upward.

What the switch does on the two Intel machines, on the shipping build. Same fixture on both (32 members, 16,384 blocks at 256 KiB, 100% parity, scattered damage), eleven depths from the first where the switch engages, three repetitions, and an A/A control at every depth so a "change" smaller than the control is noise. Medians of the three:

Both classes win at every depth, and the win grows with the depth: +2% to +20% on the i5, +4% to +25% on the Core Ultra 9, three of three repetitions at every rung, every rung clearing its own A/A. An earlier build of this switch lost on the i5's class at nine of fourteen depths (worst -12.9% at 5,120 blocks), which is why that class was the last to be switched on: the fix holds the first solver stage on the ordinary arithmetic below the depth where the new kernel starts to pay, and the ladder above is that fix measured on the same machine, fixture and seeds. The pre-fix ladder is kept with the logs.

The Apple figures do understate, by a measured amount. Same fixture, three repetitions, an A/A control at every depth:

M1 Ultra +0.4% at 256 blocks rising to +20.7% at 16,384. M3 Ultra +0.8% rising to +20.0%. Near nothing at the shallow end, about a fifth off the deep repairs.

Because the i5's loss sits between the depths those ladders sampled, the same band was re-measured on three Apple parts at half steps - an M5 Max and two M1 Ultras, nine depths from 2,048 to 6,144, three repetitions each, an A/A control at every depth - looking for a loss the coarser ladders could have stepped over. There is none: across 27 depth-medians the worst is -2.6% (0.1 s, on the M5 at 4,608) and the best +5.3%, and every negative sits inside its own machine's A/A of ±0.8 to 1.8%. The Apple default holds where the AVX2 one does not.

That is not a licence to add 20% to the deepest rows. Those ladders stop at 16,384 blocks and the Apple tables run to 32,177; on the one machine measured past that band, an AMD Zen 5, the gain fell from +32.7% at 12,288 to +4.5% at 16,384. Nothing on this page has been adjusted upward.

Where this comes from

A PAR2 set is capped at 32,768 input slices, and a slice must be a multiple of 4 bytes (MAX_INPUT_SLICES, crates/nzbkit-base/src/par2gen.rs). The cap is on the count, not the size, so past a certain set size the slice has to grow - there is no other way to stay under it.

Damage on Usenet arrives one article at a time, and a poster sizes an article at roughly 768,000 bytes. A block is only repairable whole, so one lost article destroys one whole slice, however big that slice is. While a slice is one article, that is an even trade. Once a slice is four articles, every single lost article burns four articles' worth of recovery capacity.

So the share of articles you can afford to lose is not the parity percentage. It is parity x (article / slice), and past the knee it falls in exact proportion to the set size.

23.44 GiB is where it starts to bite

32,768 slices x 768,000 bytes = 25,165,824,000 bytes = 23.44 GiB (25.2 GB). Up to there a slice can stay the size of one article and 20% parity really does mean you can lose 20% of the articles. One byte past it, the slice has to grow and the protection starts falling. It is the largest set that gets the protection its parity setting advertises - which is exactly why the full-range benchmark fixture is 23 GiB.

What this calculation does not cover

It describes scattered damage - articles going missing here and there, which is the common case and the one that gets worse. It does not describe a contiguous loss, such as one whole file of a set being unavailable: there, r% parity covers r% of the volume whatever the slice size, because the lost articles fall inside the same slices rather than each spoiling a fresh one. The collapse is a granularity effect, not a capacity one.

It also assumes a poster who matched the slice to the article, which is the good case. A slice that straddles article boundaries does slightly worse than these lines, never better.

What it means for the numbers on this page

Every answer to the collapse makes the repair job harder, not easier. More parity means more recovery blocks to compute on create and more to solve on repair. Splitting into more sets means more sets to verify. Accepting the loss in protection means that when damage does arrive it is more likely to need a deep repair rather than a shallow one. That is the work the ladders measure, and it is why stopping the measurements at 20% redundancy and 2,098 blocks would have left out the cases people will actually hit.

Every repair is checked, never trusted

Every repair is checked by SHA-256 of all members against a gold manifest taken before damage. An exit code cannot stand in for it: par2j returns 16 on a successful repair, so anything that treats a zero exit as success silently discards every par2j result while appearing to work.

Every tool got the same command, and its best settings

Both tools get the same -t and -T on every leg. -T is file-hash parallelism and it cannot profitably exceed min(cores, files); parfast takes that by default, turbo has to be told. A flat -T16 was used at first, which capped turbo on the two boxes with more than 16 cores. That runs the dangerous way - it understates the competitor and inflates our ratio - so both affected rounds were re-run uncapped and the re-runs are what this page shows: the M3 full range at -T23 (turbo's deepest leg went from 1,122 s to 1,072 s, our ratio from 10.3x to 9.8x) and the M3 create sweep at -T32. The capped logs are kept on disk and not published. The ladders are unaffected: with ten members, 16 cannot bind.

Medians, and the spread behind them

Every figure is the median of the repetitions, and where the repetitions differ by 5% or more the spread is printed next to it in red, so a median cannot hide a swing. Reading those spreads turned up one real bias: the first repetition is systematically slower than the second on the boxes short of memory (1.66x over the first four rungs on the 64 GB i5, 1.14x on the 31 GB laptop, flat on the 128 GB and 256 GB Apple machines). That is a cold page cache and the write-back from set creation, not the tool. It runs in our favour to report it, since the cold pass costs parfast more than it costs turbo, so the small-job ratios are conservative; it is still wrong, and every round now starts with an untimed warm-up pass and repeats any rung whose repetitions disagree by 10% or more, up to five times while its legs stay under two minutes.

How much the repetitions disagree

Counted for every machine from the same rows the tables show:

The i5's worst is the cold first repetition at 64 blocks. A leg counts only under the round's declared command line, so a six-leg par2j thread-count probe on that machine is excluded from its 1,024 row. One lock holds each machine, and every leg samples the CPU used outside its own process tree and refuses to start above 10% of the box, waiting and retrying before it gives up; a leg measured beside somebody else's work is not slow, it is wrong, and the hash gate cannot tell.

What is not here

No extrapolated points, no fitted curves and no gap filling. A rung with no measurement is absent rather than estimated, and a round that was spoiled by another job on the same machine is thrown away rather than published with a caveat.