fix(bin): benchmark HPS without zero RTT

[mxinden]

The "100-seq-conn/1-1b-resp (aka. HPS)" benchmark supposedly measures the time for 100 uniform handshakes. Though neqo-client by default uses a resumption token for zero RTT if available. Thus the first is a normal and all following 99 handshakes are a zero RTT handshake.

To simplify the benchmark and to measure the same handshake type, establish a single connection within the benchmark function and have criterion do all the iterating.

---

Benchmark introduced in #1758.

[github-actions]

Benchmark results

Performance differences relative to 5dfe106.

• coalesce_acked_from_zero 1+1 entries
  time: [197.99 ns 198.45 ns 198.95 ns]
  change: [-1.1228% -0.8056% -0.4875%] (p = 0.00 < 0.05)
  Change within noise threshold.

• coalesce_acked_from_zero 3+1 entries
  time: [239.92 ns 240.61 ns 241.30 ns]
  change: [-0.5675% -0.1494% +0.2677%] (p = 0.48 > 0.05)
  No change in performance detected.

• coalesce_acked_from_zero 10+1 entries
  time: [238.10 ns 239.31 ns 240.74 ns]
  change: [-1.1814% -0.5837% +0.0046%] (p = 0.05 > 0.05)
  No change in performance detected.

• coalesce_acked_from_zero 1000+1 entries
  time: [219.54 ns 226.90 ns 243.65 ns]
  change: [-5.5494% +0.3998% +7.0335%] (p = 0.85 > 0.05)
  No change in performance detected.

• RxStreamOrderer::inbound_frame()
  time: [119.22 ms 119.29 ms 119.36 ms]
  change: [-0.0279% +0.0652% +0.1595%] (p = 0.18 > 0.05)
  No change in performance detected.

• transfer/Run multiple transfers with varying seeds
  time: [125.08 ms 125.36 ms 125.63 ms]
  thrpt: [31.840 MiB/s 31.909 MiB/s 31.980 MiB/s]
  change:
  time: [+4.4985% +4.8232% +5.1618%] (p = 0.00 < 0.05)
  thrpt: [-4.9085% -4.6013% -4.3049%]
  💔 Performance has regressed.

• transfer/Run multiple transfers with the same seed
  time: [126.10 ms 126.25 ms 126.41 ms]
  thrpt: [31.642 MiB/s 31.682 MiB/s 31.722 MiB/s]
  change:
  time: [+4.4303% +4.6164% +4.8141%] (p = 0.00 < 0.05)
  thrpt: [-4.5930% -4.4127% -4.2424%]
  💔 Performance has regressed.

• 1-conn/1-100mb-resp (aka. Download)/client
  time: [1.0709 s 1.1040 s 1.1385 s]
  thrpt: [87.836 MiB/s 90.576 MiB/s 93.382 MiB/s]
  change:
  time: [-3.1625% +1.1589% +5.2645%] (p = 0.61 > 0.05)
  thrpt: [-5.0012% -1.1456% +3.2658%]
  No change in performance detected.

• 1-conn/10_000-1b-seq-resp (aka. RPS)/client
  time: [385.44 ms 387.96 ms 390.46 ms]
  thrpt: [25.611 Kelem/s 25.776 Kelem/s 25.944 Kelem/s]
  change:
  time: [-1.1191% -0.0983% +0.8882%] (p = 0.85 > 0.05)
  thrpt: [-0.8804% +0.0984% +1.1318%]
  No change in performance detected.

• 1-conn/1-1b-resp (aka. HPS)/client
  time: [42.782 ms 42.942 ms 43.126 ms]
  thrpt: [23.188 elem/s 23.287 elem/s 23.375 elem/s]

Client/server transfer results

Transfer of 134217728 bytes over loopback.

Client	Server	CC	Pacing	Mean [ms]	Min [ms]	Max [ms]	Relative
msquic	msquic			757.9 ± 301.8	391.4	1077.6	1.00
neqo	msquic	reno	on	2103.4 ± 262.8	1873.8	2632.4	1.00
neqo	msquic	reno		1996.4 ± 101.4	1894.8	2196.3	1.00
neqo	msquic	cubic	on	1992.4 ± 186.5	1821.8	2471.5	1.00
neqo	msquic	cubic		2097.0 ± 233.5	1808.2	2467.3	1.00
msquic	neqo	reno	on	3286.9 ± 190.6	3138.0	3656.4	1.00
msquic	neqo	reno		3312.9 ± 242.2	3061.6	3781.5	1.00
msquic	neqo	cubic	on	3405.2 ± 215.8	3264.8	3843.6	1.00
msquic	neqo	cubic		3323.8 ± 174.3	3135.7	3785.3	1.00
neqo	neqo	reno	on	3125.1 ± 220.1	2972.3	3557.6	1.00
neqo	neqo	reno		3061.7 ± 134.4	2952.0	3383.3	1.00
neqo	neqo	cubic	on	3397.7 ± 366.9	3137.1	4208.6	1.00
neqo	neqo	cubic		3265.3 ± 173.4	3037.5	3599.8	1.00

⬇️ Download logs

[martinthomson]

This looks fine, but I noticed that these benchmarks are awfully slow. It might pay to adjust the time or the number of iterations in criterion.

[mxinden]

This looks fine, but I noticed that these benchmarks are awfully slow. It might pay to adjust the time or the number of iterations in criterion.

👍 this pull request significantly reduces the HPS benchmark run time. Will keep this in mind for the RPS benchmark, which is still slow (~40s).

Benchmarking 1-conn/1-100mb-resp (aka. Download)/client: Collecting 10 samples in estimated 10.844 s (10 iterations)
Benchmarking 1-conn/10_000-1b-seq-resp (aka. RPS)/client: Collecting 100 samples in estimated 39.564 s (100 iterations)
Benchmarking 1-conn/1-1b-resp (aka. HPS)/client: Collecting 100 samples in estimated 8.5971 s (200 iterations)