Hi David, On Mon, 2025-11-17 at 16:13 +1100, David Gibson wrote:
Well, that sure is a confusing pattern. Looks like tcp_notsent_lowat is the biggest culprit... except for the outlier where it seems ok with both custom notsent_lowat and custom congestion control.
How repeatable are each of these results?
Very repeatable. I reran the tests from a different host (my laptop over home Wifi) targeting different servers (two servers that I control) and I got similar results.
Actually, is it just pasta that's changed in that time? Or could it be a new kernel version or something else?
I'm always just using the most recent packages in Fedora, so both pasta and the kernel would have upgraded in that time. Specifically: - Around June 6th, I didn't notice any issues; I was using "kernel-6.14.9-300.fc42.x86_64" and "passt-0^20250512.g8ec1341-1.fc42.x86_64". - Around August 20th, I started noticing intermittent issues; I was using "kernel-6.15.9-201.fc42.x86_64" and "passt-0^20250805.g309eefd-2.fc42.x86_64". - Around September 26th, the problem became persistent; I was using "kernel-6.16.8-200.fc42.x86_64" and "passt-0^20250919.g623dbf6-1.fc42.x86_64". - I ran the tests in this email with "kernel-6.17.8-300.fc43.x86_64" and "passt-0^20250919.g623dbf6-1.fc43.x86_64".
I also find it quite interesting that setting any of the sysctls individually or in pairs improves the upload speeds, but setting all 3 at once slows it down drastically.
I'm curious about the case with default congestion control, but custom lowat and mem_max. I think that one's missing from the table above.
Ok, I ran some more tests. The problematic combination seems to be network=pasta direction=upload rtt/ping=large net.core.wmem_max=7500000 net.ipv4.tcp_notsent_lowat=131072 net.ipv4.tcp_congestion_control=bbr and all other parameters seem to not make much difference. The data is somewhat noisy, but still fairly consistent. I ran the tests for all combinations for 10MB twice and 100MB once. The full data is below: network ping size wmem rmem lowat cngst qdisc down up pasta 170ms 100MB # . # # # 17.69 180.00 pasta 170ms 100MB # # # # . 15.41 180.00 pasta 170ms 100MB . # # . . 15.23 180.00 pasta 170ms 100MB # . # # . 11.86 180.00 pasta 170ms 100MB # # # # # 10.90 180.00 pasta 170ms 100MB # . # . . 23.91 128.18 pasta 170ms 10MB . . . # # 6.98 120.00 pasta 170ms 10MB # # # # # 6.26 120.00 pasta 170ms 10MB # . # # # 6.19 120.00 pasta 170ms 10MB # . # # # 6.10 120.00 pasta 170ms 10MB # . # # . 6.05 120.00 pasta 170ms 10MB # . # # . 5.93 120.00 pasta 170ms 10MB # # # # . 5.57 120.00 pasta 170ms 10MB . # . # . 4.45 120.00 pasta 170ms 100MB . # # . # 17.46 118.70 pasta 50ms 100MB # . # # # 6.57 117.59 pasta 170ms 100MB . . # . # 12.91 94.23 pasta 170ms 100MB . . # . . 12.46 89.22 pasta 170ms 100MB . # # # # 15.05 85.79 pasta 170ms 100MB . . . # # 12.92 82.36 pasta 170ms 100MB . . # # . 15.03 76.69 pasta 170ms 100MB . # . # . 18.91 74.79 pasta 170ms 100MB . . # # # 21.75 66.75 pasta 170ms 100MB # # # . . 23.02 66.36 pasta 50ms 100MB . . # # # 7.57 58.58 pasta 50ms 100MB . . # # . 8.46 58.24 pasta 170ms 100MB # # # . # 12.15 55.27 pasta 170ms 100MB . # # # . 17.17 54.86 pasta 50ms 100MB . # # . . 6.80 51.66 pasta 170ms 100MB # . # . # 11.89 51.40 pasta 170ms 10MB . . . # . 7.25 51.27 pasta 170ms 10MB . . # # # 6.98 49.34 pasta 50ms 100MB . # # # # 6.79 47.94 pasta 170ms 100MB . . . # . 11.98 45.94 pasta 50ms 100MB . . # . . 8.62 45.85 pasta 50ms 100MB # . # # . 7.05 45.24 pasta 50ms 100MB . . # . # 8.25 44.98 pasta 170ms 10MB # # # # # 6.91 44.95 pasta 170ms 10MB # # . . . 6.16 44.43 pasta 50ms 100MB . # # . # 8.47 44.18 pasta 50ms 100MB # . # . . 6.97 43.38 pasta 170ms 10MB # . # . . 5.27 42.54 pasta 170ms 10MB . # # . . 4.92 42.46 pasta 170ms 10MB # # # . # 6.15 42.36 pasta 170ms 10MB . # # . # 7.06 41.61 pasta 50ms 100MB . # # # . 6.66 40.83 pasta 170ms 100MB . # . # # 14.50 40.45 pasta 170ms 10MB # # # . # 7.10 39.84 pasta 170ms 10MB . # # . # 5.76 38.34 pasta 50ms 100MB # # # . # 6.81 37.96 pasta 170ms 10MB # # . # . 8.50 37.88 pasta 170ms 10MB . # # # . 5.47 37.85 pasta 170ms 10MB # . # . # 3.82 37.46 pasta 170ms 10MB # # # . . 6.79 37.19 pasta 170ms 10MB . # # # . 5.34 37.03 host 170ms 100MB # . # # # 15.09 36.65 pasta 170ms 10MB # # # . . 7.19 36.25 pasta 50ms 100MB # . # . # 7.63 35.72 pasta 170ms 10MB # . # . # 6.53 35.69 pasta 170ms 10MB # # # # . 8.35 35.69 pasta 170ms 100MB # . . # # 13.78 35.65 pasta 170ms 100MB # # . # # 16.32 35.20 host 170ms 100MB . . # # # 13.61 33.89 host 170ms 100MB . . . # . 14.76 33.87 host 170ms 100MB # . . # # 16.60 33.79 host 170ms 100MB # . # # . 12.71 33.41 pasta 170ms 100MB # # . # . 15.82 33.28 pasta 170ms 100MB . # . . # 13.55 33.25 host 170ms 100MB # # . # . 13.97 33.19 host 170ms 100MB . . # # . 17.49 33.15 host 170ms 100MB . # # # . 14.68 32.89 host 170ms 100MB . # # # # 17.12 32.71 host 170ms 100MB # . . # . 22.21 32.71 host 170ms 100MB # # # # . 12.18 32.44 pasta 170ms 100MB # . . # . 16.30 32.23 host 170ms 100MB # # . # # 15.46 32.08 host 170ms 100MB . # . # # 14.47 31.97 host 170ms 100MB . . . # # 16.80 31.84 host 170ms 100MB . # . # . 13.04 31.77 pasta 170ms 100MB . . . . # 15.85 31.74 host 170ms 100MB # # # # # 19.09 31.57 pasta 170ms 10MB . . # # . 5.62 30.59 pasta 170ms 10MB . # # # # 5.58 30.46 pasta 170ms 100MB . . . . . 11.22 30.22 pasta 170ms 100MB # . . . # 18.42 30.17 pasta 170ms 100MB . # . . . 14.93 30.09 pasta 170ms 100MB # # . . # 14.24 29.86 pasta 50ms 100MB # # # . . 6.57 29.81 host 170ms 100MB . # . . . 15.52 29.10 pasta 170ms 10MB . . # # . 5.11 28.77 host 170ms 100MB . # . . # 15.73 28.63 host 170ms 100MB # # # . . 16.57 28.54 host 170ms 100MB . # # . # 15.03 28.41 host 170ms 100MB . # # . . 14.29 28.19 host 170ms 100MB . . . . . 15.26 28.04 pasta 170ms 100MB # # . . . 14.35 27.76 pasta 170ms 10MB . . # . # 5.60 27.74 pasta 50ms 10MB # . # # . 1.37 27.68 pasta 170ms 10MB . # . # # 4.47 27.56 host 170ms 100MB . . . . # 16.84 27.43 host 170ms 100MB # # # . # 14.88 27.27 host 170ms 100MB . . # . # 17.53 26.75 host 170ms 100MB # . # . # 15.83 26.69 pasta 170ms 10MB . # . # # 5.29 26.13 host 170ms 100MB # # . . # 16.73 26.04 host 170ms 100MB # # . . . 19.30 25.79 pasta 170ms 100MB # . . . . 14.31 25.78 host 170ms 100MB # . # . . 20.52 25.61 pasta 170ms 10MB # . # . . 6.37 24.39 pasta 170ms 10MB . . # . . 7.07 24.33 host 170ms 100MB # . . . # 17.69 24.23 host 170ms 100MB # . . . . 19.94 23.75 pasta 50ms 100MB # # # # # 6.37 23.43 pasta 50ms 10MB # # # # . 1.52 22.99 pasta 50ms 100MB # # # # . 6.31 22.65 pasta 170ms 10MB . . # . # 6.52 22.01 pasta 50ms 10MB # . # # . 1.35 21.59 pasta 170ms 10MB . . . # . 7.15 21.35 pasta 170ms 10MB . . # . . 6.40 20.86 pasta 50ms 100MB . . . # . 8.43 20.49 pasta 50ms 100MB # # . # . 6.50 19.70 host 170ms 10MB # . . # # 4.71 19.61 pasta 50ms 100MB . # . # # 8.03 19.47 pasta 50ms 100MB . . . # # 8.40 19.29 pasta 170ms 10MB # # . # # 7.63 18.92 pasta 50ms 100MB . # . # . 7.09 18.88 host 50ms 100MB . . # # . 7.94 18.85 host 50ms 100MB . # . # . 7.15 18.74 host 170ms 100MB . . # . . 13.75 18.71 host 170ms 10MB . . . # . 4.52 18.57 host 170ms 10MB . . . # . 6.24 18.52 pasta 170ms 10MB . . # # # 7.04 18.48 host 50ms 100MB # . . # . 6.38 18.45 host 50ms 100MB . # . # # 7.37 17.89 pasta 170ms 10MB . # . # . 7.35 17.88 host 170ms 10MB # # . # . 6.11 17.76 host 50ms 100MB . . # . . 8.20 17.56 host 170ms 10MB . . # # . 6.56 17.51 pasta 170ms 10MB # . . # # 6.39 17.51 host 170ms 10MB . # # # # 4.86 17.46 host 170ms 10MB # # . # # 6.91 17.45 host 50ms 100MB # # # # # 7.11 17.45 host 170ms 10MB . # . # . 4.93 17.45 host 170ms 10MB . # # # . 5.14 17.44 pasta 170ms 10MB . # # # # 5.69 17.41 host 170ms 10MB # . # # # 3.64 17.39 pasta 170ms 10MB # # . # # 6.59 17.33 host 170ms 10MB . # . # # 4.83 17.33 pasta 170ms 10MB # . . # . 5.46 17.30 pasta 50ms 100MB # . . # # 6.23 17.30 pasta 170ms 10MB # . . # . 4.30 17.25 host 170ms 10MB . . # # # 5.64 17.25 pasta 50ms 100MB # . . # . 6.76 17.20 pasta 170ms 10MB . . . # # 7.77 17.19 host 50ms 100MB . . . # # 7.61 17.18 host 170ms 10MB . . . # # 6.01 17.16 host 50ms 100MB . # # # # 6.44 17.04 host 170ms 10MB # # # # . 5.08 17.04 host 170ms 10MB . # # # . 4.62 17.02 host 50ms 100MB # . # # # 6.51 17.02 host 50ms 100MB # . . # # 6.20 16.99 host 50ms 100MB # . # # . 6.43 16.98 pasta 170ms 10MB # # . # . 5.87 16.88 pasta 50ms 100MB # # . # # 6.36 16.86 host 170ms 10MB . . . # # 5.03 16.85 host 170ms 10MB # # # # # 3.44 16.83 host 170ms 10MB # # . # # 6.12 16.83 host 170ms 10MB # . # # . 5.62 16.83 pasta 170ms 10MB # . . # # 14.82 16.83 host 170ms 10MB # . . # . 5.14 16.82 host 170ms 10MB # . # # . 6.47 16.81 host 170ms 10MB . # . # # 5.13 16.79 host 170ms 10MB # . . # . 5.72 16.78 host 170ms 10MB # # # # # 5.49 16.78 host 170ms 10MB # # # # . 4.72 16.77 host 170ms 10MB # # . # . 5.79 16.76 host 50ms 100MB . . # # # 7.41 16.71 host 170ms 10MB . . # # # 6.10 16.70 host 50ms 100MB . # . . # 6.69 16.68 host 170ms 10MB . # # # # 6.06 16.67 host 170ms 10MB # . . # # 6.01 16.65 host 170ms 10MB . # . # . 5.32 16.58 host 170ms 10MB . . # # . 4.90 16.55 host 50ms 100MB # # . # # 6.47 16.40 pasta 170ms 10MB . # # . . 4.81 16.19 host 170ms 10MB # . # # # 5.77 16.06 host 50ms 100MB . # # # . 6.67 15.97 pasta 170ms 10MB # # . . # 6.78 15.23 host 50ms 100MB # # # # . 6.56 14.97 pasta 50ms 100MB . . . . . 8.19 14.95 pasta 170ms 10MB . . . . . 6.10 14.69 pasta 170ms 10MB # . . . . 5.78 14.51 pasta 170ms 10MB . # . . # 6.42 14.22 pasta 170ms 10MB . # . . . 6.30 14.20 host 50ms 100MB # # . # . 6.12 14.08 pasta 170ms 10MB . . . . # 6.63 14.07 pasta 170ms 10MB . # . . # 4.93 13.53 pasta 50ms 10MB # # # # # 1.29 13.41 pasta 170ms 10MB # . . . # 6.31 13.11 host 50ms 100MB . # # . # 7.62 13.07 pasta 170ms 10MB # # . . . 6.41 13.03 host 50ms 10MB # # # # . 1.80 12.98 host 50ms 100MB . . . # . 7.67 12.88 pasta 170ms 10MB # . . . . 5.65 12.76 pasta 170ms 10MB . . . . # 5.88 12.59 pasta 170ms 10MB # # . . # 5.69 12.55 host 50ms 100MB # . # . . 6.67 12.52 pasta 50ms 10MB # . # # # 1.22 12.50 host 170ms 10MB # . # . . 5.41 12.41 pasta 50ms 100MB # # . . . 6.58 12.28 pasta 50ms 10MB # # # # # 2.12 12.20 pasta 170ms 10MB . # . . . 6.56 12.16 pasta 170ms 10MB . . . . . 5.38 12.11 host 170ms 10MB # # # . # 5.60 12.00 pasta 170ms 10MB # . . . # 10.77 11.98 pasta 50ms 10MB # # # # . 1.19 11.92 pasta 50ms 100MB # . . . . 6.63 11.81 host 170ms 10MB . . # . # 5.55 11.80 host 50ms 100MB . # . . . 6.64 11.71 host 170ms 10MB # # # . # 5.35 11.70 pasta 50ms 100MB . # . . # 7.40 11.69 host 170ms 10MB # . # . # 6.18 11.66 host 50ms 100MB # # # . . 6.40 11.60 pasta 50ms 10MB . # # # . 1.24 11.51 host 170ms 10MB . # . . # 5.54 11.40 pasta 50ms 10MB . . # # # 1.35 11.38 host 170ms 10MB . # . . . 6.33 11.23 host 170ms 10MB # . # . # 4.25 11.22 host 50ms 100MB # # . . . 6.90 11.19 host 170ms 10MB . # # . . 6.59 11.16 host 50ms 100MB . # # . . 7.29 11.09 host 50ms 100MB . . # . # 8.63 11.07 host 170ms 10MB . . . . # 6.00 10.99 pasta 50ms 100MB # . . . # 7.90 10.86 host 170ms 10MB . . . . . 6.66 10.86 pasta 50ms 10MB # . # . # 2.61 10.84 host 170ms 10MB # . # . . 5.41 10.83 host 170ms 10MB # . . . # 5.80 10.76 host 170ms 10MB . . # . . 5.38 10.68 pasta 50ms 100MB . # . . . 7.13 10.68 host 170ms 10MB # # . . . 8.34 10.68 pasta 50ms 10MB . # # # # 1.39 10.66 pasta 50ms 10MB . # # # . 1.39 10.64 host 170ms 10MB # . . . . 4.92 10.64 host 170ms 10MB # # . . # 5.70 10.60 pasta 50ms 100MB . . . . # 6.92 10.56 host 50ms 100MB . . . . # 6.86 10.56 pasta 50ms 100MB # # . . # 6.32 10.53 host 170ms 10MB . . # . # 6.53 10.52 host 50ms 100MB # . . . . 6.67 10.52 host 170ms 10MB . . . . # 5.04 10.48 host 50ms 100MB # # # . # 6.39 10.45 host 170ms 10MB . # . . # 5.09 10.36 host 170ms 10MB # # # . . 6.01 10.29 host 50ms 100MB # . # . # 8.37 10.19 host 170ms 10MB . # # . . 7.35 10.14 host 170ms 10MB . # . . . 5.03 10.13 host 170ms 10MB # # # . . 6.11 10.08 host 170ms 10MB . # # . # 6.63 10.04 host 50ms 100MB # . . . # 7.07 10.04 host 50ms 100MB . . . . . 7.68 10.01 host 170ms 10MB # # . . # 4.41 9.96 host 170ms 10MB . . . . . 6.56 9.92 host 170ms 10MB # . . . . 5.72 9.88 pasta 50ms 10MB # . # . # 1.64 9.82 host 50ms 100MB # # . . # 6.32 9.72 host 170ms 10MB # . . . # 6.53 9.60 pasta 50ms 10MB # # # . # 1.53 9.25 host 170ms 10MB # # . . . 5.07 9.16 pasta 50ms 10MB . . # # # 1.41 9.10 host 170ms 10MB . . # . . 6.06 8.95 host 170ms 10MB . # # . # 6.71 8.83 pasta 50ms 10MB . . # . . 1.42 8.81 pasta 50ms 10MB . . # # . 1.45 8.65 pasta 50ms 10MB . # # # # 1.28 8.35 pasta 50ms 10MB # . # . . 1.47 7.92 pasta 50ms 10MB . # # . # 1.51 7.78 pasta 50ms 10MB # # # . . 1.87 7.61 pasta 50ms 10MB . . # # . 1.53 7.32 pasta 50ms 10MB # . # # # 1.33 7.22 pasta 50ms 10MB # # # . # 1.31 7.15 pasta 50ms 10MB . . . . . 1.34 7.12 pasta 50ms 10MB # # # . . 1.42 6.95 pasta 50ms 10MB . # # . . 1.51 6.83 pasta 50ms 10MB . # . . # 1.79 6.76 host 50ms 10MB . . . # # 1.42 6.66 pasta 50ms 10MB # . # . . 1.60 6.26 pasta 50ms 10MB # . . # # 1.55 5.66 pasta 50ms 10MB . # # . . 1.55 5.62 pasta 50ms 10MB . # . . . 1.44 5.53 pasta 50ms 10MB # # . # # 1.30 5.52 pasta 50ms 10MB . . # . # 1.43 5.28 pasta 50ms 10MB . . . . # 1.35 5.20 pasta 50ms 10MB . . . . # 1.49 5.15 pasta 50ms 10MB . # # . # 1.47 5.13 pasta 50ms 10MB . . # . . 1.40 5.10 pasta 50ms 10MB . . . . . 1.35 5.10 pasta 50ms 10MB . . # . # 1.40 5.08 host 50ms 10MB . # . # # 1.64 4.99 pasta 50ms 10MB # . . . # 1.44 4.69 pasta 50ms 10MB . # . # # 2.60 4.67 pasta 50ms 10MB . # . . . 1.45 4.63 pasta 50ms 10MB . # . . # 1.34 4.63 host 50ms 10MB . . # # . 1.31 4.57 host 50ms 10MB # . . # # 1.37 4.51 host 50ms 10MB # # . # # 1.24 4.48 pasta 50ms 10MB # # . . # 1.55 4.44 host 50ms 10MB # # . . # 1.49 4.42 pasta 50ms 10MB # # . . . 1.22 4.35 host 50ms 10MB . # . # . 1.29 4.33 host 50ms 10MB . # # # . 1.36 4.27 host 50ms 10MB # # # # # 1.32 4.04 pasta 50ms 10MB . . . # # 1.31 3.89 pasta 50ms 10MB . # . # . 1.33 3.88 pasta 50ms 10MB . . . # # 1.50 3.85 host 50ms 10MB # . # # . 1.43 3.78 host 50ms 10MB . . . # . 1.28 3.75 host 50ms 10MB # . # . # 1.52 3.74 pasta 50ms 10MB # . . # . 1.38 3.73 host 50ms 10MB . # # # # 1.36 3.68 host 50ms 10MB # # . # . 1.21 3.68 host 50ms 10MB . # . # # 1.41 3.66 pasta 50ms 10MB # . . # . 1.43 3.66 host 50ms 10MB . . . # # 1.62 3.65 pasta 50ms 10MB # # . # . 1.51 3.64 host 50ms 10MB . . # # # 1.31 3.64 host 50ms 10MB . . # . . 1.48 3.61 pasta 50ms 10MB # . . . . 1.30 3.60 pasta 50ms 10MB # . . . # 1.49 3.56 host 50ms 10MB . . . . # 1.30 3.54 pasta 50ms 10MB # # . # # 1.33 3.53 host 50ms 10MB # # . # . 1.38 3.52 host 50ms 10MB # . # # # 1.18 3.51 host 50ms 10MB . . . # . 1.59 3.51 pasta 50ms 10MB . . . # . 1.58 3.49 host 50ms 10MB . . . . . 1.48 3.48 host 50ms 10MB . # . . # 1.57 3.46 pasta 50ms 10MB # . . . . 1.49 3.45 pasta 50ms 10MB . # . # # 1.46 3.44 host 50ms 10MB . # # . . 1.41 3.41 host 50ms 10MB # # # . . 1.34 3.40 host 50ms 10MB # # # . # 1.34 3.40 host 50ms 10MB # # # # . 1.29 3.39 host 50ms 10MB # . # . . 1.87 3.39 host 50ms 10MB # # # . # 1.38 3.38 host 50ms 10MB . . . . . 1.43 3.37 pasta 50ms 10MB # # . . . 1.29 3.36 host 50ms 10MB # . # . # 1.59 3.33 host 50ms 10MB . . # . # 1.46 3.32 host 50ms 10MB # . . . . 1.47 3.29 host 50ms 10MB . # . . . 1.49 3.28 host 50ms 10MB . # . . # 1.38 3.28 host 50ms 10MB . . # . . 1.39 3.28 host 50ms 10MB # . . # . 1.43 3.27 host 50ms 10MB # . . # . 1.35 3.27 pasta 50ms 10MB # . . # # 1.39 3.26 host 50ms 10MB # # . . . 1.28 3.24 host 50ms 10MB . # # . # 1.43 3.24 host 50ms 10MB # . . . # 1.57 3.19 host 50ms 10MB . # # # . 1.27 3.19 host 50ms 10MB . # # # # 1.54 3.18 host 50ms 10MB . . # # . 1.34 3.14 host 50ms 10MB # . # # . 1.32 3.14 pasta 50ms 10MB . . . # . 1.58 3.12 host 50ms 10MB . # # . . 1.59 3.09 host 50ms 10MB . # . # . 1.43 3.08 pasta 50ms 10MB # # . . # 1.30 3.07 pasta 50ms 10MB # # . # . 1.35 3.01 host 50ms 10MB # # # # # 1.53 2.99 host 50ms 10MB . # # . # 1.54 2.94 host 50ms 10MB # . # . . 1.58 2.94 host 50ms 10MB . # . . . 1.43 2.87 host 50ms 10MB . . # . # 1.46 2.87 host 50ms 10MB # . # # # 1.36 2.87 host 50ms 10MB # # # . . 1.40 2.84 host 50ms 10MB # # . . # 1.29 2.79 host 50ms 10MB # . . . . 1.32 2.78 host 50ms 10MB # . . . # 1.37 2.68 host 50ms 10MB # # . . . 1.21 2.67 host 50ms 10MB . . . . # 1.37 2.66 host 50ms 10MB # # . # # 1.34 2.39 host 50ms 10MB # . . # # 1.43 2.15 pasta 50ms 10MB . # . # . 1.45 2.04 host 50ms 10MB . . # # # 1.54 1.93 column description network podman --network= argument ping the RTT to the remote server size the size of the file {up,down}loaded wmem sysctl rmem sysctl lowat sysctl cngst sysctl qdisc sysctl down download time in seconds up upload time in seconds (timed out at 120s for the 10MB tests and 180s for the 100MB tests ) column sysctl "#" (custom) "." (default) wmem net.core.wmem_max 7500000 212992 rmem net.core.rmem_max 7500000 212992 lowat net.ipv4.tcp_notsent_lowat 131072 4294967295 cngst net.ipv4.tcp_congestion_control bbr cubic qdisc net.core.default_qdisc cake fq_codel And this is the messy script that I used to generate this data (I'm hosting both servers myself, so feel free to use them for testing): #!/usr/bin/bash set -euo pipefail # The location of the files script_dir="$(dirname "$(realpath "$0")")" output_file="$script_dir/pasta-upload-test-results.tsv" # The size of the file to upload/download file_size="100M" # Define the sysctl parameters declare -a -r sysctl_keys=( "custom" "default" ) declare -A -r wmem_maxes=( [default]=212992 [custom]=7500000 ) declare -A -r rmem_maxes=( [default]=212992 [custom]=7500000 ) declare -A -r tcp_notsent_lowats=( [default]=4294967295 [custom]=131072 ) declare -A -r tcp_congestion_controls=( [default]="cubic" [custom]="bbr" ) declare -A -r default_qdiscs=( [default]="fq_codel" [custom]="cake" ) # Define the network parameters declare -a -r podman_networks=("host" "pasta") declare -A -r servers=( [50ms]="https://www.maxchernoff.ca/tools/speedtest" [170ms]="https://svn.tug.org:8369/speedtest" ) # Initialize the temporary directory temp_dir=$(mktemp -d) cd "$temp_dir" trap 'rm -rf "$temp_dir"' EXIT # Define the network test commands download() { local network=$1 local server=$2 podman run \ --quiet \ --rm \ --pull="never" \ --volume="$temp_dir:/srv/:Z" \ --workdir="/srv/" \ --network="$network" \ quay.io/fedora/fedora-minimal \ timeout --signal=SIGINT 180s \ curl \ --silent \ --output /dev/null \ --write-out '%{time_total}\n' \ "$server?bytes=$file_size" \ || echo "180.000000" } upload() { local network=$1 local server=$2 podman run \ --quiet \ --rm \ --pull="never" \ --volume="$temp_dir:/srv/:Z" \ --workdir="/srv/" \ --network="$network" \ quay.io/fedora/fedora-minimal \ timeout --signal=SIGINT 180s \ curl \ --silent \ --output /dev/null \ --write-out '%{time_total}\n' \ -X POST \ --upload-file test.bin \ "$server" \ || echo "180.000000" } # Pull the container podman pull --quiet quay.io/fedora/fedora-minimal >/dev/null 2>&1 # Create a sample file to upload head --bytes="$file_size" /dev/urandom > test.bin # Run the tests printf 'network\tping_time\twmem_max\trmem_max\ttcp_notsent_lowat\ttcp_congestion_control\tdefault_qdisc\tdownload_time\tupload_time\n' \ | tee "$output_file" for wmem_max in "${sysctl_keys[@]}"; do for rmem_max in "${sysctl_keys[@]}"; do for tcp_notsent_lowat in "${sysctl_keys[@]}"; do for tcp_congestion_control in "${sysctl_keys[@]}"; do for default_qdisc in "${sysctl_keys[@]}"; do sudo sysctl -w \ net.core.wmem_max="${wmem_maxes[$wmem_max]}" \ net.core.rmem_max="${rmem_maxes[$rmem_max]}" \ net.ipv4.tcp_notsent_lowat="${tcp_notsent_lowats[$tcp_notsent_lowat]}" \ net.ipv4.tcp_congestion_control="${tcp_congestion_controls[$tcp_congestion_control]}" \ net.core.default_qdisc="${default_qdiscs[$default_qdisc]}" \ > /dev/null for server in "${!servers[@]}"; do for network in "${podman_networks[@]}"; do printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \ "$network" \ "$server" \ "$wmem_max" \ "$rmem_max" \ "$tcp_notsent_lowat" \ "$tcp_congestion_control" \ "$default_qdisc" \ "$(download "$network" "${servers[$server]}")" \ "$(upload "$network" "${servers[$server]}")" \ | tee -a "$output_file" done done done done done done done # Sort the lines cat "$output_file" | \ sort -k9rn -k8rn | \ column --table --table-right=2,8,9 --output-width=999 | \ tac | awk 'NR==1 {line=$0; next} 1; END{print line}' | tac | \ sponge "$output_file"
I bisected a kernel a few weeks ago, so I can try that here if you think that this is a kernel bug and not a pasta bug.
Could be either, I have no real intuition either way so far.
Ok, I'll try testing with some older versions of both and see if that narrows down the issue. Thanks, -- Max