On Wed, 12 Nov 2025 04:22:32 -0700
Max Chernoff
Hi Stefano,
On Wed, 2025-11-12 at 11:32 +0100, Stefano Brivio wrote:
Hmm, actually, I have a hack that's not quite correct (we should make ACK_INTERVAL adaptive instead, which is one of the other bits I'm working on), but if it fixes the issue for you, it should at least mean that we're talking about the same issue.
Patch attached. Can you give that a try?
That seemed to help quite a bit---it's now 200x faster than before, but still 10x slower than --network=host:
Thanks for testing. I'm fairly sure it's that problem, then. Setting 1 ms as interval between checks for socket-side ACKs (reported by the kernel) as my hack does is not the appropriate solution, I'm implementing something based on the reported round-trip-time (RTT) instead. As a further hack, you could probably do something like this on top: --- diff --git a/tcp.c b/tcp.c index 697f80d..8c50ee0 100644 --- a/tcp.c +++ b/tcp.c @@ -339,7 +339,7 @@ enum { #define MSS_DEFAULT 536 #define WINDOW_DEFAULT 14600 /* RFC 6928 */ -#define ACK_INTERVAL 1 /* ms */ +#define ACK_INTERVAL 200 /* us */ #define SYN_TIMEOUT 10 /* s */ #define ACK_TIMEOUT 2 #define FIN_TIMEOUT 60 @@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn) } if (conn->flags & ACK_TO_TAP_DUE) { - it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000 * 1000; + it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000; } else if (conn->flags & ACK_FROM_TAP_DUE) { if (!(conn->events & ESTABLISHED)) it.it_value.tv_sec = SYN_TIMEOUT; --- ...but again, I'm going to fix that properly in a bit.
Also, I should mention that I'm using the following networking-related sysctls:
net.core.wmem_max=7500000 net.core.rmem_max=7500000
Those were settings we recommended for KubeVirt until https://github.com/kubevirt/user-guide/pull/933, but they don't seem to necessarily make sense as we seem to have made peace with the TCP auto-tuning mechanism in Linux meanwhile. See also https://bugs.passt.top/show_bug.cgi?id=138 and commit 71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to maximum values"). As the issue here is about socket (kernel) buffers being "too small" for a while, I guess that those settings plus reverting that commit would "fix" the issue entirely for you. But it's impractical to rely on users to set those, that's why I'm looking for something adaptive which still plays nicely with TCP auto-tuning instead.
net.ipv4.tcp_notsent_lowat=131072 net.core.default_qdisc=cake net.ipv4.tcp_congestion_control=bbr
I'm not sure if those really matter for pasta, but I haven't really thought about them.
I read some articles that suggested that those were a good idea, and I've been using them for about a year now, but I can disable those for testing if you want. I'm also using systemd's IPAddressAllow/IPAddressDeny/RestrictAddressFamilies and some SELinux port restrictions; I can easily disable those too.
No need, I don't think those make a difference. -- Stefano