- user - passt

Re: Connecting back to the host through a dummy veth interface
by Stefano Brivio 23 Dec '25

23 Dec '25

Hi Felix, On Thu, 18 Dec 2025 13:32:36 +0100 Felix Rubio <felix(a)kngnt.org> wrote: > Hi everybody, > > I am trying to run a number of rootless podman pods and containers by different > users, while still being able to talk to each other. To this end I am creating > a dummy veth interface and publishing all the exposed ports there (this works: > I can communicate from other host services with those containers), and I am > also trying to set that dummy veth interface as the default gateway for the > pods/containers (with the expectation that then they will be able to reach > each other). However, this is not working... and I am pretty lost. > > For example, I am running the following command, trying to connect a ldap > client container to a ldap server container, unsuccessfully. > > podman run --rm --dns=10.255.255.1 --network=pasta:--outbound- > if4=cluster_dns0,--gateway=10.255.255.1 --add-host=ldap.host.internal:host-gateway sh -c "ip add && ip route && ldapwhoami -H ldaps:// > ldap.host.internal:1636" > > Is this something impossible to do, or am I doing something wrong? Sorry, I'm a bit swamped at the moment, and I plan to get back to you in a bit, but meanwhile, I think the dummy veth trick is unnecessarily complicated. I think you could simply connect "to the host" and redirect from there to the containers by means of mapped ports. See: https://blog.podman.io/2024/10/podman-5-3-changes-for-improved-networking-e… for a couple of details. But I'll try to come up with a full example next. -- Stefano

2 9

Connecting back to the host through a dummy veth interface
by Felix Rubio 18 Dec '25

18 Dec '25

Hi everybody, I am trying to run a number of rootless podman pods and containers by different users, while still being able to talk to each other. To this end I am creating a dummy veth interface and publishing all the exposed ports there (this works: I can communicate from other host services with those containers), and I am also trying to set that dummy veth interface as the default gateway for the pods/containers (with the expectation that then they will be able to reach each other). However, this is not working... and I am pretty lost. For example, I am running the following command, trying to connect a ldap client container to a ldap server container, unsuccessfully. podman run --rm --dns=10.255.255.1 --network=pasta:--outbound- if4=cluster_dns0,--gateway=10.255.255.1 --add-host=ldap.host.internal:host-gateway sh -c "ip add && ip route && ldapwhoami -H ldaps:// ldap.host.internal:1636" Is this something impossible to do, or am I doing something wrong? Thank you very much for any help you can provide! -- Felix Felix Rubio

1 0

passt: new version 2025_12_15.b40f5cd available
by Stefano Brivio 15 Dec '25

15 Dec '25

The new version with tag 2025_12_15.b40f5cd includes the following changes: b40f5cd tcp: Use less-than-MSS window on no queued data, or no data sent recently 35fa86a conf, fwd: Move initialisation of auto port scanning out of conf() 5be1a22 tcp: Remove extra space from TCP_INFO debug messages (trivial) 87f1a91 pasta: Clean up waiting pasta child on failures e6612fe treewide: Introduce passt_exit() helper d6c5b6e tcp: Suppress new instance of cppcheck bug 14191 https://passt.top/passt/log/?qt=range&q=2025_12_10.d04c480..2025_12_15.b40f… Packages: - Alpine Linux: https://pkgs.alpinelinux.org/packages?name=passt - Arch Linux: https://www.archlinux.org/packages/extra/x86_64/passt/ https://archlinuxarm.org/packages/aarch64/passt https://archlinuxarm.org/packages/armv7h/passt - Chimera: https://pkgs.chimera-linux.org/packages?name=passt - Clear Linux: https://github.com/clearlinux-pkgs/passt/ - Copr (CentOS Stream, EPEL, Fedora, Mageia): https://copr.fedorainfracloud.org/coprs/sbrivio/passt/build/9911424/ permanent mirror: https://passt.top/builds/copr/0^20251215.gb40f5cd/ - Debian tracker: https://tracker.debian.org/pkg/passt - Fedora updates: https://bodhi.fedoraproject.org/updates/?packages=passt - Gentoo versions: https://packages.gentoo.org/packages/net-misc/passt - GNU Guix: https://packages.guix.gnu.org/packages/passt/ - Homebrew: https://formulae.brew.sh/formula/passt - NixOS: https://github.com/NixOS/nixpkgs/tree/nixos-unstable/pkgs/by-name/pa/passt - openSUSE: https://software.opensuse.org/package/passt - OpenMandriva: https://github.com/OpenMandrivaAssociation/passt/tree/master - PLD Linux: https://git.pld-linux.org/cgi-bin/gitweb.cgi?p=packages/passt.git - Solus: https://github.com/getsolus/packages/tree/main/packages/p/passt - Ubuntu tracker: https://launchpad.net/ubuntu/+source/passt - Void Linux: https://voidlinux.org/packages/?q=passt - Static builds: - Package for other RPM-based distributions, x86_64 only: https://passt.top/builds/latest/x86_64/passt-gb40f5cd-1.x86_64.rpm - x86_64 static binaries: https://passt.top/builds/latest/x86_64/ - Debian package, from x86_64 static build: https://passt.top/builds/latest/x86_64/passt_0b40f5cd-1_all.deb -- Stefano

1 0

passt: new version 2025_12_10.d04c480 available
by Stefano Brivio 10 Dec '25

10 Dec '25

Subject: passt: new version 2025_12_10.d04c480 available The new version with tag 2025_12_10.d04c480 includes the following changes: d04c480 pif: Correctly set scope_id for guest-side link local addresses 696709d tcp: Correct timer expiry value in trace message https://passt.top/passt/log/?qt=range&q=2025_12_09.c3f1ba7..2025_12_10.d04c… Packages: - Alpine Linux: https://pkgs.alpinelinux.org/packages?name=passt - Arch Linux: https://www.archlinux.org/packages/extra/x86_64/passt/ https://archlinuxarm.org/packages/aarch64/passt https://archlinuxarm.org/packages/armv7h/passt - Chimera: https://pkgs.chimera-linux.org/packages?name=passt - Clear Linux: https://github.com/clearlinux-pkgs/passt/ - Copr (CentOS Stream, EPEL, Fedora, Mageia): https://copr.fedorainfracloud.org/coprs/sbrivio/passt/build/9896122/ permanent mirror: https://passt.top/builds/copr/0^20251210.gd04c480/ - Debian tracker: https://tracker.debian.org/pkg/passt - Fedora updates: https://bodhi.fedoraproject.org/updates/?packages=passt - Gentoo versions: https://packages.gentoo.org/packages/net-misc/passt - GNU Guix: https://packages.guix.gnu.org/packages/passt/ - Homebrew: https://formulae.brew.sh/formula/passt - NixOS: https://github.com/NixOS/nixpkgs/tree/nixos-unstable/pkgs/by-name/pa/passt - openSUSE: https://software.opensuse.org/package/passt - OpenMandriva: https://github.com/OpenMandrivaAssociation/passt/tree/master - PLD Linux: https://git.pld-linux.org/cgi-bin/gitweb.cgi?p=packages/passt.git - Solus: https://github.com/getsolus/packages/tree/main/packages/p/passt - Ubuntu tracker: https://launchpad.net/ubuntu/+source/passt - Void Linux: https://voidlinux.org/packages/?q=passt - Static builds: - Package for other RPM-based distributions, x86_64 only: https://passt.top/builds/latest/x86_64/passt-gd04c480-1.x86_64.rpm - x86_64 static binaries: https://passt.top/builds/latest/x86_64/ - Debian package, from x86_64 static build: https://passt.top/builds/latest/x86_64/passt_0d04c480-1_all.deb -- Stefano

1 0

passt: new version 2025_12_09.c3f1ba7 available
by Stefano Brivio 09 Dec '25

09 Dec '25

The new version with tag 2025_12_09.c3f1ba7 includes the following changes: c3f1ba7 tcp_splice, flow: Add socket to epoll set before connect(), drop assert e8b56a3 fedora: Fix build on Fedora 43, selinux_requires_min not available on Copr builders c93515c tcp: Skip redundant ACK on partial sendmsg() failure e0f1330 tcp: Send a duplicate ACK also on complete sendmsg() failure 2b5c906 tcp: Allow exceeding the available sending buffer size in window advertisements cf1925f tcp: Don't limit window to less-than-MSS values, use zero instead 9139e60 tcp: Acknowledge everything if it looks like bulk traffic, not interactive 28f413d tcp: Don't clear ACK_TO_TAP_DUE if we're advertising a zero-sized window 000601b tcp: Adaptive interval based on RTT for socket-side acknowledgement checks 920a479 tcp: Limit advertised window to available, not total sending buffer size f423e12 tcp: Change usage factor of sending buffer in tcp_get_sndbuf() to 75% 6305b6c tcp, util: Add function for scaling to linearly interpolated factor, use it 5d83844 iov: Fix coding style of basic (non-IOV_TAIL) parts 0cb8f90 tcp, udp: Pad batched frames for vhost-user modes to 60 bytes (802.3 minimum) 68b0a36 tcp, udp: Pad batched frames to 60 bytes (802.3 minimum) in non-vhost-user modes b9cd36f udp: Fix coding style for comment to enum udp_iov_idx e3e8af7 tcp: Fix coding style for comment to enum tcp_iov_parts 1e51971 tap: Pad non-batched frames to 802.3 minimum (60 bytes) if needed 822ce09 test: Update Makefile to avoid failing on missing images 4ddd59b conf: Separate local mode for each IP version, don't enable disabled IP version 834982c vu_common: Clarify prototype of vu_collect() 7c7be67 test: Expand tmux right status bar to fit pass/fail/skipped counter and time aa809d3 tcp: Enable SO_KEEPALIVE if we see keep-alive segments from container / guest 9e2e381 seccomp: Fix build and operation on 32-bit musl targets fdbb4ef fwd: Preserve non-standard loopback address when splice forwarding e3c4c41 tcp: Always populate oaddr field for socket initiated flows cec1ca8 util: Rename sock_l4_dualstack() to sock_l4_dualstack_any() 653705c tcp, udp: Bind outbound listening sockets by interface instead of address e77cbf0 tcp, udp: Remove fallback if creating dual stack socket fails e6f6eb2 util: Fix setting of IPV6_V6ONLY socket option 74e6f12 udp: Move udp_sock_init() special case to its caller c8d4683 udp: Unify some more inbound/outbound parts of udp_sock_init() 860d3b1 tcp: Merge tcp_ns_sock_init[46]() into tcp_sock_init_one() 3b8b834 util, flow, pif: Simplify sock_l4_sa() interface b0523f6 inany: Let length of sockaddr_inany be implicit from the family 484dcfa flow: Remove bogus @path field from flowside_sock_args f37db87 conf: More useful errors for kernels without SO_BINDTODEVICE 0ae7b8c util: Extend sock_probe_mem() to sock_probe_features() 10e850a util: Correct error message on SO_BINDTODEVICE failure 1a83487 tcp: Clamp the retry timeout 3dde0e0 tcp: Update data retransmission timeout 0e8df13 tcp: Resend SYN for inbound connections ffecfaf util: Introduce read_file() and read_file_integer() function 785214c tcp: Rename "retrans" to "retries" 2002c7d arp/ndp: don't send messages on uninitialized tap interface 391c15a test: Fix IPv6 address/prefix mismatch error 95ab87b spec: use %selinux_requires_min macro, drop overlapping dependencies be1583f fwd: Don't explicitly exclude reverse-direction TCP ports for UDP fd3fc8d fwd: Exclude ports based on prior mapping state bdbdf4e Revert "fwd: Update all port maps before applying exclusions" 2c6590d udp: Use IP_FREEBIND for flow sockets as well as listening sockets 75b8bb9 tcp: Properly remove sockets from epoll loop when connection is closed a36031a seccomp.sh: Quote tr character ranges to prevent glob expansion 1d16439 contrib/selinux: use regex instead of SELinux template f6b6118 tcp, udp: Don't exclude ports in {tcp,udp}_port_rebind() 81942a2 fwd: Update all port maps before applying exclusions 942bfdb fwd: Check forwarding mode in fwd_scan_ports_*() rather than caller 06c3dcc fwd: Share port scanning logic between init and timer cases 1754f61 fwd: Move port exclusion handling from procfs_scan_listen() to callers 1bc7d54 fwd: Consolidate scans (not rebinds) in fwd.c 260075b tcp, udp, fwd: Run all port scanning from a single timer 1f2c08b icmp: Remove vestiges of ICMP timer 57446ca passt: Move main event loop processing into passt_worker() aaa8f34 udp: Use epoll instance management for UDP flows ca93124 icmp: Use epoll instance management for ICMP flows dd5302d tcp, flow: Replace per-connection in_epoll flag with an epollid in flow_common 05972c7 util: Move epoll registration out of sock_l4_sa() 965ea66 epoll_ctl: Extract epoll operations 8bfa47a util: Simplify epoll_del() interface to take epollfd directly 3a9dbe0 icmp: let icmp use mac address from flowside structure ad72098 tap: change signature of function tap_push_l2h() 860a487 tcp: forward external source MAC address through tap interface 627d5f7 udp: forward external source MAC address through tap interface 5d35ac3 flow: add MAC address of LAN local remote hosts to flow 7917155 arp/ndp: send ARP announcement / unsolicited NA when neigbour entry added e456c02 arp/ndp: respond with true MAC address of LAN local remote hosts 45869d6 fwd: Add cache table for ARP/NDP contents 3c46901 netlink: add subscription on changes in NDP/ARP table 3b9c7f8 Add reverse Christmas tree to CONTRIBUTING.md 7bb150c fwd: Fix misspelling b2ccb6d test: Fix the escaping issue in memory/passt test 812a7c6 test: Update the threshold value for some perf tests 4fb4c31 tap: Update some function comments for accuracy 8a6def6 passt: Rename EPOLL_EVENTS to NUM_EPOLL_EVENTS 720cc0b Fix the wrong command in CONTRIBUTING.md 6d84c90 test: For missing static checkers, skip rather than failing tests 5149f34 test: Add some missing quoting in exeter runner 898cf50 test: Use ${} consistently in lib/exeter 5da0316 isolation: keep CAP_DAC_OVERRIDE initially 518d8c1 tcp: Clarify logic calculating how much guest data to ack a947839 tcp: On partial send (incomplete sendmsg()), request a retransmission right away b145441 tcp: Don't consider FIN flags with mismatching sequence 8efa80b tcp: Completely ignore data segment in CLOSE-WAIT state, log a message b3217aa tcp: Fix ACK sequence on FIN to tap 2a16cdf test: Add linting of Python test scripts 81fd66a test: Don't delete exetool on make clean 2274c3a cppcheck: Suppress variable scope warnings in dhcpv6() ee9b236 cppcheck: Suppress a buggy cppcheck warning 065d199 cppcheck: Suppress the suppression of a suppression b4b3b08 clang-tidy: Suppress redundant expression warning 4d9c35b test: Update passt.mbuto and passt.mem.mbuto a96a962 netlink: Don't require address to be global, just not link local 7086781 test: Fix printf error when debug is enabled c54ef9e test: Update README.md f4729be test: Update mbuto profile to fix the symlink of /bin de28c20 test: Update lib/term for clearer output when DEBUG is enabled f07d7b0 test: fix 'make assets' failure as root e81c692 tap: Drop frames if no client connected https://passt.top/passt/log/?qt=range&q=2025_09_19.623dbf6..2025_12_09.c3f1… Packages: - Alpine Linux: https://pkgs.alpinelinux.org/packages?name=passt - Arch Linux: https://www.archlinux.org/packages/extra/x86_64/passt/ https://archlinuxarm.org/packages/aarch64/passt https://archlinuxarm.org/packages/armv7h/passt - Chimera: https://pkgs.chimera-linux.org/packages?name=passt - Clear Linux: https://github.com/clearlinux-pkgs/passt/ - Copr (CentOS Stream, EPEL, Fedora, Mageia): https://copr.fedorainfracloud.org/coprs/sbrivio/passt/build/9888020/ permanent mirror: https://passt.top/builds/copr/0^20251209.gc3f1ba7/ - Debian tracker: https://tracker.debian.org/pkg/passt - Fedora updates: https://bodhi.fedoraproject.org/updates/?packages=passt - Gentoo versions: https://packages.gentoo.org/packages/net-misc/passt - GNU Guix: https://packages.guix.gnu.org/packages/passt/ - Homebrew: https://formulae.brew.sh/formula/passt - NixOS: https://github.com/NixOS/nixpkgs/tree/nixos-unstable/pkgs/by-name/pa/passt - openSUSE: https://software.opensuse.org/package/passt - OpenMandriva: https://github.com/OpenMandrivaAssociation/passt/tree/master - PLD Linux: https://git.pld-linux.org/cgi-bin/gitweb.cgi?p=packages/passt.git - Solus: https://github.com/getsolus/packages/tree/main/packages/p/passt - Ubuntu tracker: https://launchpad.net/ubuntu/+source/passt - Void Linux: https://voidlinux.org/packages/?q=passt - Static builds: - Package for other RPM-based distributions, x86_64 only: https://passt.top/builds/latest/x86_64/passt-gc3f1ba7-1.x86_64.rpm - x86_64 static binaries: https://passt.top/builds/latest/x86_64/ - Debian package, from x86_64 static build: https://passt.top/builds/latest/x86_64/passt_0c3f1ba7-1_all.deb -- Stefano

1 0

do *not* upgrade to 2025_12_08.e8b56a3
by Stefano Brivio 08 Dec '25

08 Dec '25

...or derived packages. Fedora's OpenQA just caught a breaking issue: https://bodhi.fedoraproject.org/updates/FEDORA-2025-93b4eb64c3#comment-4473… which I'm investigating right now. -- Stefano

1 0

Help with pasta usage
by Ayon T 05 Dec '25

05 Dec '25

Hi, I've been using pasta as a network driver for rootless docker and I've been running into a couple of issues for a while now. I hope this is where I can find some help troubleshooting. The issue is that when I use pasta as the network driver as opposed to slirp4netns, I'm unable to access the internet through rootless docker or use ping (or traceroute) through its containers. So if I run "docker pull <image-name>" I get a timeout error: > Using default tag: latest Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 169.254.2.1:58905 ->10.0.2.3:53: i/o timeout I'm running pasta version 0.0~git20250217.a1e48a0-1 on Ubuntu 24.04.2 LTS with docker v27.3.1 build ce12230. Please let me know what other details I could provide to troubleshoot this issue. Thanks and regards, Ayon Tarafdar

3 10

Port Forwarding between host-guest via passt using QEMU-libvirt
by James Sinton 04 Dec '25

04 Dec '25

Hi All, I'm having trouble getting host-to-guest port forwarding to work via passt for a userspace connection: Please see my post on Superuser for more details: https://superuser.com/q/1927306/3115375?sem=2 Thank you. Best, James

3 6

Auto forwarding ports, but only to localhost
by Jan Wrobel 01 Dec '25

01 Dec '25

Hi, For pasta, would you consider an option to enable automatic forwarding of ports bound in a namespace, but make the forwarded ports available only via localhost, not all addresses? I'm working on a sandboxing program which uses pasta. The option -t "auto" is super convenient, but requires extra care, without proper firewall setup bound ports become automatically available to outside world. For a sandboxing program like mine, it is not a safe default to run with, because the program shouldn't assume the user will have a firewall configured. If something like "localhost/auto" was supported, it would match the convenience of "auto", no manual port mapping config would be needed, but would be safer for uses cases where exposing ports to outside world is problematic. Cheers, Jan

3 7

Re: pasta slow at HTTP upload
by David Gibson 23 Nov '25

23 Nov '25

On Wed, Nov 12, 2025 at 05:30:35PM -0700, Max Chernoff via user wrote: > Date: Wed, 12 Nov 2025 17:30:35 -0700 > From: Max Chernoff <git(a)maxchernoff.ca> > To: Stefano Brivio <sbrivio(a)redhat.com> > CC: passt-user(a)passt.top > Subject: Re: pasta slow at HTTP upload > List-Id: "For passt users: support, questions and answers" > <passt-user.passt.top> > > Hi Stefano, > > On Wed, 2025-11-12 at 13:53 +0100, Stefano Brivio wrote: > > As a further hack, you could probably do something like this on top: > > > > --- > > diff --git a/tcp.c b/tcp.c > > index 697f80d..8c50ee0 100644 > > --- a/tcp.c > > +++ b/tcp.c > > @@ -339,7 +339,7 @@ enum { > > #define MSS_DEFAULT 536 > > #define WINDOW_DEFAULT 14600 /* > > RFC 6928 */ > > > > -#define ACK_INTERVAL 1 /* ms */ > > +#define ACK_INTERVAL 200 /* us */ > > #define SYN_TIMEOUT 10 /* s */ > > #define ACK_TIMEOUT 2 > > #define FIN_TIMEOUT 60 > > @@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn) > > } > > > > if (conn->flags & ACK_TO_TAP_DUE) { > > - it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000 * 1000; > > + it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000; > > } else if (conn->flags & ACK_FROM_TAP_DUE) { > > if (!(conn->events & ESTABLISHED)) > > it.it_value.tv_sec = SYN_TIMEOUT; > > --- > > That actually makes it worse again, about as bad as before the patch. > But I've just tried rebuilding with original patch again, and also with > the exact same binary that I used yesterday, and that's slow now too. > I've verified with pgrep that Podman is using the correct pasta version, > so I have no idea what's happening. > > However, I do remember that for the past few months, some uploads would > randomly go really quickly, so maybe the problem happens sporadically, > and when I was testing the patched version I just happened to get > (un)lucky? > > > > net.core.wmem_max=7500000 > > > net.core.rmem_max=7500000 > > > > Those were settings we recommended for KubeVirt until > > https://github.com/kubevirt/user-guide/pull/933, but they don't seem to > > necessarily make sense as we seem to have made peace with the TCP > > auto-tuning mechanism in Linux meanwhile. > > > > See also https://bugs.passt.top/show_bug.cgi?id=138 and commit > > 71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to > > maximum values"). > > > > As the issue here is about socket (kernel) buffers being "too small" for > > a while, I guess that those settings plus reverting that commit would > > "fix" the issue entirely for you. But it's impractical to rely on users > > to set those, that's why I'm looking for something adaptive which still > > plays nicely with TCP auto-tuning instead. > > Ah, I didn't know that those (used to be) recommended for pasta; I set > those for Caddy since it complains on startup if those aren't set > > https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes > > > > net.ipv4.tcp_notsent_lowat=131072 > > > net.core.default_qdisc=cake > > > net.ipv4.tcp_congestion_control=bbr > > I set those as a general performance tuning thing (not for pasta > specifically) based off of > > https://blog.cloudflare.com/optimizing-tcp-for-high-throughput-and-low-late… > > https://grapheneos.org/articles/server-traffic-shaping > > > I'm not sure if those really matter for pasta, but I haven't really > > thought about them. > > Aha, those do seem to be the issue. Using the original (unpatched) > pasta: Huh, interesting. > (Set everything to my original settings) > $ sudo sysctl -w net.core.wmem_max=7500000 net.core.rmem_max=7500000 net.ipv4.tcp_notsent_lowat=131072 net.core.default_qdisc=cake net.ipv4.tcp_congestion_control=bbr > > (Test with --network=host) > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 20.4M > > (Test with --network=pasta) > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 4 100M 0 0 4 5120k 0 42440 0:41:10 0:02:03 0:39:07 26188 > (Stopped early since I got sick of waiting) > > (Set everything to the kernel defaults) > $ sudo sysctl -w net.core.wmem_max=212992 net.core.rmem_max=212992 net.ipv4.tcp_notsent_lowat=4294967295 net.core.default_qdisc=fq_codel net.ipv4.tcp_congestion_control=cubic > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 4 1431k 0:01:11 0:01:11 --:--:-- 1399k > > (tcp_congestion_control=default, tcp_notsent_lowat=custom, [rw]mem_max=default) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 5 1484k 0:01:08 0:01:08 --:--:-- 1400k > > (tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=default) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=212992 net.core.wmem_max=212992 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 12 3576k 0:00:28 0:00:28 --:--:-- 14.9M > > (tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=default) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 8 2595k 0:00:39 0:00:39 --:--:-- 17.1M > > (tcp_congestion_control=default, tcp_notsent_lowat=default, [rw]mem_max=custom) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 7 2107k 0:00:48 0:00:48 --:--:-- 8310k > > (tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=custom) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 15 4620k 0:00:22 0:00:22 --:--:-- 12.1M > > (tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=custom) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=7500000 net.core.wmem_max=7500000 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 5 100M 0 0 5 5376k 0 29899 0:58:27 0:03:04 0:55:23 0 > (Stopped early since I got sick of waiting) > > (Test with --network=host again) > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 24.3M Well, that sure is a confusing pattern. Looks like tcp_notsent_lowat is the biggest culprit... except for the outlier where it seems ok with both custom notsent_lowat and custom congestion control. How repeatable are each of these results? > I know fairly little about networking and the kernel, so if the answer > is just "don't set those sysctls together", I'd be okay with that. But I > haven't changed these sysctls since February, my upload speeds via pasta > were fine up until a few months ago, and the upload speeds are still > okay with --network=host, so I suspect that this is a bug somewhere. I have a vague theory why tcp_notsent_lowat might be bad for pasta but not --network=host. I'm far from certain about it though, and I don't have a good idea what change on the pasta side might have changed. Actually, is it just pasta that's changed in that time? Or could it be a new kernel version or something else? I'm having some trouble wrapping my head around what tcp_notsent_lowat does, but I _think_ the upshot of it will be to make the socket send path more "bursty": we'll fill the socket send buffer (which could be very large), then stop. We won't wake up again to refill until the amount in the socket buffer drops below tcp_notsend_lowat. Assuming we've been getting plenty of data from the other side, we'll then refill the buffer very quickly. Less wakups sounds like a good thing in the abstract, but I'm wondering if this burstiness is messing with the flow control on the other side, causing the guest side peer to slow down to nothing. If that is the case, it would make sense that it gets even worse with large socket buffer sizes, since that would make the "bursts" even bigger. Likewise I was wondering if having different congestion control algorithms on each side might somehow conflict causing a slowdown. I don't think the data you have above really suggest that though. > I also find it quite interesting that setting any of the sysctls > individually or in pairs improves the upload speeds, but setting all 3 > at once slows it down drastically. I'm curious about the case with default congestion control, but custom lowat and mem_max. I think that one's missing from the table above. > I bisected a kernel a few weeks ago, > so I can try that here if you think that this is a kernel bug and not a > pasta bug. Could be either, I have no real intuition either way so far. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson

2 1