September 2024 - user

Re: Rootless Podman with VRRP
by David Gibson 19 Sep '24

19 Sep '24

On Mon, Sep 16, 2024 at 02:46:51PM +0000, Castelli, Anton via user wrote: > Date: Mon, 16 Sep 2024 14:46:51 +0000 > From: "Castelli, Anton" <anton.castelli(a)siu.edu> > To: "passt-user(a)passt.top" <passt-user(a)passt.top> > Subject: Rootless Podman with VRRP > List-Id: "For passt users: support, questions and answers" > <passt-user.passt.top> > > I'm trying to get a service in a rootless Podman container (BIND DNS > server) to respond correctly when using VRRP (via keeepalived) on > the host. It seems like Pasta will forward the inbound traffic to > the container from the VRRP address, but the responses will be from > the regular IP address instead of the VRRP address, which causes the > client to ignore the response. I've tried adding Pasta network > options to the container, but the behavior seems to be the same. I'm not familiar with VRRP beyond a quick skim of the wikipedia article just now. I think the (only) relevant thing is that the host has multiple addresses, but it's possible there's some other complexity on the host that I'm not factoring in yet. > OS: Centos Stream 9 > Podman: 5.2.2 > Pasta: 0^20240806.gee36266-2.el9.x86_64-pasta This version includes UDP flow tracking, which fixes some of the worst bugs with UDP forwarding / addressing, but there are still some edge cases as you've discovered. I think I know what's going on here. With TCP, once we accept() a connection, it's local address on the host is part of the accept()ed socket, so we'll always use the same address for reply packets. With UDP, however, this doesn't happen automatically: the local address for replies is controlled by the bound address of the socket we use to send them out. In this instance that will be the same socket as is listening for the incoming requests. I'm assuming that socket will be set to listen to DNS traffic on any address, so it will be bound to 0.0.0.0:53 on the host. That will see the traffic to the VRRP address, but when we go to send the reply the kernel will pick the local address according to its routing tables and it seems it's picking the default address. That is a bug, at least in principle: we should remember the local address for a UDP flow and use the same local address for replies. With the flow table we now have the means of correctly tracking this, however, we haven't implemented this (yet) because it's kind of fiddly: we'd need to use additional getsockopt() calls to determine and control the local address for datagrams on a per-packet basis. > Outside interface: > > ens18 > 10.1.1.1/24 (main IP) > 10.1.1.2/32 (VRRP IP) > > TCPdump shows the problem (note that the reply packet has source as the main IP, not the VRRP IP: > > IP 10.2.2.2.37392 > 10.1.1.2.53: 60211+ [1au] A? www.example.com. (56) > IP 10.1.1.1.53 > 10.2.2.2.37392: 60211*- 1/0/1 A 192.168.254.7 (88) > > Tried starting the container with non-default pasta options, but the result is the same: > > --network pasta:-I,tap0,-o,10.1.1.2,--ipv4-only,-a,10.0.2.0,-n,24,-g,10.0.2.2,--dns-forward,10.0.2.3,--no-ndp,--no-dhcpv6,--no-dhcp Most of those options are not relevant. --dns-forward is set by podman and won't change anything (it's about DNS requests _out_ from the container). Similarly -g, --ipv4-only and the --no-* options are unlikely to be relevant here. -a is also probably not relevant: it controls the guest's address, but this is a host side addressing problem. That said, -a 10.0.2.0 -n 24 is probably not a good idea, since it's setting the guest's address to the network address. -o is the one that you'd think is relevant, since it's supposed to control the outbound local address. The catch here is that while it will control the local address for new sockets we create to handle flows initiated by the guest, here the flow is initiated by the host and so goes through the listening socket, which is still bound to 0.0.0.0:53. > Any help with possible solutions would be greatly appreciated. I believe I have a workaround. The trick here is we want the listening socket to be bound to the VRRP address, this can be done by setting that specific IP for publishing the port so, for example: podman run --publish 10.1.1.2:53:53/udp ... (which should translate to to the pasta option "-u 10.1.1.2/53:53"). The tradeoff for this workaround is that the container will now *only* respond to DNS traffic to the VRRP address, but I'm guessing that might be what you want anyway. You could also explicitly publish any additional addresses you want to respond on. But, longer term, this is a bug that we'd like to fix when we have time. Would you mind filing a ticket on bugs.passt.top so we don't forget it? -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson

2 5

Re: Rootless Podman with VRRP
by Stefano Brivio 17 Sep '24

17 Sep '24

On Tue, 17 Sep 2024 18:53:24 +0000 "Castelli, Anton" <anton.castelli(a)siu.edu> wrote: > Stefano, > > I've encountered an error when trying to submit the bug report. I tried twice with the same result. See attached screenshot. Sorry, my bad. Can you please try again? -- Stefano

2 1

Re: Rootless Podman with VRRP
by Stefano Brivio 17 Sep '24

17 Sep '24

Hi Anton, On Tue, 17 Sep 2024 15:22:04 +0000 "Castelli, Anton via user" <passt-user(a)passt.top> wrote: > I will be happy to submit a bug report. Unfortunately, I'm having > trouble getting signed up. I've tried to send the new account email > to both my work email address and a personal Gmail address. I have > not received the email in either case (I've checked the spam folders > too). You should have the email now, I'm currently reviewing subscription requests manually because of an influx of malicious automated requests. -- Stefano

2 1

Rootless Podman with VRRP
by Castelli, Anton 16 Sep '24

16 Sep '24

I'm trying to get a service in a rootless Podman container (BIND DNS server) to respond correctly when using VRRP (via keeepalived) on the host. It seems like Pasta will forward the inbound traffic to the container from the VRRP address, but the responses will be from the regular IP address instead of the VRRP address, which causes the client to ignore the response. I've tried adding Pasta network options to the container, but the behavior seems to be the same. OS: Centos Stream 9 Podman: 5.2.2 Pasta: 0^20240806.gee36266-2.el9.x86_64-pasta Outside interface: ens18 10.1.1.1/24 (main IP) 10.1.1.2/32 (VRRP IP) TCPdump shows the problem (note that the reply packet has source as the main IP, not the VRRP IP: IP 10.2.2.2.37392 > 10.1.1.2.53: 60211+ [1au] A? www.example.com. (56) IP 10.1.1.1.53 > 10.2.2.2.37392: 60211*- 1/0/1 A 192.168.254.7 (88) Tried starting the container with non-default pasta options, but the result is the same: --network pasta:-I,tap0,-o,10.1.1.2,--ipv4-only,-a,10.0.2.0,-n,24,-g,10.0.2.2,--dns-forward,10.0.2.3,--no-ndp,--no-dhcpv6,--no-dhcp Any help with possible solutions would be greatly appreciated. Thanks, -- ANTON CASTELLI Network Engineer V OFFICE OF INFORMATION TECHNOLOGY MAIL CODE 4622 SOUTHERN ILLINOIS UNIVERSITY 625 WHAM DRIVE ROOM B15 CARBONDALE, ILLINOIS 62901 anton.castelli(a)siu.edu P: 618.453.6424 OIT.SIU.EDU

1 0

passt: new version 2024_09_06.6b38f07 available
by Stefano Brivio 06 Sep '24

06 Sep '24

The new version with tag 2024_09_06.6b38f07 includes the following changes: 6b38f07 apparmor: Allow read access to /proc/sys/net/ipv4/ip_local_port_range 116bc82 selinux: Allow read access to /proc/sys/net/ipv4/ip_local_port_range a33ecaf tap: Don't risk truncating frames on full buffer in tap_pasta_input() d2a1dc7 tap: Restructure in tap_pasta_input() 11e2905 tap: Improve handling of EINTR in tap_passt_input() 49fc4e0 tap: Split out handling of EPOLLIN events 63513e5 util: Fix order of operands and carry of one second in timespec_diff_us() 748ef4c cppcheck: Work around some cppcheck 2.15.0 redundantInitialization warnings afedc24 tcp: Use EPOLLET for any state of not established connections aff5a49 udp: Handle more error conditions in udp_sock_errs() bd99f02 udp: Treat errors getting errors as unrecoverable bd092ca udp: Split socket error handling out from udp_sock_recv() 88bfa38 flow: Helpers to log details of a flow 1166401 udp: Allow UDP flows to be prematurely closed 7ad9f9b flow: Fix incorrect hash probe in flowside_lookup() 0ea60e5 log: Don't prefix log file messages with time and severity if they're continuations 3836396 Makefile: Enable _FORTIFY_SOURCE iff needed eedc81b fwd, conf: Probe host's ephemeral ports 4a41dc5 conf, fwd: Don't attempt to forward port 0 1daf6f4 conf, fwd: Make ephemeral port logic more flexible 712ca32 seccomp.sh: Try to account for terminal width while formatting list of system calls e0be6bc udp: Use dual stack sockets for port forwarding when possible c78b194 udp: Remove unnnecessary local from udp_sock_init() 620e19a udp: Merge udp[46]_mh_recv arrays 418feb3 test: Look for possible sshd-session paths (if it's there at all) in mbuto's profile https://passt.top/passt/log/?qt=range&q=2024_08_21.1d6142f..2024_09_06.6b38… Packages: - Alpine Linux: https://pkgs.alpinelinux.org/packages?name=passt - Arch Linux: https://www.archlinux.org/packages/extra/x86_64/passt/ https://archlinuxarm.org/packages/aarch64/passt https://archlinuxarm.org/packages/armv7h/passt - Chimera: https://pkgs.chimera-linux.org/packages?name=passt - Clear Linux: https://github.com/clearlinux-pkgs/passt/ - Debian tracker: https://tracker.debian.org/pkg/passt - Copr (CentOS Stream, EPEL, Fedora, Mageia): https://copr.fedorainfracloud.org/coprs/sbrivio/passt/build/7989588/ permanent mirror: https://passt.top/builds/copr/0^20240906.g6b38f07/ - Fedora updates: https://bodhi.fedoraproject.org/updates/?packages=passt - Gentoo versions: https://packages.gentoo.org/packages/net-misc/passt - GNU Guix: https://packages.guix.gnu.org/packages/passt/ - Homebrew: https://formulae.brew.sh/formula/passt - NixOS: https://github.com/NixOS/nixpkgs/tree/nixos-unstable/pkgs/by-name/pa/passt - openSUSE: https://software.opensuse.org/package/passt - PLD Linux: https://git.pld-linux.org/cgi-bin/gitweb.cgi?p=packages/passt.git - Solus: https://github.com/getsolus/packages/tree/main/packages/p/passt - Ubuntu tracker: https://launchpad.net/ubuntu/+source/passt - Void Linux: https://voidlinux.org/packages/?q=passt - Static builds: - Package for other RPM-based distributions, x86_64 only: https://passt.top/builds/latest/x86_64/passt-g6b38f07-1.x86_64.rpm - x86_64 static binaries: https://passt.top/builds/latest/x86_64/ - Debian package, from x86_64 static build: https://passt.top/builds/latest/x86_64/passt_6b38f07-1_all.deb -- Stefano

1 0

2024

2023

2022

2021

user September 2024