February 2025 - dev

Migration testing with different host IPs
by David Gibson 16 Mar '25

16 Mar '25

I finally managed to set up tests simulating migration where the source and target host are different machines which may have different IPs. I'm getting some weird failures, but first the setup itself, since it might be useful. I'm attaching nsqemu.sh, which is based on my earlier qemu.sh. # Steps to run 1. In terminal 1, run $ nsqemu.sh box 2. In terminal 2, run $ nsqemu.sh source 3. In terminal 3, run $ nsqemu.sh target 4. In terminal 2 set up guest network # ip link set eth0 up # dhclient eth0 5. In terminal 2 (source) connect to the server running terminal 1 # socat STDIO TCP4:172.16.1.1:5555 6. Send some data back and forth between terminal 2 (source guest) and terminal 1 (external peer) 7. Initiate the migration in terminal 2 C-a c (qemu) migrate tcp:172.16.55.22:4444 8. Wait for migration to complete, watching terminal 3 (target guest) 9. Attempt to send data back and forth between terminal 3 (target guest) and terminal 1 (external peer) This isn't quite working for me, for reasons I haven't figured out yet # Testing with different source and target host IPs To test this change the IP= lines in the source and target sections so that they're different. This test is mostly working for me; the target drops the connection without breaking continuing connectivity on the target. # Network model The "box" creates an L1 namespace to host everything. It contains a bridge device "br0" with address 172.16.1.1. It's not connected to the outside internet. The peer server runs here. The source and target "hosts" are L2 namespaces, each with a veth attached to br0, and *both* with address 172.16.1.111, default route via 172.16.1.1 Additionally there's a veth directly between the two L2 spaces, used for the actual migration stream. It has addresses 172.16.55.11 and 172.16.55.22. # Weird failure For some reason, after the migration with this setup the target guest receives further incoming data from the peer server, but further outbound data seems to reach the bridge.. but not the server process. br.pcap attached. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson

2 2

Apparmor (and other) Issues
by Prafulla Giri 06 Mar '25

06 Mar '25

Esteemed maintainer, First and foremost, thank you very much for your hard work: passt is awesome and allows one to run more useful user-space VM-s. I have encountered 2 particular issues with the usage of passt with Debian, and wanted to bring them to your attention as I think you are probably the best person to deal with this. I do plan on sending a report to the Debian team afterwards. For reference, I tested these on Debian Testing Daily Image dated 28 January 2025, with updates, and the version of passt available with it is passt 0.0~git20250121.4f2c8e7-1 - Passt's default Apparmor config needs to allow writes to $XDG_RUNTIME_DIR (which is at /run/user/$UID). Currently it doesn't. Virt-manager, at least, tries to create the necessary sockets in the directory but apparmor prevents that from happening (and the error message Virt-Manager gives isn't helpful either: the first time around I falsely believed it was a segfault or similar issue). I managed to get passt working past this flaw (pun intended) by manually disabling apparmor for the binary. Passt works just fine in Fedora 41 as it doesn't use Apparmor but uses SELinux, and thus the configs don't affect it. - This second issue is perhaps a bit more Debian-specific, but I am going to mention it so that you might drop some hints for the Debian maintainers to debug this: Once Apparmor is disabled and a VM is configured to work with passt, DNS resolution doesn't work in the VM (IP Addresses work just fine) i.e. ping fsf.org doesn't work but `ping 209.51.188.174` does. The hypervisor details follow: $ virsh version # on Debian Testing a.k.a. 'Trixie' Compiled against library: libvirt 11.0.0 Using library: libvirt 11.0.0 Using API: QEMU 11.0.0Running hypervisor: QEMU 9.2.0 This, again, isn't an issue with Fedora 41, where everything just works. The hypervisor details for Fedora 41 are: $ virsh version # on Fedora 41 Compiled against library: libvirt 10.6.0 Using library: libvirt 10.6.0 Using API: QEMU 10.6.0 Running hypervisor: QEMU 9.1.2 Again, I will be making a report to the Debian maintainers, should they wish to chime in regarding Apparmor configs or the DNS resolution issue. Thank you once again for this awesome tool.

4 32

[PATCH 0/1] Send RST for guest packets with no flow
by David Gibson 03 Mar '25

03 Mar '25

As we discussed on email, this adds support for sending an RST in response to packets from the guest which don't match an existing flow and are neither SYN (requesting a new connection) nor themselves RST. This is a sligjhtly larger patch than I'd like, but I can't really see a way to simplify it without making fairly extensive reworks to share more code with paths for RST where there is a known connection. That would end up being more churn. David Gibson (1): tcp: Send RST in response to guest packets that match no connection tap.c | 13 +++++------ tap.h | 6 +++++ tcp.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 84 insertions(+), 7 deletions(-) -- 2.48.1

2 3

[PATCH v8 0/4] Reconstruct incoming ICMP headers for failed UDP connect and forward back
by Jon Maloy 03 Mar '25

03 Mar '25

v2: - Added patch breaking out udp header creation from function tap_udp4_send(). - Updated the ICMP creation by using the new function. - Added logics to find correct flow, depending on origin. - All done after feedback from David Gibson. v3: - More changes after feedback from David Gibson. v4: - Even more changes after feedback from D. Gibson v5: - Added corresponding patches for IPv6 v6: - Fixed some small nits after comments from D. Gibson. v7: - Added handling of all rejected ICMP messages - Returning correct user data amount if IPv6 as per RFC 4884. v8: - Added MTU to ICMPv4 ICMP_FRAG_NEEDED messages. - Added ASSERT() validation to message creation functions. Jon Maloy (4): tap: break out building of udp header from tap_udp4_send function udp: create and send ICMPv4 to local peer when applicable tap: break out building of udp header from tap_udp6_send function udp: create and send ICMPv6 to local peer when applicable tap.c | 86 +++++++++++++++++++++++--------- tap.h | 15 ++++++ udp.c | 132 ++++++++++++++++++++++++++++++++++++++++++++----- udp_internal.h | 2 +- udp_vu.c | 4 +- 5 files changed, 200 insertions(+), 39 deletions(-) -- 2.48.1

2 5

[PATCH] pasta.te: fix demo.sh and remove one duplicate rule
by 7ppKb5bW 01 Mar '25

01 Mar '25

On Fedora 41, without "allow pasta_t unconfined_t:dir read" /usr/bin/pasta can't open /proc/[pid]/ns , which is required by pasta_netns_quit_init() . This patch also remove one duplicate rule "allow pasta_t nsfs_t:file read;" , "allow pasta_t nsfs_t:file { open read };" at line 123 is enough. --- contrib/selinux/pasta.te | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/contrib/selinux/pasta.te b/contrib/selinux/pasta.te index 69be081..d0ff0cc 100644 --- a/contrib/selinux/pasta.te +++ b/contrib/selinux/pasta.te @@ -171,7 +171,7 @@ allow pasta_t init_t:lnk_file read; allow pasta_t init_t:unix_stream_socket connectto; allow pasta_t init_t:dbus send_msg; allow pasta_t init_t:system status; -allow pasta_t unconfined_t:dir search; +allow pasta_t unconfined_t:dir { read search }; allow pasta_t unconfined_t:file read; allow pasta_t unconfined_t:lnk_file read; allow pasta_t self:process { setpgid setcap }; @@ -192,8 +192,6 @@ allow pasta_t sysctl_net_t:dir search; allow pasta_t sysctl_net_t:file { open read write }; allow pasta_t kernel_t:system module_request; -allow pasta_t nsfs_t:file read; - allow pasta_t proc_t:dir mounton; allow pasta_t proc_t:filesystem mount; allow pasta_t net_conf_t:lnk_file read; --

2 2

[PATCH v7 0/4] Reconstruct incoming ICMP headers for failed UDP connect and forward back
by Jon Maloy 28 Feb '25

28 Feb '25

v2: - Added patch breaking out udp header creation from function tap_udp4_send(). - Updated the ICMP creation by using the new function. - Added logics to find correct flow, depending on origin. - All done after feedback from David Gibson. v3: - More changes after feedback from David Gibson. v4: - Even more changes after feedback from D. Gibson v5: - Added corresponding patches for IPv6 v6: - Fixed some small nits after comments from D. Gibson. v7: - Added handling of all rejected ICMP messages - Returning correct user data amount if IPv6 as per RFC 4884. Jon Maloy (4): tap: break out building of udp header from tap_udp4_send function udp: create and send ICMPv4 to local peer when applicable tap: break out building of udp header from tap_udp6_send function udp: create and send ICMPv6 to local peer when applicable tap.c | 86 +++++++++++++++++++++++--------- tap.h | 15 ++++++ udp.c | 132 ++++++++++++++++++++++++++++++++++++++++++++----- udp_internal.h | 2 +- udp_vu.c | 4 +- 5 files changed, 200 insertions(+), 39 deletions(-) -- 2.48.1

2 6

[PATCH v4 0/5] Improve robustness of migration
by David Gibson 28 Feb '25

28 Feb '25

From Red Hat internal testing we've had some reports that if attempting to migrate without passt-repair, the failure mode is uglier than we'd like. The migration fails, which is somewhat expected, but we don't correctly roll things back on the source, so it breaks network there as well. Address this and several other fragilities in the migration. Everything tested with the basic test suite, plus some additional testing for the specific functionality of the patches: For patches 1..2: * I get a clean migration if there are now active flows * Migration completes, although connections are broken if passt-repair isn't connected For patches 3..5: * Migration completes ok if the source and destination hosts have different IPs * Target correctly sees the bind() failure and abandons the flow * Unfortunately, target-side client doesn't get a reset, it just sits there not working. This is because a) the RST we try to send is lost because the queue is still inactive over the migration and b) we don't send RSTs or ICMPs for packets from the guest which no longer match a flow (I hope to tackle this soon) * After manually quitting the stuck client on the target, other connectivity works There are more fragile cases that I'm looking to fix, particularly the die()s in flow_migrate_source_rollback() and elsewhere. David Gibson (5): migrate, flow: Trivially succeed if migrating with no flows migrate, flow: Don't attempt to migrate TCP flows without passt-repair tcp: Correct error code handling from tcp_flow_repair_socket() tcp: Unconditionally move to CLOSED state on tcp_rst() migrate, tcp: Don't flow_alloc_cancel() during incoming migration flow.c | 17 +++++++++++++++-- tcp.c | 28 +++++++++++++++++++++------- 2 files changed, 36 insertions(+), 9 deletions(-) -- 2.48.1

2 6

[PATCH] selinux: Fixes/workarounds for passt and passt-repair, mostly for libvirt usage
by Stefano Brivio 28 Feb '25

28 Feb '25

Here are a bunch of workarounds and a couple of fixes for libvirt usage which are rather hard to split into single logical patches as there appear to be some obscure dependencies between some of them: - passt-repair needs to have an exec_type typeattribute (otherwise the policy for lsmd(1) causes a violation on getattr on its executable) file, and that typeattribute just happened to be there for passt as a result of init_daemon_domain(), but passt-repair isn't a daemon, so we need an explicit corecmd_executable_file() - passt-repair needs a workaround, which I'll revisit once https://github.com/fedora-selinux/selinux-policy/issues/2579 is solved, for usage with libvirt: allow it to use qemu_var_run_t and virt_var_run_t sockets - add 'bpf' and 'dac_read_search' capabilities for passt-repair: they are needed (for whatever reason I didn't investigate) to actually receive socket files via SCM_RIGHTS - passt needs further workarounds in the sense of https://github.com/fedora-selinux/selinux-policy/issues/2579: allow it to use map and use svirt_tmpfs_t (not just svirt_image_t): it depends on where the libvirt guest image is - ...it also needs to map /dev/null if <access mode='shared'/> is enabled in libvirt's XML for the memoryBacking object, for vhost-user operation - and 'ioctl' on the TCP socket appears to be actually needed, on top of 'getattr', to dump some socket parameters Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- contrib/selinux/passt-repair.te | 33 +++++++++++++++++++++++++++++++-- contrib/selinux/passt.te | 9 +++++++-- 2 files changed, 38 insertions(+), 4 deletions(-) diff --git a/contrib/selinux/passt-repair.te b/contrib/selinux/passt-repair.te index e3ffbcd..f171be6 100644 --- a/contrib/selinux/passt-repair.te +++ b/contrib/selinux/passt-repair.te @@ -28,12 +28,22 @@ require { type console_device_t; type user_devpts_t; type user_tmp_t; + + # Workaround: passt-repair needs to needs to access socket files + # that passt, started by libvirt, might create under different + # labels, depending on whether passt is started as root or not. + # + # However, libvirt doesn't maintain its own policy, which makes + # updates particularly complicated. To avoid breakage in the short + # term, deal with that in passt's own policy. + type qemu_var_run_t; + type virt_var_run_t; } type passt_repair_t; domain_type(passt_repair_t); type passt_repair_exec_t; -files_type(passt_repair_exec_t); +corecmd_executable_file(passt_repair_exec_t); role unconfined_r types passt_repair_t; @@ -41,7 +51,8 @@ allow passt_repair_t passt_repair_exec_t:file { read execute execute_no_trans en type_transition unconfined_t passt_repair_exec_t:process passt_repair_t; allow unconfined_t passt_repair_t:process transition; -allow passt_repair_t self:capability { dac_override net_admin net_raw }; +allow passt_repair_t self:capability { dac_override dac_read_search net_admin net_raw }; +allow passt_repair_t self:capability2 bpf; allow passt_repair_t console_device_t:chr_file { append open getattr read write ioctl }; allow passt_repair_t user_devpts_t:chr_file { append open getattr read write ioctl }; @@ -50,9 +61,27 @@ allow passt_repair_t unconfined_t:unix_stream_socket { connectto read write }; allow passt_repair_t passt_t:unix_stream_socket { connectto read write }; allow passt_repair_t user_tmp_t:unix_stream_socket { connectto read write }; +allow passt_repair_t user_tmp_t:dir search; + allow passt_repair_t unconfined_t:sock_file { read write }; allow passt_repair_t passt_t:sock_file { read write }; allow passt_repair_t user_tmp_t:sock_file { read write }; allow passt_repair_t unconfined_t:tcp_socket { read setopt write }; allow passt_repair_t passt_t:tcp_socket { read setopt write }; + +# Workaround: passt-repair needs to needs to access socket files +# that passt, started by libvirt, might create under different +# labels, depending on whether passt is started as root or not. +# +# However, libvirt doesn't maintain its own policy, which makes +# updates particularly complicated. To avoid breakage in the short +# term, deal with that in passt's own policy. +allow passt_repair_t qemu_var_run_t:unix_stream_socket { connectto read write }; +allow passt_repair_t virt_var_run_t:unix_stream_socket { connectto read write }; + +allow passt_repair_t qemu_var_run_t:dir search; +allow passt_repair_t virt_var_run_t:dir search; + +allow passt_repair_t qemu_var_run_t:sock_file { read write }; +allow passt_repair_t virt_var_run_t:sock_file { read write }; diff --git a/contrib/selinux/passt.te b/contrib/selinux/passt.te index f595079..f8ea672 100644 --- a/contrib/selinux/passt.te +++ b/contrib/selinux/passt.te @@ -29,6 +29,9 @@ require { # particularly complicated. To avoid breakage in the short term, # deal with it in passt's own policy. type svirt_image_t; + type svirt_tmpfs_t; + type svirt_t; + type null_device_t; class file { ioctl getattr setattr create read write unlink open relabelto execute execute_no_trans map }; class dir { search write add_name remove_name mounton }; @@ -45,7 +48,7 @@ require { type net_conf_t; type proc_net_t; type node_t; - class tcp_socket { create accept listen name_bind name_connect getattr }; + class tcp_socket { create accept listen name_bind name_connect getattr ioctl }; class udp_socket { create accept listen }; class icmp_socket { bind create name_bind node_bind setopt read write }; class sock_file { create unlink write }; @@ -129,7 +132,7 @@ corenet_udp_sendrecv_all_ports(passt_t) allow passt_t node_t:icmp_socket { name_bind node_bind }; allow passt_t port_t:icmp_socket name_bind; -allow passt_t self:tcp_socket { create getopt setopt connect bind listen accept shutdown read write getattr }; +allow passt_t self:tcp_socket { create getopt setopt connect bind listen accept shutdown read write getattr ioctl }; allow passt_t self:udp_socket { create getopt setopt connect bind read write }; allow passt_t self:icmp_socket { bind create setopt read write }; @@ -143,3 +146,5 @@ allow passt_t unconfined_t:unix_stream_socket { read write }; # particularly complicated. To avoid breakage in the short term, # deal with it in passt's own policy. allow passt_t svirt_image_t:file { read write map }; +allow passt_t svirt_tmpfs_t:file { read write map }; +allow passt_t null_device_t:chr_file map; -- 2.43.0

1 0

tcp_rst() complications
by David Gibson 27 Feb '25

27 Feb '25

Amongst other things, I spotted some additional complications using tcp_rst() in the migration path (some of which might also have implications in other contexts). These might be things we can safely ignore, at least for now, but I haven't thought through them enough to be sure. 1) Sending RST to guest during migration The first issue is that tcp_rst() will send an actual RST to the guest on the tap interface. During migration, that means we're sending to the guest while it's suspended. At the very least that means we probably have a much higher that usual chance of getting a queue full failure writing to the tap interface, which could hit problem (2). But, beyond that, with vhost-user that means we're writing to guest memory while the guest is suspended. Kind of the whole point of the suspended time is that the guest memory doesn't change during it, so I'm not sure what the consequences will be. Now, at the moment I think all our tcp_rst() calls are either on the source during rollback (i.e. we're committed to resuming only on the source) or on the target past the point of no return (i.e. we're committed to resuming only on the target). I suspect that means we can get away with it, but I do worry this could break something in qeme by violating that assumption. 2) tcp_rst() failures tcp_rst() can fail if tcp_send_flag() fails. In this case we *don't* change the events to CLOSED. I _think_ that's a bug: even if we weren't able to send the RST to the guest, we've already closed the socket so the flow is dead. Moving to CLOSED state (and then removing the flow entirely) should mean that we'll resend an RST if the guest attempts to use the flow again later. But.. I was worried there might be some subtle reason for not changing the event state in that case. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson

2 5

[PATCH v2 0/2] More graceful handling of migration without passt-repair
by David Gibson 27 Feb '25

27 Feb '25

From Red Hat internal testing we've had some reports that if attempting to migrate without passt-repair, the failure mode is uglier than we'd like. The migration fails, which is somewhat expected, but we don't correctly roll things back on the source, so it breaks network there as well. Handle this more gracefully allowing the migration to proceed in this case, but allow TCP connections to break I've now tested this reasonably: * I get a clean migration if there are now active flows * Migration completes, although connections are broken if passt-repair isn't connected * Basic test suite (minus perf) I didn't manage to test with libvirt yet, but I'm pretty convinced the behaviour should be better than it was. There are more fragile cases that I'm looking to fix, particularly the die()s in flow_migrate_source_rollback() and elsewhere, however I ran into various complications that I didn't manage to sort out today. I'll continue looking at those tomorrow. I'm now pretty confident that those additional fixes won't entirely supersede the changes in this series, so it should be fine to apply these on their own. David Gibson (2): migrate, flow: Trivially succeed if migrating with no flows migrate, flow: Don't attempt to migrate TCP flows without passt-repair flow.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) -- 2.48.1

2 9