- dev - passt

[PATCH v6 0/7] Add vhost-user support to passt. (part 3)
by Laurent Vivier 07 Oct '24

07 Oct '24

This series of patches adds vhost-user support to passt and then allows passt to connect to QEMU network backend using virtqueue rather than a socket. With QEMU, rather than using to connect: -netdev stream,id=s,server=off,addr.type=unix,addr.path=/tmp/passt_1.socket we will use: -chardev socket,id=chr0,path=/tmp/passt_1.socket -netdev vhost-user,id=netdev0,chardev=chr0 -device virtio-net,netdev=netdev0 -object memory-backend-memfd,id=memfd0,share=on,size=$RAMSIZE -numa node,memdev=memfd0 The memory backend is needed to share data between passt and QEMU. Performance comparison between "-netdev stream" and "-netdev vhost-user": $ iperf3 -c localhost -p 10001 -t 60 -6 -u -b 50G socket: [ 5] 0.00-60.05 sec 95.6 GBytes 13.7 Gbits/sec 0.017 ms 6998988/10132413 (69%) receiver vhost-user: [ 5] 0.00-60.04 sec 237 GBytes 33.9 Gbits/sec 0.006 ms 53673/7813770 (0.69%) receiver $ iperf3 -c localhost -p 10001 -t 60 -4 -u -b 50G socket: [ 5] 0.00-60.05 sec 98.9 GBytes 14.1 Gbits/sec 0.018 ms 6260735/9501832 (66%) receiver vhost-user: [ 5] 0.00-60.05 sec 235 GBytes 33.7 Gbits/sec 0.008 ms 37581/7752699 (0.48%) receiver $ iperf3 -c localhost -p 10001 -t 60 -6 socket: [ 5] 0.00-60.00 sec 17.3 GBytes 2.48 Gbits/sec 0 sender [ 5] 0.00-60.06 sec 17.3 GBytes 2.48 Gbits/sec receiver vhost-user: [ 5] 0.00-60.00 sec 191 GBytes 27.4 Gbits/sec 0 sender [ 5] 0.00-60.05 sec 191 GBytes 27.3 Gbits/sec receiver $ iperf3 -c localhost -p 10001 -t 60 -4 socket: [ 5] 0.00-60.00 sec 15.6 GBytes 2.24 Gbits/sec 0 sender [ 5] 0.00-60.06 sec 15.6 GBytes 2.24 Gbits/sec receiver vhost-user: [ 5] 0.00-60.00 sec 189 GBytes 27.1 Gbits/sec 0 sender [ 5] 0.00-60.04 sec 189 GBytes 27.0 Gbits/sec receiver v5: - rebase on top of 2024_09_06.6b38f07 - rework udp_vu.c as ref.udp.v6 has been removed and we need to know if we receive IPv4 or IPv6 frame when we prepare the guest buffers for recvmsg() - remove vnet->hdrlen as the size is always the same with virtio-net v1 - address comments from David and Stefano v6: - rebase - extract 3 patches from "vhost-user: add vhost-user": passt: rename tap_sock_init() to tap_backend_init() tcp: Export headers functions udp: Prepare udp.c to be shared with vhost-user - introduce new functions vu_collect_one_frame(), vu_collect(), vu_set_vnethdr(), vu_flush(), vu_send_single() to be called from tcp_vu.c, udp_vu.c and ICMP/DHCP where vhost-user code was duplicated. v4: - rebase on top of 2024_08_21.1d6142f (rebasing on top of 620e19a1b48a ("udp: Merge udp[46]_mh_recv arrays") introduces a regression in the measure of the latency with UDP because I think I don't replace correctly ref.udp.v6 that is removed by this commit) - Addressed most of the comments from David and Stefano (I didn't want to postpone this version to next week, so I'll address the remaining comments in the next version). v3: - rebase on top of flow table - update tcp_vu.c to look like udp_vu.c (recv()/prepare()/send_frame()) - address comments from Stefano and David on version 2 v2: - remove PATCH 4 - rewrite PATCH 2 and 3 to follow passt coding style - move some code from PATCH 3 to PATCH 4 (previously PATCH 5) - partially addressed David's comment on PATCH 5 Laurent Vivier (7): packet: replace struct desc by struct iovec vhost-user: introduce virtio API vhost-user: introduce vhost-user API udp: Prepare udp.c to be shared with vhost-user tcp: Export headers functions passt: rename tap_sock_init() to tap_backend_init() vhost-user: add vhost-user Makefile | 6 +- conf.c | 22 +- epoll_type.h | 4 + iov.c | 1 - isolation.c | 15 +- packet.c | 91 +++-- packet.h | 22 +- passt.1 | 10 +- passt.c | 11 +- passt.h | 6 + pcap.c | 1 - tap.c | 128 ++++-- tap.h | 7 +- tcp.c | 37 +- tcp_internal.h | 15 + tcp_vu.c | 476 +++++++++++++++++++++++ tcp_vu.h | 12 + udp.c | 84 ++-- udp_internal.h | 34 ++ udp_vu.c | 332 ++++++++++++++++ udp_vu.h | 13 + util.h | 8 + vhost_user.c | 1005 ++++++++++++++++++++++++++++++++++++++++++++++++ vhost_user.h | 206 ++++++++++ virtio.c | 660 +++++++++++++++++++++++++++++++ virtio.h | 184 +++++++++ vu_common.c | 327 ++++++++++++++++ vu_common.h | 47 +++ 28 files changed, 3626 insertions(+), 138 deletions(-) create mode 100644 tcp_vu.c create mode 100644 tcp_vu.h create mode 100644 udp_internal.h create mode 100644 udp_vu.c create mode 100644 udp_vu.h create mode 100644 vhost_user.c create mode 100644 vhost_user.h create mode 100644 virtio.c create mode 100644 virtio.h create mode 100644 vu_common.c create mode 100644 vu_common.h -- 2.46.2

1 7

[PATCH v2] test: Add tests for passt in vhost-user mode
by Stefano Brivio 07 Oct '24

07 Oct '24

Run functional and performance tests for vhost-user mode as well. For functional tests, we add passt_vu and passt_vu_in_ns as symbolic links to their non-vhost-user counterparts, as no differences are intended but we want to distinguish them in test logs. For performance tests, instead, we add separate perf/passt_vu_tcp and perf/passt_vu_udp files, as we need longer test duration, as well as higher UDP sending bandwidths and larger TCP windows, to actually get the highest throughput vhost-user mode offers. For valgrind tests, vhost-user mode needs two extra system calls: statx and readlink. Add them as EXTRA_SYSCALLS for the valgrind target. Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com> --- v2: - oops, add all the files I forgot in v1: test/perf/passt_vu_tcp, test/perf/passt_vu_udp, and symbolic links - also run the 'two_guests' case in vhost-user mode Makefile | 3 +- test/lib/perf_report | 15 +++ test/lib/setup | 77 ++++++++++++--- test/passt_vu | 1 + test/passt_vu_in_ns | 1 + test/perf/passt_vu_tcp | 211 +++++++++++++++++++++++++++++++++++++++++ test/perf/passt_vu_udp | 159 +++++++++++++++++++++++++++++++ test/run | 25 +++++ test/two_guests_vu | 1 + 9 files changed, 478 insertions(+), 15 deletions(-) create mode 120000 test/passt_vu create mode 120000 test/passt_vu_in_ns create mode 100644 test/perf/passt_vu_tcp create mode 100644 test/perf/passt_vu_udp create mode 120000 test/two_guests_vu diff --git a/Makefile b/Makefile index 74a9513..197729f 100644 --- a/Makefile +++ b/Makefile @@ -136,7 +136,8 @@ qrap: $(QRAP_SRCS) passt.h valgrind: EXTRA_SYSCALLS += rt_sigprocmask rt_sigtimedwait rt_sigaction \ rt_sigreturn getpid gettid kill clock_gettime mmap \ - mmap2 munmap open unlink gettimeofday futex + mmap2 munmap open unlink gettimeofday futex statx \ + readlink valgrind: FLAGS += -g -DVALGRIND valgrind: all diff --git a/test/lib/perf_report b/test/lib/perf_report index d1ef50b..c4ec817 100755 --- a/test/lib/perf_report +++ b/test/lib/perf_report @@ -49,6 +49,21 @@ td:empty { visibility: hidden; } __passt_tcp_LINE__ __passt_udp_LINE__ </table> +</li><li><p>passt with vhost-user support</p> +<table class="passt" width="70%"> + <tr> + <th/> + <th id="perf_passt_vu_tcp" colspan="__passt_vu_tcp_cols__">TCP, __passt_vu_tcp_threads__ at __passt_vu_tcp_freq__ GHz</th> + <th id="perf_passt_vu_udp" colspan="__passt_vu_udp_cols__">UDP, __passt_vu_udp_threads__ at __passt_vu_udp_freq__ GHz</th> + </tr> + <tr> + <td align="right">MTU:</td> + __passt_vu_tcp_header__ + __passt_vu_udp_header__ + </tr> + __passt_vu_tcp_LINE__ __passt_vu_udp_LINE__ +</table> + <style type="text/CSS"> table.pasta_local td { border: 0px solid; padding: 6px; line-height: 1; } table.pasta_local td { text-align: right; } diff --git a/test/lib/setup b/test/lib/setup index 5338393..3409bd2 100755 --- a/test/lib/setup +++ b/test/lib/setup @@ -15,8 +15,7 @@ INITRAMFS="${BASEPATH}/mbuto.img" VCPUS="$( [ $(nproc) -ge 8 ] && echo 6 || echo $(( $(nproc) / 2 + 1 )) )" -__mem_kib="$(sed -n 's/MemTotal:[ ]*$[0-9]*$ kB/\1/p' /proc/meminfo)" -VMEM="$((${__mem_kib} / 1024 / 4))" +MEM_KIB="$(sed -n 's/MemTotal:[ ]*$[0-9]*$ kB/\1/p' /proc/meminfo)" QEMU_ARCH="$(uname -m)" [ "${QEMU_ARCH}" = "i686" ] && QEMU_ARCH=i386 @@ -46,6 +45,7 @@ setup_passt() { [ ${PCAP} -eq 1 ] && __opts="${__opts} -p ${LOGDIR}/passt.pcap" [ ${DEBUG} -eq 1 ] && __opts="${__opts} -d" [ ${TRACE} -eq 1 ] && __opts="${__opts} --trace" + [ ${VHOST_USER} -eq 1 ] && __opts="${__opts} --vhost-user" context_run passt "make clean" context_run passt "make valgrind" @@ -54,16 +54,29 @@ setup_passt() { # pidfile isn't created until passt is listening wait_for [ -f "${STATESETUP}/passt.pid" ] + __vmem="$((${MEM_KIB} / 1024 / 4))" + if [ ${VHOST_USER} -eq 1 ]; then + __vmem="$(((${__vmem} + 500) / 1000))G" + __qemu_netdev=" \ + -chardev socket,id=c,path=${STATESETUP}/passt.socket \ + -netdev vhost-user,id=v,chardev=c \ + -device virtio-net,netdev=v \ + -object memory-backend-memfd,id=m,share=on,size=${__vmem} \ + -numa node,memdev=m" + else + __qemu_netdev="-device virtio-net-pci,netdev=s \ + -netdev stream,id=s,server=off,addr.type=unix,addr.path=${STATESETUP}/passt.socket" + fi + GUEST_CID=94557 context_run_bg qemu 'qemu-system-'"${QEMU_ARCH}" \ ' -machine accel=kvm' \ - ' -m '${VMEM}' -cpu host -smp '${VCPUS} \ + ' -m '${__vmem}' -cpu host -smp '${VCPUS} \ ' -kernel '"${KERNEL}" \ ' -initrd '${INITRAMFS}' -nographic -serial stdio' \ ' -nodefaults' \ ' -append "console=ttyS0 mitigations=off apparmor=0" ' \ - ' -device virtio-net-pci,netdev=s0 ' \ - " -netdev stream,id=s0,server=off,addr.type=unix,addr.path=${STATESETUP}/passt.socket " \ + " ${__qemu_netdev}" \ " -pidfile ${STATESETUP}/qemu.pid" \ " -device vhost-vsock-pci,guest-cid=$GUEST_CID" @@ -142,6 +155,7 @@ setup_passt_in_ns() { [ ${PCAP} -eq 1 ] && __opts="${__opts} -p ${LOGDIR}/passt_in_pasta.pcap" [ ${DEBUG} -eq 1 ] && __opts="${__opts} -d" [ ${TRACE} -eq 1 ] && __opts="${__opts} --trace" + [ ${VHOST_USER} -eq 1 ] && __opts="${__opts} --vhost-user" if [ ${VALGRIND} -eq 1 ]; then context_run passt "make clean" @@ -154,17 +168,30 @@ setup_passt_in_ns() { fi wait_for [ -f "${STATESETUP}/passt.pid" ] + __vmem="$((${MEM_KIB} / 1024 / 4))" + if [ ${VHOST_USER} -eq 1 ]; then + __vmem="$(((${__vmem} + 500) / 1000))G" + __qemu_netdev=" \ + -chardev socket,id=c,path=${STATESETUP}/passt.socket \ + -netdev vhost-user,id=v,chardev=c \ + -device virtio-net,netdev=v \ + -object memory-backend-memfd,id=m,share=on,size=${__vmem} \ + -numa node,memdev=m" + else + __qemu_netdev="-device virtio-net-pci,netdev=s \ + -netdev stream,id=s,server=off,addr.type=unix,addr.path=${STATESETUP}/passt.socket" + fi + GUEST_CID=94557 context_run_bg qemu 'qemu-system-'"${QEMU_ARCH}" \ ' -machine accel=kvm' \ ' -M accel=kvm:tcg' \ - ' -m '${VMEM}' -cpu host -smp '${VCPUS} \ + ' -m '${__vmem}' -cpu host -smp '${VCPUS} \ ' -kernel '"${KERNEL}" \ ' -initrd '${INITRAMFS}' -nographic -serial stdio' \ ' -nodefaults' \ ' -append "console=ttyS0 mitigations=off apparmor=0" ' \ - ' -device virtio-net-pci,netdev=s0 ' \ - " -netdev stream,id=s0,server=off,addr.type=unix,addr.path=${STATESETUP}/passt.socket " \ + " ${__qemu_netdev}" \ " -pidfile ${STATESETUP}/qemu.pid" \ " -device vhost-vsock-pci,guest-cid=$GUEST_CID" @@ -214,6 +241,7 @@ setup_two_guests() { [ ${PCAP} -eq 1 ] && __opts="${__opts} -p ${LOGDIR}/passt_1.pcap" [ ${DEBUG} -eq 1 ] && __opts="${__opts} -d" [ ${TRACE} -eq 1 ] && __opts="${__opts} --trace" + [ ${VHOST_USER} -eq 1 ] && __opts="${__opts} --vhost-user" context_run_bg passt_1 "./passt -s ${STATESETUP}/passt_1.socket -P ${STATESETUP}/passt_1.pid -f ${__opts} -t 10001 -u 10001" wait_for [ -f "${STATESETUP}/passt_1.pid" ] @@ -222,33 +250,54 @@ setup_two_guests() { [ ${PCAP} -eq 1 ] && __opts="${__opts} -p ${LOGDIR}/passt_2.pcap" [ ${DEBUG} -eq 1 ] && __opts="${__opts} -d" [ ${TRACE} -eq 1 ] && __opts="${__opts} --trace" + [ ${VHOST_USER} -eq 1 ] && __opts="${__opts} --vhost-user" context_run_bg passt_2 "./passt -s ${STATESETUP}/passt_2.socket -P ${STATESETUP}/passt_2.pid -f ${__opts} -t 10004 -u 10004" wait_for [ -f "${STATESETUP}/passt_2.pid" ] + __vmem="$((${MEM_KIB} / 1024 / 4))" + if [ ${VHOST_USER} -eq 1 ]; then + __vmem="$(((${__vmem} + 500) / 1000))G" + __qemu_netdev1=" \ + -chardev socket,id=c,path=${STATESETUP}/passt_1.socket \ + -netdev vhost-user,id=v,chardev=c \ + -device virtio-net,netdev=v \ + -object memory-backend-memfd,id=m,share=on,size=${__vmem} \ + -numa node,memdev=m" + __qemu_netdev1=" \ + -chardev socket,id=c,path=${STATESETUP}/passt_2.socket \ + -netdev vhost-user,id=v,chardev=c \ + -device virtio-net,netdev=v \ + -object memory-backend-memfd,id=m,share=on,size=${__vmem} \ + -numa node,memdev=m" + else + __qemu_netdev1="-device virtio-net-pci,netdev=s \ + -netdev stream,id=s,server=off,addr.type=unix,addr.path=${STATESETUP}/passt_1.socket" + __qemu_netdev2="-device virtio-net-pci,netdev=s \ + -netdev stream,id=s,server=off,addr.type=unix,addr.path=${STATESETUP}/passt_2.socket" + fi + GUEST_1_CID=94557 context_run_bg qemu_1 'qemu-system-'"${QEMU_ARCH}" \ ' -M accel=kvm:tcg' \ - ' -m '${VMEM}' -cpu host -smp '${VCPUS} \ + ' -m '${__vmem}' -cpu host -smp '${VCPUS} \ ' -kernel '"${KERNEL}" \ ' -initrd '${INITRAMFS}' -nographic -serial stdio' \ ' -nodefaults' \ ' -append "console=ttyS0 mitigations=off apparmor=0" ' \ - ' -device virtio-net-pci,netdev=s0 ' \ - " -netdev stream,id=s0,server=off,addr.type=unix,addr.path=${STATESETUP}/passt_1.socket " \ + " ${__qemu_netdev1}" \ " -pidfile ${STATESETUP}/qemu_1.pid" \ " -device vhost-vsock-pci,guest-cid=$GUEST_1_CID" GUEST_2_CID=94558 context_run_bg qemu_2 'qemu-system-'"${QEMU_ARCH}" \ ' -M accel=kvm:tcg' \ - ' -m '${VMEM}' -cpu host -smp '${VCPUS} \ + ' -m '${__vmem}' -cpu host -smp '${VCPUS} \ ' -kernel '"${KERNEL}" \ ' -initrd '${INITRAMFS}' -nographic -serial stdio' \ ' -nodefaults' \ ' -append "console=ttyS0 mitigations=off apparmor=0" ' \ - ' -device virtio-net-pci,netdev=s0 ' \ - " -netdev stream,id=s0,server=off,addr.type=unix,addr.path=${STATESETUP}/passt_2.socket " \ + " ${__qemu_netdev2}" \ " -pidfile ${STATESETUP}/qemu_2.pid" \ " -device vhost-vsock-pci,guest-cid=$GUEST_2_CID" diff --git a/test/passt_vu b/test/passt_vu new file mode 120000 index 0000000..22f1840 --- /dev/null +++ b/test/passt_vu @@ -0,0 +1 @@ +passt \ No newline at end of file diff --git a/test/passt_vu_in_ns b/test/passt_vu_in_ns new file mode 120000 index 0000000..3ff479e --- /dev/null +++ b/test/passt_vu_in_ns @@ -0,0 +1 @@ +passt_in_ns \ No newline at end of file diff --git a/test/perf/passt_vu_tcp b/test/perf/passt_vu_tcp new file mode 100644 index 0000000..b434008 --- /dev/null +++ b/test/perf/passt_vu_tcp @@ -0,0 +1,211 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# PASST - Plug A Simple Socket Transport +# for qemu/UNIX domain socket mode +# +# PASTA - Pack A Subtle Tap Abstraction +# for network namespace/tap device mode +# +# test/perf/passt_vu_tcp - Check TCP performance in passt vhost-user mode +# +# Copyright (c) 2021 Red Hat GmbH +# Author: Stefano Brivio <sbrivio(a)redhat.com> + +gtools /sbin/sysctl ip jq nproc seq sleep iperf3 tcp_rr tcp_crr # From neper +nstools /sbin/sysctl ip jq nproc seq sleep iperf3 tcp_rr tcp_crr +htools bc head sed seq + +set MAP_NS4 192.0.2.2 +set MAP_NS6 2001:db8:9a55::2 + +test passt: throughput and latency + +guest /sbin/sysctl -w net.core.rmem_max=536870912 +guest /sbin/sysctl -w net.core.wmem_max=536870912 +guest /sbin/sysctl -w net.core.rmem_default=33554432 +guest /sbin/sysctl -w net.core.wmem_default=33554432 +guest /sbin/sysctl -w net.ipv4.tcp_rmem="4096 131072 268435456" +guest /sbin/sysctl -w net.ipv4.tcp_wmem="4096 131072 268435456" +guest /sbin/sysctl -w net.ipv4.tcp_timestamps=0 + +ns /sbin/sysctl -w net.ipv4.tcp_rmem="4096 524288 134217728" +ns /sbin/sysctl -w net.ipv4.tcp_wmem="4096 524288 134217728" +ns /sbin/sysctl -w net.ipv4.tcp_timestamps=0 + +gout IFNAME ip -j link show | jq -rM '.[] | select(.link_type == "ether").ifname' + +hout FREQ_PROCFS (echo "scale=1"; sed -n 's/cpu MHz.*: $[0-9]*$\..*$/(\1+10^2\/2)\/10^3/p' /proc/cpuinfo) | bc -l | head -n1 +hout FREQ_CPUFREQ (echo "scale=1"; printf '( %i + 10^5 / 2 ) / 10^6\n' $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq) ) | bc -l +hout FREQ [ -n "__FREQ_CPUFREQ__" ] && echo __FREQ_CPUFREQ__ || echo __FREQ_PROCFS__ + +set THREADS 4 +set TIME 5 +set OMIT 0.1 +set OPTS -Z -P __THREADS__ -l 1M -O__OMIT__ -N + +info Throughput in Gbps, latency in µs, __THREADS__ threads at __FREQ__ GHz +report passt_vu tcp __THREADS__ __FREQ__ + +th MTU 256B 576B 1280B 1500B 9000B 65520B + + +tr TCP throughput over IPv6: guest to host +iperf3s ns 10002 + +bw - +bw - +guest ip link set dev __IFNAME__ mtu 1280 +iperf3 BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -w 16M +bw __BW__ 1.2 1.5 +guest ip link set dev __IFNAME__ mtu 1500 +iperf3 BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -w 32M +bw __BW__ 1.6 1.8 +guest ip link set dev __IFNAME__ mtu 9000 +iperf3 BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -w 64M +bw __BW__ 4.0 5.0 +guest ip link set dev __IFNAME__ mtu 65520 +iperf3 BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -w 64M +bw __BW__ 7.0 8.0 + +iperf3k ns + +tl TCP RR latency over IPv6: guest to host +lat - +lat - +lat - +lat - +lat - +nsb tcp_rr --nolog -6 +gout LAT tcp_rr --nolog -l1 -6 -c -H __MAP_NS6__ | sed -n 's/^throughput=$.*$/\1/p' +lat __LAT__ 200 150 + +tl TCP CRR latency over IPv6: guest to host +lat - +lat - +lat - +lat - +lat - +nsb tcp_crr --nolog -6 +gout LAT tcp_crr --nolog -l1 -6 -c -H __MAP_NS6__ | sed -n 's/^throughput=$.*$/\1/p' +lat __LAT__ 500 400 + +tr TCP throughput over IPv4: guest to host +iperf3s ns 10002 + +guest ip link set dev __IFNAME__ mtu 256 +iperf3 BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 2M +bw __BW__ 0.2 0.3 +guest ip link set dev __IFNAME__ mtu 576 +iperf3 BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 4M +bw __BW__ 0.5 0.8 +guest ip link set dev __IFNAME__ mtu 1280 +iperf3 BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 8M +bw __BW__ 1.2 1.5 +guest ip link set dev __IFNAME__ mtu 1500 +iperf3 BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 16M +bw __BW__ 1.6 1.8 +guest ip link set dev __IFNAME__ mtu 9000 +iperf3 BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 64M +bw __BW__ 4.0 5.0 +guest ip link set dev __IFNAME__ mtu 65520 +iperf3 BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 64M +bw __BW__ 7.0 8.0 + +iperf3k ns + +# Reducing MTU below 1280 deconfigures IPv6, get our address back +guest dhclient -6 -x +guest dhclient -6 __IFNAME__ + +tl TCP RR latency over IPv4: guest to host +lat - +lat - +lat - +lat - +lat - +nsb tcp_rr --nolog -4 +gout LAT tcp_rr --nolog -l1 -4 -c -H __MAP_NS4__ | sed -n 's/^throughput=$.*$/\1/p' +lat __LAT__ 200 150 + +tl TCP CRR latency over IPv4: guest to host +lat - +lat - +lat - +lat - +lat - +nsb tcp_crr --nolog -4 +gout LAT tcp_crr --nolog -l1 -4 -c -H __MAP_NS4__ | sed -n 's/^throughput=$.*$/\1/p' +lat __LAT__ 500 400 + +tr TCP throughput over IPv6: host to guest +iperf3s guest 10001 + +bw - +bw - +bw - +bw - +bw - +iperf3 BW ns ::1 10001 __TIME__ __OPTS__ -w 32M +bw __BW__ 6.0 6.8 + +iperf3k guest + +tl TCP RR latency over IPv6: host to guest +lat - +lat - +lat - +lat - +lat - +guestb tcp_rr --nolog -P 10001 -C 10011 -6 +sleep 1 +nsout LAT tcp_rr --nolog -l1 -P 10001 -C 10011 -6 -c -H ::1 | sed -n 's/^throughput=$.*$/\1/p' +lat __LAT__ 200 150 + +tl TCP CRR latency over IPv6: host to guest +lat - +lat - +lat - +lat - +lat - +guestb tcp_crr --nolog -P 10001 -C 10011 -6 +sleep 1 +nsout LAT tcp_crr --nolog -l1 -P 10001 -C 10011 -6 -c -H ::1 | sed -n 's/^throughput=$.*$/\1/p' +lat __LAT__ 500 350 + + +tr TCP throughput over IPv4: host to guest +iperf3s guest 10001 + +bw - +bw - +bw - +bw - +bw - +iperf3 BW ns 127.0.0.1 10001 __TIME__ __OPTS__ -w 32M +bw __BW__ 6.0 6.8 + +iperf3k guest + +tl TCP RR latency over IPv4: host to guest +lat - +lat - +lat - +lat - +lat - +guestb tcp_rr --nolog -P 10001 -C 10011 -4 +sleep 1 +nsout LAT tcp_rr --nolog -l1 -P 10001 -C 10011 -4 -c -H 127.0.0.1 | sed -n 's/^throughput=$.*$/\1/p' +lat __LAT__ 200 150 + +tl TCP CRR latency over IPv6: host to guest +lat - +lat - +lat - +lat - +lat - +guestb tcp_crr --nolog -P 10001 -C 10011 -4 +sleep 1 +nsout LAT tcp_crr --nolog -l1 -P 10001 -C 10011 -4 -c -H 127.0.0.1 | sed -n 's/^throughput=$.*$/\1/p' +lat __LAT__ 500 300 + +te diff --git a/test/perf/passt_vu_udp b/test/perf/passt_vu_udp new file mode 100644 index 0000000..943ac11 --- /dev/null +++ b/test/perf/passt_vu_udp @@ -0,0 +1,159 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# PASST - Plug A Simple Socket Transport +# for qemu/UNIX domain socket mode +# +# PASTA - Pack A Subtle Tap Abstraction +# for network namespace/tap device mode +# +# test/perf/passt_vu_udp - Check UDP performance in passt vhost-user mode +# +# Copyright (c) 2021 Red Hat GmbH +# Author: Stefano Brivio <sbrivio(a)redhat.com> + +gtools /sbin/sysctl ip jq nproc sleep iperf3 udp_rr # From neper +nstools ip jq sleep iperf3 udp_rr +htools bc head sed + +set MAP_NS4 192.0.2.2 +set MAP_NS6 2001:db8:9a55::2 + +test passt: throughput and latency + +guest /sbin/sysctl -w net.core.rmem_max=16777216 +guest /sbin/sysctl -w net.core.wmem_max=16777216 +guest /sbin/sysctl -w net.core.rmem_default=16777216 +guest /sbin/sysctl -w net.core.wmem_default=16777216 + +hout FREQ_PROCFS (echo "scale=1"; sed -n 's/cpu MHz.*: $[0-9]*$\..*$/(\1+10^2\/2)\/10^3/p' /proc/cpuinfo) | bc -l | head -n1 +hout FREQ_CPUFREQ (echo "scale=1"; printf '( %i + 10^5 / 2 ) / 10^6\n' $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq) ) | bc -l +hout FREQ [ -n "__FREQ_CPUFREQ__" ] && echo __FREQ_CPUFREQ__ || echo __FREQ_PROCFS__ + +set THREADS 2 +set TIME 1 +set OPTS -u -P __THREADS__ --pacing-timer 1000 + +info Throughput in Gbps, latency in µs, __THREADS__ threads at __FREQ__ GHz + +report passt_vu udp __THREADS__ __FREQ__ + +th pktlen 256B 576B 1280B 1500B 9000B 65520B + +tr UDP throughput over IPv6: guest to host +iperf3s ns 10002 +# (datagram size) = (packet size) - 48: 40 bytes of IPv6 header, 8 of UDP header + +bw - +bw - +iperf3 BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -b 3G -l 1232 +bw __BW__ 0.8 1.2 +iperf3 BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -b 4G -l 1452 +bw __BW__ 1.0 1.5 +iperf3 BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -b 10G -l 8952 +bw __BW__ 4.0 5.0 +iperf3 BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -b 20G -l 64372 +bw __BW__ 4.0 5.0 + +iperf3k ns + +tl UDP RR latency over IPv6: guest to host +lat - +lat - +lat - +lat - +lat - +nsb udp_rr --nolog -6 +gout LAT udp_rr --nolog -6 -c -H __MAP_NS6__ | sed -n 's/^throughput=$.*$/\1/p' +lat __LAT__ 200 150 + + +tr UDP throughput over IPv4: guest to host +iperf3s ns 10002 +# (datagram size) = (packet size) - 28: 20 bytes of IPv4 header, 8 of UDP header + +iperf3 BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -b 1G -l 228 +bw __BW__ 0.0 0.0 +iperf3 BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -b 2G -l 548 +bw __BW__ 0.4 0.6 +iperf3 BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -b 3G -l 1252 +bw __BW__ 0.8 1.2 +iperf3 BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -b 4G -l 1472 +bw __BW__ 1.0 1.5 +iperf3 BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -b 10G -l 8972 +bw __BW__ 4.0 5.0 +iperf3 BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -b 20G -l 65492 +bw __BW__ 4.0 5.0 + +iperf3k ns + +tl UDP RR latency over IPv4: guest to host +lat - +lat - +lat - +lat - +lat - +nsb udp_rr --nolog -4 +gout LAT udp_rr --nolog -4 -c -H __MAP_NS4__ | sed -n 's/^throughput=$.*$/\1/p' +lat __LAT__ 200 150 + + +tr UDP throughput over IPv6: host to guest +iperf3s guest 10001 +# (datagram size) = (packet size) - 48: 40 bytes of IPv6 header, 8 of UDP header + +bw - +bw - +iperf3 BW ns ::1 10001 __TIME__ __OPTS__ -b 3G -l 1232 +bw __BW__ 0.8 1.2 +iperf3 BW ns ::1 10001 __TIME__ __OPTS__ -b 4G -l 1452 +bw __BW__ 1.0 1.5 +iperf3 BW ns ::1 10001 __TIME__ __OPTS__ -b 10G -l 8952 +bw __BW__ 3.0 4.0 +iperf3 BW ns ::1 10001 __TIME__ __OPTS__ -b 20G -l 64372 +bw __BW__ 3.0 4.0 + +iperf3k guest + +tl UDP RR latency over IPv6: host to guest +lat - +lat - +lat - +lat - +lat - +guestb udp_rr --nolog -P 10001 -C 10011 -6 +sleep 1 +nsout LAT udp_rr --nolog -P 10001 -C 10011 -6 -c -H ::1 | sed -n 's/^throughput=$.*$/\1/p' +lat __LAT__ 200 150 + + +tr UDP throughput over IPv4: host to guest +iperf3s guest 10001 +# (datagram size) = (packet size) - 28: 20 bytes of IPv4 header, 8 of UDP header + +iperf3 BW ns 127.0.0.1 10001 __TIME__ __OPTS__ -b 1G -l 228 +bw __BW__ 0.0 0.0 +iperf3 BW ns 127.0.0.1 10001 __TIME__ __OPTS__ -b 2G -l 548 +bw __BW__ 0.4 0.6 +iperf3 BW ns 127.0.0.1 10001 __TIME__ __OPTS__ -b 3G -l 1252 +bw __BW__ 0.8 1.2 +iperf3 BW ns 127.0.0.1 10001 __TIME__ __OPTS__ -b 4G -l 1472 +bw __BW__ 1.0 1.5 +iperf3 BW ns 127.0.0.1 10001 __TIME__ __OPTS__ -b 10G -l 8972 +bw __BW__ 3.0 4.0 +iperf3 BW ns 127.0.0.1 10001 __TIME__ __OPTS__ -b 20G -l 65492 +bw __BW__ 3.0 4.0 + +iperf3k guest + +tl UDP RR latency over IPv4: host to guest +lat - +lat - +lat - +lat - +lat - +guestb udp_rr --nolog -P 10001 -C 10011 -4 +sleep 1 +nsout LAT udp_rr --nolog -P 10001 -C 10011 -4 -c -H 127.0.0.1 | sed -n 's/^throughput=$.*$/\1/p' +lat __LAT__ 200 150 + +te diff --git a/test/run b/test/run index 547a729..f188d8e 100755 --- a/test/run +++ b/test/run @@ -93,6 +93,7 @@ run() { test memory/passt teardown memory + VHOST_USER=0 setup passt test passt/ndp test passt/dhcp @@ -115,7 +116,22 @@ run() { test two_guests/basic teardown two_guests + VHOST_USER=1 + setup passt_in_ns + test passt_vu/ndp + test passt_vu_in_ns/dhcp + test passt_vu_in_ns/icmp + test passt_vu_in_ns/tcp + test passt_vu_in_ns/udp + test passt_vu_in_ns/shutdown + teardown passt_in_ns + + setup two_guests + test two_guests_vu/basic + teardown two_guests + VALGRIND=0 + VHOST_USER=0 setup passt_in_ns test passt/ndp test passt_in_ns/dhcp @@ -126,6 +142,15 @@ run() { test passt_in_ns/shutdown teardown passt_in_ns + VHOST_USER=1 + setup passt_in_ns + test passt_vu/ndp + test passt_vu_in_ns/dhcp + test perf/passt_vu_tcp + test perf/passt_vu_udp + test passt_vu_in_ns/shutdown + teardown passt_in_ns + # TODO: Make those faster by at least pre-installing gcc and make on # non-x86 images, then re-enable. skip_distro() { diff --git a/test/two_guests_vu b/test/two_guests_vu new file mode 120000 index 0000000..144b7ca --- /dev/null +++ b/test/two_guests_vu @@ -0,0 +1 @@ +test/two_guests \ No newline at end of file -- 2.43.0

2 2

[PATCH v6 0/5] tcp: use csum_iov() in tcp_update_check_tcp[4|6]()
by Laurent Vivier 04 Oct '24

04 Oct '24

For vhost-user, we will need to spread TCP payload over several buffers. To re-use tcp_update_check_tcp[4|6](), provide an iovec rather than a pointer to a buffer. This series updates also csum_iov() and pcap_iov() to add an offset of bytes to skip in the iovec array. It's based on top of "tcp: Use tcp_payload_t rather than tcphdr" that is added in the series for convenience. v6: update tcp.c with fix for clang-tidy error and add missing static Laurent Vivier (5): tcp: Use tcp_payload_t rather than tcphdr pcap: Add an offset argument in pcap_iov() checksum: Add an offset argument in csum_iov() tcp: Update TCP checksum using an iovec array udp: Update UDP checksum using an iovec array checksum.c | 46 +++++++++++----- checksum.h | 7 +-- iov.c | 1 - pcap.c | 5 +- pcap.h | 2 +- tap.c | 14 +++-- tap.h | 2 +- tcp.c | 140 +++++++++++++++++++++++++++++++++++++++---------- tcp_buf.c | 29 ---------- tcp_internal.h | 29 ++++++++++ udp.c | 17 ++++-- 11 files changed, 206 insertions(+), 86 deletions(-) -- 2.46.0

2 8

[PATCH v5 0/5] tcp: use csum_iov() in tcp_update_check_tcp[4|6]()
by Laurent Vivier 03 Oct '24

03 Oct '24

For vhost-user, we will need to spread TCP payload over several buffers. To re-use tcp_update_check_tcp[4|6](), provide an iovec rather than a pointer to a buffer. This series updates also csum_iov() and pcap_iov() to add an offset of bytes to skip in the iovec array. It's based on top of "tcp: Use tcp_payload_t rather than tcphdr" that is added in the series for convenience. v3 adds "udp: Update UDP checksum using an iovec array" that modifies csum_udp4() and csum_udp6() in the same way. Laurent Vivier (5): tcp: Use tcp_payload_t rather than tcphdr pcap: Add an offset argument in pcap_iov() checksum: Add an offset argument in csum_iov() tcp: Update TCP checksum using an iovec array udp: Update UDP checksum using an iovec array checksum.c | 46 +++++++++++----- checksum.h | 7 +-- iov.c | 1 - pcap.c | 5 +- pcap.h | 2 +- tap.c | 14 +++-- tap.h | 2 +- tcp.c | 140 +++++++++++++++++++++++++++++++++++++++---------- tcp_buf.c | 29 ---------- tcp_internal.h | 29 ++++++++++ udp.c | 17 ++++-- 11 files changed, 206 insertions(+), 86 deletions(-) -- 2.46.0

3 15

[PATCH] conf: Add --dns-host option to configure host side nameserver
by David Gibson 03 Oct '24

03 Oct '24

When redirecting DNS queries with the --dns-forward option, passt/pasta needs a host side nameserver to redirect the queries to. This is controlled by the c->ip[46].dns_host variables. This is set to the first first nameserver listed in the host's /etc/resolv.conf, and there isn't currently a way to override it from the command line. Prior to 0b25cac9 ("conf: Treat --dns addresses as guest visible addresses") it was possible to alter this with the -D/--dns option. However, doing so was confusing and had some nonsensical edge cases because -D generally takes guest side addresses, rather than host side addresses. Add a new --dns-host option to restore this functionality in a more sensible way. Link: https://bugs.passt.top/show_bug.cgi?id=102 Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- conf.c | 16 ++++++++++++++++ passt.1 | 17 +++++++++++++---- 2 files changed, 29 insertions(+), 4 deletions(-) Based on the earlier patch adding the --freebind option. diff --git a/conf.c b/conf.c index e617bb6..b3b5342 100644 --- a/conf.c +++ b/conf.c @@ -829,6 +829,9 @@ static void usage(const char *name, FILE *f, int status) " --dns-forward ADDR Forward DNS queries sent to ADDR\n" " can be specified zero to two times (for IPv4 and IPv6)\n" " default: don't forward DNS queries\n" + " --dns-host ADDR Host nameserver to direct queries to\n" + " can be specified zero to two times (for IPv4 and IPv6)\n" + " default: first nameserver from host's /etc/resolv.conf\n" " --no-tcp Disable TCP protocol handler\n" " --no-udp Disable UDP protocol handler\n" " --no-icmp Disable ICMP/ICMPv6 protocol handler\n" @@ -1290,6 +1293,7 @@ void conf(struct ctx *c, int argc, char **argv) {"map-host-loopback", required_argument, NULL, 21 }, {"map-guest-addr", required_argument, NULL, 22 }, {"host-lo-to-ns-lo", no_argument, NULL, 23 }, + {"dns-host", required_argument, NULL, 24 }, { 0 }, }; const char *logname = (c->mode == MODE_PASTA) ? "pasta" : "passt"; @@ -1472,6 +1476,18 @@ void conf(struct ctx *c, int argc, char **argv) die("--host-lo-to-ns-lo is for pasta mode only"); c->host_lo_to_ns_lo = 1; break; + case 24: + if (inet_pton(AF_INET6, optarg, &c->ip6.dns_host) && + !IN6_IS_ADDR_UNSPECIFIED(&c->ip6.dns_host)) + break; + + if (inet_pton(AF_INET, optarg, &c->ip4.dns_host) && + !IN4_IS_ADDR_UNSPECIFIED(&c->ip4.dns_host) && + !IN4_IS_ADDR_BROADCAST(&c->ip4.dns_host)) + break; + + die("Invalid host nameserver address: %s", optarg); + break; case 'd': c->debug = 1; c->quiet = 0; diff --git a/passt.1 b/passt.1 index 1386f29..f084978 100644 --- a/passt.1 +++ b/passt.1 @@ -249,10 +249,19 @@ the host. .TP .BR \-\-dns-forward " " \fIaddr Map \fIaddr\fR (IPv4 or IPv6) as seen from guest or namespace to the -first configured DNS resolver (with corresponding IP version). Maps -only UDP and TCP traffic to port 53 or port 853. Replies are -translated back with a reverse mapping. This option can be specified -zero to two times (once for IPv4, once for IPv6). +nameserver (with corresponding IP version) specified by the +\fB\-\-dns-host\fR option. Maps only UDP and TCP traffic to port 53 or +port 853. Replies are translated back with a reverse mapping. This +option can be specified zero to two times (once for IPv4, once for +IPv6). + +.TP +.BR \-\-dns-host " " \fIaddr +Configure the host nameserver which guest or namespace queries to the +\fB\-\-dns-forward\fR address will be redirected to. This option can +be specified zero to two times (once for IPv4, once for IPv6). +By default, the first nameserver from the host's +\fI/etc/resolv.conf\fR. .TP .BR \-S ", " \-\-search " " \fIlist -- 2.46.2

1 0

[PATCH v4] conf: Add command line switch to enable IP_FREEBIND socket option
by David Gibson 03 Oct '24

03 Oct '24

In a couple of recent reports, we've seen that it can be useful for pasta to forward ports from addresses which are not currently configured on the host, but might be in future. That can be done with the sysctl net.ipv4.ip_nonlocal_bind, but that does require CAP_NET_ADMIN to set in the first place. We can allow the same thing on a per-socket basis with the IP_FREEBIND (or IPV6_FREEBIND) socket option. Add a --freebind command line argument to enable this socket option on all listening sockets. Link: https://bugs.passt.top/show_bug.cgi?id=101 Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- conf.c | 2 ++ passt.1 | 10 ++++++++++ passt.h | 2 ++ util.c | 16 ++++++++++++++++ 4 files changed, 30 insertions(+) Changes since v3: * Rebased on the fixes for podman bug 24045, which is likely to merge first and has a textual conflict with this one. Changes since v2: * Actually committed all the changes that were supposed to be in v2 :/ Changes since v1: * Reworded --help and man page additions for clarity * Added missing structure comment * Fixed some incorrect indentation diff --git a/conf.c b/conf.c index b5318f3..e617bb6 100644 --- a/conf.c +++ b/conf.c @@ -836,6 +836,7 @@ static void usage(const char *name, FILE *f, int status) " --no-ndp Disable NDP responses\n" " --no-dhcpv6 Disable DHCPv6 server\n" " --no-ra Disable router advertisements\n" + " --freebind Bind to any address for forwarding\n" " --no-map-gw Don't map gateway address to host\n" " -4, --ipv4-only Enable IPv4 operation only\n" " -6, --ipv6-only Enable IPv6 operation only\n"); @@ -1258,6 +1259,7 @@ void conf(struct ctx *c, int argc, char **argv) {"no-dhcpv6", no_argument, &c->no_dhcpv6, 1 }, {"no-ndp", no_argument, &c->no_ndp, 1 }, {"no-ra", no_argument, &c->no_ra, 1 }, + {"freebind", no_argument, &c->freebind, 1 }, {"no-map-gw", no_argument, &no_map_gw, 1 }, {"ipv4-only", no_argument, NULL, '4' }, {"ipv6-only", no_argument, NULL, '6' }, diff --git a/passt.1 b/passt.1 index 332384c..1386f29 100644 --- a/passt.1 +++ b/passt.1 @@ -327,6 +327,16 @@ namespace will be silently dropped. Disable Router Advertisements. Router Solicitations coming from guest or target namespace will be ignored. +.TP +.BR \-\-freebind +Allow any binding address to be specified for \fB-t\fR and \fB-u\fR +options. Usually binding addresses must be addresses currently +configured on the host. With \fB\-\-freebind\fR, the +\fBIP_FREEBIND\fR or \fBIPV6_FREEBIND\fR socket option is enabled +allowing any address to be used. This is typically used to bind +addresses which might be configured on the host in future, at which +point the forwarding will immediately start operating. + .TP .BR \-\-map-host-loopback " " \fIaddr Translate \fIaddr\fR to refer to the host. Packets from the guest to diff --git a/passt.h b/passt.h index f7b7a58..72c7f72 100644 --- a/passt.h +++ b/passt.h @@ -226,6 +226,7 @@ struct ip6_ctx { * @no_ndp: Disable NDP handler altogether * @no_ra: Disable router advertisements * @host_lo_to_ns_lo: Map host loopback addresses to ns loopback addresses + * @freebind: Allow binding of non-local addresses for forwarding * @low_wmem: Low probed net.core.wmem_max * @low_rmem: Low probed net.core.rmem_max */ @@ -286,6 +287,7 @@ struct ctx { int no_ndp; int no_ra; int host_lo_to_ns_lo; + int freebind; int low_wmem; int low_rmem; diff --git a/util.c b/util.c index ebd93ed..eba7d52 100644 --- a/util.c +++ b/util.c @@ -52,6 +52,7 @@ int sock_l4_sa(const struct ctx *c, enum epoll_type type, { sa_family_t af = ((const struct sockaddr *)sa)->sa_family; union epoll_ref ref = { .type = type, .data = data }; + bool freebind = false; struct epoll_event ev; int fd, y = 1, ret; uint8_t proto; @@ -61,8 +62,11 @@ int sock_l4_sa(const struct ctx *c, enum epoll_type type, case EPOLL_TYPE_TCP_LISTEN: proto = IPPROTO_TCP; socktype = SOCK_STREAM | SOCK_NONBLOCK; + freebind = c->freebind; break; case EPOLL_TYPE_UDP_LISTEN: + freebind = c->freebind; + /* fallthrough */ case EPOLL_TYPE_UDP_REPLY: proto = IPPROTO_UDP; socktype = SOCK_DGRAM | SOCK_NONBLOCK; @@ -127,6 +131,18 @@ int sock_l4_sa(const struct ctx *c, enum epoll_type type, } } + if (freebind) { + int level = af == AF_INET ? IPPROTO_IP : IPPROTO_IPV6; + int opt = af == AF_INET ? IP_FREEBIND : IPV6_FREEBIND; + + if (setsockopt(fd, level, opt, &y, sizeof(y))) { + err_perror("Failed to set %s on socket %i", + af == AF_INET ? "IP_FREEBIND" + : "IPV6_FREEBIND", + fd); + } + } + if (bind(fd, sa, sl) < 0) { /* We'll fail to bind to low ports if we don't have enough * capabilities, and we'll fail to bind on already bound ports, -- 2.46.2

1 0

[PATCH v3] conf: Add command line switch to enable IP_FREEBIND socket option
by David Gibson 03 Oct '24

03 Oct '24

In a couple of recent reports, we've seen that it can be useful for pasta to forward ports from addresses which are not currently configured on the host, but might be in future. That can be done with the sysctl net.ipv4.ip_nonlocal_bind, but that does require CAP_NET_ADMIN to set in the first place. We can allow the same thing on a per-socket basis with the IP_FREEBIND (or IPV6_FREEBIND) socket option. Add a --freebind command line argument to enable this socket option on all listening sockets. Link: https://bugs.passt.top/show_bug.cgi?id=101 Changes since v2: * Actually committed all the changes that were supposed to be in v2 :/ Changes since v1: * Reworded --help and man page additions for clarity * Added missing structure comment * Fixed some incorrect indentation Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- conf.c | 2 ++ passt.1 | 10 ++++++++++ passt.h | 2 ++ util.c | 16 ++++++++++++++++ 4 files changed, 30 insertions(+) diff --git a/conf.c b/conf.c index 6e62510..e360fb9 100644 --- a/conf.c +++ b/conf.c @@ -836,6 +836,7 @@ static void usage(const char *name, FILE *f, int status) " --no-ndp Disable NDP responses\n" " --no-dhcpv6 Disable DHCPv6 server\n" " --no-ra Disable router advertisements\n" + " --freebind Bind to any address for forwarding\n" " --no-map-gw Don't map gateway address to host\n" " -4, --ipv4-only Enable IPv4 operation only\n" " -6, --ipv6-only Enable IPv6 operation only\n"); @@ -1255,6 +1256,7 @@ void conf(struct ctx *c, int argc, char **argv) {"no-dhcpv6", no_argument, &c->no_dhcpv6, 1 }, {"no-ndp", no_argument, &c->no_ndp, 1 }, {"no-ra", no_argument, &c->no_ra, 1 }, + {"freebind", no_argument, &c->freebind, 1 }, {"no-map-gw", no_argument, &no_map_gw, 1 }, {"ipv4-only", no_argument, NULL, '4' }, {"ipv6-only", no_argument, NULL, '6' }, diff --git a/passt.1 b/passt.1 index 79d134d..5ac2962 100644 --- a/passt.1 +++ b/passt.1 @@ -327,6 +327,16 @@ namespace will be silently dropped. Disable Router Advertisements. Router Solicitations coming from guest or target namespace will be ignored. +.TP +.BR \-\-freebind +Allow any binding address to be specified for \fB-t\fR and \fB-u\fR +options. Usually binding addresses must be addresses currently +configured on the host. With \fB\-\-freebind\fR, the +\fBIP_FREEBIND\fR or \fBIPV6_FREEBIND\fR socket option is enabled +allowing any address to be used. This is typically used to bind +addresses which might be configured on the host in future, at which +point the forwarding will immediately start operating. + .TP .BR \-\-map-host-loopback " " \fIaddr Translate \fIaddr\fR to refer to the host. Packets from the guest to diff --git a/passt.h b/passt.h index 031c9b6..4908ed9 100644 --- a/passt.h +++ b/passt.h @@ -225,6 +225,7 @@ struct ip6_ctx { * @no_dhcpv6: Disable DHCPv6 server * @no_ndp: Disable NDP handler altogether * @no_ra: Disable router advertisements + * @freebind: Allow binding of non-local addresses for forwarding * @low_wmem: Low probed net.core.wmem_max * @low_rmem: Low probed net.core.rmem_max */ @@ -284,6 +285,7 @@ struct ctx { int no_dhcpv6; int no_ndp; int no_ra; + int freebind; int low_wmem; int low_rmem; diff --git a/util.c b/util.c index ebd93ed..eba7d52 100644 --- a/util.c +++ b/util.c @@ -52,6 +52,7 @@ int sock_l4_sa(const struct ctx *c, enum epoll_type type, { sa_family_t af = ((const struct sockaddr *)sa)->sa_family; union epoll_ref ref = { .type = type, .data = data }; + bool freebind = false; struct epoll_event ev; int fd, y = 1, ret; uint8_t proto; @@ -61,8 +62,11 @@ int sock_l4_sa(const struct ctx *c, enum epoll_type type, case EPOLL_TYPE_TCP_LISTEN: proto = IPPROTO_TCP; socktype = SOCK_STREAM | SOCK_NONBLOCK; + freebind = c->freebind; break; case EPOLL_TYPE_UDP_LISTEN: + freebind = c->freebind; + /* fallthrough */ case EPOLL_TYPE_UDP_REPLY: proto = IPPROTO_UDP; socktype = SOCK_DGRAM | SOCK_NONBLOCK; @@ -127,6 +131,18 @@ int sock_l4_sa(const struct ctx *c, enum epoll_type type, } } + if (freebind) { + int level = af == AF_INET ? IPPROTO_IP : IPPROTO_IPV6; + int opt = af == AF_INET ? IP_FREEBIND : IPV6_FREEBIND; + + if (setsockopt(fd, level, opt, &y, sizeof(y))) { + err_perror("Failed to set %s on socket %i", + af == AF_INET ? "IP_FREEBIND" + : "IPV6_FREEBIND", + fd); + } + } + if (bind(fd, sa, sl) < 0) { /* We'll fail to bind to low ports if we don't have enough * capabilities, and we'll fail to bind on already bound ports, -- 2.46.2

1 0

[PATCH v2] conf: Add command line switch to enable IP_FREEBIND socket option
by David Gibson 03 Oct '24

03 Oct '24

In a couple of recent reports, we've seen that it can be useful for pasta to forward ports from addresses which are not currently configured on the host, but might be in future. That can be done with the sysctl net.ipv4.ip_nonlocal_bind, but that does require CAP_NET_ADMIN to set in the first place. We can allow the same thing on a per-socket basis with the IP_FREEBIND (or IPV6_FREEBIND) socket option. Add a --freebind command line argument to enable this socket option on all listening sockets. Link: https://bugs.passt.top/show_bug.cgi?id=101 Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- conf.c | 2 ++ passt.1 | 10 ++++++++++ passt.h | 2 ++ util.c | 15 +++++++++++++++ 4 files changed, 29 insertions(+) Changes since v1: * Reworded --help and man page additions for clarity * Added missing structure comment * Fixed some incorrect indentation diff --git a/conf.c b/conf.c index 6e62510..e360fb9 100644 --- a/conf.c +++ b/conf.c @@ -836,6 +836,7 @@ static void usage(const char *name, FILE *f, int status) " --no-ndp Disable NDP responses\n" " --no-dhcpv6 Disable DHCPv6 server\n" " --no-ra Disable router advertisements\n" + " --freebind Bind to any address for forwarding\n" " --no-map-gw Don't map gateway address to host\n" " -4, --ipv4-only Enable IPv4 operation only\n" " -6, --ipv6-only Enable IPv6 operation only\n"); @@ -1255,6 +1256,7 @@ void conf(struct ctx *c, int argc, char **argv) {"no-dhcpv6", no_argument, &c->no_dhcpv6, 1 }, {"no-ndp", no_argument, &c->no_ndp, 1 }, {"no-ra", no_argument, &c->no_ra, 1 }, + {"freebind", no_argument, &c->freebind, 1 }, {"no-map-gw", no_argument, &no_map_gw, 1 }, {"ipv4-only", no_argument, NULL, '4' }, {"ipv6-only", no_argument, NULL, '6' }, diff --git a/passt.1 b/passt.1 index 79d134d..5ac2962 100644 --- a/passt.1 +++ b/passt.1 @@ -327,6 +327,16 @@ namespace will be silently dropped. Disable Router Advertisements. Router Solicitations coming from guest or target namespace will be ignored. +.TP +.BR \-\-freebind +Allow any binding address to be specified for \fB-t\fR and \fB-u\fR +options. Usually binding addresses must be addresses currently +configured on the host. With \fB\-\-freebind\fR, the +\fBIP_FREEBIND\fR or \fBIPV6_FREEBIND\fR socket option is enabled +allowing any address to be used. This is typically used to bind +addresses which might be configured on the host in future, at which +point the forwarding will immediately start operating. + .TP .BR \-\-map-host-loopback " " \fIaddr Translate \fIaddr\fR to refer to the host. Packets from the guest to diff --git a/passt.h b/passt.h index 031c9b6..4908ed9 100644 --- a/passt.h +++ b/passt.h @@ -225,6 +225,7 @@ struct ip6_ctx { * @no_dhcpv6: Disable DHCPv6 server * @no_ndp: Disable NDP handler altogether * @no_ra: Disable router advertisements + * @freebind: Allow binding of non-local addresses for forwarding * @low_wmem: Low probed net.core.wmem_max * @low_rmem: Low probed net.core.rmem_max */ @@ -284,6 +285,7 @@ struct ctx { int no_dhcpv6; int no_ndp; int no_ra; + int freebind; int low_wmem; int low_rmem; diff --git a/util.c b/util.c index ebd93ed..96e3de8 100644 --- a/util.c +++ b/util.c @@ -52,6 +52,7 @@ int sock_l4_sa(const struct ctx *c, enum epoll_type type, { sa_family_t af = ((const struct sockaddr *)sa)->sa_family; union epoll_ref ref = { .type = type, .data = data }; + bool freebind = false; struct epoll_event ev; int fd, y = 1, ret; uint8_t proto; @@ -61,8 +62,11 @@ int sock_l4_sa(const struct ctx *c, enum epoll_type type, case EPOLL_TYPE_TCP_LISTEN: proto = IPPROTO_TCP; socktype = SOCK_STREAM | SOCK_NONBLOCK; + freebind = c->freebind; break; case EPOLL_TYPE_UDP_LISTEN: + freebind = c->freebind; + /* fallthrough */ case EPOLL_TYPE_UDP_REPLY: proto = IPPROTO_UDP; socktype = SOCK_DGRAM | SOCK_NONBLOCK; @@ -127,6 +131,17 @@ int sock_l4_sa(const struct ctx *c, enum epoll_type type, } } + if (freebind) { + int level = af == AF_INET ? IPPROTO_IP : IPPROTO_IPV6; + int opt = af == AF_INET ? IP_FREEBIND : IPV6_FREEBIND; + + if (setsockopt(fd, level, opt, &y, sizeof(y))) { + err_perror("Failed to set %s on socket %i", + af == AF_INET ? "IP_FREEBIND" : "IPV6_FREEBIND", + fd); + } + } + if (bind(fd, sa, sl) < 0) { /* We'll fail to bind to low ports if we don't have enough * capabilities, and we'll fail to bind on already bound ports, -- 2.46.2

1 0

[PATCH] conf: Add command line switch to enable IP_FREEBIND socket option
by David Gibson 03 Oct '24

03 Oct '24

In a couple of recent reports, we've seen that it can be useful for pasta to forward ports from addresses which are not currently configured on the host, but might be in future. That can be done with the sysctl net.ipv4.ip_nonlocal_bind, but that does require CAP_NET_ADMIN to set in the first place. We can allow the same thing on a per-socket basis with the IP_FREEBIND (or IPV6_FREEBIND) socket option. Add a --freebind command line argument to enable this socket option on all listening sockets. Link: https://bugs.passt.top/show_bug.cgi?id=101 Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au> --- conf.c | 2 ++ passt.1 | 6 ++++++ passt.h | 1 + util.c | 15 +++++++++++++++ 4 files changed, 24 insertions(+) diff --git a/conf.c b/conf.c index 6e62510..84aa89d 100644 --- a/conf.c +++ b/conf.c @@ -836,6 +836,7 @@ static void usage(const char *name, FILE *f, int status) " --no-ndp Disable NDP responses\n" " --no-dhcpv6 Disable DHCPv6 server\n" " --no-ra Disable router advertisements\n" + " --freebind Allow forwarding from any address\n" " --no-map-gw Don't map gateway address to host\n" " -4, --ipv4-only Enable IPv4 operation only\n" " -6, --ipv6-only Enable IPv6 operation only\n"); @@ -1255,6 +1256,7 @@ void conf(struct ctx *c, int argc, char **argv) {"no-dhcpv6", no_argument, &c->no_dhcpv6, 1 }, {"no-ndp", no_argument, &c->no_ndp, 1 }, {"no-ra", no_argument, &c->no_ra, 1 }, + {"freebind", no_argument, &c->freebind, 1 }, {"no-map-gw", no_argument, &no_map_gw, 1 }, {"ipv4-only", no_argument, NULL, '4' }, {"ipv6-only", no_argument, NULL, '6' }, diff --git a/passt.1 b/passt.1 index 79d134d..a2547f8 100644 --- a/passt.1 +++ b/passt.1 @@ -327,6 +327,12 @@ namespace will be silently dropped. Disable Router Advertisements. Router Solicitations coming from guest or target namespace will be ignored. +.TP +.BR \-\-freebind +Allow forwarding from addresses which are not configured on the host, +but might be in future. This enables the \fBIP_FREEBIND\fR or +\fBIPB6_FREEBIND\fR option on listening sockets. + .TP .BR \-\-map-host-loopback " " \fIaddr Translate \fIaddr\fR to refer to the host. Packets from the guest to diff --git a/passt.h b/passt.h index 031c9b6..e00049e 100644 --- a/passt.h +++ b/passt.h @@ -284,6 +284,7 @@ struct ctx { int no_dhcpv6; int no_ndp; int no_ra; + int freebind; int low_wmem; int low_rmem; diff --git a/util.c b/util.c index ebd93ed..96e3de8 100644 --- a/util.c +++ b/util.c @@ -52,6 +52,7 @@ int sock_l4_sa(const struct ctx *c, enum epoll_type type, { sa_family_t af = ((const struct sockaddr *)sa)->sa_family; union epoll_ref ref = { .type = type, .data = data }; + bool freebind = false; struct epoll_event ev; int fd, y = 1, ret; uint8_t proto; @@ -61,8 +62,11 @@ int sock_l4_sa(const struct ctx *c, enum epoll_type type, case EPOLL_TYPE_TCP_LISTEN: proto = IPPROTO_TCP; socktype = SOCK_STREAM | SOCK_NONBLOCK; + freebind = c->freebind; break; case EPOLL_TYPE_UDP_LISTEN: + freebind = c->freebind; + /* fallthrough */ case EPOLL_TYPE_UDP_REPLY: proto = IPPROTO_UDP; socktype = SOCK_DGRAM | SOCK_NONBLOCK; @@ -127,6 +131,17 @@ int sock_l4_sa(const struct ctx *c, enum epoll_type type, } } + if (freebind) { + int level = af == AF_INET ? IPPROTO_IP : IPPROTO_IPV6; + int opt = af == AF_INET ? IP_FREEBIND : IPV6_FREEBIND; + + if (setsockopt(fd, level, opt, &y, sizeof(y))) { + err_perror("Failed to set %s on socket %i", + af == AF_INET ? "IP_FREEBIND" : "IPV6_FREEBIND", + fd); + } + } + if (bind(fd, sa, sl) < 0) { /* We'll fail to bind to low ports if we don't have enough * capabilities, and we'll fail to bind on already bound ports, -- 2.46.2

2 2

[PATCH v3 0/4] Don't expose container loopback services to the host
by David Gibson 02 Oct '24

02 Oct '24

podman issue #24045 pointed out that pasta's spliced forwarding logic can expose services within the namespace bound only to 127.0.0.1 or ::1 to the host. However, the namespace probably expects those to only be accessible to itself, so that's probably not what we want. This changes our forwarding logic so as not to do this. Note that the podman tests will currently fail with this series, I've submitted podman PR https://github.com/containers/podman/pull/24064 to fix that. Link: https://github.com/containers/podman/issues/24045 Changes since v2: * Add new field do structure comment Changes since v1: * Add --host-lo-to-ns-lo option to preserve the old behaviour * Clarify the new behaviour in the man page * Add some extra patches making some other corrections to the man page David Gibson (4): passt.1: Mark --stderr as deprecated more prominently passt.1: Clarify and update "Handling of local addresses" section test: Clarify test for spliced inbound transfers fwd: Direct inbound spliced forwards to the guest's external address conf.c | 9 ++++++ fwd.c | 31 +++++++++++++----- passt.1 | 75 +++++++++++++++++++++++++++----------------- passt.h | 2 ++ test/passt_in_ns/tcp | 8 ++--- test/passt_in_ns/udp | 4 +-- test/pasta/tcp | 16 +++++----- test/pasta/udp | 8 ++--- 8 files changed, 98 insertions(+), 55 deletions(-) -- 2.46.2

1 4

2024

2023

2022

2021

dev