On Thu, 30 Jan 2025 09:32:36 +0100 Stefano Brivio <sbrivio(a)redhat.com> wrote:I would like to quickly complete the whole flow first, because I think we can inform design and implementation decisions much better at that pointSo, there seems to be a problem with (testing?) this. I couldn't quite understand the root cause yet, and it doesn't happen with the reference source.c and target.c implementations I shared. Let's assume I have a connection in the source guest to 127.0.0.1:9091, from 127.0.0.1:56350. After the migration, in the target, I get: --- socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 79 setsockopt(79, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 bind(79, {sa_family=AF_INET, sin_port=htons(56350), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 sendmsg(72, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1", iov_len=1}], msg_iovlen=1, msg_control=[{cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, cmsg_data=[79]}], msg_controllen=24, msg_flags=0}, 0) = 1 recvfrom(72, "\1", 1, 0, NULL, NULL) = 1 setsockopt(79, SOL_TCP, TCP_REPAIR_QUEUE, [2], 4) = 0 setsockopt(79, SOL_TCP, TCP_QUEUE_SEQ, [1788468535], 4) = 0 write(2, "77.6923: ", 977.6923: ) = 9 write(2, "Set send queue sequence for sock"..., 51Set send queue sequence for socket 79 to 1788468535) = 51 write(2, "\n", 1 ) = 1 setsockopt(79, SOL_TCP, TCP_REPAIR_QUEUE, [1], 4) = 0 setsockopt(79, SOL_TCP, TCP_QUEUE_SEQ, [115288604], 4) = 0 write(2, "77.6924: ", 977.6924: ) = 9 write(2, "Set receive queue sequence for s"..., 53Set receive queue sequence for socket 79 to 115288604) = 53 write(2, "\n", 1 ) = 1 connect(79, {sa_family=AF_INET, sin_port=htons(9091), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EADDRNOTAVAIL (Cannot assign requested address) --- EADDRNOTAVAIL, according to the documentation, which seems to be consistent with a glance at the implementation (that is, I must be missing some issue in the kernel), should be returned on connect() if: EADDRNOTAVAIL (Internet domain sockets) The socket referred to by sockfd had not previously been bound to an address and, upon attempting to bind it to an ephemeral port, it was determined that all port numbers in the ephemeral port range are currently in use. See the discussion of /proc/sys/net/ipv4/ip_local_port_range in ip(7). but well, of course it was bound. To a port, indeed, not a full address, that is, any (0.0.0.0) and address port, but I think for the purposes of this description that bind() call is enough. Is this related to SO_REUSEADDR? I need it (on both source and target) because, at least in my tests, source and target are on the same machine, in the same namespace. If I drop it: --- bind(79, {sa_family=AF_INET, sin_port=htons(46280), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use) --- as expected. However, in my reference implementation, with a connection from 127.0.0.1:9998 to 127.0.0.1:9091, this is what the target does: --- socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 bind(3, {sa_family=AF_INET, sin_port=htons(9998), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 socket(AF_UNIX, SOCK_STREAM, 0) = 4 unlink("/tmp/repair.sock") = 0 bind(4, {sa_family=AF_UNIX, sun_path="/tmp/repair.sock"}, 110) = 0 listen(4, 1) = 0 accept(4, NULL, NULL) = 5 sendmsg(5, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1", iov_len=1}], msg_iovlen=1, msg_control=[{cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, cmsg_data=[3]}], msg_controllen=24, msg_flags=0}, 0) = 1 recvfrom(5, "\1", 1, 0, NULL, NULL) = 1 setsockopt(3, SOL_TCP, TCP_REPAIR_QUEUE, [2], 4) = 0 setsockopt(3, SOL_TCP, TCP_QUEUE_SEQ, [1612504019], 4) = 0 setsockopt(3, SOL_TCP, TCP_REPAIR_QUEUE, [1], 4) = 0 setsockopt(3, SOL_TCP, TCP_QUEUE_SEQ, [1756508956], 4) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(9091), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 --- The only obvious difference is that, here, I'm not binding to an ephemeral port: the source port (in both source and target "guests") is 9998. Fine, so I tried forcing a lower port in passt (source) as well, and this is what I get in the target now: --- socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 79 setsockopt(79, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 bind(79, {sa_family=AF_INET, sin_port=htons(9000), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 sendmsg(72, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\1", iov_len=1}], msg_iovlen=1, msg_control=[{cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, cmsg_data=[79]}], msg_controllen=24, msg_flags=0}, 0) = 1 recvfrom(72, "\1", 1, 0, NULL, NULL) = 1 setsockopt(79, SOL_TCP, TCP_REPAIR_QUEUE, [2], 4) = 0 setsockopt(79, SOL_TCP, TCP_QUEUE_SEQ, [-348109334], 4) = 0 write(2, "46.9751: ", 946.9751: ) = 9 write(2, "Set send queue sequence for sock"..., 51Set send queue sequence for socket 79 to 3946857962) = 51 write(2, "\n", 1 ) = 1 setsockopt(79, SOL_TCP, TCP_REPAIR_QUEUE, [1], 4) = 0 setsockopt(79, SOL_TCP, TCP_QUEUE_SEQ, [-1820322671], 4) = 0 write(2, "46.9752: ", 946.9752: ) = 9 write(2, "Set receive queue sequence for s"..., 54Set receive queue sequence for socket 79 to 2474644625) = 54 write(2, "\n", 1 ) = 1 connect(79, {sa_family=AF_INET, sin_port=htons(9091), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EADDRNOTAVAIL (Cannot assign requested address) --- no obvious difference. I'll try binding to an explicit address, next, but I have no idea why 1. we get EADDRNOTAVAIL after a bind() and 2. it works with the reference implementation. Yes, I explicitly close() the socket in the source passt now, but that doesn't change things. This is presumably just an issue with testing, because in real use cases source and target guests would be on different machines. Another idea could be separating the namespaces. I can't just run source and target passt in two instances of pasta --config-net, because pasta would run into the same issue, but I could isolate one namespace with it, then add two network namespaces inside that, and connect them with veth pairs. -- Stefano