On Wed, Oct 16, 2024 at 05:26:48PM +0200, Stefano Brivio wrote:On Wed, 16 Oct 2024 19:39:40 +1100 David Gibson <david(a)gibson.dropbear.id.au> wrote:Hrm... I guess. Counterpoints.. - Most other failures to get a usable address will result in a visible error - dhclient has a --dad-wait-time option which seems to imply that the script should wait for DAD - The upstream script version waits for DAD In any case I filed a report for it https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1085231On Wed, Oct 16, 2024 at 04:46:52PM +1100, David Gibson wrote:Oops. On one hand, I would feel inclined to propose a fix for the Debian and Ubuntu packages. On the other hand, I wonder if it's universally considered a bug: the DHCPv6 client did its job at that point, and it's debatable whether dhclient should wait for the address to be usable before forking to background. That is, arguably, the job of dhclient's is to request and configure an address. It's not a network configuration daemon. There might be many other reasons why that address is unusable, and yet dhclient is not responsible for them.On Wed, Oct 16, 2024 at 02:15:19PM +1100, David Gibson wrote:Ok.. I've understood a bit more. While timing is a factor here, it looks like the main reason I wasn't seeing it on my machine is what I'd consider a bug in the Debian version of the dhclient-script: when adding an IPv6 address, it returns without waiting for DAD to complete (i.e. for the address to be non-tentative).On Thu, Oct 10, 2024 at 04:57:32PM +1100, David Gibson wrote: > On Wed, Oct 09, 2024 at 10:44:33PM +0200, Stefano Brivio wrote: > > On Wed, 9 Oct 2024 15:07:21 +0200 > > Stefano Brivio <sbrivio(a)redhat.com> wrote: [snip] > > > > @@ -447,20 +447,35 @@ uint8_t fwd_nat_from_host(const struct ctx *c, uint8_t proto, > > > > (proto == IPPROTO_TCP || proto == IPPROTO_UDP)) { > > > > /* spliceable */ > > > > > > > > - /* Preserve the specific loopback adddress used, but let the > > > > - * kernel pick a source port on the target side > > > > + /* The traffic will go over the guest's 'lo' interface, but by > > > > + * default use its external address, so we don't inadvertently > > > > + * expose services that listen only on the guest's loopback > > > > + * address. That can be overridden by --host-lo-to-ns-lo which > > > > + * will instead forward to the loopback address in the guest. > > > > + * > > > > + * In either case, let the kernel pick the source address to > > > > + * match. > > > > */ > > > > - tgt->oaddr = ini->eaddr; > > > > + if (inany_v4(&ini->eaddr)) { > > > > + if (c->host_lo_to_ns_lo) > > > > + tgt->eaddr = inany_loopback4; > > > > + else > > > > + tgt->eaddr = inany_from_v4(c->ip4.addr_seen); > > > > + tgt->oaddr = inany_any4; > > > > + } else { > > > > + if (c->host_lo_to_ns_lo) > > > > + tgt->eaddr = inany_loopback6; > > > > + else > > > > + tgt->eaddr.a6 = c->ip6.addr_seen; > > > > > > Either this... > > > > > > > + tgt->oaddr = inany_any6; > > > > > > or this (and not something before this patch, up to 3/4) make the > > > "TCP/IPv6: host to ns (spliced): big transfer" test in pasta/tcp hang, > > > sometimes (about one in three/four runs), that's what I mistakenly > > > reported as coming from Laurent's series at: > > Huh, interesting. Just got back from my leave and ran that group of > tests in a loop this afternoon, but didn't manage to reproduce. I > have administrivia that will probably fill the rest of this week, but > I'll look into this as soon as I can. I reproduced the problem on passt.top, and I have a partial idea what's going on. As you say it's seeming like the address (addr_seen == addr in this case) isn't properly ready. This is over splice, but on the tap interface, I see the container sending NS messages for its own address - seems like it's doing DAD. But more importantly, we're answering those NS messages with NA messages, because we answer all NS. i.e. we're making the DAD fail. What I'm not sure of is how this ever worked at all. --config-net makes sense, since we disable DAD, but our test suite has always been using NDP+DHCP instead of --config-net. So, AFACT, we'll always fail guest DAD attempts, both IPv6, which happens most of the time and for IPv4 via ARP, which is used much more rarely. I think we need to be more selective in what NS or ARP lookups we resopnd to. The question is what approach to take:Hmm... no.. there's more to this. Usually DAD requests have :: as the source address, and we *do* exclude those from getting replies. In this case though, we're getting NS requests for the assigned address from what looks like the SLAAC address. So, I do think it would be wise to explicitly exclude these: we shouldn't be giving NA responses for an address that ought to belong to the guest, even if it doesn't look like a DAD. But, I'm not sure what's triggering this. Is for some reason the DHCP address not "taking", so the container is trying to locate it on the network instead? Or _is_ this DAD, but under some circumstances rather than using :: as the source address it uses another configured address.By the way, I guess it's just an issue for test scripts like this one.Why do you guess that?Here I think it's a much clearer argument that it's a real bug. We play fast and loose with it for mbuto, but dhclient can typically be called on an interface that isn't even up: the PREINIT/PREINIT6 script actions are specifically for this, they'll bring the interface up ready for the client to do its thing. I'd say the script is failing to do its job for PREINIT6 if there isn't a usable link-local address at the end. I filed a report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1085229There's also an additional bug, which doesn't cause this problem, I think, but caused some problems when I was investigating. DHCPv6 needs the link-local SLAAC address already configured and non-tentative. The Fedora dhclient-script waits for that too at the PREINIT6 stage, but the Debian one doesn't, meaning if you attempt dhclient -6 immediately after starting the namespace it will fail to bind the UDP address it needs.Right, and that's fine for us because we have a 2-second delay after SLAAC. This looks to me a bit more like a real bug, but again, there might be many other reasons why dhclient can't use a link-local address. One might argue that it's the responsibility of the user/caller to invoke dhclient when appropriate.In that sense, you might be wondering why there's a 2-second delay after SLAAC, but no delay after invoking dhclient -6: the reason is that I was convinced that one wouldn't need DAD once a DHCPv6 client configures an address. The server is already checking that, I thought. Well, no. RFC 8415 18.2.10.1: https://datatracker.ietf.org/doc/html/rfc8415#section-18.2.10.1 says: If the client can operate with the addresses and/or prefixes obtained from the server: [...] - The client MUST perform duplicate address detection as per Section 5.4 of [RFC4862], which does list some exceptions, on each of the received addresses in any IAs on which it has not performed duplicate address detection during processing of any of the previous Reply messages from the server. The client performs the duplicate address detection before using the received addresses for any traffic. If any of the addresses are found to be in use on the link, the client sends a Decline message to the server for those addresses as described in Section 18.2.8.Indeed.Hrm. I suppose. Fwiw we already make the equivalent exclusion for ARPI still think it's a good idea not to give NA messages for the guest assigned address, but we'll need a different workaround for this issue.I read the rest of your reasoning about it, but the nice thing of the current behaviour (and that's why I added that single check on the source address in ndp()) is that the guest can really use whatever address it wants, regardless of what we tried to configure, and we'll resolve any other address.If we receive a neighbour solicitation for the guest assigned address, and the source address is not unspecified, it means that the guest is _not_ using the assigned address, and it's actually trying to reach it.Ok. More complex, but faster, I guess. I'll try implementing this. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibsonI guess we'll have to manually wait for DAD to complete in the DHCP tests, which will be kind of mucky :/Alternatively, we could use the same trick as added by commit f4e9f26480ef ("pasta: Disable neighbour solicitations on device up to prevent DAD"): disable neighbour solicitations, run dhclient -6, set 'nodad' on the address, and re-enable neighbour solicitations. This works for me: