Re: qemu couldn't connect the unix domain socket

On Fri, Oct 29, 2021 at 3:44 PM Stefano Brivio wrote:

Hi Feng Li,

On Fri, 29 Oct 2021 13:27:44 +0800 Li Feng wrote:

...

Hi Stefano,

I got the coredump file, it reports the `fork` syscall is bad:

Program terminated with signal SIGSYS, Bad system call. #0 __GI__Fork () at ../sysdeps/nptl/_Fork.c:50 50 return pid; (gdb) bt #0 __GI__Fork () at ../sysdeps/nptl/_Fork.c:50 #1 0x00007f8c04fdc02a in __libc_fork () at fork.c:73 #2 0x00007f8c05009f8b in daemon (nochdir=0, noclose=0) at daemon.c:48 #3 0x000000000040c1e9 in main (argc=1, argv=0x7ffd10b5cd78) at passt.c:368 quit)

That's not necessarily because of fork() -- fork() is already in the list of allowed syscalls. The signal is asynchronous, it might be received a bit before or after passt is executing what you see in gdb.

This is probably another syscall triggered by daemon() in the specific glibc version (2.34-7.fc35) on your system -- I haven't tested Fedora 35 yet. An easy way to find out which one is the syscall causing this is using strace.

For example, suppose I forgot to add listen() to the list of allowed syscalls:

diff --git a/passt.c b/passt.c index 6436a45..43249cf 100644 --- a/passt.c +++ b/passt.c @@ -277,3 +277,3 @@ static void pid_file(struct ctx *c) { * #syscalls read write open close fork dup2 exit chdir ioctl writev syslog - * #syscalls prlimit64 epoll_ctl epoll_create1 epoll_wait accept4 accept listen + * #syscalls prlimit64 epoll_ctl epoll_create1 epoll_wait accept4 accept * #syscalls socket bind connect getsockopt setsockopt recvfrom sendto shutdown

Then:

$ strace ./passt [...] setsockopt(6, SOL_SOCKET, SO_SNDBUF, [1073741823], 4) = 0 getsockopt(6, SOL_SOCKET, SO_SNDBUF, [268435456], [4]) = 0 setsockopt(6, SOL_SOCKET, SO_RCVBUF, [1073741823], 4) = 0 getsockopt(6, SOL_SOCKET, SO_RCVBUF, [268435456], [4]) = 0 close(6) = 0 socket(AF_UNIX, SOCK_STREAM, 0) = 6 socket(AF_UNIX, SOCK_STREAM|SOCK_NONBLOCK, 0) = 7 connect(7, {sa_family=AF_UNIX, sun_path="/tmp/passt_1.socket"}, 110) = -1 ENOENT (No such file or directory) close(7) = 0 unlink("/tmp/passt_1.socket") = -1 ENOENT (No such file or directory) bind(6, {sa_family=AF_UNIX, sun_path="/tmp/passt_1.socket"}, 110) = 0 write(2, "UNIX domain socket bound at /tmp"..., 48UNIX domain socket bound at /tmp/passt_1.socket ) = 48 write(2, "\n", 1 ) = 1 listen(6, 0) = ? +++ killed by SIGSYS +++ Bad system call

you would see that listen() is the first syscall not returning here (strace can't see a return from there).

It's around daemon(), and the process might have forked already, so you should run strace with the -f option, which also traces child processes:

strace -f ./passt

the missing syscall should now be obvious from the output.

Thanks for the detailed explanation. I finally found out that the `qrap` was the root cause. I patched the qemu, and it works well. In VM，I got the ip 192.168.64.217, which is the same to host. This is in VM: root@192.168.64.217 08:42:35 /tmp $ ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 65520 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff altname enp0s2 inet 192.168.64.217/20 brd 192.168.79.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever inet6 fe80::9fa9:7232:8d10:96be/64 scope link noprefixroute valid_lft forever preferred_lft forever I have tested the ping and curl, it works, amazing!

...

Looks like the seccomp is still badly configured. I have little knowledge about the seccomp.

Short summary: this is seccomp in filter mode (seccomp-bpf), it's a mechanism to block the system call (terminating the process, here) in case it's a syscall we didn't expect to be executed.

It's a security feature: that's to avoid that an attacker, who already gained some control on the process execution, is able to potentially exploit a further vulnerability (e.g. in the kernel) by executing a particular syscall. This is a relatively famous example of it:

https://reverse.put.as/2017/11/07/exploiting-cve-2017-5123/

passt implements this as a list of syscalls in code comments, those are translated by seccomp.sh into a BPF program, which is then loaded by seccomp() in passt.c. If a syscall not included in the resulting list is triggered, the kernel will terminate the process with a SYGSYS signal.

However, different C libraries (on different architectures) might issue different syscalls to implement the same function (daemon(), here), and the list I made was just tested on the systems I use and the reports of a few other users, so some are surely missing right now.

While adding tests for OpenSUSE and Debian, I already found a few alternative syscalls for some functions (I'll prepare a patch soon) -- I haven't started with Fedora 35 tests yet.

This background knowledge helped me a lot. The seccomp works well with Fedora 35 without the `qrap`. Thanks again for your great work.

-- Stefano