[PATCH passt v2 0/7] Add fuzzing
With this series, fuzzing actually works, albeit slowly. More on that below. Patches 1 & 2 are the same as before. Patch 3 is Stefano Brivio's modified patch (with some changes that we discussed together on IRC but otherwise unchanged). Patch 4 is new, but discussed already upstream: It changes the order in which EPOLLIN and EPOLLRDHUP events are processed, so that we don't drop packets when the socket is closed. Patches 5 & 6 are the hacks that were needed to get fuzzing to work. Patch 6 removes all seccomp and other isolation stuff because for unclear reasons that breaks AFL instrumentation. AFL appears to fork off a second process, and somehow strace cannot follow that process, but the second process fails, and that breaks AFL completely. Without strace data it's rather hard to see what's going on so I didn't investigate this further. Patch 7 adds the fuzzing wrapper and is not greatly changed from before. However I did have to disable the AFL "fork server" optimization which somehow doesn't work with passt (it does work fine with libnbd & nbdkit). Speed is not great. I'm getting ~ 75-80 execs/second. Really we want this to be much higher, since that ultimately governs how fast we can explore new code paths and find bugs. Ideally well over 1000 execs/s (per fuzzing process) would be a good target to aim for. (Of course this depends on the hardware as well.) We could try to find out why the fork server is not compatible with passt, but I don't think we'd gain very much performance there. To explore this I ran a dummy fuzzed process from the same wrapper, and it was hardly any faster. I think the real gains are going to come from reducing the overhead of starting passt. There seem to be some netlink messages sent during start up and maybe if those could be reduced or eliminated we might see better performance. The other factor is fuzzing stability, which hovers around 87-90%. While this isn't necessarily bad, I wonder where the non-determinism is coming from [ideal figures would be 95-100%]. Passt doesn't appear to use threads. It does call getrandom (for TCP sequence numbers) so it'd be good to have a way to bypass that. However I don't fully understand what's going on here. Rich.
If you run the build several times it will fail unnecessarily with:
ln -s passt pasta
ln: failed to create symbolic link 'pasta': File exists
make: *** [Makefile:134: pasta] Error 1
Signed-off-by: Richard W.M. Jones
On Thu, Nov 17, 2022 at 06:49:32PM +0000, Richard W.M. Jones wrote:
If you run the build several times it will fail unnecessarily with:
ln -s passt pasta ln: failed to create symbolic link 'pasta': File exists make: *** [Makefile:134: pasta] Error 1
Signed-off-by: Richard W.M. Jones
Reviewed-by: David Gibson
--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Makefile b/Makefile index 1dc2df5..f8ecaea 100644 --- a/Makefile +++ b/Makefile @@ -133,7 +133,7 @@ passt.avx2: $(PASST_SRCS) $(HEADERS) passt.avx2: passt
pasta.avx2 pasta.1 pasta: pasta%: passt% - ln -s $< $@ + ln -sf $< $@
qrap: $(QRAP_SRCS) passt.h $(CC) $(FLAGS) $(CFLAGS) $(CPPFLAGS) $(QRAP_SRCS) -o qrap $(LDFLAGS)
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
On Fri, 18 Nov 2022 12:30:25 +1100
David Gibson
On Thu, Nov 17, 2022 at 06:49:32PM +0000, Richard W.M. Jones wrote:
If you run the build several times it will fail unnecessarily with:
ln -s passt pasta ln: failed to create symbolic link 'pasta': File exists make: *** [Makefile:134: pasta] Error 1
Signed-off-by: Richard W.M. Jones
Reviewed-by: David Gibson
I'm not sure why I haven't hit that before.
I'm also a bit puzzled: after all, if 'pasta' exists, the 'pasta' target is up to date. Then I hit this while doing stuff on Debian packages, I don't remember what the exact sequence was. Well, in any case, it looks reasonable. -- Stefano
These files are left around by emacs amongst other editors.
Signed-off-by: Richard W.M. Jones
On Thu, Nov 17, 2022 at 06:49:33PM +0000, Richard W.M. Jones wrote:
These files are left around by emacs amongst other editors.
Signed-off-by: Richard W.M. Jones
Heh, I've been meaning to add that, but didn't get to it yet. Thanks!
Reviewed-by: David Gibson
--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Makefile b/Makefile index f8ecaea..ced7af2 100644 --- a/Makefile +++ b/Makefile @@ -146,7 +146,7 @@ valgrind: all
.PHONY: clean clean: - $(RM) $(BIN) *.o seccomp.h pasta.1 \ + $(RM) $(BIN) *~ *.o seccomp.h pasta.1 \ passt.tar passt.tar.gz *.deb *.rpm \ passt.pid README.plain.md
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
This passes a fully connected stream socket to passt.
Signed-off-by: Richard W.M. Jones
In the case where the client writes a packet and then closes the
socket, because we receive EPOLLIN|EPOLLRDHUP together we have a
choice of whether to close the socket immediately, or read the packet
and then close the socket. Choose the latter.
This should improve fuzzing coverage and arguably is a better choice
even for regular use since dropping packets on close is bad.
See-also: https://archives.passt.top/passt-dev/20221117171805.3746f53a@elisabeth/
Signed-off-by: Richard W.M. Jones
On Thu, Nov 17, 2022 at 06:49:35PM +0000, Richard W.M. Jones wrote:
In the case where the client writes a packet and then closes the socket, because we receive EPOLLIN|EPOLLRDHUP together we have a choice of whether to close the socket immediately, or read the packet and then close the socket. Choose the latter.
This should improve fuzzing coverage and arguably is a better choice even for regular use since dropping packets on close is bad.
See-also: https://archives.passt.top/passt-dev/20221117171805.3746f53a@elisabeth/ Signed-off-by: Richard W.M. Jones
Reviewed-by: David Gibson
--- tap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/tap.c b/tap.c index 9998127..d97af6a 100644 --- a/tap.c +++ b/tap.c @@ -1106,13 +1106,13 @@ void tap_handler(struct ctx *c, int fd, uint32_t events, return; }
- if (events & (EPOLLRDHUP | EPOLLHUP | EPOLLERR)) - goto reinit; - if ((c->mode == MODE_PASST && tap_handler_passt(c, now)) || (c->mode == MODE_PASTA && tap_handler_pasta(c, now))) goto reinit;
+ if (events & (EPOLLRDHUP | EPOLLHUP | EPOLLERR)) + goto reinit; + return; reinit: if (c->one_off) {
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
This is a hack. Ideally there'd be a way to build a "non-production" build of passt which would turn off all the encapsulation features. They are not relevant for fuzzing and simply add overhead. --- Makefile | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/Makefile b/Makefile index ced7af2..ee496fe 100644 --- a/Makefile +++ b/Makefile @@ -119,6 +119,19 @@ all: $(BIN) $(MANPAGES) docs static: FLAGS += -static -DGLIBC_NO_STATIC_NSS static: clean all +# XXX Hack for AFL instrumentation +EXTRA_SYSCALLS += \ + clone \ + getpid \ + gettid \ + madvise \ + mmap \ + mprotect \ + prctl \ + rt_sigprocmask \ + sched_yield \ + sigaltstack + seccomp.h: seccomp.sh $(PASST_SRCS) $(PASST_HEADERS) @ EXTRA_SYSCALLS="$(EXTRA_SYSCALLS)" ./seccomp.sh $(PASST_SRCS) $(PASST_HEADERS) -- 2.37.0.rc2
I suspect these interfere in some way with AFL instrumentation. The instrumented binary deadlocks without this patch, but it's hard to understand why just looking at strace output. --- conf.c | 2 +- passt.c | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/conf.c b/conf.c index 9ec346f..fcc5664 100644 --- a/conf.c +++ b/conf.c @@ -1655,7 +1655,7 @@ void conf(struct ctx *c, int argc, char **argv) usage(argv[0]); } - isolate_user(uid, gid, !netns_only, userns, c->mode); + //isolate_user(uid, gid, !netns_only, userns, c->mode); if (c->pasta_conf_ns) c->no_ra = 1; diff --git a/passt.c b/passt.c index 8b2c50d..2a4e65a 100644 --- a/passt.c +++ b/passt.c @@ -185,7 +185,7 @@ int main(int argc, char **argv) arch_avx2_exec(argv); - isolate_initial(); + //isolate_initial(); c.pasta_netns_fd = c.fd_tap = c.fd_tap_listen = -1; @@ -291,17 +291,19 @@ int main(int argc, char **argv) } } +#if 0 if (isolate_prefork(&c)) { err("Failed to sandbox process, exiting\n"); exit(EXIT_FAILURE); } +#endif if (!c.foreground) __daemon(pidfile_fd, devnull_fd); else write_pidfile(pidfile_fd, getpid()); - isolate_postfork(&c); + //isolate_postfork(&c); timer_init(&c, &now); -- 2.37.0.rc2
And adjust it so it can be used to fuzz passt. Follow the
instructions in README.fuzzing.md
Signed-off-by: Richard W.M. Jones
I forgot to mention that I didn't sort out test cases yet. The current test case (single zero length packet) is not ideal. We might have a single test case, say a single TCP SYN packet. However if there are distinct areas of functionality in passt (eg. TCP connect, ARP, DNS, DHCP, ...), *and* if those are geometrically very far apart in the search space, then you could argue for having one test case per major feature. Capturing that into tap files will be a fun afternoon for someone. More on this topic here: https://aflplus.plus/docs/fuzzing_in_depth/#2-preparing-the-fuzzing-campaign - - - Also ... while we do have --fd functionality, even better would be to modify passt so it can read input from stdin and drop output. See: https://aflplus.plus/docs/best_practices/#fuzzing-a-network-service The socketpair / --fd approach is a bit cleaner. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
I was wondering if --fd 0 would work, but it doesn't. This hangs: $ ../passt -1 -f -e --fd 0 < testcase_dir/empty_tap It's clear from strace why this is. Trying to add fd 0 to epoll returns -EPERM. The epoll_ctl(2) manual says: EPERM The target file fd does not support epoll. This error can occur if fd refers to, for example, a regular file or a directory. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Thu, 17 Nov 2022 18:49:31 +0000
"Richard W.M. Jones"
With this series, fuzzing actually works, albeit slowly. More on that below.
Patches 1 & 2 are the same as before.
Patch 3 is Stefano Brivio's modified patch (with some changes that we discussed together on IRC but otherwise unchanged).
Patch 4 is new, but discussed already upstream: It changes the order in which EPOLLIN and EPOLLRDHUP events are processed, so that we don't drop packets when the socket is closed.
Patches 5 & 6 are the hacks that were needed to get fuzzing to work. Patch 6 removes all seccomp and other isolation stuff because for unclear reasons that breaks AFL instrumentation. AFL appears to fork off a second process, and somehow strace cannot follow that process, but the second process fails, and that breaks AFL completely. Without strace data it's rather hard to see what's going on so I didn't investigate this further.
Patch 7 adds the fuzzing wrapper and is not greatly changed from before. However I did have to disable the AFL "fork server" optimization which somehow doesn't work with passt (it does work fine with libnbd & nbdkit).
Speed is not great. I'm getting ~ 75-80 execs/second. Really we want this to be much higher, since that ultimately governs how fast we can explore new code paths and find bugs. Ideally well over 1000 execs/s (per fuzzing process) would be a good target to aim for. (Of course this depends on the hardware as well.)
Applied up to 4/7, thanks! For the rest: I have a local branch with 5/7 and 6/7 fixed up: the 'fuzzing' Makefile target enables the syscalls you listed and avoids prctl(PR_SET_DUMPABLE, 0) in isolation_postfork() via -DFUZZING. I managed to speed things up by skipping some operations not needed for fuzzing, but just a tiny bit (~200 execs/s). I'm looking into switching to persistent mode: https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/READM... and introducing frames with special values, as you hinted on IRC, for example one-byte frames with commands such as "go ahead with socket processing then come back to 'tap' frames", so that passt has a chance to do some meaningful socket-side operations before getting back to fuzz input. Patch 7/7 is very useful and appreciated anyway as it demystifies the whole topic for me, and we can probably recycle most of the documentation. I'm not sure yet how/if the wrapper still fits with the stuff I'm looking into. -- Stefano
On Fri, Nov 25, 2022 at 10:23:54AM +0100, Stefano Brivio wrote:
and introducing frames with special values, as you hinted on IRC, for example one-byte frames with commands such as "go ahead with socket processing then come back to 'tap' frames", so that passt has a chance to do some meaningful socket-side operations before getting back to fuzz input.
You can improve the chance that the fuzzer will find these frames by either including them in test cases (we need better test cases, which is separate issue), or by making the encoding such that they are easy to find. eg. if you had four possible values, encode them only in the bottom two bits and ignore the higher bits. Since these frames are only used for fuzzing you can change the meaning of them later, so exact encoding isn't an ABI issue.
Patch 7/7 is very useful and appreciated anyway as it demystifies the whole topic for me, and we can probably recycle most of the documentation. I'm not sure yet how/if the wrapper still fits with the stuff I'm looking into.
It would definitely be better to have passt itself be able to read a file off disk. For example when we fuzz nbdkit we do not used or need a wrapper, because nbdkit has an -s / --single option that reads from stdin and writes to stdout. This was originally added to inetd support. We drive nbdkit from the fuzzer directly like this: afl-fuzz -i fuzzing/testcase_dir -o fuzzing/sync_dir -M fuzz01 \ ./server/nbdkit -s -t 1 ./plugins/memory/.libs/nbdkit-memory-plugin.so 1M (https://gitlab.com/nbdkit/nbdkit/-/blob/ef035f7090d8bec2700ef1f941e371d351d6...) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Fri, 25 Nov 2022 10:11:03 +0000
"Richard W.M. Jones"
On Fri, Nov 25, 2022 at 10:23:54AM +0100, Stefano Brivio wrote:
and introducing frames with special values, as you hinted on IRC, for example one-byte frames with commands such as "go ahead with socket processing then come back to 'tap' frames", so that passt has a chance to do some meaningful socket-side operations before getting back to fuzz input.
You can improve the chance that the fuzzer will find these frames by either including them in test cases (we need better test cases, which is separate issue), or by making the encoding such that they are easy to find. eg. if you had four possible values, encode them only in the bottom two bits and ignore the higher bits. Since these frames are only used for fuzzing you can change the meaning of them later, so exact encoding isn't an ABI issue.
Right, yes, I was thinking we could, under #ifdef FUZZING, accept frames that are shorter than a 802.3 header (up to 13 bytes), and take their length (in network order, but I guess AFL++ can easily get familiar with it) as fuzzing flow commands. About test cases, I'm not sure this should be included in regular test runs, because there's no reasonable definition for a test duration. I'd rather have a separate script which keeps running indefinitely, updating sources as they become available.
Patch 7/7 is very useful and appreciated anyway as it demystifies the whole topic for me, and we can probably recycle most of the documentation. I'm not sure yet how/if the wrapper still fits with the stuff I'm looking into.
It would definitely be better to have passt itself be able to read a file off disk.
For example when we fuzz nbdkit we do not used or need a wrapper, because nbdkit has an -s / --single option that reads from stdin and writes to stdout. This was originally added to inetd support.
We drive nbdkit from the fuzzer directly like this:
afl-fuzz -i fuzzing/testcase_dir -o fuzzing/sync_dir -M fuzz01 \ ./server/nbdkit -s -t 1 ./plugins/memory/.libs/nbdkit-memory-plugin.so 1M
(https://gitlab.com/nbdkit/nbdkit/-/blob/ef035f7090d8bec2700ef1f941e371d351d6...)
Ah, I didn't know, interesting. On the other hand, would it really make sense to add support for that (which has probably no use other than fuzzing) instead of a hack with __AFL_FUZZ_TESTCASE_BUF and mmap()? According to AFL++ documentation that should speed things up considerably: https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/READM... -- Stefano
On Tue, Nov 29, 2022 at 02:34:42PM +0100, Stefano Brivio wrote:
On Fri, 25 Nov 2022 10:11:03 +0000 "Richard W.M. Jones"
wrote: On Fri, Nov 25, 2022 at 10:23:54AM +0100, Stefano Brivio wrote:
and introducing frames with special values, as you hinted on IRC, for example one-byte frames with commands such as "go ahead with socket processing then come back to 'tap' frames", so that passt has a chance to do some meaningful socket-side operations before getting back to fuzz input.
You can improve the chance that the fuzzer will find these frames by either including them in test cases (we need better test cases, which is separate issue), or by making the encoding such that they are easy to find. eg. if you had four possible values, encode them only in the bottom two bits and ignore the higher bits. Since these frames are only used for fuzzing you can change the meaning of them later, so exact encoding isn't an ABI issue.
Right, yes, I was thinking we could, under #ifdef FUZZING, accept frames that are shorter than a 802.3 header (up to 13 bytes), and take their length (in network order, but I guess AFL++ can easily get familiar with it) as fuzzing flow commands.
About test cases, I'm not sure this should be included in regular test runs, because there's no reasonable definition for a test duration. I'd rather have a separate script which keeps running indefinitely, updating sources as they become available.
Yes for libnbd/nbdkit/hivex we don't (can't) fuzz as part of regular tests. It's a separate process. You might be able to apply to this programme: https://google.github.io/oss-fuzz/ (You'll still need to add the fuzzing support upstream.)
Patch 7/7 is very useful and appreciated anyway as it demystifies the whole topic for me, and we can probably recycle most of the documentation. I'm not sure yet how/if the wrapper still fits with the stuff I'm looking into.
It would definitely be better to have passt itself be able to read a file off disk.
For example when we fuzz nbdkit we do not used or need a wrapper, because nbdkit has an -s / --single option that reads from stdin and writes to stdout. This was originally added to inetd support.
We drive nbdkit from the fuzzer directly like this:
afl-fuzz -i fuzzing/testcase_dir -o fuzzing/sync_dir -M fuzz01 \ ./server/nbdkit -s -t 1 ./plugins/memory/.libs/nbdkit-memory-plugin.so 1M
(https://gitlab.com/nbdkit/nbdkit/-/blob/ef035f7090d8bec2700ef1f941e371d351d6...)
Ah, I didn't know, interesting. On the other hand, would it really make sense to add support for that (which has probably no use other than fuzzing) instead of a hack with __AFL_FUZZ_TESTCASE_BUF and mmap()?
According to AFL++ documentation that should speed things up considerably: https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/READM...
Yes if you can get __AFL_FUZZ_TESTCASE_BUF working that would be even better. For nbdkit it seemed like it would be quite difficult because we would need to ensure that the binary can be completely "reset" between runs without being re-exec'd, so we'd need to chase down every possible allocation / initialization and make sure it's reversed. We get pretty good fuzzing performance by the method above so it was never a priority. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Tue, Nov 29, 2022 at 02:34:42PM +0100, Stefano Brivio wrote:
On Fri, 25 Nov 2022 10:11:03 +0000 "Richard W.M. Jones"
wrote: On Fri, Nov 25, 2022 at 10:23:54AM +0100, Stefano Brivio wrote:
and introducing frames with special values, as you hinted on IRC, for example one-byte frames with commands such as "go ahead with socket processing then come back to 'tap' frames", so that passt has a chance to do some meaningful socket-side operations before getting back to fuzz input.
You can improve the chance that the fuzzer will find these frames by either including them in test cases (we need better test cases, which is separate issue), or by making the encoding such that they are easy to find. eg. if you had four possible values, encode them only in the bottom two bits and ignore the higher bits. Since these frames are only used for fuzzing you can change the meaning of them later, so exact encoding isn't an ABI issue.
Right, yes, I was thinking we could, under #ifdef FUZZING, accept frames that are shorter than a 802.3 header (up to 13 bytes), and take their length (in network order, but I guess AFL++ can easily get familiar with it) as fuzzing flow commands.
About test cases, I'm not sure this should be included in regular test runs, because there's no reasonable definition for a test duration. I'd rather have a separate script which keeps running indefinitely, updating sources as they become available.
I agree we don't want to do fuzzing per se as part of the regular test. However, it would be nice to replay specific fuzz-generated scenarios where we've previously found bugs, to avoid regression. I'm not really sure what would be involved in that.
Patch 7/7 is very useful and appreciated anyway as it demystifies the whole topic for me, and we can probably recycle most of the documentation. I'm not sure yet how/if the wrapper still fits with the stuff I'm looking into.
It would definitely be better to have passt itself be able to read a file off disk.
For example when we fuzz nbdkit we do not used or need a wrapper, because nbdkit has an -s / --single option that reads from stdin and writes to stdout. This was originally added to inetd support.
We drive nbdkit from the fuzzer directly like this:
afl-fuzz -i fuzzing/testcase_dir -o fuzzing/sync_dir -M fuzz01 \ ./server/nbdkit -s -t 1 ./plugins/memory/.libs/nbdkit-memory-plugin.so 1M
(https://gitlab.com/nbdkit/nbdkit/-/blob/ef035f7090d8bec2700ef1f941e371d351d6...)
Ah, I didn't know, interesting. On the other hand, would it really make sense to add support for that (which has probably no use other than fuzzing) instead of a hack with __AFL_FUZZ_TESTCASE_BUF and mmap()?
According to AFL++ documentation that should speed things up considerably: https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/READM...
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
participants (3)
-
David Gibson
-
Richard W.M. Jones
-
Stefano Brivio