As discussed on calls, I've been investigating Avocado as a possible
replacement for our hand-rolled shell test framework. This is a
second draft proof-of-concept conversion of a few of the tests to the
Avocado framework. Specifically, the static checkers, the build tests
and the pasta transfer tests are converted.
In this second version, I've done a lot of rework (maybe too much) on
the helper library. It's got a lot more capabilities and is more
consistent.
The new tests will get failures if you don't also apply the fix I send
recently to prevent updating ip6.addr_seen to ::.
Promising things:
* I remain happy with the speed improvement (under half the time of
the equivalent old tests)
* The fact that it's no longer dependent on the host network is great
* I'm cautiously optimistic about the helper library making it
relatively easy to add a bunch more tests with further coverage
Concerning things:
* I'm coming to hate OO and jUnit patterns more and more, these can
make it pretty non-obvious what each test relies on in terms of setup
Changes since RFCv1:
* Sweeping rework of the test library
* Started using tags to split between the "real" tests (run with
"make avocado") and "meta" tests (run with "make
avocado-meta").
The latter are tests for the test infrastructure and helpers
themselves, rather than testing anything in passt or pasta
David Gibson (21):
test/nstool: Provide useful error if given a path that's too long
test/nstool: Fix fd leak in accept() loop
avocado: Make a duplicate copy of testsuite for comparison purposes
avocado: Don't double download assets for test/ and oldtest/
avocado: Move static checkers to avocado
avocado/tasst: Helper functions for executing commands in different
places
avocado: Convert build tests to avocado
avocado/tasst: Add helpers for running background commands on sites
avocado/tasst: Add helper to get network interface names for a site
avocado/tasst: Add helpers to run commands with nstool
avocado/tasst: Add ifup and network address helpers to Site
avocado/tasst: Helper for creating veth devices between namespaces
avocado/tasst: Add helper for getting MTU of a network interface
avocado/tasst: Add helper to wait for IP address to appear
avocado/tasst: Add helpers for getting a site's routes
avocado/tasst: Helpers for test transferring data between sites
avocado/tasst: IP address allocation helpers
avocado/tasst: Helpers for testing NDP behaviour
avocado/tasst: Helpers for testing DHCP & DHCPv6 behaviour
avocado/tasst: Helpers to construct a simple network environment for
tests
avocado: Convert basic pasta tests
Makefile | 19 +
avocado/.gitignore | 1 +
avocado/build.py | 75 +++
avocado/pasta.py | 183 ++++++
avocado/static_checkers/clang-tidy.sh | 3 +
avocado/static_checkers/cppcheck.sh | 3 +
avocado/tasst/__init__.py | 32 +
avocado/tasst/address.py | 89 +++
avocado/tasst/dhcp.py | 133 +++++
avocado/tasst/dhcpv6.py | 123 ++++
avocado/tasst/meta/__init__.py | 16 +
avocado/tasst/meta/static_ifup.py | 60 ++
avocado/tasst/meta/veth.py | 95 +++
avocado/tasst/ndp.py | 129 ++++
avocado/tasst/nstool.py | 214 +++++++
avocado/tasst/scenario/__init__.py | 11 +
avocado/tasst/scenario/simple.py | 96 +++
avocado/tasst/site.py | 207 +++++++
avocado/tasst/transfer.py | 228 +++++++
oldtest/.gitignore | 11 +
oldtest/Makefile | 119 ++++
oldtest/README.md | 137 +++++
{test => oldtest}/build/all | 0
{test => oldtest}/build/clang_tidy | 0
{test => oldtest}/build/cppcheck | 0
oldtest/ci | 1 +
oldtest/demo/passt | 245 ++++++++
oldtest/demo/pasta | 274 +++++++++
oldtest/demo/podman | 819 ++++++++++++++++++++++++++
oldtest/distro/debian | 252 ++++++++
oldtest/distro/fedora | 396 +++++++++++++
oldtest/distro/opensuse | 208 +++++++
oldtest/distro/ubuntu | 216 +++++++
oldtest/env/mate-terminal.profile | 42 ++
oldtest/find-arm64-firmware.sh | 13 +
oldtest/lib/context | 130 ++++
oldtest/lib/layout | 259 ++++++++
oldtest/lib/layout_ugly | 113 ++++
oldtest/lib/perf_report | 272 +++++++++
oldtest/lib/setup | 385 ++++++++++++
oldtest/lib/setup_ugly | 58 ++
oldtest/lib/term | 750 +++++++++++++++++++++++
oldtest/lib/test | 398 +++++++++++++
oldtest/lib/util | 133 +++++
oldtest/lib/video | 152 +++++
oldtest/memory/passt | 187 ++++++
oldtest/nstool.c | 565 ++++++++++++++++++
oldtest/passt.mbuto | 83 +++
oldtest/passt.mem.mbuto | 44 ++
oldtest/passt/dhcp | 70 +++
oldtest/passt/ndp | 33 ++
oldtest/passt/shutdown | 19 +
oldtest/passt/tcp | 76 +++
oldtest/passt/udp | 46 ++
oldtest/passt_in_ns/icmp | 32 +
oldtest/passt_in_ns/shutdown | 19 +
oldtest/passt_in_ns/tcp | 256 ++++++++
oldtest/passt_in_ns/udp | 138 +++++
{test => oldtest}/pasta/dhcp | 0
{test => oldtest}/pasta/ndp | 0
{test => oldtest}/pasta/tcp | 0
{test => oldtest}/pasta/udp | 0
oldtest/pasta_options/log_to_file | 93 +++
oldtest/perf/passt_tcp | 215 +++++++
oldtest/perf/passt_udp | 165 ++++++
oldtest/perf/pasta_tcp | 300 ++++++++++
oldtest/perf/pasta_udp | 219 +++++++
oldtest/prepare-distro-img.sh | 18 +
oldtest/run | 238 ++++++++
oldtest/run_demo | 1 +
oldtest/two_guests/basic | 80 +++
oldtest/valgrind.supp | 9 +
test/lib/layout | 31 -
test/lib/setup | 47 --
test/nstool.c | 24 +-
test/run | 14 -
76 files changed, 10022 insertions(+), 100 deletions(-)
create mode 100644 avocado/.gitignore
create mode 100644 avocado/build.py
create mode 100644 avocado/pasta.py
create mode 100755 avocado/static_checkers/clang-tidy.sh
create mode 100755 avocado/static_checkers/cppcheck.sh
create mode 100644 avocado/tasst/__init__.py
create mode 100644 avocado/tasst/address.py
create mode 100644 avocado/tasst/dhcp.py
create mode 100644 avocado/tasst/dhcpv6.py
create mode 100644 avocado/tasst/meta/__init__.py
create mode 100644 avocado/tasst/meta/static_ifup.py
create mode 100644 avocado/tasst/meta/veth.py
create mode 100644 avocado/tasst/ndp.py
create mode 100644 avocado/tasst/nstool.py
create mode 100644 avocado/tasst/scenario/__init__.py
create mode 100644 avocado/tasst/scenario/simple.py
create mode 100644 avocado/tasst/site.py
create mode 100644 avocado/tasst/transfer.py
create mode 100644 oldtest/.gitignore
create mode 100644 oldtest/Makefile
create mode 100644 oldtest/README.md
rename {test => oldtest}/build/all (100%)
rename {test => oldtest}/build/clang_tidy (100%)
rename {test => oldtest}/build/cppcheck (100%)
create mode 120000 oldtest/ci
create mode 100644 oldtest/demo/passt
create mode 100644 oldtest/demo/pasta
create mode 100644 oldtest/demo/podman
create mode 100644 oldtest/distro/debian
create mode 100644 oldtest/distro/fedora
create mode 100644 oldtest/distro/opensuse
create mode 100644 oldtest/distro/ubuntu
create mode 100644 oldtest/env/mate-terminal.profile
create mode 100755 oldtest/find-arm64-firmware.sh
create mode 100644 oldtest/lib/context
create mode 100644 oldtest/lib/layout
create mode 100644 oldtest/lib/layout_ugly
create mode 100755 oldtest/lib/perf_report
create mode 100755 oldtest/lib/setup
create mode 100755 oldtest/lib/setup_ugly
create mode 100755 oldtest/lib/term
create mode 100755 oldtest/lib/test
create mode 100755 oldtest/lib/util
create mode 100755 oldtest/lib/video
create mode 100644 oldtest/memory/passt
create mode 100644 oldtest/nstool.c
create mode 100755 oldtest/passt.mbuto
create mode 100755 oldtest/passt.mem.mbuto
create mode 100644 oldtest/passt/dhcp
create mode 100644 oldtest/passt/ndp
create mode 100644 oldtest/passt/shutdown
create mode 100644 oldtest/passt/tcp
create mode 100644 oldtest/passt/udp
create mode 100644 oldtest/passt_in_ns/icmp
create mode 100644 oldtest/passt_in_ns/shutdown
create mode 100644 oldtest/passt_in_ns/tcp
create mode 100644 oldtest/passt_in_ns/udp
rename {test => oldtest}/pasta/dhcp (100%)
rename {test => oldtest}/pasta/ndp (100%)
rename {test => oldtest}/pasta/tcp (100%)
rename {test => oldtest}/pasta/udp (100%)
create mode 100644 oldtest/pasta_options/log_to_file
create mode 100644 oldtest/perf/passt_tcp
create mode 100644 oldtest/perf/passt_udp
create mode 100644 oldtest/perf/pasta_tcp
create mode 100644 oldtest/perf/pasta_udp
create mode 100755 oldtest/prepare-distro-img.sh
create mode 100755 oldtest/run
create mode 120000 oldtest/run_demo
create mode 100644 oldtest/two_guests/basic
create mode 100644 oldtest/valgrind.supp
--
2.40.1