As discussed on calls, I've been investigating Avocado as a possible
replacement for our hand-rolled shell test framework. This is a third
draft proof-of-concept conversion of a few of the tests to the Avocado
framework. Specifically, the static checkers, the build tests and the
pasta transfer tests are converted.
This third version completely avoids the avocado/unittest built in
setUp() and tearDown() functions. Instead it uses Python's with
statement and context managers. This is more Pythonic, and avoids
some really cryptic multiple inheritance confusion when we want to mix
and match tests with different environment setups.
Promising things:
* I remain happy with the speed improvement (under half the time of
the equivalent old tests)
* The fact that it's no longer dependent on the host network is great
* I'm cautiously optimistic about the helper library making it
relatively easy to add a bunch more tests with further coverage
Concerning things:
* Using context managers instead of setUp/tearDown significantly
mitigates it, but the OO/jUnit patterns still make things uglier.
Changes since RFCv2:
* Remove all setUp() and tearDown() implementations
* Remove the weird 'subsetup' stuff we were using along with those
Changes since RFCv1:
* Sweeping rework of the test library
* Started using tags to split between the "real" tests (run with
"make avocado") and "meta" tests (run with "make
avocado-meta").
The latter are tests for the test infrastructure and helpers
themselves, rather than testing anything in passt or pasta
David Gibson (20):
avocado: Make a duplicate copy of testsuite for comparison purposes
avocado: Don't double download assets for test/ and oldtest/
avocado: Move static checkers to avocado
avocado/tasst: Helper functions for executing commands in different
places
avocado/tasst: Type checking helpers
avocado: Convert build tests to avocado
avocado/tasst: Add helpers for running background commands on sites
avocado/tasst: Add helper to get network interface names for a site
avocado/tasst: Add helpers to run commands with nstool
avocado/tasst: Add ifup and network address helpers to Site
avocado/tasst: Helper for creating veth devices between namespaces
avocado/tasst: Add helper for getting MTU of a network interface
avocado/tasst: Add helper to wait for IP address to appear
avocado/tasst: Add helpers for getting a site's routes
avocado/tasst: Helpers for test transferring data between sites
avocado/tasst: IP address allocation helpers
avocado/tasst: Helpers for testing NDP behaviour
avocado/tasst: Helpers for testing DHCP & DHCPv6 behaviour
avocado/tasst: Helpers to construct a simple network environment for
tests
avocado: Convert basic pasta tests
Makefile | 19 +
avocado/.gitignore | 1 +
avocado/build.py | 81 +++
avocado/pasta.py | 195 ++++++
avocado/static_checkers/clang-tidy.sh | 3 +
avocado/static_checkers/cppcheck.sh | 3 +
avocado/tasst/__init__.py | 22 +
avocado/tasst/address.py | 89 +++
avocado/tasst/dhcp.py | 152 +++++
avocado/tasst/dhcpv6.py | 135 +++++
avocado/tasst/meta/__init__.py | 16 +
avocado/tasst/meta/static_ifup.py | 59 ++
avocado/tasst/meta/veth.py | 95 +++
avocado/tasst/ndp.py | 137 +++++
avocado/tasst/nstool.py | 199 +++++++
avocado/tasst/scenario/__init__.py | 11 +
avocado/tasst/scenario/simple.py | 98 +++
avocado/tasst/site.py | 226 +++++++
avocado/tasst/transfer.py | 224 +++++++
avocado/tasst/typing.py | 23 +
oldtest/.gitignore | 11 +
oldtest/Makefile | 119 ++++
oldtest/README.md | 137 +++++
{test => oldtest}/build/all | 0
{test => oldtest}/build/clang_tidy | 0
{test => oldtest}/build/cppcheck | 0
oldtest/ci | 1 +
oldtest/demo/passt | 245 ++++++++
oldtest/demo/pasta | 274 +++++++++
oldtest/demo/podman | 819 ++++++++++++++++++++++++++
oldtest/distro/debian | 252 ++++++++
oldtest/distro/fedora | 396 +++++++++++++
oldtest/distro/opensuse | 208 +++++++
oldtest/distro/ubuntu | 216 +++++++
oldtest/env/mate-terminal.profile | 42 ++
oldtest/find-arm64-firmware.sh | 13 +
oldtest/lib/context | 130 ++++
oldtest/lib/layout | 259 ++++++++
oldtest/lib/layout_ugly | 113 ++++
oldtest/lib/perf_report | 272 +++++++++
oldtest/lib/setup | 385 ++++++++++++
oldtest/lib/setup_ugly | 58 ++
oldtest/lib/term | 750 +++++++++++++++++++++++
oldtest/lib/test | 398 +++++++++++++
oldtest/lib/util | 133 +++++
oldtest/lib/video | 152 +++++
oldtest/memory/passt | 187 ++++++
oldtest/nstool.c | 565 ++++++++++++++++++
oldtest/passt.mbuto | 83 +++
oldtest/passt.mem.mbuto | 44 ++
oldtest/passt/dhcp | 70 +++
oldtest/passt/ndp | 33 ++
oldtest/passt/shutdown | 19 +
oldtest/passt/tcp | 76 +++
oldtest/passt/udp | 46 ++
oldtest/passt_in_ns/icmp | 32 +
oldtest/passt_in_ns/shutdown | 19 +
oldtest/passt_in_ns/tcp | 256 ++++++++
oldtest/passt_in_ns/udp | 138 +++++
{test => oldtest}/pasta/dhcp | 0
{test => oldtest}/pasta/ndp | 0
{test => oldtest}/pasta/tcp | 0
{test => oldtest}/pasta/udp | 0
oldtest/pasta_options/log_to_file | 93 +++
oldtest/perf/passt_tcp | 215 +++++++
oldtest/perf/passt_udp | 165 ++++++
oldtest/perf/pasta_tcp | 300 ++++++++++
oldtest/perf/pasta_udp | 219 +++++++
oldtest/prepare-distro-img.sh | 18 +
oldtest/run | 238 ++++++++
oldtest/run_demo | 1 +
oldtest/two_guests/basic | 80 +++
oldtest/valgrind.supp | 9 +
test/lib/layout | 31 -
test/lib/setup | 47 --
test/run | 14 -
76 files changed, 10077 insertions(+), 92 deletions(-)
create mode 100644 avocado/.gitignore
create mode 100644 avocado/build.py
create mode 100644 avocado/pasta.py
create mode 100755 avocado/static_checkers/clang-tidy.sh
create mode 100755 avocado/static_checkers/cppcheck.sh
create mode 100644 avocado/tasst/__init__.py
create mode 100644 avocado/tasst/address.py
create mode 100644 avocado/tasst/dhcp.py
create mode 100644 avocado/tasst/dhcpv6.py
create mode 100644 avocado/tasst/meta/__init__.py
create mode 100644 avocado/tasst/meta/static_ifup.py
create mode 100644 avocado/tasst/meta/veth.py
create mode 100644 avocado/tasst/ndp.py
create mode 100644 avocado/tasst/nstool.py
create mode 100644 avocado/tasst/scenario/__init__.py
create mode 100644 avocado/tasst/scenario/simple.py
create mode 100644 avocado/tasst/site.py
create mode 100644 avocado/tasst/transfer.py
create mode 100644 avocado/tasst/typing.py
create mode 100644 oldtest/.gitignore
create mode 100644 oldtest/Makefile
create mode 100644 oldtest/README.md
rename {test => oldtest}/build/all (100%)
rename {test => oldtest}/build/clang_tidy (100%)
rename {test => oldtest}/build/cppcheck (100%)
create mode 120000 oldtest/ci
create mode 100644 oldtest/demo/passt
create mode 100644 oldtest/demo/pasta
create mode 100644 oldtest/demo/podman
create mode 100644 oldtest/distro/debian
create mode 100644 oldtest/distro/fedora
create mode 100644 oldtest/distro/opensuse
create mode 100644 oldtest/distro/ubuntu
create mode 100644 oldtest/env/mate-terminal.profile
create mode 100755 oldtest/find-arm64-firmware.sh
create mode 100644 oldtest/lib/context
create mode 100644 oldtest/lib/layout
create mode 100644 oldtest/lib/layout_ugly
create mode 100755 oldtest/lib/perf_report
create mode 100755 oldtest/lib/setup
create mode 100755 oldtest/lib/setup_ugly
create mode 100755 oldtest/lib/term
create mode 100755 oldtest/lib/test
create mode 100755 oldtest/lib/util
create mode 100755 oldtest/lib/video
create mode 100644 oldtest/memory/passt
create mode 100644 oldtest/nstool.c
create mode 100755 oldtest/passt.mbuto
create mode 100755 oldtest/passt.mem.mbuto
create mode 100644 oldtest/passt/dhcp
create mode 100644 oldtest/passt/ndp
create mode 100644 oldtest/passt/shutdown
create mode 100644 oldtest/passt/tcp
create mode 100644 oldtest/passt/udp
create mode 100644 oldtest/passt_in_ns/icmp
create mode 100644 oldtest/passt_in_ns/shutdown
create mode 100644 oldtest/passt_in_ns/tcp
create mode 100644 oldtest/passt_in_ns/udp
rename {test => oldtest}/pasta/dhcp (100%)
rename {test => oldtest}/pasta/ndp (100%)
rename {test => oldtest}/pasta/tcp (100%)
rename {test => oldtest}/pasta/udp (100%)
create mode 100644 oldtest/pasta_options/log_to_file
create mode 100644 oldtest/perf/passt_tcp
create mode 100644 oldtest/perf/passt_udp
create mode 100644 oldtest/perf/pasta_tcp
create mode 100644 oldtest/perf/pasta_udp
create mode 100755 oldtest/prepare-distro-img.sh
create mode 100755 oldtest/run
create mode 120000 oldtest/run_demo
create mode 100644 oldtest/two_guests/basic
create mode 100644 oldtest/valgrind.supp
--
2.40.1