New subject: [PATCH v2 01/21] test/nstool: Provide useful error if given a path that's too long

16 May 2023

As discussed on calls, I've been investigating Avocado as a possible
replacement for our hand-rolled shell test framework.  This is a
second draft proof-of-concept conversion of a few of the tests to the
Avocado framework.  Specifically, the static checkers, the build tests
and the pasta transfer tests are converted.

In this second version, I've done a lot of rework (maybe too much) on
the helper library.  It's got a lot more capabilities and is more
consistent.

The new tests will get failures if you don't also apply the fix I send
recently to prevent updating ip6.addr_seen to ::.

Promising things:
  * I remain happy with the speed improvement (under half the time of
    the equivalent old tests)
  * The fact that it's no longer dependent on the host network is great
  * I'm cautiously optimistic about the helper library making it
    relatively easy to add a bunch more tests with further coverage

Concerning things:
  * I'm coming to hate OO and jUnit patterns more and more, these can
    make it pretty non-obvious what each test relies on in terms of setup

Changes since RFCv1:
  * Sweeping rework of the test library
  * Started using tags to split between the "real" tests (run with
    "make avocado") and "meta" tests (run with "make
avocado-meta").
    The latter are tests for the test infrastructure and helpers
    themselves, rather than testing anything in passt or pasta

David Gibson (21):
  test/nstool: Provide useful error if given a path that's too long
  test/nstool: Fix fd leak in accept() loop
  avocado: Make a duplicate copy of testsuite for comparison purposes
  avocado: Don't double download assets for test/ and oldtest/
  avocado: Move static checkers to avocado
  avocado/tasst: Helper functions for executing commands in different
    places
  avocado: Convert build tests to avocado
  avocado/tasst: Add helpers for running background commands on sites
  avocado/tasst: Add helper to get network interface names for a site
  avocado/tasst: Add helpers to run commands with nstool
  avocado/tasst: Add ifup and network address helpers to Site
  avocado/tasst: Helper for creating veth devices between namespaces
  avocado/tasst: Add helper for getting MTU of a network interface
  avocado/tasst: Add helper to wait for IP address to appear
  avocado/tasst: Add helpers for getting a site's routes
  avocado/tasst: Helpers for test transferring data between sites
  avocado/tasst: IP address allocation helpers
  avocado/tasst: Helpers for testing NDP behaviour
  avocado/tasst: Helpers for testing DHCP & DHCPv6 behaviour
  avocado/tasst: Helpers to construct a simple network environment for
    tests
  avocado: Convert basic pasta tests

 Makefile                              |  19 +
 avocado/.gitignore                    |   1 +
 avocado/build.py                      |  75 +++
 avocado/pasta.py                      | 183 ++++++
 avocado/static_checkers/clang-tidy.sh |   3 +
 avocado/static_checkers/cppcheck.sh   |   3 +
 avocado/tasst/__init__.py             |  32 +
 avocado/tasst/address.py              |  89 +++
 avocado/tasst/dhcp.py                 | 133 +++++
 avocado/tasst/dhcpv6.py               | 123 ++++
 avocado/tasst/meta/__init__.py        |  16 +
 avocado/tasst/meta/static_ifup.py     |  60 ++
 avocado/tasst/meta/veth.py            |  95 +++
 avocado/tasst/ndp.py                  | 129 ++++
 avocado/tasst/nstool.py               | 214 +++++++
 avocado/tasst/scenario/__init__.py    |  11 +
 avocado/tasst/scenario/simple.py      |  96 +++
 avocado/tasst/site.py                 | 207 +++++++
 avocado/tasst/transfer.py             | 228 +++++++
 oldtest/.gitignore                    |  11 +
 oldtest/Makefile                      | 119 ++++
 oldtest/README.md                     | 137 +++++
 {test => oldtest}/build/all           |   0
 {test => oldtest}/build/clang_tidy    |   0
 {test => oldtest}/build/cppcheck      |   0
 oldtest/ci                            |   1 +
 oldtest/demo/passt                    | 245 ++++++++
 oldtest/demo/pasta                    | 274 +++++++++
 oldtest/demo/podman                   | 819 ++++++++++++++++++++++++++
 oldtest/distro/debian                 | 252 ++++++++
 oldtest/distro/fedora                 | 396 +++++++++++++
 oldtest/distro/opensuse               | 208 +++++++
 oldtest/distro/ubuntu                 | 216 +++++++
 oldtest/env/mate-terminal.profile     |  42 ++
 oldtest/find-arm64-firmware.sh        |  13 +
 oldtest/lib/context                   | 130 ++++
 oldtest/lib/layout                    | 259 ++++++++
 oldtest/lib/layout_ugly               | 113 ++++
 oldtest/lib/perf_report               | 272 +++++++++
 oldtest/lib/setup                     | 385 ++++++++++++
 oldtest/lib/setup_ugly                |  58 ++
 oldtest/lib/term                      | 750 +++++++++++++++++++++++
 oldtest/lib/test                      | 398 +++++++++++++
 oldtest/lib/util                      | 133 +++++
 oldtest/lib/video                     | 152 +++++
 oldtest/memory/passt                  | 187 ++++++
 oldtest/nstool.c                      | 565 ++++++++++++++++++
 oldtest/passt.mbuto                   |  83 +++
 oldtest/passt.mem.mbuto               |  44 ++
 oldtest/passt/dhcp                    |  70 +++
 oldtest/passt/ndp                     |  33 ++
 oldtest/passt/shutdown                |  19 +
 oldtest/passt/tcp                     |  76 +++
 oldtest/passt/udp                     |  46 ++
 oldtest/passt_in_ns/icmp              |  32 +
 oldtest/passt_in_ns/shutdown          |  19 +
 oldtest/passt_in_ns/tcp               | 256 ++++++++
 oldtest/passt_in_ns/udp               | 138 +++++
 {test => oldtest}/pasta/dhcp          |   0
 {test => oldtest}/pasta/ndp           |   0
 {test => oldtest}/pasta/tcp           |   0
 {test => oldtest}/pasta/udp           |   0
 oldtest/pasta_options/log_to_file     |  93 +++
 oldtest/perf/passt_tcp                | 215 +++++++
 oldtest/perf/passt_udp                | 165 ++++++
 oldtest/perf/pasta_tcp                | 300 ++++++++++
 oldtest/perf/pasta_udp                | 219 +++++++
 oldtest/prepare-distro-img.sh         |  18 +
 oldtest/run                           | 238 ++++++++
 oldtest/run_demo                      |   1 +
 oldtest/two_guests/basic              |  80 +++
 oldtest/valgrind.supp                 |   9 +
 test/lib/layout                       |  31 -
 test/lib/setup                        |  47 --
 test/nstool.c                         |  24 +-
 test/run                              |  14 -
 76 files changed, 10022 insertions(+), 100 deletions(-)
 create mode 100644 avocado/.gitignore
 create mode 100644 avocado/build.py
 create mode 100644 avocado/pasta.py
 create mode 100755 avocado/static_checkers/clang-tidy.sh
 create mode 100755 avocado/static_checkers/cppcheck.sh
 create mode 100644 avocado/tasst/__init__.py
 create mode 100644 avocado/tasst/address.py
 create mode 100644 avocado/tasst/dhcp.py
 create mode 100644 avocado/tasst/dhcpv6.py
 create mode 100644 avocado/tasst/meta/__init__.py
 create mode 100644 avocado/tasst/meta/static_ifup.py
 create mode 100644 avocado/tasst/meta/veth.py
 create mode 100644 avocado/tasst/ndp.py
 create mode 100644 avocado/tasst/nstool.py
 create mode 100644 avocado/tasst/scenario/__init__.py
 create mode 100644 avocado/tasst/scenario/simple.py
 create mode 100644 avocado/tasst/site.py
 create mode 100644 avocado/tasst/transfer.py
 create mode 100644 oldtest/.gitignore
 create mode 100644 oldtest/Makefile
 create mode 100644 oldtest/README.md
 rename {test => oldtest}/build/all (100%)
 rename {test => oldtest}/build/clang_tidy (100%)
 rename {test => oldtest}/build/cppcheck (100%)
 create mode 120000 oldtest/ci
 create mode 100644 oldtest/demo/passt
 create mode 100644 oldtest/demo/pasta
 create mode 100644 oldtest/demo/podman
 create mode 100644 oldtest/distro/debian
 create mode 100644 oldtest/distro/fedora
 create mode 100644 oldtest/distro/opensuse
 create mode 100644 oldtest/distro/ubuntu
 create mode 100644 oldtest/env/mate-terminal.profile
 create mode 100755 oldtest/find-arm64-firmware.sh
 create mode 100644 oldtest/lib/context
 create mode 100644 oldtest/lib/layout
 create mode 100644 oldtest/lib/layout_ugly
 create mode 100755 oldtest/lib/perf_report
 create mode 100755 oldtest/lib/setup
 create mode 100755 oldtest/lib/setup_ugly
 create mode 100755 oldtest/lib/term
 create mode 100755 oldtest/lib/test
 create mode 100755 oldtest/lib/util
 create mode 100755 oldtest/lib/video
 create mode 100644 oldtest/memory/passt
 create mode 100644 oldtest/nstool.c
 create mode 100755 oldtest/passt.mbuto
 create mode 100755 oldtest/passt.mem.mbuto
 create mode 100644 oldtest/passt/dhcp
 create mode 100644 oldtest/passt/ndp
 create mode 100644 oldtest/passt/shutdown
 create mode 100644 oldtest/passt/tcp
 create mode 100644 oldtest/passt/udp
 create mode 100644 oldtest/passt_in_ns/icmp
 create mode 100644 oldtest/passt_in_ns/shutdown
 create mode 100644 oldtest/passt_in_ns/tcp
 create mode 100644 oldtest/passt_in_ns/udp
 rename {test => oldtest}/pasta/dhcp (100%)
 rename {test => oldtest}/pasta/ndp (100%)
 rename {test => oldtest}/pasta/tcp (100%)
 rename {test => oldtest}/pasta/udp (100%)
 create mode 100644 oldtest/pasta_options/log_to_file
 create mode 100644 oldtest/perf/passt_tcp
 create mode 100644 oldtest/perf/passt_udp
 create mode 100644 oldtest/perf/pasta_tcp
 create mode 100644 oldtest/perf/pasta_udp
 create mode 100755 oldtest/prepare-distro-img.sh
 create mode 100755 oldtest/run
 create mode 120000 oldtest/run_demo
 create mode 100644 oldtest/two_guests/basic
 create mode 100644 oldtest/valgrind.supp

-- 
2.40.1

[PATCH v2 00/21] RFCv2: Proof-of-concept conversion of some tests to Avocado framework