[PATCH v3 00/15] RFC: Proof-of-concept based exeter+Avocado tests
Here's another draft of my work on testing passt with Avocado and the exeter library I recently created. It includes Cleber's patch adding some basic Avocado tests and builds on that. This draft is IMO significantly less janky than the last one, but certainly not jank-free: * We create the Avocado job files from the exeter sources in the Makefile. Ideally Avocado would eventually be extended to handle this itself * The names that Avocado sees for each test aren't great Stefano, If you could look particularly at 7/15 and 15/15 which add the real tests for passt/pasta, that would be great. The more specific you can be about what you find ugly about how the tests are written, then better I can try to address that. I suspect it will be easier to actually apply the series, then look at the new test files (test/build/build.py, and test/pasta/pasta.py particularly). From there you can look at as much of the support library as you need to, rather than digging through the actual patches to look for that. Cleber, If you could look at 4..7/22 particularly to review how I'm connecting the actual tests to the Avocado runner, that would be helpful. Changes since v2: * Added mypy type checking throughout * Use exeter "scenarios" to reduce boilerplate * Folded together a number of closely related patches (from 22 patches down to 15) * Entirely avoid open-coded Python context managers in favour of contextlib.contextmanager * No longer accidentally run a lot of the "meta" tests in the "real" testsuite * So, so many assorted cleanups Cleber Rosa (1): test: run static checkers with Avocado and JSON definitions David Gibson (14): test: Adjust how we invoke tests with run_avocado test: Extend make targets to run Avocado tests test: Exeter based static tests tasst: Support library and linters for tests in Python tasst/cmdsite: Base helpers for running shell commands in various places test: Add exeter+Avocado based build tests tasst/unshare: Add helpers to run commands in Linux unshared namespaces tasst/ip: Helpers for configuring IPv4 and IPv6 tasst/veth: Helpers for constructing veth devices between namespaces tasst: Helpers to test transferring data between sites tasst: Helpers for testing NDP behaviour tasst: Helpers for testing DHCP & DHCPv6 behaviour tasst: Helpers to construct a simple network environment for tests avocado: Convert basic pasta tests test/.gitignore | 2 + test/Makefile | 59 +++++++- test/avocado/static_checkers.json | 12 ++ test/build/.gitignore | 2 + test/build/build.py | 86 +++++++++++ test/build/static_checkers.sh | 28 ++++ test/meta/.gitignore | 1 + test/meta/lint.sh | 28 ++++ test/pasta/.gitignore | 1 + test/pasta/pasta.py | 130 +++++++++++++++++ test/run_avocado | 52 +++++++ test/tasst/.gitignore | 1 + test/tasst/__init__.py | 11 ++ test/tasst/__main__.py | 40 +++++ test/tasst/cmdsite.py | 193 ++++++++++++++++++++++++ test/tasst/dhcp.py | 190 ++++++++++++++++++++++++ test/tasst/ip.py | 234 ++++++++++++++++++++++++++++++ test/tasst/ndp.py | 106 ++++++++++++++ test/tasst/pasta.py | 48 ++++++ test/tasst/scenario/__init__.py | 12 ++ test/tasst/scenario/simple.py | 108 ++++++++++++++ test/tasst/selftest/__init__.py | 16 ++ test/tasst/transfer.py | 193 ++++++++++++++++++++++++ test/tasst/unshare.py | 166 +++++++++++++++++++++ test/tasst/veth.py | 104 +++++++++++++ 25 files changed, 1822 insertions(+), 1 deletion(-) create mode 100644 test/avocado/static_checkers.json create mode 100644 test/build/.gitignore create mode 100644 test/build/build.py create mode 100644 test/build/static_checkers.sh create mode 100644 test/meta/.gitignore create mode 100644 test/meta/lint.sh create mode 100644 test/pasta/.gitignore create mode 100644 test/pasta/pasta.py create mode 100755 test/run_avocado create mode 100644 test/tasst/.gitignore create mode 100644 test/tasst/__init__.py create mode 100644 test/tasst/__main__.py create mode 100644 test/tasst/cmdsite.py create mode 100644 test/tasst/dhcp.py create mode 100644 test/tasst/ip.py create mode 100644 test/tasst/ndp.py create mode 100644 test/tasst/pasta.py create mode 100644 test/tasst/scenario/__init__.py create mode 100644 test/tasst/scenario/simple.py create mode 100644 test/tasst/selftest/__init__.py create mode 100644 test/tasst/transfer.py create mode 100644 test/tasst/unshare.py create mode 100644 test/tasst/veth.py -- 2.46.0
From: Cleber Rosa
Currently the Avocado test cases expect to be run from the base dir of
the passt repo. At least for the time being, it turns out to be more
convenient to structure the tests to run from the test/ subdirectory. So,
adjust them to do so.
We make some changes to run_avocado to work better with this too:
* It appeared to have one too many os.path.dirname() calls, so it
set repo_root_path to the parent of the passt tree, rather than the
tree itself
* We add an os.chdir(), so the tests will be invoked from the test
directory regardles of where we invoke run_avocado
* We adjust the runner.identifier config parameter so we get distinct
(although very verbose) names for the more complex tests we're going
to add.
Signed-off-by: David Gibson
Add a new 'avocado' target to the test/ Makefile, which will install
avocado into a Python venv, and run the Avocado based tests with it.
Signed-off-by: David Gibson
Introduce some trivial testcases based on the exeter library. These run
the C static checkers, which is redundant with the included Avocado json
file, but are useful as an example. We extend the make avocado target to
generate Avocado job files from the exeter tests and include them in the
test run.
Signed-off-by: David Gibson
Create a Python package "tasst" with common helper code for use in
passt and pasta tests. Initially it just has a placeholder selftest.
Add a "make meta" target to run selftests for the test infrastructure
itself. For now this runs the dummy selftest, the flake8 Python
linter and the mypy static type checker.
Signed-off-by: David Gibson
Signed-off-by: David Gibson
Add a new test script to run the equivalent of the tests in build/all
using exeter and Avocado. This new version of the tests is more robust
than the original, since it makes a temporary copy of the source tree so
will not be affected by concurrent manual builds.
Signed-off-by: David Gibson
Use our existing nstool C helper, add python wrappers to easily run
commands in various namespaces.
Signed-off-by: David Gibson
Specifically these operate to configure IP using ip(8) running within
any CmdSite. We also include things to allocate addresses in example
networks.
Signed-off-by: David Gibson
Signed-off-by: David Gibson
Many of our existing tests are based on using socat to transfer between
various locations connected via pasta or passt. Add helpers to make
avocado tests performing similar transfers. Add selftests to verify those
work as expected when we don't have pasta or passt involved yet.
Signed-off-by: David Gibson
Signed-iff-by: David Gibson
Signed-off-by: David Gibson
This constructs essentially the simplest sensible network for passt/pasta
to operate in. We have one netns "simhost" to represent the host where we
will run passt or pasta, and a second "gw" to represent its default
gateway.
Signed-off-by: David Gibson
Convert the old-style tests for pasta (DHCP, NDP, TCP and UDP transfers)
to using avocado. There are a few differences in what we test, but this
should generally improve coverage:
* We run in a constructed network environment, so we no longer depend on
the real host's networking configuration
* We do independent setup for each individual test
* We add explicit tests for --config-net, which we use to accelerate that
setup for the TCP and UDP tests
* The TCP and UDP tests now test transfers between the guest and a
(simulated) remote site that's on a different network from the simulated
pasta host. Thus testing the no NAT case that passt/pasta emphasizes.
(We need to add tests for the NAT cases back in).
Signed-off-by: David Gibson
participants (1)
-
David Gibson