[PATCH v6 0/4] New proof-of-concept based exeter tests

David Gibson

1 Sep 2025 1 Sep '25

6:25 a.m.

Here's a new approach to building passt tests with exeter. This new one no longer uses Avocado in the default case, although it would still be possible to manually run the exeter based tests with Avocado. Here's another draft of my work on testing passt with Avocado and the exeter library I recently created. It includes Cleber's patch adding some basic Avocado tests and builds on that. For now this only does simple tests, to show how the integration could work. It adds some new trivial "smoke tests" and converts the linter and build checks to exeter. More complex tests will require building the sinte/pesto library we've discussed. A lot of the work for that already exists in my earlier exeter test series, but it will need some rework to split it into a separate component. v6: * Use exeter's new metadata support to print nicer test names v5: * Updated according to Stefano's review - Fixed a number of whitespace errors - Improved many comments and variable names to make things clearer * New patch adding parallel test execution with BATS * Improved autodetection of exeter tests using "exetool probe" David Gibson (4): test: Extend test scripts to allow running exeter tests. test: Run static checkers as exeter tests test: Convert build tests to exeter test: Allow exeter & podman tests to be parallel executed with BATS test/.gitignore | 2 + test/Makefile | 22 ++++++- test/build/all | 61 ------------------- test/build/build.py | 109 ++++++++++++++++++++++++++++++++++ test/build/clang_tidy | 17 ------ test/build/cppcheck | 17 ------ test/build/static_checkers.sh | 26 ++++++++ test/lib/exeter | 57 ++++++++++++++++++ test/run | 18 ++++-- test/smoke/smoke.sh | 33 ++++++++++ 10 files changed, 260 insertions(+), 102 deletions(-) delete mode 100644 test/build/all create mode 100755 test/build/build.py delete mode 100644 test/build/clang_tidy delete mode 100644 test/build/cppcheck create mode 100755 test/build/static_checkers.sh create mode 100644 test/lib/exeter create mode 100755 test/smoke/smoke.sh -- 2.51.0

Show replies by date

David Gibson

1 Sep 1 Sep

6:25 a.m.

New subject: [PATCH v6 1/4] test: Extend test scripts to allow running exeter tests.

Introduce some trivial testcases based on the exeter library. These run passt and pasta with --help and --version options. Extend our test scripts to run these tests. Signed-off-by: David Gibson --- test/.gitignore | 1 + test/Makefile | 5 +++- test/lib/exeter | 57 +++++++++++++++++++++++++++++++++++++++++++++ test/run | 9 ++++++- test/smoke/smoke.sh | 33 ++++++++++++++++++++++++++ 5 files changed, 103 insertions(+), 2 deletions(-) create mode 100644 test/lib/exeter create mode 100755 test/smoke/smoke.sh diff --git a/test/.gitignore b/test/.gitignore index 3573444f..cf48b885 100644 --- a/test/.gitignore +++ b/test/.gitignore @@ -11,3 +11,4 @@ nstool rampstream guest-key guest-key.pub +/exeter/ diff --git a/test/Makefile b/test/Makefile index bf63db87..332f3f3e 100644 --- a/test/Makefile +++ b/test/Makefile @@ -50,7 +50,7 @@ UBUNTU_NEW_IMGS = xenial-server-cloudimg-powerpc-disk1.img \ jammy-server-cloudimg-s390x.img UBUNTU_IMGS = $(UBUNTU_OLD_IMGS) $(UBUNTU_NEW_IMGS) -DOWNLOAD_ASSETS = mbuto podman \ +DOWNLOAD_ASSETS = exeter mbuto podman \ $(DEBIAN_IMGS) $(FEDORA_IMGS) $(OPENSUSE_IMGS) $(UBUNTU_IMGS) TESTDATA_ASSETS = small.bin big.bin medium.bin \ rampstream @@ -70,6 +70,9 @@ assets: $(ASSETS) pull-%: % git -C $* pull +exeter: + git clone https://gitlab.com/dgibson/exeter.git + mbuto: git clone git://mbuto.sh/mbuto diff --git a/test/lib/exeter b/test/lib/exeter new file mode 100644 index 00000000..e313abc5 --- /dev/null +++ b/test/lib/exeter @@ -0,0 +1,57 @@ +#!/bin/sh +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# PASST - Plug A Simple Socket Transport +# for qemu/UNIX domain socket mode +# +# PASTA - Pack A Subtle Tap Abstraction +# for network namespace/tap device mode +# +# test/lib/exeter - Run exeter tests within the rest of passt's tests +# +# Copyright Red Hat +# Author: David Gibson + +EXETOOL="$BASEPATH/exeter/exetool/exetool" + +# is_exeter() - Determine if a test file is an exeter program +# $@: Command line to invoke test program +is_exeter() { + $EXETOOL probe -- "$@" +} + +# exeter() - Run each test in an exeter program, logging each test separately +# $@: Command line to invoke exeter test program +exeter() { + STATESETUP="${STATEBASE}/$1" + mkdir -p "${STATESETUP}" + + context_setup_host host + layout_host + + cd test + + __ntests=$("$@" --list | wc -l) + if [ $? != 0 ]; then + info "Failed to get exeter manifest for $@" + pause_continue \ + "Press any key to pause test session" \ + "Resuming in " \ + "Paused, press any key to continue" \ + 5 + return + fi + + status_file_start "$* (exeter)" ${__ntests} + + for __testid in $("$@" --list); do + __desc="$($EXETOOL desc -- "$@" -- ${__testid})" + status_test_start "${__desc}" + context_run host "$@" "${__testid}" && status_test_ok || status_test_fail + done + + cd .. + + teardown_context_watch ${PANE_HOST} host +} diff --git a/test/run b/test/run index f73c3119..745a09fa 100755 --- a/test/run +++ b/test/run @@ -53,6 +53,7 @@ COMMIT="$(git log --oneline --no-decorate -1)" . lib/layout_ugly . lib/test . lib/video +. lib/exeter # cleanup() - Remove temporary files cleanup() { @@ -67,6 +68,8 @@ run() { perf_init [ ${CI} -eq 1 ] && video_start ci + exeter smoke/smoke.sh + setup build test build/all test build/cppcheck @@ -223,6 +226,10 @@ run_selected() { __setup= for __test; do + if is_exeter "test/${__test}"; then + exeter "${__test}" + continue + fi # HACK: the migrate tests need the setup repeated for # each test if [ "${__test%%/*}" != "${__setup}" -o \ @@ -234,7 +241,7 @@ run_selected() { test "${__test}" done - teardown "${__setup}" + [ -n "${__setup}" ] && teardown "${__setup}" log "PASS: ${STATUS_PASS}, FAIL: ${STATUS_FAIL}, SKIPPED: ${STATUS_SKIPPED}" diff --git a/test/smoke/smoke.sh b/test/smoke/smoke.sh new file mode 100755 index 00000000..a642fb96 --- /dev/null +++ b/test/smoke/smoke.sh @@ -0,0 +1,33 @@ +#! /bin/sh +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# PASST - Plug A Simple Socket Transport +# for qemu/UNIX domain socket mode +# +# PASTA - Pack A Subtle Tap Abstraction +# for network namespace/tap device mode +# +# test/smoke/smoke.sh - Basic smoke tests +# +# Copyright Red Hat +# Author: David Gibson + +. $(dirname $0)/../exeter/sh/exeter.sh + +PASST=$(dirname $0)/../../passt +PASTA=$(dirname $0)/../../pasta + +exeter_register passt_version $PASST --version +exeter_set_description passt_version "Check passt --version works" + +exeter_register pasta_version $PASTA --version +exeter_set_description pasta_version "Check pasta --version works" + +exeter_register passt_help $PASST --help +exeter_set_description passt_help "Check passt --help works" + +exeter_register pasta_help $PASTA --help +exeter_set_description pasta_help "Check pasta --help works" + +exeter_main "$@" -- 2.51.0

David Gibson

6:25 a.m.

New subject: [PATCH v6 2/4] test: Run static checkers as exeter tests

Move the static checkers from the current DSL to exeter. Signed-off-by: David Gibson --- test/build/clang_tidy | 17 ----------------- test/build/cppcheck | 17 ----------------- test/build/static_checkers.sh | 26 ++++++++++++++++++++++++++ test/run | 3 +-- 4 files changed, 27 insertions(+), 36 deletions(-) delete mode 100644 test/build/clang_tidy delete mode 100644 test/build/cppcheck create mode 100755 test/build/static_checkers.sh diff --git a/test/build/clang_tidy b/test/build/clang_tidy deleted file mode 100644 index 40573bfd..00000000 --- a/test/build/clang_tidy +++ /dev/null @@ -1,17 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-or-later -# -# PASST - Plug A Simple Socket Transport -# for qemu/UNIX domain socket mode -# -# PASTA - Pack A Subtle Tap Abstraction -# for network namespace/tap device mode -# -# test/build/clang_tidy - Run source through clang-tidy(1) linter -# -# Copyright (c) 2021 Red Hat GmbH -# Author: Stefano Brivio - -htools clang-tidy - -test Run clang-tidy -host make clang-tidy diff --git a/test/build/cppcheck b/test/build/cppcheck deleted file mode 100644 index 0e1dbced..00000000 --- a/test/build/cppcheck +++ /dev/null @@ -1,17 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-or-later -# -# PASST - Plug A Simple Socket Transport -# for qemu/UNIX domain socket mode -# -# PASTA - Pack A Subtle Tap Abstraction -# for network namespace/tap device mode -# -# test/build/cppcheck - Run source through cppcheck(1) linter -# -# Copyright (c) 2021 Red Hat GmbH -# Author: Stefano Brivio - -htools cppcheck - -test Run cppcheck -host make cppcheck diff --git a/test/build/static_checkers.sh b/test/build/static_checkers.sh new file mode 100755 index 00000000..42806e79 --- /dev/null +++ b/test/build/static_checkers.sh @@ -0,0 +1,26 @@ +#! /bin/sh +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# PASST - Plug A Simple Socket Transport +# for qemu/UNIX domain socket mode +# +# PASTA - Pack A Subtle Tap Abstraction +# for network namespace/tap device mode +# +# test/build/static_checkers.sh - Run static checkers +# +# Copyright Red Hat +# Author: David Gibson + +. $(dirname $0)/../exeter/sh/exeter.sh + +exeter_register cppcheck make -C .. cppcheck +exeter_set_description cppcheck "passt sources pass cppcheck" + +exeter_register clang_tidy make -C .. clang-tidy +exeter_set_description clang_tidy "passt sources pass clang-tidy" + +exeter_main "$@" + + diff --git a/test/run b/test/run index 745a09fa..9e26952c 100755 --- a/test/run +++ b/test/run @@ -69,11 +69,10 @@ run() { [ ${CI} -eq 1 ] && video_start ci exeter smoke/smoke.sh + exeter build/static_checkers.sh setup build test build/all - test build/cppcheck - test build/clang_tidy teardown build setup pasta -- 2.51.0

David Gibson

6:25 a.m.

New subject: [PATCH v6 3/4] test: Convert build tests to exeter

Convert the tests in build/all to be based on exeter. The new version of the tests is more robust than the original, since it makes a temporary copy of the source tree so will not be affected by concurrent manual builds. Signed-off-by: David Gibson --- test/build/all | 61 ------------------------- test/build/build.py | 109 ++++++++++++++++++++++++++++++++++++++++++++ test/run | 8 ++-- 3 files changed, 113 insertions(+), 65 deletions(-) delete mode 100644 test/build/all create mode 100755 test/build/build.py diff --git a/test/build/all b/test/build/all deleted file mode 100644 index 1f79e0d8..00000000 --- a/test/build/all +++ /dev/null @@ -1,61 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-or-later -# -# PASST - Plug A Simple Socket Transport -# for qemu/UNIX domain socket mode -# -# PASTA - Pack A Subtle Tap Abstraction -# for network namespace/tap device mode -# -# test/build/all - Build targets, one by one, then all together, check output -# -# Copyright (c) 2021 Red Hat GmbH -# Author: Stefano Brivio - -htools make cc rm uname getconf mkdir cp rm man - -test Build passt -host make clean -check ! [ -e passt ] -host CFLAGS="-Werror" make passt -check [ -f passt ] - -test Build pasta -host make clean -check ! [ -e pasta ] -host CFLAGS="-Werror" make pasta -check [ -h pasta ] - -test Build qrap -host make clean -check ! [ -e qrap ] -host CFLAGS="-Werror" make qrap -check [ -f qrap ] - -test Build all -host make clean -check ! [ -e passt ] -check ! [ -e pasta ] -check ! [ -e qrap ] -host CFLAGS="-Werror" make -check [ -f passt ] -check [ -h pasta ] -check [ -f qrap ] - -test Install -host mkdir __STATEDIR__/prefix -host prefix=__STATEDIR__/prefix make install -check [ -f __STATEDIR__/prefix/bin/passt ] -check [ -h __STATEDIR__/prefix/bin/pasta ] -check [ -f __STATEDIR__/prefix/bin/qrap ] -check man -M __STATEDIR__/prefix/share/man -W passt -check man -M __STATEDIR__/prefix/share/man -W pasta -check man -M __STATEDIR__/prefix/share/man -W qrap - -test Uninstall -host prefix=__STATEDIR__/prefix make uninstall -check ! [ -f __STATEDIR__/prefix/bin/passt ] -check ! [ -h __STATEDIR__/prefix/bin/pasta ] -check ! [ -f __STATEDIR__/prefix/bin/qrap ] -check ! man -M __STATEDIR__/prefix/share/man -W passt 2>/dev/null -check ! man -M __STATEDIR__/prefix/share/man -W pasta 2>/dev/null -check ! man -M __STATEDIR__/prefix/share/man -W qrap 2>/dev/null diff --git a/test/build/build.py b/test/build/build.py new file mode 100755 index 00000000..e49287c9 --- /dev/null +++ b/test/build/build.py @@ -0,0 +1,109 @@ +#! /usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +# PASST - Plug A Simple Socket Transport +# for qemu/UNIX domain socket mode +# +# PASTA - Pack A Subtle Tap Abstraction +# for network namespace/tap device mode +# +# test/build/build.py - Test build and install targets +# +# Copyright Red Hat +# Author: David Gibson + +import contextlib +import os +from pathlib import Path +import subprocess +import tempfile +from typing import Iterable, Iterator + +import exeter + +def sh(cmd): + """Run given command in a shell""" + subprocess.run(cmd, shell=True) + + +@contextlib.contextmanager +def clone_sources() -> Iterator[str]: + """Create a temporary copy of the passt sources. + + When the context enters create a temporary directory and copy the + passt sources into it. Clean it up when the context exits. + """ + + os.chdir('..') # Move from test/ to repo base + with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as tmpdir: + sh(f"cp --parents -d $(git ls-files) {tmpdir}") + os.chdir(tmpdir) + yield tmpdir + + +def test_make(target: str, expected_files: list[str]) -> None: + """Test `make {target}` + + Arguments: + target -- make target to invoke + expected_files -- files make is expected to create + + Verifies that + 1) `make target` completes successfully + 2) expected_files care created by `make target` + 3) expected_files are removed by `make clean` + """ + + ex_paths = [Path(f) for f in expected_files] + with clone_sources(): + for p in ex_paths: + assert not p.exists(), f"{p} existed before make" + sh(f'make {target} CFLAGS="-Werror"') + for p in ex_paths: + assert p.exists(), f"{p} wasn't made" + sh('make clean') + for p in ex_paths: + assert not p.exists(), f"{p} existed after make clean" + + +exeter.register('make_passt', test_make, 'passt', ['passt']) +exeter.register('make_pasta', test_make, 'pasta', ['pasta']) +exeter.register('make_qrap', test_make, 'qrap', ['qrap']) +exeter.register('make_all', test_make, 'all', ['passt', 'pasta', 'qrap']) + + +@exeter.test +def test_install_uninstall() -> None: + """Test `make install` and `make uninstall` + + Tests that `make install` installs the expected files to the + install prefix, and that `make uninstall` removes them again. + """ + + with clone_sources(): + with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) \ + as prefix: + bindir = Path(prefix) / 'bin' + mandir = Path(prefix) / 'share/man' + progs = ['passt', 'pasta', 'qrap'] + + # Install + sh(f'make install CFLAGS="-Werror" prefix={prefix}') + + for prog in progs: + exe = bindir / prog + assert exe.is_file(), f"{exe} does not exist as a regular file" + sh(f'man -M {mandir} -W {prog}') + + # Uninstall + sh(f'make uninstall prefix={prefix}') + + for prog in progs: + exe = bindir / prog + assert not exe.exists(), f"{exe} exists after uninstall" + sh(f'! man -M {mandir} -W {prog}') + + +if __name__ == '__main__': + exeter.main() diff --git a/test/run b/test/run index 9e26952c..52507f1d 100755 --- a/test/run +++ b/test/run @@ -43,6 +43,9 @@ KERNEL=${KERNEL:-"/boot/vmlinuz-$(uname -r)"} COMMIT="$(git log --oneline --no-decorate -1)" +# Let exeter tests written in Python find their modules +export PYTHONPATH=${BASEPATH}/exeter/py3 + . lib/util . lib/context . lib/setup @@ -69,12 +72,9 @@ run() { [ ${CI} -eq 1 ] && video_start ci exeter smoke/smoke.sh + exeter build/build.py exeter build/static_checkers.sh - setup build - test build/all - teardown build - setup pasta test pasta/ndp test pasta/dhcp -- 2.51.0

Stefano Brivio

2 Sep 2 Sep

9:39 a.m.

New subject: [PATCH v6 3/4] test: Convert build tests to exeter

On Mon, 1 Sep 2025 14:25:14 +1000 David Gibson wrote:

...

+def test_make(target: str, expected_files: list[str]) -> None: + """Test `make {target}` + + Arguments: + target -- make target to invoke + expected_files -- files make is expected to create + + Verifies that + 1) `make target` completes successfully + 2) expected_files care created by `make target` + 3) expected_files are removed by `make clean` + """ + + ex_paths = [Path(f) for f in expected_files] + with clone_sources(): + for p in ex_paths: + assert not p.exists(), f"{p} existed before make" + sh(f'make {target} CFLAGS="-Werror"') + for p in ex_paths: + assert p.exists(), f"{p} wasn't made" + sh('make clean') + for p in ex_paths: + assert not p.exists(), f"{p} existed after make clean" + + +exeter.register('make_passt', test_make, 'passt', ['passt']) +exeter.register('make_pasta', test_make, 'pasta', ['pasta']) +exeter.register('make_qrap', test_make, 'qrap', ['qrap']) +exeter.register('make_all', test_make, 'all', ['passt', 'pasta', 'qrap'])

I guess I'm missing something, but how do you set descriptions from Python? By the way, the shell script interface you added for that looks quite convenient and obvious to me. -- Stefano

David Gibson

12:41 p.m.

New subject: [PATCH v6 3/4] test: Convert build tests to exeter

On Tue, Sep 02, 2025 at 09:39:53AM +0200, Stefano Brivio wrote:

...

On Mon, 1 Sep 2025 14:25:14 +1000 David Gibson wrote:

...
+def test_make(target: str, expected_files: list[str]) -> None: + """Test `make {target}` + + Arguments: + target -- make target to invoke + expected_files -- files make is expected to create + + Verifies that + 1) `make target` completes successfully + 2) expected_files care created by `make target` + 3) expected_files are removed by `make clean` + """ + + ex_paths = [Path(f) for f in expected_files] + with clone_sources(): + for p in ex_paths: + assert not p.exists(), f"{p} existed before make" + sh(f'make {target} CFLAGS="-Werror"') + for p in ex_paths: + assert p.exists(), f"{p} wasn't made" + sh('make clean') + for p in ex_paths: + assert not p.exists(), f"{p} existed after make clean" + + +exeter.register('make_passt', test_make, 'passt', ['passt']) +exeter.register('make_pasta', test_make, 'pasta', ['pasta']) +exeter.register('make_qrap', test_make, 'qrap', ['qrap']) +exeter.register('make_all', test_make, 'all', ['passt', 'pasta', 'qrap'])

I guess I'm missing something, but how do you set descriptions from Python?

There are two ways: 1) foo = exeter.register(...) foo.set_description("test that does the thing") 2) By default exeter will take it from the first line of the test function's docstring.

...

By the way, the shell script interface you added for that looks quite convenient and obvious to me.

-- Stefano

-- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson

Stefano Brivio

2:23 p.m.

New subject: [PATCH v6 3/4] test: Convert build tests to exeter

On Tue, 2 Sep 2025 20:41:41 +1000 David Gibson wrote:

...

On Tue, Sep 02, 2025 at 09:39:53AM +0200, Stefano Brivio wrote:

...
On Mon, 1 Sep 2025 14:25:14 +1000 David Gibson wrote:

...
+def test_make(target: str, expected_files: list[str]) -> None: + """Test `make {target}` + + Arguments: + target -- make target to invoke + expected_files -- files make is expected to create + + Verifies that + 1) `make target` completes successfully + 2) expected_files care created by `make target` + 3) expected_files are removed by `make clean` + """ + + ex_paths = [Path(f) for f in expected_files] + with clone_sources(): + for p in ex_paths: + assert not p.exists(), f"{p} existed before make" + sh(f'make {target} CFLAGS="-Werror"') + for p in ex_paths: + assert p.exists(), f"{p} wasn't made" + sh('make clean') + for p in ex_paths: + assert not p.exists(), f"{p} existed after make clean" + + +exeter.register('make_passt', test_make, 'passt', ['passt']) +exeter.register('make_pasta', test_make, 'pasta', ['pasta']) +exeter.register('make_qrap', test_make, 'qrap', ['qrap']) +exeter.register('make_all', test_make, 'all', ['passt', 'pasta', 'qrap'])

I guess I'm missing something, but how do you set descriptions from Python?

There are two ways: 1)

foo = exeter.register(...) foo.set_description("test that does the thing")

2)

By default exeter will take it from the first line of the test function's docstring.

Ah, this is really convenient. I mean I saw it happening but I wasn't sure why. :) -- Stefano

David Gibson

3 Sep 3 Sep

2:54 a.m.

New subject: [PATCH v6 3/4] test: Convert build tests to exeter

On Tue, Sep 02, 2025 at 02:23:23PM +0200, Stefano Brivio wrote:

...

On Tue, 2 Sep 2025 20:41:41 +1000 David Gibson wrote:

...
On Tue, Sep 02, 2025 at 09:39:53AM +0200, Stefano Brivio wrote:

...
On Mon, 1 Sep 2025 14:25:14 +1000 David Gibson wrote:

...
+def test_make(target: str, expected_files: list[str]) -> None: + """Test `make {target}` + + Arguments: + target -- make target to invoke + expected_files -- files make is expected to create + + Verifies that + 1) `make target` completes successfully + 2) expected_files care created by `make target` + 3) expected_files are removed by `make clean` + """ + + ex_paths = [Path(f) for f in expected_files] + with clone_sources(): + for p in ex_paths: + assert not p.exists(), f"{p} existed before make" + sh(f'make {target} CFLAGS="-Werror"') + for p in ex_paths: + assert p.exists(), f"{p} wasn't made" + sh('make clean') + for p in ex_paths: + assert not p.exists(), f"{p} existed after make clean" + + +exeter.register('make_passt', test_make, 'passt', ['passt']) +exeter.register('make_pasta', test_make, 'pasta', ['pasta']) +exeter.register('make_qrap', test_make, 'qrap', ['qrap']) +exeter.register('make_all', test_make, 'all', ['passt', 'pasta', 'qrap'])

I guess I'm missing something, but how do you set descriptions from Python?

There are two ways: 1)

foo = exeter.register(...) foo.set_description("test that does the thing")

2)

By default exeter will take it from the first line of the test function's docstring.

Ah, this is really convenient. I mean I saw it happening but I wasn't sure why. :)

To be a little more specific here, exeter.register will introspect the docstring and put it in the description. @exeter.test calls exeter.register, so it does the same thing. If you register with parameters it will expand Python-style formatting parameters in the docstring. This is useful for something like: def test_n(n : int): """Test the thing {n} times""" ... for i in range(3): exeter.register(f'test_{i}', test_n, i) You can always overwrite the default description afterwards with set_description. register_pipe() won't set a description automatically, but you can still set one explicitly afterwards. The Scenario stuff... I haven't checked, it's probably needs some work. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson

David Gibson

1 Sep 1 Sep

6:25 a.m.

New subject: [PATCH v6 4/4] test: Allow exeter & podman tests to be parallel executed with BATS

As a proof-of-concept, this adds a "make bats" target which will execute both the podman and exeter based testcases in parallel using BATS. Signed-off-by: David Gibson --- test/.gitignore | 1 + test/Makefile | 17 +++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/test/.gitignore b/test/.gitignore index cf48b885..9412f0d3 100644 --- a/test/.gitignore +++ b/test/.gitignore @@ -12,3 +12,4 @@ rampstream guest-key guest-key.pub /exeter/ +*.bats diff --git a/test/Makefile b/test/Makefile index 332f3f3e..1853fea3 100644 --- a/test/Makefile +++ b/test/Makefile @@ -5,7 +5,9 @@ # Copyright Red Hat # Author: David Gibson +BATS = bats -j $(shell nproc) WGET = wget -c +EXETOOL = ./exeter/exetool/exetool DEBIAN_IMGS = debian-8.11.0-openstack-amd64.qcow2 \ debian-10-nocloud-amd64.qcow2 \ @@ -62,6 +64,12 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \ ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS) +EXETER_PYPATH = exeter/py3 +EXETER_BATS = smoke/smoke.sh.bats \ + build/build.py.bats build/static_checkers.sh.bats +BATS_FILES = $(EXETER_BATS) \ + podman/test/system/505-networking-pasta.bats + CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99 assets: $(ASSETS) @@ -73,6 +81,8 @@ pull-%: % exeter: git clone https://gitlab.com/dgibson/exeter.git +$(EXETOOL): pull-exeter + mbuto: git clone git://mbuto.sh/mbuto @@ -118,6 +128,12 @@ medium.bin: big.bin: dd if=/dev/urandom bs=1M count=10 of=$@ +$(EXETER_BATS): %.bats: % $(EXETOOL) + PYTHONPATH=$(EXETER_PYPATH) $(EXETOOL) bats -- $< > $@ + +bats: $(BATS_FILES) pull-podman + PYTHONPATH=$(EXETER_PYPATH) CONTAINERS_HELPER_BINARY_DIR=.. $(BATS) $(BATS_FILES) + check: assets ./run @@ -127,6 +143,7 @@ debug: assets clean: rm -f perf.js *~ rm -f $(LOCAL_ASSETS) + rm -f $(EXETER_BATS) rm -rf test_logs rm -f prepared-*.qcow2 prepared-*.img -- 2.51.0

Stefano Brivio

3 Sep 3 Sep

11:14 p.m.

On Mon, 1 Sep 2025 14:25:11 +1000 David Gibson wrote:

...

Here's a new approach to building passt tests with exeter. This new one no longer uses Avocado in the default case, although it would still be possible to manually run the exeter based tests with Avocado.

Here's another draft of my work on testing passt with Avocado and the exeter library I recently created. It includes Cleber's patch adding some basic Avocado tests and builds on that.

For now this only does simple tests, to show how the integration could work. It adds some new trivial "smoke tests" and converts the linter and build checks to exeter. More complex tests will require building the sinte/pesto library we've discussed. A lot of the work for that already exists in my earlier exeter test series, but it will need some rework to split it into a separate component.

v6: * Use exeter's new metadata support to print nicer test names

Thanks, it looks much more readable now. And to me it looks ready to merge, but I hit something during testing that I'm not quite sure how to solve, assuming it can be considered an issue at all. Initially, I hadn't upgraded my local copy of exeter, so even smoke tests would fail, until it occurred to me that of course I needed to drop the 'exeter' directory and 'make exeter' again. It's an issue we conceptually already have with mbuto (even though it didn't get breaking changes for months now) and with Podman to some extent. With exeter, I guess we're going to hit that kind of issue pretty frequently at least in the near future. So I was wondering: should we enforce some form of up-to-date check from test/Makefile? I'm personally fine without it and, given that I plan to update test/README.md soon anyway ('make' under test/ is not mentioned at all), I can mention this kind of problem as well, so it shouldn't be a big deal for others. But I wanted to know if you have thoughts or proposals on the matter, before applying this. -- Stefano

David Gibson

4 Sep 4 Sep

4:42 a.m.

On Wed, Sep 03, 2025 at 11:14:35PM +0200, Stefano Brivio wrote:

...

On Mon, 1 Sep 2025 14:25:11 +1000 David Gibson wrote:

...
Here's a new approach to building passt tests with exeter. This new one no longer uses Avocado in the default case, although it would still be possible to manually run the exeter based tests with Avocado.

Here's another draft of my work on testing passt with Avocado and the exeter library I recently created. It includes Cleber's patch adding some basic Avocado tests and builds on that.

For now this only does simple tests, to show how the integration could work. It adds some new trivial "smoke tests" and converts the linter and build checks to exeter. More complex tests will require building the sinte/pesto library we've discussed. A lot of the work for that already exists in my earlier exeter test series, but it will need some rework to split it into a separate component.

v6: * Use exeter's new metadata support to print nicer test names

Thanks, it looks much more readable now.

And to me it looks ready to merge, but I hit something during testing that I'm not quite sure how to solve, assuming it can be considered an issue at all.

Initially, I hadn't upgraded my local copy of exeter, so even smoke tests would fail, until it occurred to me that of course I needed to drop the 'exeter' directory and 'make exeter' again. It's an issue we conceptually already have with mbuto (even though it didn't get breaking changes for months now) and with Podman to some extent.

Yes.

...

With exeter, I guess we're going to hit that kind of issue pretty frequently at least in the near future. So I was wondering: should we enforce some form of up-to-date check from test/Makefile?

Good point. We actually already have a mechanism for this: the pull-% targets, which we already use for mbuto and podman. I pull a dependency on this for the "make bats" target, but not for regular "make assets" so we didn't update exeter. I'll respin for this, because I have one other trivial change to make. It does occur to me that there's another version of the problem too. Once again we already conceptually have it with mbuto and podman, but it will be much worse with exeter due to pace of development: If (for debugging) we check out an old version of the passt source tree, it will still pull the latest version of exeter, which it might not be compatible with[0]. The obvious solution - but you might not like it much - is git submodules. They (kind of rightly) have a reputation as being a pain to work with, but they do solve exactly this problem - qemu uses them heavily and pretty successfully for this.

...

I'm personally fine without it and, given that I plan to update test/README.md soon anyway ('make' under test/ is not mentioned at all), I can mention this kind of problem as well, so it shouldn't be a big deal for others. But I wanted to know if you have thoughts or proposals on the matter, before applying this.

-- Stefano

[0] At some point, I do want to stabilise the exeter interfaces, and be concerned with backwards compatibility, which would largely fix this. However, at present I'm still finding stuff that wants to change at the exeter level often enough that I think the effort of maintaining compatibility would be prohibitive. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson

Stefano Brivio

5 Sep 5 Sep

1:14 a.m.

On Thu, 4 Sep 2025 12:42:57 +1000 David Gibson wrote:

...

On Wed, Sep 03, 2025 at 11:14:35PM +0200, Stefano Brivio wrote:

...
On Mon, 1 Sep 2025 14:25:11 +1000 David Gibson wrote:

...
Here's a new approach to building passt tests with exeter. This new one no longer uses Avocado in the default case, although it would still be possible to manually run the exeter based tests with Avocado.

Here's another draft of my work on testing passt with Avocado and the exeter library I recently created. It includes Cleber's patch adding some basic Avocado tests and builds on that.

For now this only does simple tests, to show how the integration could work. It adds some new trivial "smoke tests" and converts the linter and build checks to exeter. More complex tests will require building the sinte/pesto library we've discussed. A lot of the work for that already exists in my earlier exeter test series, but it will need some rework to split it into a separate component.

v6: * Use exeter's new metadata support to print nicer test names

Thanks, it looks much more readable now.

And to me it looks ready to merge, but I hit something during testing that I'm not quite sure how to solve, assuming it can be considered an issue at all.

Initially, I hadn't upgraded my local copy of exeter, so even smoke tests would fail, until it occurred to me that of course I needed to drop the 'exeter' directory and 'make exeter' again. It's an issue we conceptually already have with mbuto (even though it didn't get breaking changes for months now) and with Podman to some extent.

Yes.

...
With exeter, I guess we're going to hit that kind of issue pretty frequently at least in the near future. So I was wondering: should we enforce some form of up-to-date check from test/Makefile?

Good point. We actually already have a mechanism for this: the pull-% targets, which we already use for mbuto and podman. I pull a dependency on this for the "make bats" target, but not for regular "make assets" so we didn't update exeter.

Ah, right, of course. That actually looks good enough to me.

...

I'll respin for this, because I have one other trivial change to make.

It does occur to me that there's another version of the problem too. Once again we already conceptually have it with mbuto and podman, but it will be much worse with exeter due to pace of development:

If (for debugging) we check out an old version of the passt source tree, it will still pull the latest version of exeter, which it might not be compatible with[0]. The obvious solution - but you might not like it much - is git submodules. They (kind of rightly) have a reputation as being a pain to work with, but they do solve exactly this problem - qemu uses them heavily and pretty successfully for this.

To me it looks like a solution adding permanent complexity to solve a problem that's, in practice, temporary. Once exeter is stable enough, we'll just require a recent version. And before that day, I doubt we'll want to debug / bisect issues by running any of the exeter-based tests. Every time I'm debugging a possible regression I come up with some stand-alone reproducer or manual test. If the test suite had caught the regression at a given point, then we wouldn't have hit the regression in the first place.

...

...
I'm personally fine without it and, given that I plan to update test/README.md soon anyway ('make' under test/ is not mentioned at all), I can mention this kind of problem as well, so it shouldn't be a big deal for others. But I wanted to know if you have thoughts or proposals on the matter, before applying this.

-- Stefano

[0] At some point, I do want to stabilise the exeter interfaces, and be concerned with backwards compatibility, which would largely fix this. However, at present I'm still finding stuff that wants to change at the exeter level often enough that I think the effort of maintaining compatibility would be prohibitive.

Right, I wouldn't care, I really can't see any practical need. In practice, the kind of breakages we have right now are something much more basic: outdated test/README.md and broken test/Makefile which I couldn't find a moment to take care of, yet. And I still need to killall -9 nstool every time. -- Stefano

Age (days ago)

Last active (days ago)

List overview

Download

11 comments

2 participants

participants (2)

David Gibson
Stefano Brivio

[PATCH v6 0/4] New proof-of-concept based exeter tests

tags

participants (2)