Re: [PATCH v6 0/4] New proof-of-concept based exeter tests

On Wed, Sep 03, 2025 at 11:14:35PM +0200, Stefano Brivio wrote:

On Mon, 1 Sep 2025 14:25:11 +1000 David Gibson wrote:

...

Here's a new approach to building passt tests with exeter. This new one no longer uses Avocado in the default case, although it would still be possible to manually run the exeter based tests with Avocado.

Here's another draft of my work on testing passt with Avocado and the exeter library I recently created. It includes Cleber's patch adding some basic Avocado tests and builds on that.

For now this only does simple tests, to show how the integration could work. It adds some new trivial "smoke tests" and converts the linter and build checks to exeter. More complex tests will require building the sinte/pesto library we've discussed. A lot of the work for that already exists in my earlier exeter test series, but it will need some rework to split it into a separate component.

v6: * Use exeter's new metadata support to print nicer test names

Thanks, it looks much more readable now.

And to me it looks ready to merge, but I hit something during testing that I'm not quite sure how to solve, assuming it can be considered an issue at all.

Initially, I hadn't upgraded my local copy of exeter, so even smoke tests would fail, until it occurred to me that of course I needed to drop the 'exeter' directory and 'make exeter' again. It's an issue we conceptually already have with mbuto (even though it didn't get breaking changes for months now) and with Podman to some extent.

Yes.

With exeter, I guess we're going to hit that kind of issue pretty frequently at least in the near future. So I was wondering: should we enforce some form of up-to-date check from test/Makefile?

Good point. We actually already have a mechanism for this: the pull-% targets, which we already use for mbuto and podman. I pull a dependency on this for the "make bats" target, but not for regular "make assets" so we didn't update exeter. I'll respin for this, because I have one other trivial change to make. It does occur to me that there's another version of the problem too. Once again we already conceptually have it with mbuto and podman, but it will be much worse with exeter due to pace of development: If (for debugging) we check out an old version of the passt source tree, it will still pull the latest version of exeter, which it might not be compatible with[0]. The obvious solution - but you might not like it much - is git submodules. They (kind of rightly) have a reputation as being a pain to work with, but they do solve exactly this problem - qemu uses them heavily and pretty successfully for this.

I'm personally fine without it and, given that I plan to update test/README.md soon anyway ('make' under test/ is not mentioned at all), I can mention this kind of problem as well, so it shouldn't be a big deal for others. But I wanted to know if you have thoughts or proposals on the matter, before applying this.

-- Stefano

[0] At some point, I do want to stabilise the exeter interfaces, and be concerned with backwards compatibility, which would largely fix this. However, at present I'm still finding stuff that wants to change at the exeter level often enough that I think the effort of maintaining compatibility would be prohibitive. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson