[LTP] [Automated-testing] [RFC PATCH 1/3] runltp: Deprecate, add info about kirk

[Richard Purdie]

On Fri, 2024-06-07 at 17:45 +0200, Cyril Hrubis wrote:
> Hi!
> > > kirk is not perfect but already much better than old runltp script.
> > > Let's deprecate runltp and propagate kirk.
> > > 
> > > Signed-off-by: Petr Vorel <pvorel@suse.cz>
> > > ---
> > >  runltp | 13 +++++++++++++
> > >  1 file changed, 13 insertions(+)
> > 
> > I'd note that Yocto Project's CI is still using runltp and we have no
> > recipe for kirk, or any experience of using it.
> 
> That's why runltp isn't going to disappear without a deprecation period,
> the idea is to add the deprecation and wait a few releases before the
> final removal, that in practical terms means at least a year, possibly
> two for users to explore the replacement and give us feedback.

It helps to know that, it wasn't clear how quickly you planed to remove
runltp!

> > This does therefore worry me a little bit, there appears to be a lot
> > of complexity in kirk we don't need.
> 
> I would say that there is a complexity that you do not think that you
> need but in reality you do. First of all the assumption that you can
> have the test runner that keeps the results and overall state on the
> same machine that runs the tests is the most flawed of them all. So
> running the tests over some kind of connection is the basis design
> principle of kirk. That allows us to easily and safely detect when we
> crash kernels with our tests, which tend to happen more often than most
> people think. And I can go about all the things that are there because
> of a good reasons for hours.

I think you misunderstand my point. Yocto project already has code to
handle setting up qemu instances, connecting to them, collecting data
from them etc. and we use that with ltp in the same way we use it for
lots of other tests. So yes, I agree with you that you need a
connection but we already have a solution for that. 

We probably don't want some tests doing this with kirk and everything
else doing it differently. I suspect we wouldn't want to switch
everything we're doing over to kirk either as that wouldn't work for us
or the kirk maintainers due to differing needs and expectations.

> That being said, the current kirk implementation ended up more complex
> than I would like it, and that is something to improve over the
> deprecation period. The general idea is to allow users to experiment
> with kirk, even when it's not perfect to get feedback and ideally make
> it usable for most usecases before we get rid of runltp for good.

It sounds like we need to switch to kirk and use it simply as a direct
run host driver, but we are going to have a lot of complexity in there
we aren't in need of.

Cheers,

Richard