[LTP] [Automated-testing] [RFC PATCH 1/3] runltp: Deprecate, add info about kirk

[Bird, Tim]

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> On Fri, 2024-06-07 at 17:45 +0200, Cyril Hrubis wrote:
> > Hi!
> > > > kirk is not perfect but already much better than old runltp script.
> > > > Let's deprecate runltp and propagate kirk.
> > > >
> > > > Signed-off-by: Petr Vorel <pvorel@suse.cz>
> > > > ---
> > > >  runltp | 13 +++++++++++++
> > > >  1 file changed, 13 insertions(+)
> > >
> > > I'd note that Yocto Project's CI is still using runltp and we have no
> > > recipe for kirk, or any experience of using it.
> >
> > That's why runltp isn't going to disappear without a deprecation period,
> > the idea is to add the deprecation and wait a few releases before the
> > final removal, that in practical terms means at least a year, possibly
> > two for users to explore the replacement and give us feedback.
> 
> It helps to know that, it wasn't clear how quickly you planed to remove
> runltp!
> 
> > > This does therefore worry me a little bit, there appears to be a lot
> > > of complexity in kirk we don't need.
> >
> > I would say that there is a complexity that you do not think that you
> > need but in reality you do. First of all the assumption that you can
> > have the test runner that keeps the results and overall state on the
> > same machine that runs the tests is the most flawed of them all. So
> > running the tests over some kind of connection is the basis design
> > principle of kirk. That allows us to easily and safely detect when we
> > crash kernels with our tests, which tend to happen more often than most
> > people think. And I can go about all the things that are there because
> > of a good reasons for hours.
> 
> I think you misunderstand my point. Yocto project already has code to
> handle setting up qemu instances, connecting to them, collecting data
> from them etc. and we use that with ltp in the same way we use it for
> lots of other tests. So yes, I agree with you that you need a
> connection but we already have a solution for that.
> 
> We probably don't want some tests doing this with kirk and everything
> else doing it differently. I suspect we wouldn't want to switch
> everything we're doing over to kirk either as that wouldn't work for us
> or the kirk maintainers due to differing needs and expectations.
> 
> > That being said, the current kirk implementation ended up more complex
> > than I would like it, and that is something to improve over the
> > deprecation period. The general idea is to allow users to experiment
> > with kirk, even when it's not perfect to get feedback and ideally make
> > it usable for most usecases before we get rid of runltp for good.
> 
> It sounds like we need to switch to kirk and use it simply as a direct
> run host driver, but we are going to have a lot of complexity in there
> we aren't in need of.

I'm in the same position as Richard here.  Fuego uses runltp on the target (device under test),
and has it's own mechanisms for detecting timeouts or kernel crashes, gathering test output,
restarting targets, etc. from a test host.  These same mechanisms are used for other tests.
Fuego also has mechanisms for running individual LTP tests on the device (by installing the
individual test, executing it, gathering results remotely, and removing the test and test artifacts).
This is used in cases where the overhead of installing and running runltp is too big.  I haven't
investigated kirk yet (but it was on my list of things to do).

Will kirk and/or LTP provide a simple mechanism for quick install and execution of
individual tests or small sets of tests (and itself)?  Fuego's model of testing is geared towards
testing of production devices, where no test artifacts are pre-installed on the target,
and full cleanup (removal of tests and test artifacts) is done between tests.

 -- Tim