[LTP] [Automated-testing] [RFC PATCH 1/3] runltp: Deprecate, add info about kirk

[Richard Purdie]

On Fri, 2024-06-07 at 18:35 +0200, Cyril Hrubis wrote:
> Hi!
> > > > This does therefore worry me a little bit, there appears to be a lot
> > > > of complexity in kirk we don't need.
> > > 
> > > I would say that there is a complexity that you do not think that you
> > > need but in reality you do. First of all the assumption that you can
> > > have the test runner that keeps the results and overall state on the
> > > same machine that runs the tests is the most flawed of them all. So
> > > running the tests over some kind of connection is the basis design
> > > principle of kirk. That allows us to easily and safely detect when we
> > > crash kernels with our tests, which tend to happen more often than most
> > > people think. And I can go about all the things that are there because
> > > of a good reasons for hours.
> > 
> > I think you misunderstand my point. Yocto project already has code to
> > handle setting up qemu instances, connecting to them, collecting data
> > from them etc. and we use that with ltp in the same way we use it for
> > lots of other tests. So yes, I agree with you that you need a
> > connection but we already have a solution for that. 
> > 
> > We probably don't want some tests doing this with kirk and everything
> > else doing it differently. I suspect we wouldn't want to switch
> > everything we're doing over to kirk either as that wouldn't work for us
> > or the kirk maintainers due to differing needs and expectations.
> 
> This may actually work, since we are trying to make kirk flexible enough
> to run other testsuites, I think that we already run subset of selftest
> with kirk in our environment.

I'm not convinced that would be a great fit for either project to be
honest. Reading more below, you have a very specific idea of how
communication should happen and many of our test workflow needs are
going to be outside of scope of a single binary communicating over
virtio. It makes sense for kernel testing but we have other needs.

> > > That being said, the current kirk implementation ended up more complex
> > > than I would like it, and that is something to improve over the
> > > deprecation period. The general idea is to allow users to experiment
> > > with kirk, even when it's not perfect to get feedback and ideally make
> > > it usable for most usecases before we get rid of runltp for good.
> > 
> > It sounds like we need to switch to kirk and use it simply as a direct
> > run host driver, but we are going to have a lot of complexity in there
> > we aren't in need of.
> 
> I'm afraid that's not a good solution either. The end goal for kirk is
> to have a small binary locked in RAM and with realtime priority to
> execute tests and send back logs, in case of qemu over virtio, to the
> kirk. That is to make sure that logs are collected properly even when
> kernel is out of memory and in a similar situations.
> 
> If you run kirk on the VM, reporting is not going to be reliable.

This means you're effectively mandating how ltp be run and the only
variable would be the kernel binary. Whilst I can understand that, I'm
not sure how useful us testing with this would be.

> My initial idea was to build the new generation testrunner as a set of
> building blocks, that could be reused separately, but the current desing
> does not make it easy.
> 
> We do have the ltx binary, which is the small dispatcher supposed to run
> on the VM. And in an ideal world we would have a python library that
> talks to it on the other end, as a part of kirk, that could be reused
> separately. And the same for building lists of test to execute, ideally
> we would have a python library that would export a simple interface so
> that everyone could integrate the blocks that they really need into
> their solution.

Automated testing is a hard problem to solve generically and even if
you do manage that, this all looks like a lot of work even just to
reproduce what works today :/.

I do understand the idea but in practise, I don't have the time or know
anyone with the time to put into something like this. I can barely keep
the tests/infra we have running.

Cheers,

Richard