[LTP] [Automated-testing] [RFC PATCH 1/3] runltp: Deprecate, add info about kirk

[Cyril Hrubis]

Hi!
> > This may actually work, since we are trying to make kirk flexible enough
> > to run other testsuites, I think that we already run subset of selftest
> > with kirk in our environment.
> 
> I'm not convinced that would be a great fit for either project to be
> honest. Reading more below, you have a very specific idea of how
> communication should happen and many of our test workflow needs are
> going to be outside of scope of a single binary communicating over
> virtio. It makes sense for kernel testing but we have other needs.

Fair enough.

> > > > That being said, the current kirk implementation ended up more complex
> > > > than I would like it, and that is something to improve over the
> > > > deprecation period. The general idea is to allow users to experiment
> > > > with kirk, even when it's not perfect to get feedback and ideally make
> > > > it usable for most usecases before we get rid of runltp for good.
> > > 
> > > It sounds like we need to switch to kirk and use it simply as a direct
> > > run host driver, but we are going to have a lot of complexity in there
> > > we aren't in need of.
> > 
> > I'm afraid that's not a good solution either. The end goal for kirk is
> > to have a small binary locked in RAM and with realtime priority to
> > execute tests and send back logs, in case of qemu over virtio, to the
> > kirk. That is to make sure that logs are collected properly even when
> > kernel is out of memory and in a similar situations.
> > 
> > If you run kirk on the VM, reporting is not going to be reliable.
> 
> This means you're effectively mandating how ltp be run and the only
> variable would be the kernel binary. Whilst I can understand that, I'm
> not sure how useful us testing with this would be.

Not at all. As I replied to Tim, there is no secret sauce in runltp or
kirk. In the end it's a tool to execute a test binaries. If you have a
system that can execute binaries, reliably transfer logs and handle
kernel crashes you can as well just execute the tests yourself. All you
need from us is a tooling that will produce a list(s) of tests to
execute.

> > My initial idea was to build the new generation testrunner as a set of
> > building blocks, that could be reused separately, but the current desing
> > does not make it easy.
> > 
> > We do have the ltx binary, which is the small dispatcher supposed to run
> > on the VM. And in an ideal world we would have a python library that
> > talks to it on the other end, as a part of kirk, that could be reused
> > separately. And the same for building lists of test to execute, ideally
> > we would have a python library that would export a simple interface so
> > that everyone could integrate the blocks that they really need into
> > their solution.
> 
> Automated testing is a hard problem to solve generically and even if
> you do manage that, this all looks like a lot of work even just to
> reproduce what works today :/.

Indeed. However I stil think that there are reusable parts that may be
worth putting together.

> I do understand the idea but in practise, I don't have the time or know
> anyone with the time to put into something like this. I can barely keep
> the tests/infra we have running.

This reminds me:

https://en.wikipedia.org/wiki/Boots_theory

If we apply that to software not being able to inovate keeps you doing
work that shouldn't needed to be done in the first place.

-- 
Cyril Hrubis
chrubis@suse.cz