[LTP] [Automated-testing] [RFC PATCH 1/3] runltp: Deprecate, add info about kirk

[Cyril Hrubis]

Hi!
> > > This does therefore worry me a little bit, there appears to be a lot
> > > of complexity in kirk we don't need.
> > 
> > I would say that there is a complexity that you do not think that you
> > need but in reality you do. First of all the assumption that you can
> > have the test runner that keeps the results and overall state on the
> > same machine that runs the tests is the most flawed of them all. So
> > running the tests over some kind of connection is the basis design
> > principle of kirk. That allows us to easily and safely detect when we
> > crash kernels with our tests, which tend to happen more often than most
> > people think. And I can go about all the things that are there because
> > of a good reasons for hours.
> 
> I think you misunderstand my point. Yocto project already has code to
> handle setting up qemu instances, connecting to them, collecting data
> from them etc. and we use that with ltp in the same way we use it for
> lots of other tests. So yes, I agree with you that you need a
> connection but we already have a solution for that. 
> 
> We probably don't want some tests doing this with kirk and everything
> else doing it differently. I suspect we wouldn't want to switch
> everything we're doing over to kirk either as that wouldn't work for us
> or the kirk maintainers due to differing needs and expectations.

This may actually work, since we are trying to make kirk flexible enough
to run other testsuites, I think that we already run subset of selftest
with kirk in our environment.

> > That being said, the current kirk implementation ended up more complex
> > than I would like it, and that is something to improve over the
> > deprecation period. The general idea is to allow users to experiment
> > with kirk, even when it's not perfect to get feedback and ideally make
> > it usable for most usecases before we get rid of runltp for good.
> 
> It sounds like we need to switch to kirk and use it simply as a direct
> run host driver, but we are going to have a lot of complexity in there
> we aren't in need of.

I'm afraid that's not a good solution either. The end goal for kirk is
to have a small binary locked in RAM and with realtime priority to
execute tests and send back logs, in case of qemu over virtio, to the
kirk. That is to make sure that logs are collected properly even when
kernel is out of memory and in a similar situations.

If you run kirk on the VM, reporting is not going to be reliable.

My initial idea was to build the new generation testrunner as a set of
building blocks, that could be reused separately, but the current desing
does not make it easy.

We do have the ltx binary, which is the small dispatcher supposed to run
on the VM. And in an ideal world we would have a python library that
talks to it on the other end, as a part of kirk, that could be reused
separately. And the same for building lists of test to execute, ideally
we would have a python library that would export a simple interface so
that everyone could integrate the blocks that they really need into
their solution.

-- 
Cyril Hrubis
chrubis@suse.cz