[LTP] [Automated-testing] [RFC PATCH 1/3] runltp: Deprecate, add info about kirk

[Bird, Tim]

> -----Original Message-----
> From: Cyril Hrubis <chrubis@suse.cz>
> > > I'm afraid that's not a good solution either. The end goal for kirk is
> > > to have a small binary locked in RAM and with realtime priority to
> > > execute tests and send back logs, in case of qemu over virtio, to the
> > > kirk. That is to make sure that logs are collected properly even when
> > > kernel is out of memory and in a similar situations.
> > >
> > > If you run kirk on the VM, reporting is not going to be reliable.
> >
> > This means you're effectively mandating how ltp be run and the only
> > variable would be the kernel binary. Whilst I can understand that, I'm
> > not sure how useful us testing with this would be.
> 
> Not at all. As I replied to Tim, there is no secret sauce in runltp or
> kirk. In the end it's a tool to execute a test binaries. If you have a
> system that can execute binaries, reliably transfer logs and handle
> kernel crashes you can as well just execute the tests yourself. All you
> need from us is a tooling that will produce a list(s) of tests to
> execute.

I don't think it's that simple.

Currently, Fuego users can choose to either:
1. run a suite of tests (specified in the runtest file) using runltp executing on the target
2. run an individual test, not using runltp.

In the first case, since some of the suites have a large number of tests,
there are options in Fuego to convert the results into spreadsheet files
or PDF reports.  But this is based on the multi-test output from runltp.

Does kirk provide the same output formats and output options as runltp?

If runltp is eventually removed, I'll have to come up with a solution
for executing suites of tests on the target, and making sure the output
is the same as runltp (or modifying the report generation code to handle
a new output).

Fuego supports multiple "transport" layers.  ssh, serial console,
and adb transfers are supported, as well as a few weird transports
(such as ssh to a controller board that then transfers over serial).
I wouldn't request kirk to support these oddball transport mechanisms,
but if it had support for mapping it's communication mechanisms
onto these, or supporting some pluggable mechanism for transferring
files and executing programs (collecting stdout, stderr and return code),
that might be useful.

...
> > >
> > > We do have the ltx binary, which is the small dispatcher supposed to run
> > > on the VM. And in an ideal world we would have a python library that
> > > talks to it on the other end, as a part of kirk, that could be reused
> > > separately. And the same for building lists of test to execute, ideally
> > > we would have a python library that would export a simple interface so
> > > that everyone could integrate the blocks that they really need into
> > > their solution.
> >
> > Automated testing is a hard problem to solve generically and even if
> > you do manage that, this all looks like a lot of work even just to
> > reproduce what works today :/.
> 
> Indeed. However I stil think that there are reusable parts that may be
> worth putting together.

Possibly.  I tried for a few years to integrate Fuego and KernelCI,
but their architectures were too different, and I eventually gave up.
Now that KernelCI is changing, I've thought about going back
and seeing if I could transfer anything between the projects.

Usually, you have to write some kind of mapping layer, and the mappings
turn out to be harder than expected, due to assumptions baked in to
the architecture.

 -- Tim