[LTP] [Automated-testing] [RFC PATCH 1/3] runltp: Deprecate, add info about kirk

[Cyril Hrubis]

Hi!
> I don't think it's that simple.
> 
> Currently, Fuego users can choose to either:
> 1. run a suite of tests (specified in the runtest file) using runltp executing on the target

And this is something that I think is a suboptimal solution. If the run
crashes in the middle you can't easily continue with the rest of the
tests after the system under test has been rebooted. So the desing
decision for kirk was that the complete state of the testsuite execution
is being kept on the machine that drives the testing.

I did a talk at SUSE Labs conference when I tried to explain when we are
heading and it was recorded, unfortunately it wasn't uploaded to youtube
yet, I will post the link here once it's there.

The bottom line question is how much code from us can be reused in your
environment and this is something we have to figure out.

> 2. run an individual test, not using runltp.
> 
> In the first case, since some of the suites have a large number of tests,
> there are options in Fuego to convert the results into spreadsheet files
> or PDF reports.  But this is based on the multi-test output from runltp.
> 
> Does kirk provide the same output formats and output options as runltp?

We decided to go for JSON format for generic the test results for kirk.

However given that we store the results in python data structures
internally, it's easy to write an exporter for a format you want and
produce that directly. We can add CSV exporter for spreadsheets in 100
lines of python.

The JSON exported looks like this:

https://github.com/linux-test-project/kirk/blob/master/libkirk/export.py

> If runltp is eventually removed, I'll have to come up with a solution
> for executing suites of tests on the target, and making sure the output
> is the same as runltp (or modifying the report generation code to handle
> a new output).

The second one is preferable.

> Fuego supports multiple "transport" layers.  ssh, serial console,
> and adb transfers are supported, as well as a few weird transports
> (such as ssh to a controller board that then transfers over serial).
> I wouldn't request kirk to support these oddball transport mechanisms,
> but if it had support for mapping it's communication mechanisms
> onto these, or supporting some pluggable mechanism for transferring
> files and executing programs (collecting stdout, stderr and return code),
> that might be useful.

That is actually the end goal. The ltx is designed to marshal messages
over a byte oriented connection, the supported commands at the moment
can execute binaries, transfer files, etc. We intend to use it over
virtio and physical serial ports as well. So the idea is that once ltx
is started on the system uder test kirk connect to that instance to
drive the testing. It's not production ready yet, and that is something
that I have on my TODO.

> > > > We do have the ltx binary, which is the small dispatcher supposed to run
> > > > on the VM. And in an ideal world we would have a python library that
> > > > talks to it on the other end, as a part of kirk, that could be reused
> > > > separately. And the same for building lists of test to execute, ideally
> > > > we would have a python library that would export a simple interface so
> > > > that everyone could integrate the blocks that they really need into
> > > > their solution.
> > >
> > > Automated testing is a hard problem to solve generically and even if
> > > you do manage that, this all looks like a lot of work even just to
> > > reproduce what works today :/.
> > 
> > Indeed. However I stil think that there are reusable parts that may be
> > worth putting together.
> 
> Possibly.  I tried for a few years to integrate Fuego and KernelCI,
> but their architectures were too different, and I eventually gave up.
> Now that KernelCI is changing, I've thought about going back
> and seeing if I could transfer anything between the projects.
> 
> Usually, you have to write some kind of mapping layer, and the mappings
> turn out to be harder than expected, due to assumptions baked in to
> the architecture.

That's what I'm hoping to avoid to keep things in a simple modules, but
I guess that we will run into a lot of implementation details that will
make things hard or even impossible...

-- 
Cyril Hrubis
chrubis@suse.cz