[LTP] [Automated-testing] [RFC PATCH 1/3] runltp: Deprecate, add info about kirk

[Bird, Tim]

> -----Original Message-----
> From: automated-testing@lists.yoctoproject.org <automated-testing@lists.yoctoproject.org> On Behalf Of Cyril Hrubis
> Hi!
> > I don't think it's that simple.
> >
> > Currently, Fuego users can choose to either:
> > 1. run a suite of tests (specified in the runtest file) using runltp executing on the target
> 
> And this is something that I think is a suboptimal solution. If the run
> crashes in the middle you can't easily continue with the rest of the
> tests after the system under test has been rebooted. So the desing
> decision for kirk was that the complete state of the testsuite execution
> is being kept on the machine that drives the testing.

I agree with this.  That problem is actually why I developed the second
solution, and another Fuego developer added skiplist support to runltp,
several years ago.

> I did a talk at SUSE Labs conference when I tried to explain when we are
> heading and it was recorded, unfortunately it wasn't uploaded to youtube
> yet, I will post the link here once it's there.
> 
> The bottom line question is how much code from us can be reused in your
> environment and this is something we have to figure out.
> 
> > 2. run an individual test, not using runltp.
> >
> > In the first case, since some of the suites have a large number of tests,
> > there are options in Fuego to convert the results into spreadsheet files
> > or PDF reports.  But this is based on the multi-test output from runltp.
> >
> > Does kirk provide the same output formats and output options as runltp?
> 
> We decided to go for JSON format for generic the test results for kirk.
> 
> However given that we store the results in python data structures
> internally, it's easy to write an exporter for a format you want and
> produce that directly. We can add CSV exporter for spreadsheets in 100
> lines of python.

Indeed, JSON should be easier to handle.  It would still require refactoring
as we currently split the runltp output into different files.  It would likely
end up simplifying things, though.

> 
> The JSON exported looks like this:
> 
> https://github.com/linux-test-project/kirk/blob/master/libkirk/export.py
> 
> > If runltp is eventually removed, I'll have to come up with a solution
> > for executing suites of tests on the target, and making sure the output
> > is the same as runltp (or modifying the report generation code to handle
> > a new output).
> 
> The second one is preferable.
Understood.

> 
> > Fuego supports multiple "transport" layers.  ssh, serial console,
> > and adb transfers are supported, as well as a few weird transports
> > (such as ssh to a controller board that then transfers over serial).
> > I wouldn't request kirk to support these oddball transport mechanisms,
> > but if it had support for mapping it's communication mechanisms
> > onto these, or supporting some pluggable mechanism for transferring
> > files and executing programs (collecting stdout, stderr and return code),
> > that might be useful.
> 
> That is actually the end goal. The ltx is designed to marshal messages
> over a byte oriented connection, the supported commands at the moment
> can execute binaries, transfer files, etc. We intend to use it over
> virtio and physical serial ports as well. So the idea is that once ltx
> is started on the system uder test kirk connect to that instance to
> drive the testing. It's not production ready yet, and that is something
> that I have on my TODO.

In that case, would kirk or python be required on the target?
I presume that ltx runs on the target, and that is it a native app?
What are its dependencies?

> 
> > > > > We do have the ltx binary, which is the small dispatcher supposed to run
> > > > > on the VM. And in an ideal world we would have a python library that
> > > > > talks to it on the other end, as a part of kirk, that could be reused
> > > > > separately. And the same for building lists of test to execute, ideally
> > > > > we would have a python library that would export a simple interface so
> > > > > that everyone could integrate the blocks that they really need into
> > > > > their solution.
> > > >
> > > > Automated testing is a hard problem to solve generically and even if
> > > > you do manage that, this all looks like a lot of work even just to
> > > > reproduce what works today :/.
> > >
> > > Indeed. However I stil think that there are reusable parts that may be
> > > worth putting together.
> >
> > Possibly.  I tried for a few years to integrate Fuego and KernelCI,
> > but their architectures were too different, and I eventually gave up.
> > Now that KernelCI is changing, I've thought about going back
> > and seeing if I could transfer anything between the projects.
> >
> > Usually, you have to write some kind of mapping layer, and the mappings
> > turn out to be harder than expected, due to assumptions baked in to
> > the architecture.
> 
> That's what I'm hoping to avoid to keep things in a simple modules, but
> I guess that we will run into a lot of implementation details that will
> make things hard or even impossible...

Quite likely, I think.  It *is* difficult to make modular pieces that can
be used by different CI systems, but I continue to have hope. :-)

Just the fact that you are discussing this on this list is forward progress
from where we were 10 years ago.  So thank you for keeping us informed
with what's going on with LTP.
 -- Tim