[LTP] [Automated-testing] [RFC PATCH 1/3] runltp: Deprecate, add info about kirk

[Petr Vorel]

> > -----Original Message-----
> > From: ltp <ltp-bounces+tim.bird=sony.com@lists.linux.it> On Behalf Of Andrea Cervesato via ltp
> > Hi!

> > My 2 cents below.

> > On 6/7/24 23:17, Bird, Tim wrote:
> > >> -----Original Message-----
> > >> From: Cyril Hrubis <chrubis@suse.cz>
> > >>>> I'm afraid that's not a good solution either. The end goal for kirk is
> > >>>> to have a small binary locked in RAM and with realtime priority to
> > >>>> execute tests and send back logs, in case of qemu over virtio, to the
> > >>>> kirk. That is to make sure that logs are collected properly even when
> > >>>> kernel is out of memory and in a similar situations.

> > >>>> If you run kirk on the VM, reporting is not going to be reliable.
> > >>> This means you're effectively mandating how ltp be run and the only
> > >>> variable would be the kernel binary. Whilst I can understand that, I'm
> > >>> not sure how useful us testing with this would be.
> > >> Not at all. As I replied to Tim, there is no secret sauce in runltp or
> > >> kirk. In the end it's a tool to execute a test binaries. If you have a
> > >> system that can execute binaries, reliably transfer logs and handle
> > >> kernel crashes you can as well just execute the tests yourself. All you
> > >> need from us is a tooling that will produce a list(s) of tests to
> > >> execute.
> > > I don't think it's that simple.

> > > Currently, Fuego users can choose to either:
> > > 1. run a suite of tests (specified in the runtest file) using runltp executing on the target
> > > 2. run an individual test, not using runltp.

> > > In the first case, since some of the suites have a large number of tests,
> > > there are options in Fuego to convert the results into spreadsheet files
> > > or PDF reports.  But this is based on the multi-test output from runltp.

> > > Does kirk provide the same output formats and output options as runltp?
> > No, but it provides something better than that: a JSON file that can be
> > easily parsed by most of the libraries out there.
> > kirk can be literally used as it is runltp, with difference of python
> > dependency. If you have python, you can run kirk on
> > host as well as on target. All features like Qemu, LTX, SSH (etc.)
> > supports are meant to be used by host only.

> The python dependency is unfortunate.  In Fuego, we have carefully
> avoided any dependencies on the target other than those that can
> be provided by busybox (or toybox or toolbox).  This includes limiting all
> test programs to either native binaries or shell scripts using  just POSIX
> shell features.

Hi Tim,

as Cyril noted, having kirk on the host you would need only LTX on the target
(SUT). That would avoid python on the target. LTX is very small binary (given
you compile whole LTP for target). Although running kirk on the target is
possible, there is not much benefit (one can just export LTPROOT, modify PATH
and run test manually).

I'm not trying to persuade you to take kirk (understand you have strong reasons
not to take it), just FYI some test in the future will need extra tooling (e.g.
support for reboot) and that functionality you either miss or will need to
implement in Fuego (also probably Cyril already noted).

Other thing is that we all invest time to implement our own tooling (IMHO waste
of time, there is a lot of code, not just run the tests, but dmesg, coredump,
restart SUT after failure, ...) instead of reusing at least a minimal subset
and slowly improve that for the benefit of all.

Also, the more we work with *any* shell implementation the more we see it's from
the long term perspective to avoid having even slightly complex tools in shell.
Simple C code avoids many problems in shell (we know POSIX shell syntax, but
there are subtle differences in each implementation which show later on).  At
certain point we even considered writing our minimal shell implementation to
avoid shell craziness (but then we just decided to write tests mostly in C).
Also some tests are buggy when written in shell (e.g.  cgroup tests).

Kind regards,
Petr

> It sounds like kirk won't be useful to my project.
>   -- Tim