[LTP] [Automated-testing] [RFC PATCH 1/3] runltp: Deprecate, add info about kirk

[Bird, Tim]

> -----Original Message-----
> From: ltp <ltp-bounces+tim.bird=sony.com@lists.linux.it> On Behalf Of Andrea Cervesato via ltp
> Hi!
> 
> My 2 cents below.
> 
> On 6/7/24 23:17, Bird, Tim wrote:
> >> -----Original Message-----
> >> From: Cyril Hrubis <chrubis@suse.cz>
> >>>> I'm afraid that's not a good solution either. The end goal for kirk is
> >>>> to have a small binary locked in RAM and with realtime priority to
> >>>> execute tests and send back logs, in case of qemu over virtio, to the
> >>>> kirk. That is to make sure that logs are collected properly even when
> >>>> kernel is out of memory and in a similar situations.
> >>>>
> >>>> If you run kirk on the VM, reporting is not going to be reliable.
> >>> This means you're effectively mandating how ltp be run and the only
> >>> variable would be the kernel binary. Whilst I can understand that, I'm
> >>> not sure how useful us testing with this would be.
> >> Not at all. As I replied to Tim, there is no secret sauce in runltp or
> >> kirk. In the end it's a tool to execute a test binaries. If you have a
> >> system that can execute binaries, reliably transfer logs and handle
> >> kernel crashes you can as well just execute the tests yourself. All you
> >> need from us is a tooling that will produce a list(s) of tests to
> >> execute.
> > I don't think it's that simple.
> >
> > Currently, Fuego users can choose to either:
> > 1. run a suite of tests (specified in the runtest file) using runltp executing on the target
> > 2. run an individual test, not using runltp.
> >
> > In the first case, since some of the suites have a large number of tests,
> > there are options in Fuego to convert the results into spreadsheet files
> > or PDF reports.  But this is based on the multi-test output from runltp.
> >
> > Does kirk provide the same output formats and output options as runltp?
> No, but it provides something better than that: a JSON file that can be
> easily parsed by most of the libraries out there.
> kirk can be literally used as it is runltp, with difference of python
> dependency. If you have python, you can run kirk on
> host as well as on target. All features like Qemu, LTX, SSH (etc.)
> supports are meant to be used by host only.

The python dependency is unfortunate.  In Fuego, we have carefully
avoided any dependencies on the target other than those that can
be provided by busybox (or toybox or toolbox).  This includes limiting all
test programs to either native binaries or shell scripts using  just POSIX
shell features.

It sounds like kirk won't be useful to my project.
  -- Tim