[LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Richard Palethorpe]

Hello,

Jan Stancek <jstancek@redhat.com> writes:

> On Tue, Dec 21, 2021 at 6:56 PM Petr Vorel <pvorel@suse.cz> wrote:
>>
>> i all,
>>
>> [ Cc automated-testing and people who might be interested ]
>>
>> > Add an unstable kernel ABI flag and a runtest file for unstable
>> > tests. This means we can add tests which are likely to be broken by
>> > changes in the kernel ABI. Without disrupting LTP releases which are
>> > required to be stable.
>>
>> > Users who require stability can filter the tests with this flag
>> > or not schedule the unstable runtest file(s).
>>
>> I'm ok for this from a long term perspective, because agree Richie it can help
>> people to run tests on kernel from next tree or mainline rc kernel).
>>
>> It's not much work to maintain this.
>>
>> It'd also help people writing tests for  fanotify and IMA not having wait
>> several weeks.
>>
>> Yes, we could add it to fanotify22 [1], but not to ima_conditionals.sh [2],
>> which is shell. But adding support to shell is trivial.
>>
>> Acked-by: Petr Vorel <petr.vorel@gmail.com>
>>
>> ....
>> > +++ b/runtest/syscalls-unstable
>> How about having name syscalls-next? Although there can be tests which are from
>> some kernel maintainer tree (it does not have to be limited to next tree),
>> unstable can mean "tests which aren't fixed yet and thus buggy".
>
> staging?

Staging and unstable could equally mean the test itself is not fininshed
IMO. I didn't suggest next for exactly the reason mentioned, but it
might be the better choice.

>
> IMO separate runtest would be enough, any notes why and how test was developed
> could be in comments in code, where people can find it (less metadata
> to maintain),
> and those comments could stay there after feature is accepted to
> mainline, we just
> move test between runtest files.

Then the test has a useless or misleading comment saying it was
developed against a feature still in development. It's trivial to remove
such comments or meta-data. I expect test authors will do it themselves
and if they don't we can rethink accepting such tests.

Also the patch uses the meta-data to print a hint. That way we do not
need to look at the source code, runtest file and LTP version before
deciding on the severity of a problem. Doing extra work upstream saves a
lot of work downstream.

Finally note that the plan is to schedule tests without runtest files
for parallel execution. That requires meta-data.

>
>> > @@ -0,0 +1,3 @@
>> > +# Tests for kernel features which are not finalized
>> > +
>> > +fanotify22 fanotify22
>>
>> Kind regards,
>> Petr
>>
>> [1] https://patchwork.ozlabs.org/project/ltp/list/?series=272782
>> [2] https://patchwork.ozlabs.org/project/ltp/list/?series=265664
>>
>> --
>> Mailing list info: https://lists.linux.it/listinfo/ltp
>>


-- 
Thank you,
Richard.