[LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Richard Palethorpe]

Hello Li,

Li Wang <liwang@redhat.com> writes:

> Hi Richard,
>
> On Tue, Dec 21, 2021 at 7:31 PM Richard Palethorpe via ltp
> <ltp@lists.linux.it> wrote:
>>
>> Add an unstable kernel ABI flag and a runtest file for unstable
>> tests. This means we can add tests which are likely to be broken by
>> changes in the kernel ABI. Without disrupting LTP releases which are
>> required to be stable.
>>
>> Users who require stability can filter the tests with this flag
>> or not schedule the unstable runtest file(s).
>>
>> Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>
>> ---
>>
>> How about adding this to the fanotify22 patch?
>>
>>  include/tst_test.h        | 12 ++++++++++++
>>  lib/tst_test.c            |  6 ++++++
>>  runtest/syscalls-unstable |  3 +++
>>  3 files changed, 21 insertions(+)
>>  create mode 100644 runtest/syscalls-unstable
>>
>> diff --git a/include/tst_test.h b/include/tst_test.h
>> index 450ddf086..ff31e972e 100644
>> --- a/include/tst_test.h
>> +++ b/include/tst_test.h
>> @@ -174,6 +174,18 @@ struct tst_test {
>>         int skip_in_lockdown:1;
>>         int skip_in_compat:1;
>>
>> +       /*
>> +        * Test is for a feature that has not been added to the stable
>> +        * kernel ABI. That is, it's for a feature only available in
>> +        * linux-next, an RC or some other development branch.
>> +        *
>> +        * This string should be set to some text describing the
>> +        * kernel branch and version the test was developed
>> +        * against. e.g. "5.16 RC2", "linux-next-20211220",
>> +        * "net-next".
>> +        */
>> +       const char *const unstable_abi_version;
>
> I'm not sure if this will bring convenience or trouble to LTP.
>
> If a new syscall is in development (or merged in linux-next) but
> drop/change in mainline-kernel. We have to adjust the test case
> accordingly.

If someone is motivated enough to contribute tests for an unreleased
feature. There is a high chance they will make the changes. In the worst
case we just delete the test.

>
> And after the feature is go into the mainline kernel finally, should
> we move that from 'syscalls-unstable' to 'syscalls' and remove the
> tst_test->unstable_abi_version field at the same time?

Yes.

>
> Btw, why not we just keep the unstable-syscall test case stay
> in the review phase until it gets merged in mailline-kernel?
> I guess that wouldn't block anything or make anyone unhappy.

It's a lot of work to merge patches into your own branch for testing RCs
and linux-next. It makes collaboration a lot harder. It's easy for
people just to enable the unstable tests. Then new features will get
tested on a much wider range of setups.

It increases the chances of finding bugs before release.

-- 
Thank you,
Richard.