[LTP] [doc, runtest] [was: Re: [PATCH] cve: add CVE-2025-38236 test]

[Cyril Hrubis]

Hi!
> This problem happen on all runtest files, fixing just one does not fix the
> problem.

Well we can do that for any runtest file that has clear definition of
which tests belongs there. For CVE it's crystal clear, tests that have
cve tag should be there. For the rest of the runtest files, it's not so
much. Maybe for syscalls we may be able to do so.

The main thing is that we have to start somewhere got eventually get
there. I just quickly looked at the cve runtest file and figured out
that we have to add tests variants somewhere into the metadata. I.e.
quite a few of the CVE tests have command line options in the runtest
file which has to be stored somewhere else.

> Sure, it'd be possible to generate runtest/cve from metadata. Do we really want
> to implement it? (I can create a ticket). I guess we would use C and ujson to
> not require json-c or python3 for building LTP.

Or we can hook it up directly into the metadata parser, instead of
parsing the resulting JSON we can act on the data while they are in the
memory. Matching some tags and writing a test name into a file could be
easily done.

> I would be more interested to have section "CVE reproducers" in Statistics page [1].
> While the same tool could be used to do both goals, when only doc page
> implemented, it could be easily done in python3 (doc/conf.py already parses
> ltp.json).
> 
> When we are at Statistics page, also generating list of reproducers (based on
> kernel fixes) would be also nice. Because this was implemented in the previous
> asciidoctor implementation. How about having these lists Statistics, where are
> other tables already (and linking each test to "Test Catalog")?
> 
> Also I find "Statistics" name confusing. It says nothing about the content. I
> wonder if people curiously click on the page or just ignore the page (if they
> don't like math :)). Maybe "Kernel coverage" or something like that would be
> more informative.

I would put the list of reproducers and list of CVE reproducers into a
separate page that would be have "reproducers" in the name.

And statistics is probably okayish name, since coverage may mislead
people even more. For example we have a lot of tests for a write()
syscall yet coverage for all the possible write handlers in kernel is
very poor and not likely to improve.

-- 
Cyril Hrubis
chrubis@suse.cz