[LTP] Fwd: Identify current test coverage and clarify contribution opportunities

[Luigi Pellecchia]

Hi everyone,
First of all, thanks for your valuable feedback.
I'm sorry for the late reply, I was travelling.
I understand all your points and concerns.
I sent this video to you pushed by this piece of the project documentation:

https://linux-test-project.readthedocs.io/en/latest/developers/test_case_tutorial.html

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Find an untested System call
<https://linux-test-project.readthedocs.io/en/latest/developers/test_case_tutorial.html#find-an-untested-system-call>

Try to find an untested system call which has a manual page (i.e. man
syscall produces a result). It is a good idea to Git-clone the latest
kernel man-pages repository.

git clone git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git

At the time of writing, the difference between the latest man-pages release
and the HEAD of the repository (usually the latest commit) is well over 100
commits. This represents about 9 weeks of changes. If you are using a
stable Linux distribution, your man-pages package may well be years old. So
as with the kernel, it is best to have the Git repository as a reference.

You could also find a system call with untested parameters or use whatever
it is you are planning to use the LTP fo
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
from the one you as a project are asking people to discover untested
syscall.
This raised a flag in my mind because anytime someone is doing that
analysis for a selected syscall we quickly lose his contribution because
his effort is not tracked anywhere.
BASIL can help keep track of any contribution and an idea can be to start
tracking only new tests to learn ourselves if this can give some benefits
to the project.

Moreover I'm involved in the ELISA (Linux Foundation) community and we are
going to spin up an instance of this tool.
There are several companies involved in this project that are actively
working on Safety Critical applications that can take advantage from such
pedantic analysis of test coverage against man page and it could be a good
opportunity to onboard new contributors.

With that said, I understand that LTP is not only testing syscall and that
you see more value in other tests, like the regression ones.
As in ELISA we are going to start an analysis, we'd like to leverage your
expertise and suggestions to see if there is something in particular that
can give back to you some important information.
I understand that you are looking for input data and not more work to do
because of workforce constraints, so please, guide me in the right
direction.

To reply to Cyril,
yes a manual effort is required to map snippets to test cases.
To keep the mapping stable an idea can be to select a commit hash of the
man page for a selected version of the syscall.
It is also true that if a man page changes, that can be for a good reason
and probably it is better to check the mapping.
The effort of refining the effort is really limited because the User
simplifies this process.

I'm sorry for the nanosleep example. I choose a simple syscall to perform a
quick analysis.

I'll try to join the next meeting, thanks for the invitation.
I hope you see this thread as a constructive proposal and not something
that can deviate your effort from your project goal.

Cheers

--
Luigi

On Wed, Apr 17, 2024 at 1:43 AM Li Wang <liwang@redhat.com> wrote:

> On Wed, Apr 17, 2024 at 3:43 PM Andrea Cervesato via ltp <
> ltp@lists.linux.it>
> wrote:
>
> > Hi Luigi,
> >
> > On 4/17/24 08:32, Luigi Pellecchia wrote:
> > > Hi all,
> > > I'm Luigi Pellecchia, a Principal SW Quality Engineer at Red Hat.
> > > I developed an Open Source Software Quality Management Tool, named
> "BASIL
> > > The FuSa Spice" that can help the LTP keep track of the test case
> > coverage
> > > against man pages and to clarify contribution opportunities to new
> > members.
> > > I prepared an initial demo I shared on LinkedIn at
> > >
> >
> https://www.linkedin.com/posts/luigi-pellecchia_how-basil-can-help-linux-test-project-to-activity-7186248090129956864-d-vC?utm_source=share&utm_medium=member_desktop
> > Thanks for sharing. I watched the demo and it can be useful in certain
> > circumstances, but in others it could be tricky. I will explain myself
> > at the end of the email.
> > > This tool is under the hood of ELISA (Linux Foundation) github at
> > > https://github.com/elisa-tech/BASIL
> > >
> > > Any feedback will be greatly appreciated
> > >
> > > Please let me know if you have any questions, or if you'd like to see a
> > > live demo of the tool.
> > >
> > > Regards
> > >
> > > --
> > > Luigi
> > >
> > One of the things we noticed is that man pages are not updated too often
> > and some kernel features are not covered by the manuals. This would be
> > an issue when it comes to calculate tests coverage and some of us think
> > that this approach could be hard to follow.
> >
> > Also, LTP has a lack of men power nowadays (especially when it comes to
> > reviews) which means coverage report is good to track LTP status, but we
> > would take ages to implement remaining kernel tests. There's an internal
> > discussion in LTP about bringing more people inside the project and we
> > are trying to move forward to achieve this goal (see new documentation).
> > That would help the whole project.
> >
>
> So maybe giving a 'coverage report' to LTP is more valuable than the tool
> itself?
>
> At least it can reflect something we weren't aware of, we won't follow up
> the report one by one dogmatically, but pick up something useful.
>
>
>
> >
> > We have a monthly LTP meeting. The next will probably be the 8th of may
> > (I need to send the email yet). It's public, so feel free to join if you
> > want.
> >
> > Best regards,
> > Andrea Cervesato
> >
> >
> > --
> > Mailing list info: https://lists.linux.it/listinfo/ltp
> >
> >
>
> --
> Regards,
> Li Wang
>
> --
> Mailing list info: https://lists.linux.it/listinfo/ltp
>