[LTP] LTP in valgrind :)

[Bird, Tim]

> -----Original Message-----
> From: ltp <ltp-bounces+tim.bird=sony.com@lists.linux.it> On Behalf Of Mark Wielaard
> Hi Cyril, Hi ltp hackers,
> 
> On Wed, 2025-05-07 at 17:29 +0200, Cyril Hrubis wrote:
> > > > > >
> > > > > Maybe we can even add an option to the test library to supress the
> > > > > messages in output, that would be fairly simple.
> > > >
> > > > @Martin @Mark: feel free to comment what we can do for you :).
> > > > Whole thread:
> > > > https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#t
> > >
> > > Nice to see you noticed!  Primarily we're trying to identify
> > > valgrind gaps and bugs.  For our use-case it's good to have
> > > reproducible test runs in a sense of identical test results for
> > > repeated test runs.  We got pretty close to this by only taking
> > > into account the testcase summaries at the very end of the test
> > > logs.  Any steps to get closer to reproducible test runs on the
> > > LTP side are interesting for us, and we'll try to reflect those!
> >
> > I've proposed a simple solution that would cut the message from tests
> > only to contain the line numbers and results in:
> >
> > https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#m98317262b25ede62704b005fcefc63c62198bb90
> >
> > If that works for you we can get this in LTP before the next release.
> >
> 
> That looks pretty interesting, I just tried it and it turns:
> 
> $ ./testcases/kernel/syscalls/dup/dup01
> tst_tmpdir.c:316: TINFO: Using /tmp/LTP_dupbmNNGn as tmpdir (tmpfs filesystem)
> tst_test.c:1907: TINFO: LTP version: VALGRIND_3_25_0-10-g0ce068434ec3
> tst_test.c:1911: TINFO: Tested kernel: 6.13.11-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 10 19:02:09 UTC 2025 x86_64
> tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/6.13.11-200.fc41.x86_64/build/.config'
> tst_test.c:1729: TINFO: Overall timeout per run is 0h 00m 30s
> dup01.c:24: TPASS: dup(fd) returned fd 4
> dup01.c:27: TPASS: buf1.st_ino == buf2.st_ino (6921515)
> 
> Summary:
> passed   2
> failed   0
> broken   0
> skipped  0
> warnings 0
> 
> into:
> 
> $ LTP_REPRODUCIBLE_OUTPUT=1 ./testcases/kernel/syscalls/dup/dup01
> tst_tmpdir.c:316:
> tst_test.c:1907:
> tst_test.c:1911:
> tst_kconfig.c:88:
> tst_test.c:1729:
> dup01.c:24:
> dup01.c:27:

Don't you want to keep the status or result (TINFO and TPASS)?

This shouldn't change from run-to-run unless the test is flaky.

 -- Tim


> Summary:
> passed   2
> failed   0
> broken   0
> skipped  0
> warnings 0
> 
> So that still shows the "code flow" and the test results Summary (all
> output goes to stderr). I think this would indeed be useful to compare
> if a testcase runs the same "native" and under valgrind.
> 
> Thanks,
> 
> Mark
> 
> --
> Mailing list info: https://lists.linux.it/listinfo/ltp