[LTP] LTP in valgrind :)

[Mark Wielaard]

Hi Cyril, Hi ltp hackers,

On Wed, 2025-05-07 at 17:29 +0200, Cyril Hrubis wrote:
> > > > > 
> > > > Maybe we can even add an option to the test library to supress the
> > > > messages in output, that would be fairly simple.
> > > 
> > > @Martin @Mark: feel free to comment what we can do for you :).
> > > Whole thread:
> > > https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#t
> > 
> > Nice to see you noticed!  Primarily we're trying to identify
> > valgrind gaps and bugs.  For our use-case it's good to have
> > reproducible test runs in a sense of identical test results for
> > repeated test runs.  We got pretty close to this by only taking
> > into account the testcase summaries at the very end of the test
> > logs.  Any steps to get closer to reproducible test runs on the
> > LTP side are interesting for us, and we'll try to reflect those!
> 
> I've proposed a simple solution that would cut the message from tests
> only to contain the line numbers and results in:
> 
> https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#m98317262b25ede62704b005fcefc63c62198bb90
> 
> If that works for you we can get this in LTP before the next release.
> 

That looks pretty interesting, I just tried it and it turns:

$ ./testcases/kernel/syscalls/dup/dup01 
tst_tmpdir.c:316: TINFO: Using /tmp/LTP_dupbmNNGn as tmpdir (tmpfs filesystem)
tst_test.c:1907: TINFO: LTP version: VALGRIND_3_25_0-10-g0ce068434ec3
tst_test.c:1911: TINFO: Tested kernel: 6.13.11-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 10 19:02:09 UTC 2025 x86_64
tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/6.13.11-200.fc41.x86_64/build/.config'
tst_test.c:1729: TINFO: Overall timeout per run is 0h 00m 30s
dup01.c:24: TPASS: dup(fd) returned fd 4
dup01.c:27: TPASS: buf1.st_ino == buf2.st_ino (6921515)

Summary:
passed   2
failed   0
broken   0
skipped  0
warnings 0

into:

$ LTP_REPRODUCIBLE_OUTPUT=1 ./testcases/kernel/syscalls/dup/dup01 
tst_tmpdir.c:316: 
tst_test.c:1907: 
tst_test.c:1911: 
tst_kconfig.c:88: 
tst_test.c:1729: 
dup01.c:24: 
dup01.c:27: 

Summary:
passed   2
failed   0
broken   0
skipped  0
warnings 0

So that still shows the "code flow" and the test results Summary (all
output goes to stderr). I think this would indeed be useful to compare
if a testcase runs the same "native" and under valgrind.

Thanks,

Mark