[LTP] [RFC PATCH 1/1] lib: Print in summary also tests not run at all

[Li Wang]

.

On Mon, Jan 31, 2022 at 3:21 PM Petr Vorel <pvorel@suse.cz> wrote:
>
> ...
> > > +++ b/lib/tst_test.c
> > > @@ -811,6 +811,9 @@ static void do_exit(int ret)
> > >             fprintf(stderr, "broken   %d\n", results->broken);
> > >             fprintf(stderr, "skipped  %d\n", results->skipped);
> > >             fprintf(stderr, "warnings %d\n", results->warnings);
> > > +           fprintf(stderr, "not run  %d\n", tst_test->tcnt -
> > > +                           results->passed - results->failed - results->broken -
> > > +                           results->skipped - results->warnings);
> > IMO, these values are not necessary related. Refer to quotactl01.c(
> > tst_test->tcnt is less than results->passed)
> Ah, correct. Also .test_all means tst_test->tcnt to be 0.
> So my patch is utterly wrong.
>
> > Also, if you want to add a new tag, you should define the situation for it.

That sounds like a complicated method, if we go add a new tag,
we have to count the times for each arbitrary 'return' of the test.
(Because it does not use TCONF)
This obviously brings more work to the author and is not recommended IMHO.

To strictly, every test should follow the principle of new LTP API,
to list all test items in the 'struct tcase{ ... }', and with no tst_test->tcnt
defined means that is only one test.

Based on these, I think the total test number of a testcase
(with 'tst_test->tcnt' defined) should be:
    (tst_test->tcnt * test_variants)

>
> What is my point: number of the defined tests does not mean they are all run.

Agree, at least it makes sense for those tests explicitly defined tcnt.

>
> Maybe printing number of the tests in the test (tst_test->tcnt or 1 when
> tst_test->test_all)?
>
> My point is to have an idea without looking into the code know how many tests
> have been skipped on certain setup.

Yes, I understand. So how about this way:

--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -785,6 +785,8 @@ static void print_failure_hints(void)
        print_failure_hint("known-fail", "hit by known kernel failures", NULL);
 }

+static unsigned int test_variants = 1;
+
 static void do_exit(int ret)
 {
        if (results) {
@@ -811,9 +813,11 @@ static void do_exit(int ret)
                fprintf(stderr, "broken   %d\n", results->broken);
                fprintf(stderr, "skipped  %d\n", results->skipped);
                fprintf(stderr, "warnings %d\n", results->warnings);
-               fprintf(stderr, "not run  %d\n", tst_test->tcnt -
-                               results->passed - results->failed -
results->broken -
-                               results->skipped - results->warnings);
+               if (tst_test->tcnt) {
+                       fprintf(stderr, "not-run  %d\n",
(tst_test->tcnt * test_variants) -
+                                       results->passed -
results->failed - results->broken -
+                                       results->skipped - results->warnings);
+               }
        }

        do_cleanup();
@@ -1529,7 +1533,6 @@ unsigned int tst_variant;
 void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
 {
        int ret = 0;
-       unsigned int test_variants = 1;

        lib_pid = getpid();
        tst_test = self;


--
Regards,
Li Wang