[LTP] [PATCH] cve-2017-17052: tolerate ENOMEM during test

[Petr Vorel]

Hi Li,

[ Cc others as it's a pre-release ]

> On Thu, Jan 29, 2026 at 6:00 PM Li Wang via ltp <ltp@lists.linux.it> wrote:

> > As each iteration of mmap_thread() grabs a fresh 16 MiB MAP_POPULATE
> > region and never releases it. As the loop runs, those regions
> > accumulate consuming both virtual address space and committed physical
> > memory right away instead of lazily.

> > Easily mmap() fails with ENOMEM on smaller/limit RAM resource system.

> > Error Log:

> >   cve-2017-17052.c:48: TBROK: mmap((nil),16777216,PROT_READ(1),32802,-1,0) failed: ENOMEM (12)
> >   tst_test.c:479: TINFO: Child process reported TBROK killing the test
> >   tst_test.c:1909: TINFO: Killed the leftover descendant processes

> > Consider there is no practical upper bound on this allocation pattern,
> > so setting .test.min_mem_avail may not helps. Here we just tolerate
> > ENOMEM during the mmap_thread() looping.

> > Signed-off-by: Li Wang <liwang@redhat.com>
> > ---
> >  testcases/cve/cve-2017-17052.c | 9 ++++++++-
> >  1 file changed, 8 insertions(+), 1 deletion(-)

> > diff --git a/testcases/cve/cve-2017-17052.c b/testcases/cve/cve-2017-17052.c
> > index 700eb782e..61032c197 100644
> > --- a/testcases/cve/cve-2017-17052.c
> > +++ b/testcases/cve/cve-2017-17052.c
> > @@ -44,9 +44,16 @@ static void cleanup(void)

> >  static void *mmap_thread(void *arg)
> >  {
> > +       void *ptr;
> > +
> >         for (;;) {
> > -               SAFE_MMAP(NULL, 0x1000000, PROT_READ,
> > +               ptr = mmap(NULL, 0x1000000, PROT_READ,
> >                                 MAP_POPULATE|MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
> > +
> > +               if ((ptr == MAP_FAILED) && (errno == ENOMEM)) {
> > +                       usleep(1000);
> > +                       continue;
> > +               }

> Oops, I forgot to handle the other failures, so the patch should be:

> --- a/testcases/cve/cve-2017-17052.c
> +++ b/testcases/cve/cve-2017-17052.c
> @@ -44,9 +44,19 @@ static void cleanup(void)

>  static void *mmap_thread(void *arg)
>  {
> +       void *ptr;
> +
>         for (;;) {
> -               SAFE_MMAP(NULL, 0x1000000, PROT_READ,
> +               ptr = mmap(NULL, 0x1000000, PROT_READ,
>                                 MAP_POPULATE|MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
> +
> +               if (ptr == MAP_FAILED) {
> +                       if (errno == ENOMEM) {
> +                               usleep(1000);
> +                               continue;
> +                       }
> +                       tst_brk(TBROK | TTERRNO, "Unexpected mmap() error");
> +               }
>         }

>         return arg;

Patched version LGTM.
Reviewed-by: Petr Vorel <pvorel@suse.cz>

Out of curiosity how much RAM is not enough for this? I tried 400 MB + eating
other RAM but was not able to trigger.

Kind regards,
Petr