[LTP] [PATCH v2 1/1] net: tst_netload_compare(): Ignore performance failures

[Martin Doucha]

Hi,
Reviewed-by: Martin Doucha <mdoucha@suse.cz>

On 04. 01. 24 13:10, Petr Vorel wrote:
> Performance failures in tests which use tst_netload_compare() (tests in
> runtest/net.features) can hide a real error (e.g. test fails due missing
> required kernel module). Best solution would be to have feature tests
> (likely written in C API) and performance tests (the current ones).
> 
> But until it happens, allow to ignore performance failure with
> environment variable LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1.
> 
> Signed-off-by: Petr Vorel <pvorel@suse.cz>
> ---
> changes v1->v2:
> * Ignoring performance requires setting LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1
> (previously it was ignored by default, Martin), therefore rename the variable.
> * Variable uses LTP_ prefix (user defined), thus no need to whitelist it
> in tst_net.sh.
> 
> Martin also pointed out that message can be printed multiple times, but I
> don't think this is a problem (it's better that tester sees it).
> 
>   testcases/lib/tst_net.sh    | 19 +++++++++++++++----
>   testcases/network/README.md |  4 ++++
>   2 files changed, 19 insertions(+), 4 deletions(-)
> 
> diff --git a/testcases/lib/tst_net.sh b/testcases/lib/tst_net.sh
> index 2ed0e787f..6cf7f2fcb 100644
> --- a/testcases/lib/tst_net.sh
> +++ b/testcases/lib/tst_net.sh
> @@ -1,7 +1,7 @@
>   #!/bin/sh
>   # SPDX-License-Identifier: GPL-2.0-or-later
>   # Copyright (c) 2014-2017 Oracle and/or its affiliates. All Rights Reserved.
> -# Copyright (c) 2016-2023 Petr Vorel <pvorel@suse.cz>
> +# Copyright (c) 2016-2024 Petr Vorel <pvorel@suse.cz>
>   # Author: Alexey Kodanev <alexey.kodanev@oracle.com>
>   
>   [ -n "$TST_LIB_NET_LOADED" ] && return 0
> @@ -863,22 +863,33 @@ tst_netload()
>   # TIME: time that is compared to the base one
>   # THRESHOD_LOW: lower limit for TFAIL
>   # THRESHOD_HIGH: upper limit for TWARN
> +#
> +# Slow performance can be ignored with setting environment variable
> +# LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1
>   tst_netload_compare()
>   {
>   	local base_time=$1
>   	local new_time=$2
>   	local threshold_low=$3
>   	local threshold_hi=$4
> +	local ttype='TFAIL'
> +	local msg res
>   
>   	if [ -z "$base_time" -o -z "$new_time" -o -z "$threshold_low" ]; then
>   		tst_brk_ TBROK "tst_netload_compare: invalid argument(s)"
>   	fi
>   
> -	local res=$(((base_time - new_time) * 100 / base_time))
> -	local msg="performance result is ${res}%"
> +	res=$(((base_time - new_time) * 100 / base_time))
> +	msg="performance result is ${res}%"
>   
>   	if [ "$res" -lt "$threshold_low" ]; then
> -		tst_res_ TFAIL "$msg < threshold ${threshold_low}%"
> +		if [ "$LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE" = 1 ]; then
> +			ttype='TINFO';
> +			tst_res_ TINFO "WARNING: slow performance is not treated as error due LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1"
> +		else
> +			tst_res_ TINFO "Following slow performance can be ignored with LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1"
> +		fi
> +		tst_res_ $ttype "$msg < threshold ${threshold_low}%"
>   		return
>   	fi
>   
> diff --git a/testcases/network/README.md b/testcases/network/README.md
> index a0a1d3d95..e1b1296d6 100644
> --- a/testcases/network/README.md
> +++ b/testcases/network/README.md
> @@ -84,6 +84,10 @@ Where
>   Default values for all LTP network parameters are set in `testcases/lib/tst_net.sh`.
>   Network stress parameters are documented in `testcases/network/stress/README`.
>   
> +Tests which use `tst_netload_compare()` test also performance. They can fail on
> +overloaded SUT.  To ignore performance failure and test only the network functionality,
> +set `LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1` environment variable.
> +
>   ## Debugging
>   Both single and two host configurations support debugging via
>   `TST_NET_RHOST_RUN_DEBUG=1` environment variable.

-- 
Martin Doucha   mdoucha@suse.cz
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic