lib: redefine the overall timeout logic of test

This patch revises the logic for managing test timeout and runtime
limits, introducing a more structured approach. The key changes include
renaming max_runtime to runtime for clarity and adding overall_time to
represent the total time limit for a test. This ensures better separation
of test phases (setup, execution, and cleanup) and clearer timeout management.

Key Concepts:

 timeout: Defines the maximum time allowed for the entire test, including
          setup, execution, and cleanup, when no explicit runtime is set.
	  But if a runtime is explicitly defined and tst_remaining_runtime()
	  is used, the timeout applies only to the setup and cleanup phases,
	  as the runtime controls the test execution duration.

 runtime: The maximum runtime of the test's main execution loop, used
          in tests that call tst_remaining_runtime(). It ensures the
	  main execution runs for a fixed duration, regardless of kernel
	  configuration (e.g., debug kernel).

Overall timeout is structured as follows:

| -- (default_30s + timeout) * timeout_mul -- | -- runtime * runtime_mul -- |

And, the TST_UNLIMITED_RUNTIME constant has been deprecated and replaced
by TST_UNLIMITED_TIMEOUT. This change simplifies the handling of unlimited
execution scenarios while aligning the terminology with the new timeout model.

Suggested-by: Cyril Hrubis <[email protected]>
Signed-off-by: Li Wang <[email protected]>
Reviewed-by: Cyril Hrubis <[email protected]>
Reviewed-by: Petr Vorel <[email protected]>

doc/old/C-Test-API.asciidoc

@@ -94,26 +94,32 @@ in range of [0, '.tcnt' - 1].
 
 IMPORTANT: Only one of '.test' and '.test_all' can be set at a time.
 
-Each test has a limit on how long it can run and the limit composes of two
-parts max_runtime and timeout. The max_runtime is a limit for how long can the
-'.test_all' or a set of '.test' functions take and the timeout is static part
-that should cover the duration of test setup and cleanup plus some safety.
+Each test has a limit on how long it can run, composed of two parts: 'runtime'
+and 'timeout'. The 'runtime' is the limit for how long the '.test_all' or a set
+of '.test' main functions can execute. The 'timeout', on the other hand, applies
+to the total time for setup, single test function invocation, cleanup, and some
+additional safety margin. If test without an explicit 'runtime', the 'timeout'
+governs the entire test duration.
+
+The timeout timer is also reset on each subsequent iteration with the test -i
+parameter, variants or .all_filesystems.
 
 Any test that runs for more than a second or two has to make sure to:
 
-- set the runtime either by setting the '.max_runtime' in tst_test or by
-  calling 'tst_set_max_runtime()' in the test setup
+- set the runtime by setting '.runtime' in tst_test and calling 'tst_set_runtime()'
+  to monitor the remaining runtime and ensure the test exits when the runtime
+  limit is reached.
 
-- monitor remaning runtime by regular calls to 'tst_remaining_runtime()' and
-  exit when runtime has been used up
+- set the timeout by setting '.timeout' in tst_test to limit the whole test
+  run that does not use 'tst_remaining_runtime()'.
 
-Test is free to exit before max_runtime has been used up for example when
+Test is free to exit before runtime has been used up for example when
 minimal number of iteration was finished.
 
 The limit is applied to a single call of the '.test_all' function that means
 that for example when '.test_variants' or '.all_filesystems' is set the whole
-test will be limited by 'variants * (max_runtime + timeout)' seconds and the
-test runtime will be likely close to 'variants * max_runtime' seconds.
+test will be limited by 'variants * (runtime + timeout)' seconds and the
+test runtime will be likely close to 'variants * runtime' seconds.
 
 [source,c]
 -------------------------------------------------------------------------------
@@ -401,13 +407,25 @@ WARNING: This function is not thread safe.
 
 [source,c]
 -------------------------------------------------------------------------------
-void tst_set_max_runtime(int max_runtime);
+void tst_set_timeout(int timeout);
 -------------------------------------------------------------------------------
 
-Allows for setting max_runtime per test iteration dynamically in the test 'setup()',
-the timeout is specified in seconds. There are a few testcases whose runtime
-can vary arbitrarily, these can disable timeouts by setting it to
-TST_UNLIMITED_RUNTIME.
+Allows for setting the entire timeout dynamically during the test setup(). The
+timeout is specified in seconds and represents the total time allowed for a single
+test iteration, including setup, runtime, and teardown phases.
+
+[source,c]
+-------------------------------------------------------------------------------
+void tst_set_runtime(int runtime);
+-------------------------------------------------------------------------------
+
+Allows for setting the runtime per test iteration dynamically during the test 'setup()'.
+The runtime is specified in seconds and represents the duration the test is allowed
+to execute its main workload, excluding setup and teardown phases.
+
+This function is useful for tests where the duration of the main workload can be
+controlled or needs to be adjusted dynamically. For example, tests that loop until
+the runtime expires can use this function to define how long they should run.
 
 [source,c]
 -------------------------------------------------------------------------------

doc/users/setup_tests.rst

@@ -87,14 +87,14 @@ See :master:`testcases/network/README.md`.
 Test execution time and timeout
 -------------------------------
 
-The limit on how long a test can run does compose of two parts: ``max_runtime``
+The limit on how long a test can run does compose of two parts: ``runtime``
 and ``timeout``. The limit does apply to a single test variant. That means, for
 example, that tests which run for all available filesystems will apply this
 limit for a single filesystem only.
 
-The ``max_runtime`` is a cap on how long the ``run()`` function can take and for
+The ``runtime`` is a cap on how long the ``run()`` function can take and for
 most testcases this part is set to zero. For tests that do run for more than a
-second or two the ``max_runtime`` has to be defined and the ``run()`` function
+second or two the ``runtime`` has to be defined and the ``run()`` function
 has to check actively how much runtime is left.
 
 Test runtime can be scaled up and down with ``LTP_RUNTIME_MUL`` environment

include/tst_test.h

@@ -234,7 +234,7 @@ struct tst_tag {
 
 extern unsigned int tst_variant;
 
-#define TST_UNLIMITED_RUNTIME (-1)
+#define TST_UNLIMITED_TIMEOUT (-1)
 
 /**
  * struct tst_ulimit_val - An ulimit resource and value.
@@ -435,13 +435,26 @@ struct tst_fs {
  *            The directory is created by the library, the test must not create
  *            it itself.
  *
- * @max_runtime: Maximal test runtime in seconds. Any test that runs for more
- *               than a second or two should set this and also use
- *               tst_remaining_runtime() to exit when runtime was used up.
- *               Tests may finish sooner, for example if requested number of
- *               iterations was reached before the runtime runs out. If test
- *               runtime cannot be know in advance it should be set to
- *               TST_UNLIMITED_RUNTIME.
+ * @timeout: Maximum allowable time in seconds for the entire duration of a test.
+ *           By default, the timeout limits the total time for setup, single test
+ *           function invocation, and cleanup phases. However, if .runtime is
+ *           explicitly set and tst_remaining_runtime() is used in the test's
+ *           main loop, the timeout then applies only to the setup and cleanup
+ *           phases, as the runtime separately limits the main test execution.
+ *           This ensures the test does not hang indefinitely, in the rare case
+ *           that the test timeout cannot be accurately determined, it can be
+ *           set to a sufficiently large value or TST_UNLIMITED_TIMEOUT to remove
+ *           the limit.
+ *
+ * @runtime: Maximum runtime in seconds for the test's main execution loop.
+ *           This should be set for tests that are expected to run longer
+ *           than a few seconds and call tst_remaining_runtime() in their
+ *           main loop to exit gracefully when the runtime is exceeded.
+ *           Tests may finish sooner if their task completes (e.g., reaching
+ *           a requested number of iterations) before the runtime runs out.
+ *           The runtime is fixed and does not scale with timeout multipliers
+ *           (e.g., TIMEOUT_MUL), ensuring consistent test duration across
+ *           different environments (e.g., debug vs. stock kernels).
  *
  * @setup: Setup callback is called once at the start of the test in order to
  *         prepare the test environment.
@@ -565,7 +578,8 @@ struct tst_fs {
 
 	const char *mntpoint;
 
-	int max_runtime;
+	int timeout;
+	int runtime;
 
 	void (*setup)(void);
 	void (*cleanup)(void);
@@ -662,6 +676,11 @@ void tst_reinit(void);
  */
 int tst_run_script(const char *script_name, char *const params[]);
 
+/*
+ * Sets entire timeout in seconds.
+ */
+void tst_set_timeout(int timeout);
+
 unsigned int tst_multiply_timeout(unsigned int timeout);
 
 /*
@@ -676,7 +695,7 @@ unsigned int tst_remaining_runtime(void);
 /*
  * Sets maximal test runtime in seconds.
  */
-void tst_set_max_runtime(int max_runtime);
+void tst_set_runtime(int runtime);
 
 /*
  * Create and open a random file inside the given dir path.

lib/newlib_tests/test_runtime01.c

@@ -25,6 +25,6 @@ static void run(void)
 
 static struct tst_test test = {
 	.test_all = run,
-	.max_runtime = 4,
+	.runtime = 4,
 	.test_variants = 2,
 };

lib/newlib_tests/test_runtime02.c

@@ -24,5 +24,5 @@ static void run(void)
 
 static struct tst_test test = {
 	.test_all = run,
-	.max_runtime = 5,
+	.runtime = 5,
 };

lib/newlib_tests/tst_fuzzy_sync01.c

@@ -248,5 +248,5 @@ static struct tst_test test = {
 	.test = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.max_runtime = 150,
+	.runtime = 150,
 };

lib/newlib_tests/tst_fuzzy_sync02.c

@@ -225,5 +225,5 @@ static struct tst_test test = {
 	.test = run,
 	.setup = setup,
 	.cleanup = cleanup,
-	.max_runtime = 150,
+	.runtime = 150,
 };

lib/newlib_tests/tst_fuzzy_sync03.c

@@ -99,5 +99,5 @@ static struct tst_test test = {
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = run,
-	.max_runtime = 150,
+	.runtime = 150,
 };