twister: shell harness with commands alongside the harness_config

[gchwier]

Allow user to add shell commands in testcase/sample yaml file alongside the harness_config like in the console harness.

ref: #85033

Instead of having shell commands in a separate file, one can add:

        harness_config:
          shell_commands:
          - command: "kernel cycles"
            expected: "cycles: .* hw cycles"
          - command: "kernel version"
            expected: "Zephyr version .*"
          - command: "kernel sleep 100"

Added examples in samples/subsys/testsuite/pytest/shell and updated existing samples.
To test you can run:

$ZEPHYR_BASE/scripts/twister -vv -T samples/subsys/testsuite/pytest/shell -G --no-clean -ll debug

$ZEPHYR_BASE/scripts/twister -vv -T samples/drivers/flash_shell -p native_sim -c -ll debug

$ZEPHYR_BASE/scripts/twister -vv -T samples/subsys/shell/shell_module -s sample.shell.shell_module -p native_sim -c -ll debug

@nashif , @bjarki-andreasen answering that comment:
#85033 (comment)
in one sample I added example usage of anchors in yaml files, it works!

  sample.harness.shell:
    harness_config:
      shell_commands: &kernel_commands     <-------
      - command: "kernel cycles"
        expected: "cycles: .* hw cycles"
      - command: "kernel version"
        expected: "Zephyr version .*"
      - command: "kernel sleep 100"
  sample.harness.shell.vt100_colors_off:
    harness_config:
      shell_commands: *kernel_commands   <------

[nashif]

why is this being removed? what if I want to have multiple scenarios with different files, i.e not using the default test_shell.yml?

[gchwier]

You can have multiple test configurations with different shell commands in harness_config. However I understand the scenario, to override during testing the test_shell.yml (but you can also change testcase.yml).

I can restore that option to load command from given file or from harness_config.

I though also about more generic approach, to be able to include yaml snippet, e.g. using https://pypi.org/project/pyyaml-include/, than you can keep test configurations in separate files if testcase yaml is to large.

[nashif]

to override during testing the test_shell.yml (but you can also change testcase.yml).

not sure what you mean here, this is not just during testing...

test1:
  harness_config:
    shell_param_file: x.yml
test2:
  harness_config:
    shell_param_file: y.yml

[gchwier]

so as I wrote: You can have multiple test configurations with different shell commands in harness_config,
it can be replaced with:

test1:
  harness_config:
    shell_commands:
      - command1
        expected1
      - command2
test2:
  harness_config:
    shell_commands:
      - command2
        expected2
      - command3
        expected3

IMO keeping commands in a separate YAML file introduces unnecessary complexity. It will make it harder to understand what is going on in the tests. It's better to change it now, while no tests are using this feature. Later, we will need to consider backward compatibility.

[de-nordic]

Flash shell looks ok.

[bjarki-andreasen]

I like it :)

[nashif]

need to add back reading test data from files. As I said here: #85033 (comment) and here #85033 (comment) I can live with testdata in the testcase.yaml file for small sets, but this needs to scale beyond just a few entries and testdata in general should be seperated from test definition and configuration.

Now it basically supports reading from one single file that is pre-defined, which makes it worse that it was before.

[gchwier]

restored option to read commands from file. Changed shell_params_file name to shell_commands_file.
And for current implementation shell_commands option takes precedence over shell_commands_file.

@nashif Please review

[gchwier]

rebased and added new lines to twister.rst after code-block... some problems with building docs :(

[gchwier]

I don't really know why it is failing on sample.pytest.shell.vt100_colors_off on qemu_cortex_a53/qemu_cortex_a53/smp,
locally it works

[nashif]

I don't really know why it is failing on sample.pytest.shell.vt100_colors_off on qemu_cortex_a53/qemu_cortex_a53/smp,
locally it works

smp is sporadic in CI (and locally), I would just exclude this one or we will be hitting it in every other PR.

[gchwier]

smp is sporadic in CI (and locally), I would just exclude this one or we will be hitting it in every other PR.

added not CONFIG_SMP in shell sample ... however I do not understand, where is the problem (failed only on qemu_cortex_a53/smp, with vt100_colors_off .. and cannot reproduce it on local machine)