Draft Proposal of Flow Testing - Phase 1

[HiroyasuNishiyama]

Flow Testing - Phase 1

Summary

This feature allows a flow developer to create test cases for their flows that can be run within the editor or by a CLI tool.

Flow testing design has been discussed in this design note. This discussion covers wide range of topics required to make flow testing for Node-RED but takes longer time to complete detailed design and implement all the features at once.

Thus, in this document, we define the minimum required functionality that the first flow tester implementation should support. Other features discussed in the above design note will be added in following phases.

Authors

• Hiroyasu Nishiyama

  based on initial design note by following authors:

• Kazuhito Yokoi

• Yuma Matsuura

• Brian Innes

• Nick O'Leary

• Hiroyasu Nishiyama

Requirements

In traditional programming there are many test frameworks and the functionality needed to property test code is quite well understood (visit Mocha to see the typical features offered by a Javascript test framework). However, the use of test tools in low-code development environments is less well defined, but the need for a thorough testing capability still exists.

• The testing function should be able to be run from within the Node-RED editor or from a command line, so Node-RED application testing can be run as part of an automated delivery pipeline
• The tests should be created using low-code methods, similar to how applications are created using Node-RED. Ideally creating test cases should not be a coding activity
• There should be the ability to create a "test suite", containing multiple test scenarios rather than a single test
• Test definitions/implementation should not alter a flow in any way that will impact the functionality or performance during normal running
• Test artifacts and configuration should be able to be easily identified and removed from a flow when deploying to a resource constrained device, without impacting the flow.

Out of scope of this design

• Command Line testing

  This proposal does not include command line interface for flow testing. This requires discussion on headless execution of flows and tools integration.

• Single Node testing

  This proposal does not cover single node unit tests as currently implemented by node developer with mocha and node-red-node-test-helper module.

  But, with this proposal, we can treat "Single Node" as a flow consisting of one node.

• SUBFLOW testing

  Testing internal logic of SUBFLOW should also be possible from a unit
  testing perspective. However, the problem is that there is no
  mechanism for executing the SUBFLOW template in current runtime.
  So, this proposal do not address SUBFLOW internals testing.

  Testing of SUBFLOW in the main flow should be done in the same way as
  a normal node.

• Dashboard testing

  This design note do not cover testing UI and related flow of Node-RED dashboard.

Design

The rest of this document sets out the proposed design of the testing feature.

High Level Concept

This is an outline of the proposed testing approach within the Node-RED editor.

• A user can define a set of named test suites that consist of individual test cases

• Each test case can be configured with desired setup/cleanup phase:

  • setup - this phase is triggered when test starts. This phase
    can be used to setup environments for test execution. For
    example, preparing a data file used for

  • cleanup - this phase is triggered when test ends. This phase
    can be used to cleanup environments for test execution.

• For each test case, every node can be configured with a desired behaviour. This will be split into three phases:

  • beforeNode - this phase is triggered when a node receives a message, but before the node is given the message. This phase can be used to validate the message arriving. In this phase, the test can:

  • testNode - this phase, if defined, is used instead of the actual runtime node. This allows the node to be stubbed out, for example a node that writes to database or interacts with an external system.

  • afterNode - this phase is triggered whenever the node sends a message, but before the next nodes receive it. This phase can be used to validate the message(s) being sent by the node.

  These phases are defined as events for the test components. And each event defines a list of actions that are triggered by the event detection. The following can be examples of actions.

• wait - wait for specified time,

• set - set message properties, environment variables or context values,

• match - match message properties,

• send - pass input message to output port,

• recv - receive message from input port,

• function - execute JavaScript code for complex preparation and checking,

With these phases and actions, a wide range of basic test scenarios will be possible.

• the before/afterNode phases allow for checks at any point in the flow to ensure a message looks right

• the testNode phase allows for nodes to be stubbed out with test-scenario specific behaviour

• For flows that expose an API, such as HTTP In -> HTTP Response flows, there would be two different possible approaches to testing it:

  1. the testNode phase of the HTTP In node to start the test, and the beforeNode of the HTTP Response node used to verify the result.
  2. an entirely separate Test flow could be created with Inject -> HTTP Request with the testNode of the Inject node set to start the test, and the afterNode of the HTTP Request node used to verify the response.

  Both would be valid approaches to writing the test - the main point is this design has the flexibility to support either style.

• For flows that read a file, transform it, then write the output to a file. The setup phase creates a file for input, the flow is executed with check by before/afterNode for each nodes, then the resulting file will be checked by cleanup phase.

Running the tests

Running each test case will require starting from a known clean state. For example, if a flow contains a Join node and a test case causes some but not all messages to be passed to it, the node will need to be completely reset before the next test case can start.

Testing tab in the sidebar

The testing tab is located in the sidebar of the flow editor.

• Execute button: the button to start test cases

• Scenario tree: The tree view consists of a scenario and nodes which have test cases. The tree view is automatically synchronized with settings on node properties. On the UI, users can see an overview of the settings and status of the test cases.
  In scenario tree, test suites menage multiple test scenarios.

Following figure shows an example of proposed test sidebar.
New test suite and scenario can be added by +add button on tree view.

By pressing config button on a scenario, following settings panel is displayed.

In this settings panel, users can select test scenario and fill test setup and cleanup actions on each tab.

Timeout value or other configuration options may be provided in special configuration tab.

Flow testing UI for nodes

When flow testing is enabled, new Test tab will be added to settings panel of each nodes.

In this settings panel, user can select test scenario and fill receive, stub, and send actions on each tab.

Test Events & Actions

Test events are defined for test case and test target nodes. List of actions are defined for defined the test events.

Following are test events and test actions proposed as initial phase of flow testing implementation. These will be extended by the following phases to support wider aspect of flow testing.

Test Events

• Test case events - following events are defined for test case:

  • setup: executed at start of the test case,

  • cleanup: executed at end of the test case,

• Node events - following events are defined for each node:

  • receive: executed at receive of incoming message by the node,

  • stub: executed after receive event instead of message processing of the node,

  • send: executed after sending message from the node to output port.

Test Actions

• Common actions

  • wait - wait for specified time,

  • function - write complex testing using JavaScript code. JavaScript code in this action will be executed in vm environment and provide support of special functions that supports flow testing.

• Test case actions

  • set - set context or environment variables.

• Node actions

  • match - match message property with specified value, or expression,

  • set - set message property or context,

  • send - send message to output port,

  • recv - receive message from input port.

Flow data

• The test configurations are included in the main flow file. Therefore, there is no need to change the Storage API in the first implementation.
• Export dialogue will have a checkbox to indicate whether to delete the test information.

Flag to enable flow testing

There are following variety to enable flow testing to make Node-RED run in test mode.

1. Flag in an environment variable

2. Flag in settings.js

3. command line option

Implementation

Flow testing feature will be provided as an addon to Node-RED using plugin interface.

History

• 2020-06-28 - Initial proposal

[HiroyasuNishiyama]

Break down required work for implementing flow testing plugin

The following is a list of items that are expected to be required for implementing flow tester.
Please let me know if there are other items that must be added or changes on items.

• UI

  1. Sidebar UI

     • manage list of test cases

     • execute tests

     • display results & some logging info

  2. Settings UI for Settings & Cleanup

     • manage actions list for test case events

     • manage configurable options

  3. Settings UI for All Nodes in Workspace

     • manage actions list for test case events

  4. Plugin mechanism for (2)

     • need a way to add new tab for settings menu of each workspace node

  5. Action List edit UI for (2) and (3)

     • actions list edit interface

• Runtime

  1. Execution mechanism of test actions

     • send a message to/from a node

     • receive a message to/from a node

     • stub execution

     • setting context/message/env values

     • catching errors & timeout

  2. Editor API

     • run test (Editor->Runtime)

     • notify test result (Runtime->Editor)

  3. Scheme for saving testing information

     • new config nodes for holding test informationså

  4. Clearing runtime state of Node-RED
     (may be not needed in phase 1)

[knolleary]

I think there are a couple basic design choices that need agreeing before we get too deep into this work.

Where is the test configuration stored?

We are agreed it is stored in the flow file, but we have not discussed how that works. I have been assuming that each node/flow would get an optional extra property (for example t for tests). And that to do that we would need:

Adding extra properties to nodes in the flow file.
Plugins need a way to include extra properties on node objects. When the editor exports a node configuration, it needs to be told > to include the test information.
(from this comment)

Your list of items above mentions configuration nodes.

I do not have a strong preference for the approach - but we do need to identify it with a rough outline of what it looks like.

I will start a separate discussion item for this piece.

How do tests run?

This is the hardest part of the whole feature - how do the tests actually run? I suspect this will lead to needing some other hooks/apis in the core to allow the Test framework to take over properly. This is the area I will need to look at in detail.

---

At some point, we then need to create a proper task list - with the tasks being smaller pieces of deliverable work to help us track progress and see who is working on what bit. We haven't done this approach before, but I'm keen to find a way that works for us.

[knolleary]

@HiroyasuNishiyama I've started the discussion on how test configs are stored here: #3 (not sure if you're getting notified of these new discussion items...)