Proposal: Add support for TestCase.meta_data

[bryanbcook]

Several test frameworks support representing additional meta-data in the test result output. They go by different names (traits, groups, properties, etc) but would make sense to represent as a map of unique values as part of the TestCase:

declare class TestCase {
  name: string;
  total: number;
  passed: number;
  failed: number;
  errors: number;
  skipped: number;
  duration: number;
  status: string;
  failure: string;
  stack_trace: string;
  meta_data: Map<string,string>;
  steps: TestStep[];
}

My experience is mostly centered in xUnit, NUnit, MSTest and SpecFlow (Cucumber), so I would value input for the others listed here. I've done some research to illustrate how this could be included across the various platforms:

• xUnit: Traits

  Traits are set at the test level.

    <!-- testResults.xml -->
    ....
    <test>
        <traits>
            <trait name="Category" value="Bug" />
            <trait name="Bug" value="1234" />
        </traits>
    </test>
  ...       

  Represented as meta_data:

  testcase.meta_data.set("Category", "Bug");
  testcase.meta_data.set("Bug", "1234");

• jUnit: Properties

  Not all jUnit runners allow you to specify properties. Playwright and PyTest seem to do this out-of-the-box. The surefire maven plugin appears to set properties based on parsing test output.

  It would seem that properties can be set in either the testsuite or testcase level. Values at the testcase level can override the suite level. The testcase would inherit the meta-data of the parent suite.

  <!-- testResults.xml -->
  <testsuites>
      <testsuite name="Tests">
           <properties>
                <property name="key1" value="value1" />
                <property name="key2" value="value2" />
           </properties>
           <testcase name="testExample" classname="Tests">
                <properties>
                     <property name="key1" value="value1-override" />
                </properties>
           </testcase>
       </testsuite>
  </testsuites>

  Represented as meta_data:

  testcase.meta_data.set( "key1", "value1" );
  testcase.meta_data.set( "key2", "value2" );
  testcase.meta_data.set( "key1", "value1-override" );

• TestNG: Groups

  TestNG supports the concept of "groups" which allow you to categorize tests and limit execution at runtime. Groups appear in the test xml output.

  <!-- testResults.xml -->
  <testng-results>
    <suite name="Suite1">
      <groups>
        <group name="group1">
          <method signature="com.test.TestOne.test2()" name="test2" class="com.test.TestOne"/>
          <method signature="com.test.TestOne.test1()" name="test1" class="com.test.TestOne"/>
        </group>
        <group name="group2">
          <method signature="com.test.TestOne.test2()" name="test2" class="com.test.TestOne"/>
        </group>
      </groups>
      <test name="test1">
        <class name="com.test.TestOne">
          <test-method status="FAIL" signature="test1()" name="test1" ...>
          </test-method>
          <test-method status="PASS" signature="test2()" name="test2" ...>
          </test-method>
          <test-method status="PASS" signature="setUp()" name="setUp" ...>
          </test-method>
        </class>
      </test>
    </suite>
  </testng-results>

  Given that groups are not represented as a key-value-pair, they might have to be represented as a comma-delimited value, keys with empty values or both:

  testcase.meta_data.set("groups", "group1,group2");
  testcase.meta_data.set("group1", "");
  testcase.meta_data.set("group2", "");

• Cucumber: Tags

  The Cucumber JSON format includes a Tag element -- example here -- at the Feature and Scenario level. Tags are often used as runtime hooks or as categorization. The tag value in the gherkin output indicates the line number. Based on the un-informed sample I found, it would appear that the tags from the Feature are automatically propagated to the Scenario.

  [
     { 
        "id": "file-name-identifier",
        "tags": [
              {  "name": "@featureTag", "line": 1 }
         ],
         "name": "name",
         "keyword": "Feature",
         "elements": [
              {
                 "id": "file-name-identifier;scenario-name",
                 "tags": [
                     { "name": "@fast", "line": 21 },
                     { "name": "@checkout", "line": 21 },
                     { "name": "@featureTag", "line": 1 },
                     { "name": "@testCase=1234", "line": 21 }
                 ]
                 "keyword": "Scenario Outline",
                 "steps": [ 
                  ]
              }
         ]
      }
  ]

  I don't know if the "line" value would have meaning to those reading the results, but it may be valuable to include both the line number and a comma-delimited value for tags:

  Tags will be processed as follows:

  • Tags that contain an equals operator, such as @testCase=1234, will be parsed into key-value pairs.
  • Tags without an equals operator will be added to the meta with an empty value.
  • All processed tags will be added as a comma-delimited string, which allows users to iterate over the tags.
  • The original tags will be available as tagsRaw.

  testcase.meta_data.set("tags", "fast,checkout,featureTag,testCase");
  testcase.meta_data.set("tagsRaw", "@fast,@checkout,@featureTag,@testCase=1234");
  testcase.meta_data.set("fast", "");
  testcase.meta_data.set("checkout", "");
  testcase.meta_data.set("featureTag", "");
  testcase.meta_data.set("testCase", 1234");

• Mocha: Tag naming conventions

  It does not appear that mocha has a native tagging mechanism capability, but their documentation recommends using @tag or #tag declarations and using grep to perform the filtering.

   // app.specs.js
   describe('app', function() {
     describe('GET /download/:file', function() {
       it('should respond with the file @slow', function() {
         
       })
     })
   })

  Following mocha's wiki guidance, perhaps it would make sense to parse the # and @ attributes as meta-data, and repeat the above comma-delimited approach suggested above:

  testcase.meta_data.set("tags", "slow");
  testcase.meta_data.set("tagsRaw", "@slow");
  testcase.meta_data.set("slow", "");

  Tags that contain an equals operator (=) will be parsed into unique key-value pairs:

    describe('app', function() {
      describe('GET /download/:file', function() {
        it('should respond with the file @testCase=1234', function() {
          
        })
      })
    })

  testcase.meta_data.set("tags", "testCase");
  testcase.meta_data.set("tagsRaw", @testCase=1234");
  testcase.meta_data.set("testCase", "1234");

I'm happy to open or contribute to a PR for this.

[bryanbcook]

bump @ASaiAnudeep @uDuCkV @nafeger @jain0nikhil -- can we complete #35 ?