[AWS][WAF] System test failing for WAF with 8.14.0

[agithomas]

Steps to Recreate the issue

1. Update manifest.yml as

  kibana:
    version: "^8.14.0"

2. elastic-package test system -d waf -v

Output

Error: error running package system tests: could not complete test run: found ignored fields in data stream logs-aws.waf-ep: [aws.waf.terminating_rule_match_details.location aws.waf.non_terminating_matching_rules.ruleMatchDetails.location aws.waf.rule_group_list.nonTerminatingMatchingRules.ruleMatchDetails.location]. Affected documents: [
  {
    "_id": "996f88ec15-000000000000",
    "@timestamp": "2019-12-13T23:40:12.771Z",
    "ignored_field_values": null
  },
  {
    "_id": "996f88ec15-000000000826",
    "@timestamp": "2020-06-17T01:26:32.516Z",
    "ignored_field_values": null
  },
  {
    "_id": "996f88ec15-000000001695",
    "@timestamp": "2020-06-17T02:43:30.888Z",
    "ignored_field_values": null
  },
  {
    "_id": "996f88ec15-000000002947",
    "@timestamp": "2019-12-13T23:40:12.771Z",
    "ignored_field_values": null
  }
]

Initial Assumption / Observation

I guess this may be related to the recently added feature in elastic-package
- System tests fail on presence of ignored fields. cc @jsoriano

Impacted PRs

• Add AWSHealth Integration #10119 . Please refer the buildkite logs.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[agithomas]

With

skip_ignored_fields:
  - aws.waf.terminating_rule_match_details.location
  - aws.waf.non_terminating_matching_rules.ruleMatchDetails.location
  - aws.waf.rule_group_list.nonTerminatingMatchingRules.ruleMatchDetails.location

added to test-default-config.yml for AWS/WAF

Reference : PR

With this change:

--- Test results for package: aws - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬─────────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │    TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼─────────────────┤
│ aws     │ waf         │ system    │ default   │ PASS   │ 1m43.569735941s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴─────────────────╯
--- Test results for package: aws - END   ---
Done

[agithomas]

PR LInk to Resolve the issue : #10401