Receiving duplicated data from AWS cloudwatch

[stefnestor]

AWS Cloudwatch v8.6.2

Even though we now could filter the data by CloudAccountID, now when querying the data for "sum" in the 60s interval, Kibana would report double the value that shows up on Cloud watch. Looking closer at the documents we found that the documents are coming through duplicated except for one field "aws.dimensions.TransitGatewayAttachment", where one document has a value, and another document doesn't. ( We created a work-around to get the value deduplicated by adding a filter of type aws.dimensions.TransitGatewayAttachment : *. )

# only column differences are _id, aws.dimensions.TransitGatewayAttachment

@timestamp                  _id                      _index                                                     _score   agent.ephemeral_id                      agent.id                                agent.name        agent.type  agent.version   aws.cloudwatch.namespace    aws.dimensions.TransitGateway   aws.dimensions.TransitGatewayAttachment aws.transitgateway.metrics.BytesDropCountBlackhole.sum  aws.transitgateway.metrics.BytesDropCountNoRoute.sum    aws.transitgateway.metrics.BytesIn.sum  aws.transitgateway.metrics.BytesOut.sum aws.transitgateway.metrics.PacketDropCountBlackhole.sum aws.transitgateway.metrics.PacketDropCountNoRoute.sum   aws.transitgateway.metrics.PacketsIn.sum    aws.transitgateway.metrics.PacketsOut.sum   cloud.account.id      cloud.account.name    cloud.provider  cloud.region    data_stream.dataset    data_stream.namespace    data_stream.type    ecs.version elastic_agent.id                        elastic_agent.snapshot  elastic_agent.version   event.agent_id_status   event.dataset        event.duration  event.ingested              event.module    host.architecture   host.containerized  host.hostname     host.id              host.ip             host.mac           host.name         host.os.codename    host.os.family  host.os.kernel                  host.os.name    host.os.name.text   host.os.platform    host.os.type    host.os.version metricset.name  metricset.period    service.type
Mar 3, 2023 @ 12:32:00.000  42IuqYYBdg3nNnaiXK7o    .ds-metrics-aws.transitgateway-default-2023.03.01-000004    -        c99bbf44-85aa-4b10-ac28-0852d91b0667    87ecf587-2be0-44d0-9096-e84ba6798ba8    REDACTED_IP_URL_1 metricbeat  8.6.2           AWS/TransitGateway          REDACTED_TRANSITGATEWAY_1       REDACTED_TransitGatewayAttachment_1     0                                                       0                                                       7,964,366                               1,134,824,530                           0                                                       0                                                       127,879                                     795,990                                     REDACTED_CLOUDID_1    REDACTED_CLOUDID_1    aws             us-west-2       aws.transitgateway     default                  metrics             8.0.0       87ecf587-2be0-44d0-9096-e84ba6798ba8    FALSE                   8.6.2                   verified                aws.transitgateway   133,476,087     Mar 3, 2023 @ 12:33:53.000  aws             x86_64              FALSE               REDACTED_IP_URL_1 REDACTED_HOSTID_1    REDACTED_HOSTIP_1   REDACTED_HOSTMAC   REDACTED_HOSTID_1 Karoo               redhat          5.10.162-141.675.amzn2.x86_64   Amazon Linux    Amazon Linux        amzn                linux           2               cloudwatch      60,000              aws
Mar 3, 2023 @ 12:32:00.000  5GIuqYYBdg3nNnaiXK7o    .ds-metrics-aws.transitgateway-default-2023.03.01-000004    -        c99bbf44-85aa-4b10-ac28-0852d91b0667    87ecf587-2be0-44d0-9096-e84ba6798ba8    REDACTED_IP_URL_1 metricbeat  8.6.2           AWS/TransitGateway          REDACTED_TRANSITGATEWAY_1       -                                       0                                                       0                                                       7,964,366                               1,134,824,530                           0                                                       0                                                       127,879                                     795,990                                     REDACTED_CLOUDID_1    REDACTED_CLOUDID_1    aws             us-west-2       aws.transitgateway     default                  metrics             8.0.0       87ecf587-2be0-44d0-9096-e84ba6798ba8    FALSE                   8.6.2                   verified                aws.transitgateway   133,488,555     Mar 3, 2023 @ 12:33:53.000  aws             x86_64              FALSE               REDACTED_IP_URL_1 REDACTED_HOSTID_1    REDACTED_HOSTIP_1   REDACTED_HOSTMAC   REDACTED_HOSTID_1 Karoo               redhat          5.10.162-141.675.amzn2.x86_64   Amazon Linux    Amazon Linux        amzn                linux           2               cloudwatch      60,000              aws

All fields match except aws.dimensions.TransitGatewayAttachment, and this is happening with almost all documents, I did catch a few where it didn't, but they were very few of them.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!