Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] aggregate put_all not working with nested keys #3989

Open
Psych0meter opened this issue Jan 19, 2024 · 0 comments
Open

[BUG] aggregate put_all not working with nested keys #3989

Psych0meter opened this issue Jan 19, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@Psych0meter
Copy link

Describe the bug
It seems that the put_all aggregate function only works with "root" keys, and do not add new nested keys.

For example, I have these 2 postfix events that I need to aggregate, based on /host/id and /email/local_id :

{
  "@timestamp": "2024-01-19T13:25:35.503436Z",
  "postfix": {
    "nrcpt": "1",
    "size": "17603"
  },
  "application": "postfix",
  "host": {
    "id": "95e0888c1df24d79a4bc827f9636cca7"
  },
  "email": {
    "from": {
      "address": "EMAIL_FROM"
    },
    "local_id": "7A20930009B"
  },
  "message": "queue active"
}
{
  "@timestamp": "2024-01-19T13:25:35.527242Z",
  "postfix": {
    "relay": "localhost[::1]:24",
    "delays": "0/0/0/0.02",
    "dsn": "2.1.5",
    "delay": "0.03",
    "status": "sent"
  },
  "application": "postfix",
  "host": {
    "id": "95e0888c1df24d79a4bc827f9636cca7"
  },
  "email": {
    "local_id": "7A20930009B",
    "to": {
      "address": "EMAIL_TO"
    }
  },
  "test": "OK"
}

The put_all function only add keys that are at the '/' level of the JSON, and drop the other keys :

{
  "@timestamp": "2024-01-19T13:25:35.503436Z",
  "postfix": {
    "nrcpt": "1",
    "size": "17603"
  },
  "application": "postfix",
  "host": {
    "id": "95e0888c1df24d79a4bc827f9636cca7"
  },
  "email": {
    "from": {
      "address": "EMAIL_FROM"
    },
    "local_id": "7A20930009B"
  },
  "message": "queue active",
  "test": "ok"
}

To Reproduce
Steps to reproduce the behavior:

pipeline:
    processor:
    - aggregate:
        identification_keys:
          - "/host/id"
          - "/email/local_id"
        action:
          put_all:
        group_duration: "10s"
        aggregate_when: '/email/local_id != null'

Expected behavior
It needs to create the missing entries in the nested keys :

{
  "@timestamp": "2024-01-19T13:25:35.503436Z",
  "postfix": {
    "nrcpt": "1",
    "size": "17603",
    "relay": "localhost[::1]:24",
    "delays": "0/0/0/0.02",
    "dsn": "2.1.5",
    "delay": "0.03",
    "status": "sent"
  },
  "application": "postfix",
  "host": {
    "id": "95e0888c1df24d79a4bc827f9636cca7"
  },
  "email": {
    "from": {
      "address": "EMAIL_FROM"
    },
    "to": {
      "address": "EMAIL_TO"
    }
    "local_id": "7A20930009B"
  },
  "message": "queue active",
  "test": "ok"
}

Environment (please complete the following information):

  • latest dataprepper docker image (2.6.1)
@Psych0meter Psych0meter added bug Something isn't working untriaged labels Jan 19, 2024
@dlvenable dlvenable added this to the v2.8 milestone Jan 23, 2024
@dlvenable dlvenable removed this from the v2.8 milestone Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Development

No branches or pull requests

2 participants