[filebeat][okta] drop fields processor not working with okta module

[nitesh8860]

I am using Okta module for filebeat with ECK operator, logs are getting fetched and shipped to elasticsearch properly, but i am also trying to drop some fields before shipping them, it is still sending all the fields, this can be a mistake in my config also which i am pasting below. Please help on this issue

• Version: 8.3.3
• Operating System: OCI Linux 8
• Steps to Reproduce:
  Filebeat config

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat-okta
  namespace: elastic
spec:
  type: filebeat
  version: 8.3.3
  config:
    filebeat.modules:
      - module: okta
        enabled: true
        system:
          var.url: ##########
          var.api_key: ###########
          var.keep_original_message: false
    processors:
      - drop_fields:
          fields:
            [
              "agent.ephemeral_id",
              "agent.id",
              "agent.name",
              "agent.type",
              "agent.version",
              "client.as.number",
              "client.as.organization.name",
              "client.as.organization.name.text",
              "client.domain",
              "client.geo.city_name",
              "client.geo.location.lat",
              "client.geo.location.lon",
              "client.geo.region_name",
              "client.user.id",
              "client.user.full_name",
              "client.user.full_name.text",
              "client.user.name.text",
              ]

[nitesh8860]

Hello team, any help on this ?

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[nitesh8860]

Hello, any help on this one ?

[andrewkroh]

Filebeat modules use Elasticsearch Ingest Node to apply most of the transformations to the events¹. So it is likely that the field names that you are referencing do not come into existence until the data passes through Elasticsearch. The pipelines it uses is here.

One option to customize the behavior would be to add a final_pipeline to your filebeat data stream that performs the additional transformations that you want.

Another option would be to modify the included pipelines to do what you want.

Footnotes

1. https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-overview.html ↩