[Fortinet Fortigate] Split current dataset into multiple datasets

[leandrojmp]

Hello,

Currently the integration for the Fortinet Fortigate has only one dataset, fortinet_fortigate.log, which means that all logs from fortigate will be stored in the same data stream, but the integration produces different types of logs like traffic, utm and event.

The goal for this is that this would allow you to have different retention per dataset, for example, you may be required to keep audit logs for a longer time than traffic logs, audit logs have the fortinet.firewall.type value as event and traffic logs have the same field as traffic.

Since the volume of traffic logs are higher, to keep only the audit logs you do not have many options, you basically need to create a custom transform to store those events in another index, I'm not sure that you can use a reroute processor to create a custom dataset/namespace since Fleet integrations permissions are pretty limited.

Now we have this:

• fortinet_fortigate.log: all logs from fortigate

But we should have something like this:

• fortinet_fortigate.traffic: traffic logs from fortigate
• fortinet_fortigate.event: event logs from fortigate
• fortinet_fortigate.utm: utm logs from fortigate
• fortinet_fortigate.log: new/unmapped logs from fortigate

We already have separated ingest pipelines for each fortinet.firewall.type value here.

  - pipeline:
      name: '{{ IngestPipeline "event" }}'
      if: ctx.fortinet?.firewall?.type == 'event'
  - pipeline:
      name: '{{ IngestPipeline "login" }}'
      if: >-
        if (ctx.message instanceof String) {
          String normalizedMessage = ctx.message.toLowerCase();
          return (normalizedMessage.contains('login') || normalizedMessage.contains('logged in'));
        }
        return false;
  - pipeline:
      name: '{{ IngestPipeline "traffic" }}'
      if: ctx.fortinet?.firewall?.type == 'traffic'
  - pipeline:
      name: '{{ IngestPipeline "utm" }}'
      if: ctx.fortinet?.firewall?.type == 'utm' || ctx.fortinet?.firewall?.type == 'dns'

I think this helps implement separated data streams, I'm not sure how to do that, but with some orientation I could send a PR.

What we need to do to create a new dataset?

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)