Cache netflow module v9 / IPFIX templates

[111andre111]

Describe the enhancement:
At the moment there is potential data loss for events, before any device sends its template.
Describe a specific use case for the enhancement or feature:
At first a short iteration about Netflow protocols module and its circumstances:
In the documentation you find the statement, that it supports Netflow versions 1,5,6,7,8,9,IPFIX
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-netflow.html

https://en.wikipedia.org/wiki/NetFlow
Netflow versions 1, 5, 6, 7, 8 will be mapped automatically so they are static
But for version 9, IPFIX devices will send so called templates where it decides "dynamically" what fields will be sent. Otherwise it won't be possible to get events with a matching template.
https://tools.ietf.org/html/rfc3954#section-3.2

For some devices like Ciscos the send template setting is quite high like 10 or 15 minutes, so potentially you will loose 10 or 15 minutes of the flow data at the moment because beats doesn't understand these events and throws them away.
Here one of the documentation pages of how often the templates are sent:
https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/fnetflow/command/fnf-cr-book/fnf-m1.html#wp9324529020

When a new device is onboarded this is ok I think, but not for devices, that are "known" in a way, because the template that was sent 10 minutes before normally doesn't change every time, so with the help of a template cache on disk, we should be able to circumvent such scenarios. Maybe it is possible to use the registry file for that information to track at least, what was the last template for a certain device.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[andrewkroh]

This sounds like a duplicate of #14618.

[adriansr]

Yes, @111andre111 I thought the enhancement you wanted is to cache in memory the flows for which a template is not received yet, so when it arrives we can apply the template to them.

[111andre111]

You are right @adriansr , both issues have another purpose. In my case here, yes. I think about the time between the first flow are incoming until the first template comes into play.
I am not sure if the more effective was is to cache these events in memory or even cache them in a persistent queue optionally.

Another scenario would be when filebeat would be stopped and started again and new flows are incoming and then the beat would already know about the template. This is the other scenario mentioned in #14618

I hope that makes things more clear @andrewkroh

[shipler]

@andrewkroh is there any expected update for the persistant template loading in the beats netflow module? for the newer versions

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[abraxxa]

👍🏻

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!