[AWS] Introduce ignore_older & start_timestamp

[Kavindu-Dodan]

Background

This issue is to track and upgrade AWS log integrations that utilize filebeat with ignore_older & start_timestamp introduced through elastic/beats#41817.

With the mentioned properties, users can control S3 (or S3 compatible) polling based log ingestion and improve filebeat registry memory usage.

AC

Once elastic/beats#41817 is merged, expose the above-mentioned properties through relevant integrations.

Note - Configurations will be released with beats 8.18.0

[lucabelluccini]

Hello @Kavindu-Dodan

Thank you for the work on this enhancement already on Beats.

Please try to reach out integration developers who use the aws-s3 input (with polling mode) in their integrations to notify them about the new settings available in 8.18.0

One key integration which would possibly benefit from this is https://www.elastic.co/guide/en/integrations/current/cloudflare_logpush.html#cloudflare_logpush-to-collect-data-from-s3-compatible-cloudflare-r2-buckets-follow-the-below-steps

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[Kavindu-Dodan]

@lucabelluccini Good point. New configurations introduced at Beats are related to any integration that use S3 input for data ingestion. Most of the related integrations belong to my team. And I have added the team label that maintains the Cloudflare logs integrations.