Fluent-Bit kubernetes_events input plugin failed buffer write error

[Akila-I]

We are using kubernetes_events input plugin and forwarding events to OpenSearch via Fluent-Bit (version 3.16). I'm getting this error in fluent-bit pod logs continuously and the pod crashes at some point. Can someone explain why this is happening and a way to fix this please. I think its because too many events are generated in the cluster and fluent-bit pod goes out of memory. Is that the case?

error (log level - debug):

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=0

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=1

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=2

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=3

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=4

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=5

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=6

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=7

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=8

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=9

[2025/01/08 08:45:00] [error] [input:kubernetes_events:kubernetes_events.0] could not enqueue records into the ring buffer

config:

        name kubernetes_events
        kube_url https://kubernetes.default.svc/
        tag k8s_events
        tls.verify Off

[patrick-stephens]

there's a ring buffer yeah and it looks like it is filling up

@pwhelan is the ring buffer size configurable?

The alternative is might be to split into per-namespace K8S event inputs as well @Akila-I, they can all have the same tag for output