Queue filled metrics are not indexed nor displayed in Elastic Agent dashboard

[jlind23]

This issue might require changes in Beats and Integrations repositories.

Problem statement

Not all available queue filled metrics are displayed in the Elastic Agent package visualisation.

Example of metrics coming from an 8.16.1 elastic agent:

{
  "log.level": "info",
  "@timestamp": "2024-12-17T14:46:25.356Z",
  "message": "Non-zero metrics in the last 30s",
  "component": {
    "binary": "metricbeat",
    "dataset": "elastic_agent.metricbeat",
    "id": "system/metrics-monitoring",
    "type": "system/metrics"
  },
  "log": {
    "source": "system/metrics-monitoring"
  },
  "service.name": "metricbeat",
  "monitoring": {
    "ecs.version": "1.6.0",
    "metrics": {
      "beat": {
        "cpu": {
          "system": {
            "ticks": 10765
          },
          "total": {
            "ticks": 34093,
            "value": 34093
          },
          "user": {
            "ticks": 23328
          }
        },
        "info": {
          "ephemeral_id": "2a2c2bce-ce20-4ce1-ba08-238fa1eff60d",
          "uptime": {
            "ms": 81660180
          },
          "version": "8.16.1"
        },
        "memstats": {
          "gc_next": 78341208,
          "memory_alloc": 35789544,
          "memory_total": 646957088,
          "rss": 121958400
        },
        "runtime": {
          "goroutines": 50
        }
      },
      "filebeat": {
        "harvester": {
          "open_files": 0,
          "running": 0
        }
      },
      "libbeat": {
        "config": {
          "module": {
            "running": 1
          }
        },
        "output": {
          "events": {
            "active": 0
          },
          "write": {
            "latency": {
              "histogram": {
                "count": 1361,
                "max": 120,
                "mean": 22.3955078125,
                "median": 20,
                "min": 9,
                "p75": 24,
                "p95": 38,
                "p99": 62.75,
                "p999": 119.1250000000008,
                "stddev": 8.807779062155877
              }
            }
          }
        },
        "pipeline": {
          "clients": 1,
          "events": {
            "active": 0
          },
          "queue": {
            "filled": {
              "bytes": 0,
              "events": 0,
              "pct": 0
            },
            "max_bytes": 0,
            "max_events": 12800
          }
        }
      },
      "registrar": {
        "states": {
          "current": 0
        }
      }
    }
  },
  "log.logger": "monitoring",
  "log.origin": {
    "file.line": 192,
    "file.name": "log/log.go",
    "function": "github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot"
  },
  "ecs.version": "1.6.0"
}

Example of visualisation:

Definition of done

Metrics coming from this PR and this PR should be:

• Sent and indexed in ES
• Have related visualisation as part of the Elastic Agent package
• Queue usage and queue depth viz should be working again

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[jlind23]

@faec were you able to make any progress on this?