Prometheus exporter usage

[yeplaa]

I use currently the fluent component with fluent-plugin-prometheus plugin to deploy the prometheus exporter.
I generate my metrics by integrating the labels Kubernetes (retrieving from fluent-plugin-kubernetes plugin) Also i use the internal grep plugin to filter my messages. Bellow an extract of my configuration :

 <source>
      @type forward
      @log_level debug

      port 24224
      bind 0.0.0.0
    </source>

    
    
    # Cf https://docs.fluentd.org/filter/grep
    <filter {{ .Release.Namespace }}.kubernetes.var.log.containers.**>
      @type grep
      <regexp>
        key $.kubernetes.container_name
        pattern /^elasticsearch$/
      </regexp>
      
      <regexp>
        key $.kubernetes.labels.productname
        pattern /^elasticsearch$/
      </regexp>

      <regexp>
        key $.kubernetes.labels.domain
        pattern /^dbms$/
      </regexp>

      <regexp>
        # log beginning : [2018-07-04T13:34:26,108][ERROR]
        key $.message
        pattern /^\[[\d]{4}-[\d]{2}-[\d]{2}T[\d]{2}:[\d]{2}:[\d]{2},[\d].*\[ERROR\].*$/
      </regexp>

    </filter>


    <filter {{ .Release.Namespace }}.kubernetes.var.log.containers.**>
      @type prometheus
      <metric>
        name All_elasticsearch_counter
        type counter
        desc The total number of foo in message.
        <labels>
          container $.kubernetes.container_name
          namespace_name $.kubernetes.namespace_name
          pod_name $.kubernetes.pod_name
          domain $.kubernetes.labels.domain
          productname $.kubernetes.labels.productname
          setname $.kubernetes.labels.setname
        </labels>
      </metric> 
    </filter>


    # Exposition metrics promtheus / exporter
    <source>
      @type prometheus
      bind 0.0.0.0
      port 24231
      metrics_path /metrics
    </source>

Do you know if it's possible to perform the same feature with fluent-bit (and its Metrics plugins)?
Reading the doc I'm not sure

Thanks

[agup006]

Hi @yeplaa I converted this to a discussion post. This is not possible at the moment as we are mainly allowed to collect and send metrics today. As we look at more log2metric scenarios we should keep this one in mind.