Include entry metadata bytes in metrics tracking ingestion stats

[salvacorts]

What this PR does / why we need it:
In #9694 we added support for adding metadata labels to each entry in a push payload. In this PR we update the following metrics to take into account the bytes needed to store those labels:

• loki_distributor_bytes_received_total
• distributor_bytes_received

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• CHANGELOG.md updated
  • If the change is worth mentioning in the release notes, add add-to-release-notes label
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md
• For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR

[vlad-diachenko]

looks awesome )

[vlad-diachenko]

could you please also check if we have the tests and add the test-case that covers the new logic .
also, could you please update debug log that we have at the bottom of the method to print the size of non-indexed metadata separately.... it might be super useful during the investigation

[salvacorts]

could you please update debug log that we have at the bottom of the method to print the size of non-indexed metadata separately.... it might be super useful during the investigation

@vlad-diachenko Done

could you please also check if we have the tests and add the test-case that covers the new logic .

As far as I can see, we are not testing the bytesIngested and bytesReceivedStats metrics anyway. Unfortunately, I don't see a way to test out those metrics. Happy to learn about how to do so! Do we have any examples? I couldn't find any.

[vlad-diachenko]

@salvacorts

As far as I can see, we are not testing the bytesIngested and bytesReceivedStats metrics anyway. Unfortunately, I don't see a way to test out those metrics. Happy to learn about how to do so! Do we have any examples? I couldn't find any.

In general we use testutil.GatherAndCompare() to check the metrics, so, you can use unique tenant id in this test and check the metric from prometheus.DefaultRegistry to check the size is calculated correctly.

[sandeepsukhani]

As far as I can see, we are not testing the bytesIngested and bytesReceivedStats metrics anyway. Unfortunately, I don't see a way to test out those metrics. Happy to learn about how to do so! Do we have any examples? I couldn't find any.

You can also use testutil from Prometheus to get the current value from registered metrics. You can find a couple of examples in the tests here.

[sandeepsukhani]

The changes look good to me, left just a minor nit. Let us add some tests as suggested by Vlad.

[salvacorts]

Thank you @sandeepsukhani , I've added a test for the metrics :)

[sandeepsukhani]

LGTM