datagrepper stats - DB size etc.

[abitrolly]

What is the number 96,756,600 on the front page https://apps.fedoraproject.org/datagrepper/ ?

datagrepper/datagrepper/templates/index.html

Lines 70 to 74 in 3a34025

	{{message_bus_longname}}</a>: events by username, by package, by message
	source, by topic&hellip; you name it.</p>
	<span id="odometer" class="odometer centered visible-sm visible-md visible-lg">{{total}}</span>
	</div>

total of what? Helpful to have a tooltip that makes this explained.

---

And now the real question - is it possible to expose stats/metrics about fedmsg DB? Like size growth over time and current size per topic and per rate of messages per second. Maybe delays in delivering messages to subscribers and count of dropped messages per subscriber (detect dead) and topic (popularity). To estimate if this architecture is good based on real working example. I mean to estimate if this stack is better than more heavyweight Kafka/Zookeeper/...

[pypingou]

The number on the front page is the total number of fedmsg messages recorded.

Like size growth over time and current size per topic and per rate of messages per second.

This would be doable, and you can do it yourself with the DB dump that is at: https://infrastructure.fedoraproject.org/infra/db-dumps/datanommer.dump.xz (refreshed daily) just beware that this DB is pretty huge so prepare some disk space :)

[pypingou]

Maybe delays in delivering messages to subscribers and count of dropped messages per subscriber (detect dead) and topic (popularity)

Delays in delivering and dropped messages won't be computable as the messages are sent in a pub-sub mode (ie: fire and forget), there is no central broker. If nobody listens to your service, your service won't resend the messages.

[abitrolly]

DB is pretty huge so prepare some disk space :)

21GB .xz OMG. =)

Delays in delivering and dropped messages won't be computable as the messages are sent in a pub-sub mode (ie: fire and forget), there is no central broker. If nobody listens to your service, your service won't resend the messages.

That explains a lot. For me this paragraph should be present on the front page http://www.fedmsg.com/en/stable/ right after usage example. So, the datagrepper is the thing to restore the gaps if service goes down for some reason. Seems quite critical.

[pypingou]

I seem to remember this being present at one point but it seems it is no longer :(