algod: Add static EnableTelemetry retry

[urtho]

Remote telemetry with a static URI in config never gets enabled past the initial, single try at algod startup.
Remote logging to static URI never gets enabled in the event the Internet or remote service is not available during Algod startup.

There is no such issue with dynamic remote telemetry (DNS based discovery) as it retries the connection with TelemetryURIUpdateService.

This PR adds a loop that retries the static remote service every minute until it succeeds.

[urtho]

This one does not solve the HeartBeat race where HB sometimes is not being sent via telemetry.

[jannotti]

I'm ok with adding the go routine loop, but if we're doing that, let's only do that. We can remove the initial attempt to enable, and only enable in the loop.

[urtho]

This change will make static telemetry init fully async instead of sync.
Initial Start and HeartBeat events will be lost in most startup cases with static URI - same as it is now with DNS telemetry.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 7 lines in your changes missing coverage. Please review.

Project coverage is 54.90%. Comparing base (62f9082) to head (5053e2d).
Report is 16 commits behind head on master.

Files with missing lines	Patch %	Lines
cmd/algod/main.go	0.00%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6183      +/-   ##
==========================================
- Coverage   55.70%   54.90%   -0.81%     
==========================================
  Files         494      494              
  Lines       69972    69976       +4     
==========================================
- Hits        38981    38422     -559     
- Misses      28276    28875     +599     
+ Partials     2715     2679      -36     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gmalouf]

@urtho want to merge in master here and we'll get this pulled in?

[gmalouf]

Do we want this to go on indefinitely?

[jannotti]

I'm inclined to say yes. It's only once per minute. If that sends like too much, maybe double the time each time, up to, say, 10 minutes?

[jannotti]

This change will make static telemetry init fully async instead of sync.
Initial Start and HeartBeat events will be lost in most startup cases with static URI - same as it is now with DNS telemetry.

To keep those, how about a short sleep, say 2 seconds, after starting the go routine?

If you want to get very fancy, you could have the init code write to a channel when it completes, and the sleep could be a select on the channel or until a few seconds timer expires. But I would be happy with a short also and a comment explaining the reasoning.

[jannotti]

I would accept this, but I'd prefer a little delay to try to get telemetry initialized before start events.

[algorandskiy]

To keep those, how about a short sleep, say 2 seconds, after starting the go routine?

Maybe just to try a synchronous init attempt and then run a goroutine in case of failure?