chore: modify wantLastValueMetric() to verify the last value of a metric

[rhatgadkar-goog]

No description provided.

[rhatgadkar-goog]

These errors are occurring because the open_connections metric is having value 0. This could be because this metric is being updated in a goroutine inside the Dial() function: https://github.com/GoogleCloudPlatform/alloydb-go-connector/blob/623-wantlastvaluemetric/dialer.go#L377. And maybe the goroutine didn't execute yet.

[rhatgadkar-goog]

I found some interesting observation of this issue. The issue does not happen when only running the TestDialerWithMetrics test, like this: go test -run TestDialerWithMetrics. But it happens intermittently when running all the short tests, like this: go test -short. So it's flaky when running all the short tests.

I added some print statements to help in debugging this: in RecordOpenConnections(), in wantLastValueMetric(), and in the beginning and end of TestDialerWithMetrics.

When only running TestDialerWithMetrics, the value of open_connections is consistently 2 when dialing both connections. So the last value of open_connections will always be 2:

But when running all the short tests, the last value of open_connections fluctuates between 0 and 2. It's flaky:
When TestDialerWithMetrics passed:

When TestDialerWithMetrics failed:

I'm thinking that the 0 value is coming from the previous tests that ran before TestDialerWithMetrics. I tried to fix this by only processing the metrics in wantLastValueMetric() that occurred when TestDialerWithMetrics was called (by reading the timestamp of the metric), but that didn't work.

I think that this issue may take time to troubleshoot. So I think we can hold off on working on this for now, because it's not blocking anything.

[nancynh]

@rhatgadkar-goog is this ready for review? I see y'all have managed to get the tests passing :)

[rhatgadkar-goog]

No, it's not ready for review. Eno and I discussed that there's still more that we need to investigate about. We're keeping this as low priority for now

[enocom]

Going to close this since we're not actively working on it. We can re-open if we pick this back up.