feat: gateway tracking requested cids in database

[vasco-santos]

This PR adds metrics tracking whether content was stored in NFT.storage and its Pinning status characteristics. It uses nft.storage client to check the state of the content. We could go through the route of directly connecting to the DB here, but as we do this in the waitUntil it seems cleaner to rely on the client (specially as we can't access replica directly).

Prometheus metrics to be consumed are as follows:

# HELP nftgateway_errored_responses_with_known_content_total errored requests with known content in NFT.storage.
# TYPE nftgateway_errored_responses_with_known_content_total counter
nftgateway_errored_responses_with_known_content_total{env="test"} 0
# HELP nftgateway_responses_by_content_status_total total of responses by content status. Either stored or not stored.
# TYPE nftgateway_responses_by_content_status_total counter
nftgateway_responses_by_content_status_total{env="test",status="Stored"} 0
nftgateway_responses_by_content_status_total{env="test",status="NotStored"} 0
# HELP nftgateway_responses_by_pin_status_total total of responses by pin status.
# TYPE nftgateway_responses_by_pin_status_total counter
nftgateway_responses_by_pin_status_total{env="test",status="PinError"} 0
nftgateway_responses_by_pin_status_total{env="test",status="Pinned"} 0
nftgateway_responses_by_pin_status_total{env="test",status="Pinning"} 0
nftgateway_responses_by_pin_status_total{env="test",status="PinQueued"} 0
# HELP nftgateway_responses_per_time_by_status_total total of responses by status per response time bucket
# TYPE nftgateway_responses_per_time_by_status_total histogram
nftgateway_responses_per_time_by_status_total{status="PinError",le="0.05",env="test"} 0
nftgateway_responses_per_time_by_status_total{status="Pinned",le="0.05",env="test"} 1
nftgateway_responses_per_time_by_status_total{status="Pinning",le="0.05",env="test"} 0
nftgateway_responses_per_time_by_status_total{status="PinQueued",le="0.05",env="test"} 0
nftgateway_responses_per_time_by_status_total{status="PinError",le="0.1",env="test"} 0
nftgateway_responses_per_time_by_status_total{status="Pinned",le="0.1",env="test"} 2
nftgateway_responses_per_time_by_status_total{status="Pinning",le="0.1",env="test"} 0
nftgateway_responses_per_time_by_status_total{status="PinQueued",le="0.1",env="test"} 0
(...)

Follow up work (will create issue):

• improve testing by spinning up API locally with Miniflare

Should use #1391 cache in check endpoint

Closes nftstorage/nftstorage.link#14

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Pages

Latest commit:	2c437ae
Status:	✅  Deploy successful!
Preview URL:	https://8b008d6b.nft-storage-1at.pages.dev

View logs

[vasco-santos]

For just check, we don't need a token, while it is "required" in NFTstorage (but param called option)

@hugomrdias should we make it optional in the types?

[hugomrdias]

yes we should, but here if you just use it for check call might be better to not include the client.

[vasco-santos]

you mean, just HTTP the endpoint?

[olizilla]

yep, I agree, if we just want to hit the public check endpoint, maybe just add a helper function around a fetch

[olizilla]

notable that this PR is the only thing that ties the gateway impl to nftstorage. It would be neat to have a generic "racing gateway proxy" that anyone could deploy, and then have some mechinism for adding extra metrics for a given deployment.

We might want to reuse this for a web3.storage gateway

[vasco-santos]

notable that this PR is the only thing that ties the gateway impl to nftstorage. It would be neat to have a generic "racing gateway proxy" that anyone could deploy, and then have some mechinism for adding extra metrics for a given deployment.

We can have a check endpoint ENV Var to have this generic

[vasco-santos]

Closing this and we will track on issue nftstorage/nftstorage.link#14 how to gather this