feat: add gateway forbidden content types #46
Conversation
vasco-santos
force-pushed
the
feat/add-gateway-forbidden-content-types
branch
from
e1a080b to
00f0d80
Compare
Deploying with
|
| Latest commit: |
21b9408
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://fb665b9d.nftstorage-link.pages.dev |
|
I'm not sure we should block content simply because we see high traffic usage... |
I don't have a strong opinion. But, per recent meetings regarding gateway blockages one of the things we have been talking about is this. We will also consider blocking websites with JS and add other headers in the future. Let's gather feedback from @dchoi27 |
|
We're running our gateway as a public service for NFTs. For file extensions that are clearly not being used for NFT content (the referrers of this content are giant movie pirating pages), and that has small risk of false positives, we should block. If folks later start making NFTs with this sort of file extension, we can figure out how to be more precise. But we want to prevent early on being entrenched as a solution for pirates. |
|
Looking at the data, there was also a drop in cache hit rate around when the traffic spiked for this type of content, meaning that the usage of the gateway for these purposes is adversely affecting our core users. |
ACK, this makes sense - thanks for the clarification. |
packages/edge-gateway/src/gateway.js
Outdated
| `Forbidden content type: ${winnerGwResponse.response.headers.get( | ||
| 'content-type' | ||
| )}` | ||
| ) |
There was a problem hiding this comment.
We need to do this check before the waitUntil above don't we? Otherwise we cache the response and on subsequent requests it'll be served from the cache.
There was a problem hiding this comment.
yeah good catch
Co-authored-by: Alan Shaw <[email protected]>
vasco-santos
force-pushed
the
feat/add-gateway-forbidden-content-types
branch
from
e2ff7f5 to
21b9408
Compare
|
It’s very aggressive to block all binary content type returns. I also don’t think it’ll work all that well since they could change this to any other content type pretty easily, they obviously aren’t depending on the content type for any rendering since they are getting back as a binary stream. Digging into this a bit with @vasco-santos, I think that we should exhaust our other content blocking options (referrers, user-agents, etc) before resorting to this. We’re losing referrers right now on redirect to subdomains so we need to push a fix for that before we’ll have good data to work with. |
|
Closing this for now. Follow up fixing referers #68 and we will rely on CF Firewall for blocking |
This PR adds forbidden content types to avoid abuse as we were seeing ~28% of traffic with bin files from last 12 hours https://dash.cloudflare.com/fffa4b4363a7e5250af8357087263b3a/nftstorage.link/analytics/traffic?content-type=bin
This can also be problematic for gateway being blocked by other parties and we should investigate other types to block.
CF may still just put
binon other kind of types if does not know about. Anyway, we should just blockoctet-streamto start. This should likely evolve into a wrangler secret / Admin feature.