Improve Gateway TTFB

[iand]

TTFB here is defined as the 7 day mean of the 95th centile measurement of the time to first byte as reported by all Gateway NGINX servers

The TTFB encapsulates the time taken for go-ipfs to resolve and read a requested block. The block may be the root of a file dag that must be retrieved to fulfill the entire request, but this additional retrieval time may not be part of the TTFB metric if go-ipfs is able to begin streaming to the client immediately.

As a baseline reference since the start of 2022 of the TTFB metric are as follows (Grafana dashboard):

Timings are 7–day averages (in ms), request counts are 7 day totals

Week ending	Overall TTFB P95 7-day avg (ms)	/ipfs TTFB P95 7-day avg (ms)	/ipns TTFB P95 7-day avg (ms)	/ipfs requests 7-day total	/ipns requests 7 day total
2022-07-17	14465	14654	1606	1160789601	8839516
2022-07-10	15117	15398	1876	758965477	7992380
2022-07-03	14624	14847	-	758534722	6503190
2022-06-26	12137	12321	2801	681296119	6196580
2022-06-19	11783	11951	3021	676777682	5475402
2022-06-12	10202	10379	1421	663156465	6108780
2022-06-05	7695	7813	1056	679216764	6145194
2022-05-29	9569	9696	1110	750595367	5885012
2022-05-221	11321	11460	1172	684531043	6207989
2022-05-15	3930	4018	767	645623016	6210211
2022-05-08	5636	5750	805	645383820	5729412
2022-05-01	7467	7605	827	625505173	5691557
2022-04-24	6137	6273	868	674131403	6576324
2022-04-17	5828	6282	1207	579844301	6069346
2022-04-10	11320	11525	1268	611577089	6065253
2022-04-03	6383	6411	32186	490258661	6068431
2022-03-272	9117	8286	327060	530622766	4928387
2022-03-20	7842	7237	509940	564945245	4751772
2022-03-13	4602	4015	-	653778624	4720244
2022-03-06	7155	6585	492720	824065423	4292238
2022-02-27	5855	5779	410400	944414646	2349145
2022-02-20	7819	7728	432060	684044673	2578595
2022-02-13	9888	9637	452280	514643736	2573098
2022-02-06	10176	9934	464280	483058798	2568412
2022-01-30	11684	9459	508200	483914472	6096080
2022-01-23	28373	20599	541320	432975290	8736287

1 More gateway instances were added on May 24
2 Gateway scaled out on Apr 1

Background

The factors that affect TTFB include the following (in descending order of impact):

1. whether any required block is present in the local blockstore
2. the response time of the local blockstore to locate and read blocks
3. the number of intermediate nodes that must be read to locate the CID of the requested path
4. the time spent locating the provider of a block using the DHT
5. the time spent connecting to and reading from a block provider (involves Bitswap discovery)
6. For some types of requests (directory listing), the time spent enumerating the contents of a directory if a directory listing needs to be served
7. the time taken to validate the existence of the requested block (cache validation/etag check)

• Production engineering team project overview