Add MFS Support to Reprovider Strategy #10704
Conversation
Signed-off-by: Abhinav Prakash <[email protected]>
Signed-off-by: Abhinav Prakash <[email protected]>
There was a problem hiding this comment.
Hi, thank you very much. Upon looking closely at this PR, I think this could be approached better.
- The
boxo/providermodule implements the different providers and the MFS provider should be implemented there in line with the others. - The MFS provider could then be very similar to the PinnedProvider which already traverses each recursive-pin dag from its root. It probably should receive the
offlineIPLDFetcherthat Kubo can already provide (just likeIPLDFetcheris passed to the PinnedProvider.). - The MFS provider can hopefully traverse blocks recursively very similar to how it is done for the recursive pins, using the
fetcherHelpers.BlockAll(https://github.com/ipfs/boxo/blob/4645270f16bb6c0943381065596c18c667440e09/provider/provider.go#L112-L117). In principle we don't need another dag-traversal implementation. - If it is easier, we can make a general "DAGProvider" which receives a root CID and does the rest. That way we don't need to figure out the MFS root from Boxo and we can pass that from Kubo. It would be more flexible too.
- In Kubo then we make a "prioritized provider" that includes the MFS and the Pinned provider, rather than mixing the MFS part with pins.
That is how I would try this first, if you have questions or something is not clear let me know! This will require a PR in boxo first, and then a follow up in Kubo.
|
Hi @hsanjuan |
Signed-off-by: Abhinav Prakash <[email protected]>
|
I have one question.... |
hsanjuan
changed the title
You need to add a changelog entry describing the change, similar to other entries in there. You can leave that for now until things are wrapped. |
|
|
||
| if onlyPinned { | ||
| return provider.NewPinnedProvider(false, in.Pinner, in.IPLDFetcher) | ||
| mfsProvider := provider.NewDAGProvider(mfsRoot.Cid(), in.IPLDFetcher) |
There was a problem hiding this comment.
@PsychoPunkSage As noted by Hector above, this PR needs to include an end-to-end test in test/cli that ensures MFS provider is not prefetching any data that was not already in the local datastore.
Context:
- MFS is different from local blockstore and local pins in that it is lazy-loaded.
- I can do
ipfs files cp /ipfs/bafybeie6tqsijf7j6gpow5q3qdjmtqgscrmd2ijl4olpxanbbptmvh7sui/ /Tsize-testand my node won't fetch 4EiB of data unless i start walking the dag. - People can
ipfs files cp $(ipfs resolve -r /ipns/en.wikipedia-on-ipfs.org/) /wikipediaand use this to protect articles they visited from gc, without fetching entire thing. - We don't want providing to trigger fetch of any CIDs that are not already in local block-store.
- I can do
- Perhaps we should explicitly rename/replace
in.IPLDFetcherhere within.OfflineIPLDFetcher? We already initialize both in./core/node/core.goand store them separately to be extra sure offline one is used when it matters. - Separate concern: what happens once DAG walk errors on reference to a CID that is not in local store? We don't want to abort entire walk, but we should stop going deeper in this specific branch, and continue walking the DAG in other directions.
closes #10386