Replies: 3 comments 1 reply
-
Huh -- after reading #86 it seems like disabling prefeching is something that's intended to be working already. Though I seem to be running into an issue with it. I get Reproduction can be found here Perhaps @james-pre can shed some light on what the issue could be? |
Beta Was this translation helpful? Give feedback.
-
I also see in #54 that the SharedArrayBuffer method for implementing synchronous operations on async backends was already considered, but now seems to be closed as no longer necessary in light of #162. Please correct me if I'm wrong, but I don't think #162 will provide a general solution for providing synchronous operations to all innately async backends. It doesn't seem like it will solve this use-case of lazily fetching files as they're accessed. Perhaps #54 can be reconsidered? If I'm misunderstanding anything please let me know! My experience with zenfs is still only skin-deep. |
Beta Was this translation helpful? Give feedback.
-
Duplicate of #86. This is a bug, see #180. |
Beta Was this translation helpful? Give feedback.
-
First off, I just want to say that Zenfs is looking better than ever – great work! 🎉
Recently, I tried FetchFS and ran into some performance concerns. I noticed that
resolveMountConfig
is taking significantly longer than I expected (around 7 seconds on Node for ~200 files served via a CDN). Upon investigation, I realized that the issue stems from every file in the index being fetched and cached before the filesystem is even fully configured.The Problem
I initially assumed that files would be lazy-loaded when accessed through any of the filesystem methods. However, that doesn’t seem to be the case. Instead, the current implementation preloads and caches all the files, leading to:
Proposed Solution
It would be great to have a backend that supports lazy loading, allowing files to be fetched only when they are accessed. This change could drastically reduce initialization time and optimize network usage.
Here’s a potential approach:
Async Operations:
Sync Operations:
This is where things get trickier. On a cache miss, we’d need to perform an async fetch, but synchronous operations don't support async behavior directly. A possible solution could involve implementing a mechanism similar to memfs's SyncMessenger.
Reference: Memfs's Approach to running async code synchronously
Memfs utilizes a worker to handle async tasks, leveraging a SharedArrayBuffer for communication between the main thread and the worker. The main thread is temporarily blocked with a while loop that breaks once the worker updates the buffer data with the return value.
While this approach may not be ideal in all scenarios, it could be a practical solution for contexts where Zenfs operates with sparse reads or runs within a worker or iframe. Adapting this pattern could make FetchFS's sync operations compatible with lazy loading while maintaining reasonable performance.
Let me know what you think!
Beta Was this translation helpful? Give feedback.
All reactions