What are the size limits of the backends on browser?

[Davilarek]

Hi there,
My project requires storing lots of data, around 100 megabytes.
I would like to use zen-fs on browser environment.
It's important to note that the data needs to persist between reloads and even restarts of the browser, essentially, making the data kind of permanent.
Hence my question, what are the size limits of the available backends in a browser?

[james-pre]

@Davilarek,

I recommend you read this article, which explains how persistence works with the various storage APIs in the browser.

ZenFS does not implement any limits, so the only limits are those imposed by the browser.

If you are looking to store around 100 MB, I would recommend WebAccess with the Origin private file system (OPFS) or IndexedDB. Which one you choose depends on your specific use case.

Does that answer your question?

[Davilarek]

Partially. In the past, I used legacy version of BrowserFS, but I noticed some of users of my project running out of memory because of AsyncMirror, so I thought that maybe ZenFS got some upgrades

[james-pre]

AsyncMirror has been integrated with the Async mixin (doc), which is used by async file systems.

AsyncStoreOptions.sync is the file system used for synchronous operations, so it should be possible to choose a different one. As far as I know, there are not any other synchronous backends that would have more space. On most browsers, web storage (localStorage/sessionStorage) is limited to 5 MiB.

[Davilarek]

My old setup was async mirror bridging InMemory with IndexedDB, I just want to know if there were any memory efficiency improvements comparing ZenFS with old BrowserFS

[james-pre]

Potentially, since InMemoryStore changed from using a plain object to using a Map<bigint, Uint8Array>. However, I have not done any performance testing so I can not be sure.

I would love it if you would like to do performance testing and find any places where improvements can be made.

[james-pre]

@Davilarek,

I started to prototype FS size metadata last month (branch). Would you be interested in seeing that feature prioritized?

[Davilarek]

That's not necessary at the moment, however I posted a question on your Discord server, it may be worth taking a look at it

[james-pre]

Including the discord conversation here:

@Davilarek:

Is there a way I can wait for the FS to sync everything from indexedDB to InMemory? async calls work fine but calls that are sync require some time to have any content
sometimes sync calls return empty buffers even after some time... (but async still works fine)

Me:

If you wait for the IndexedDB.ready()
That should wait for the sync cache to populate

@Davilarek:

isn't awaiting configure already doing this?
https://github.com/zen-fs/core/blob/c33ad77a/src/config.ts#L61

Me:

Yes, perhaps crossCopy is not being awaited in ready then

@Davilarek:

I'm not sure

[james-pre]

@Davilarek,

With #49 opened to handle the issue you brought up on the discord, can this issue be closed?

[james-pre]

Closing this issue since the original question has been answered.