Create two LMDB libraries with different sizes from the same Arctic object (on Windows)

[levoer]

library_options = LibraryOptions()
self.arctic.create_library(lib, library_options=library_options)
database
|__arctic_cfg
|_test1
| |_data.mdb ~2GB
| |_lock.mdb
|_test2
|_data.mdb ~2GB
|_lock.mdb
As can be seen each library will take the orignal map size, whereas in my program I want some libraries to be eg 100MB, others to be 20MB and so on. How can I achieve that?

[poodlewars]

Hi @levoer there's no way to do this at the moment unfortunately. The workaround would be to create a single Arctic instance for each library you want to work with:

In [1]: from arcticdb import Arctic

In [2]: ac = Arctic("lmdb:///Users/aseaton/lmdb/one?map_size=1GB")

In [3]: ac_two = Arctic("lmdb:///Users/aseaton/lmdb/one?map_size=2GB")
20241129 11:09:48.233404 46056 W arcticdb | LMDB path at C:\Users\aseaton\lmdb\one\ has already been opened in this process which is not supported by LMDB. You should only open a single Arctic instance over a given LMDB path. To continue safely, you should delete this Arctic instance and any others over the LMDB path in this process and then try again. Current process ID=[52868]

In [4]: ac.create_library("one")
Out[4]: Library(Arctic(config=LMDB(path=C:\Users\aseaton\lmdb\one)), path=one, storage=lmdb_storage)

In [5]: ac_two.create_library("two")
Out[5]: Library(Arctic(config=LMDB(path=C:\Users\aseaton\lmdb\one)), path=two, storage=lmdb_storage)

The log warning should be OK to ignore in practice since the two Arctic objects will interact with completely different files.

Alternatively you could use LMDB on Linux (eg WSL) which will only use as much disk as your data needs, rather than creating the large file eagerly.

This is definitely something we could improve and we'll keep it on our backlog.