Excessive memory usage

[cathaysia]

When I recursively watch a folder(37323 files), the memory usage is too large. 64MB of memory needs to be allocated at a time. In addition, the vec memory growth strategy is too conservative, causing vec to keep trying to allocate memory:

please notic the slider:

[davidhalter]

If you are on Linux you should expect that a watch uses 1kB of memory (source https://unix.stackexchange.com/questions/13751/kernel-inotify-watch-limit-reached). I'm not sure this still applies to the latest kernel and you might be on MacOs, but I assume 64 MB of memory is not entirely wrong (it would be at least 32 MB), especially when you consider that inotify also uses some state internally per watch.

Other people should probably weigh in on the "vec memory growth strategy", but I doubt that this is an issue compared to the overhead of a sys call (or at least not making notify slower by a huge amount).

[cathaysia]

Actually the memory allocation happens here instead of inside inotify:

notify/notify-debouncer-full/src/lib.rs

Line 204 in 1a34031

let mut queues_remaining = HashMap::with_capacity(self.queues.len());

I tried replacing it with BTreeMap, but the memory usage was only reduced to about 57MB, and the cost was that one memory allocation was replaced by tens of thousands of memory allocations. Is there any other way to reduce the memory usage? (such as use raidx tree, etc.)

vec memory growth strategy

The memory allocation of vec is quite conservative(https://doc.rust-lang.org/src/alloc/vec/mod.rs.html#2417). I'm not sure how Rust makes such decisions in the first place, but can Vec::with_capacity be used here to reduce the number of memory allocations?

[dfaust]

@cathaysia Can you please provide more information about how you are using notify. Maybe you could create a minimal test case, that watches a folder and generates FS events (like examples/debouncer_full.rs).
Also, the memory leak issue you raised in the linked Tauri issue seems far more severe. Can you provide some more details about that (maybe even a minimal test case).
It's also very important to know what OS you are using. Since you are using Tauri, you are also using the recommended file id cache. And if you are using something other than Linux, this means the path of every file is stored in memory.
And what tools are you using to profile the memory usage?

[cathaysia]

I'm using notify in this repo: https://github.com/cathaysia/finalgit/tree/dev. If you are not in a hurry, I will try to give a minimal reproducible example

When you select a git repository, it will automatically recursively watch the entire git repository (yes, .gitignore is not followed here).

If you refresh the page a few times, you will see that the memory usage is growing.

I currently tested on Windows, Linux and MacOS. Windows performs best (the program does not lag too much), while Linux (Fedora) performs worst and is basically inoperable.

I mainly use MacOS instruments when analyzing memory. It is really convenient.

[cathaysia]

Memory leaks may be beyond the scope of this issue. The main purpose of this issue is to reduce memory allocation times and memory usage when monitoring a large number of files.