Limitations of Fuse.

[vamega]

I was wondering what advantages might be possible from having something like mergerfs as a kernel module. Are there functionality/performance limitations due to mergerfs being a fuse filesystem that wouldn't exist if it was instead implemented as a Kernel module?

[trapexit]

https://github.com/trapexit/mergerfs#why-use-fuse-why-not-a-kernel-based-solution

[trapexit]

Moved this to discussions as it isn't an "issue".

Yes. There are advantages and disadvantages to having a kernel based filesystem rather than FUSE. Do you have a more specific question about them?

[vamega]

Apologies for the late response, and marking it an issue. I wasn't aware of the discussions feature on Github.
I seem to have missed that particular section of the README. I appreciate the link.
I'm specifically wondering about whether the limitations around mmap would be lifted by using a kernel module.

I'm also curious as to whether a kernel module would remove the double caching overhead. I can't imagine that the page cache is mount point aware, or I imagine bind mounts would have a huge overhead, but if you're aware of this I'd love to know more.

I ask since I want to take on a mostly academic exercise of trying to create a loadable kernel module for some subset of mergerfs functionality as a way to get more familiar with the Kernel source code.

[trapexit]

No problem.

I'm not super familiar with the implementation of overlayfs and how it manages all forms of page caching. A quick search came up with a few articles:

https://lwn.net/Articles/636943/
https://lwn.net/Articles/717950/
https://docs.docker.com/storage/storagedriver/overlayfs-driver/#overlayfs-and-docker-performance

I ask since I want to take on a mostly academic exercise of trying to create a loadable kernel module for some subset of mergerfs functionality as a way to get more familiar with the Kernel source code.

There exists a number of such things already. Some listed in the docs. aufs, overlayfs, overlayfs2, unionfs.

The mmap situation can be handled more intelligently but needs attention. The option of direct io is per file. Not at the mount level. More logic can be added to not use direct io in certain situations. Just hasn't been a priority. In part because software either handles the error gracefully or is easily worked around.

[nohajc]

I think it's a bit misleading to say

Much lower barrier to entry (getting code into the kernel takes a lot of time and effort initially)

This would of course apply if you wanted your in-kernel solution merged into mainline linux but there's also the option to ship a kernel module separately.

I'd even say it's not so obvious why you concluded "the negatives of FUSE do not outweigh the positives".
Efficiency is important for lower power hardware for example. People often say you can't run zfs on cheap hardware and they'd probably suggest snapraid + mergerfs as a more lightweight alternative even though it eats unnecessary amounts of CPU.

Also when talking about a cross platform solution, it would be fair to say that FUSE is mostly linux only. I heard something about mergerfs on BSD but on the release page, there's only a bunch of packages for various Linux distros.

So while I'd agree the obvious advantage is that a FUSE filesystem is easier to write (and maintain), the other upsides are very much debatable. Kernel module solution would definitely be worth it. I understand it would be a huge undertaking though...

[nohajc]

Anyway, I don't want to criticize... I still think this is a great project which works perfectly fine for most people.

[trapexit]

Shipping an out of kernel driver is not easy for end users who often barely know Linux. The barrier to entry doesn't just mean "getting into mainline". It also means ease of installation by users. And what is misleading? It is harder to get into mainline. Major corps can't get their shit into mainline. Why would I be different? The point of FUSE it allow out of kernel, out of mainline filesystems to be built. Lots of major projects use it at enterprise scale. As with any technology there are tradeoffs. Companies like Google are working to resolve some of them and mergerfs will benefit from them should they come. FUSE2, passthrough, ebpf, etc. would all minimize some overhead.

It's not "unnecessary" and it is still lighter weight than some other solutions. It is called being a "tradeoff". Using more CPU than if it were a kernel solution means very little when CPU usage means very little on most systems that are 80%+ idle most of the time. Most users are IO bound. Not CPU bound.

FUSE is not Linux only but mergerfs currently is because that's the main demographic. I supported FreeBSD in a best effort form for years but certain features of the software, not just FUSE, aren't supported on FreeBSD and as I don't use the platform and no one until literally yesterday said anything... I've not spent the time to support it further. Anyone can submit patches. Same as MacOS and Windows. But no one does.

You can't have it both ways. A kernel solution would be a lot less cross compatible.

This project is my own. I wrote the docs. They are my opinions. If you want to debate them so be it but I'm not changing things. I've been running this project for several years. I'm aware of the challenges with different solutions.

[nohajc]

Well, my point was some of the benefits may be true in general but not in case of mergerfs. Sure, there is potential to make it more cross platform but does anyone care? Most people will be happy to set up a linux server to run a storage pool.

Don't agree with the ease of installation though. ZFS on Linux is also out-of-tree and all you have to do is install a package containing a dkms config. I don't think getting into mainline needs to be a goal.

Look, I'm not saying you should rewrite mergerfs just because someone else wants it. It's very much usable as it is. Obviously that kind of work would benefit from more contributors.

Now I'll go read something about those FUSE improvements. Sounds interesting.

[trapexit]

"does anyone care?" Clearly some do given the requests for MacOS, BSD, and Windows support. But not enough for me to invest the time into supporting them currently.

dkms: I never claimed there was no solution to out of tree drivers to make it easier to manage. I'm very aware of dkms and have used it probably since it was created. However, it's not perfect. It isn't supported (or even allowed to be used) in some situations (particularly in enterprise.) ZFS is naturally going to use it given the large scale of the project but most serious usage of it is probably from prepackaged setups by the OS. There also just isn't much else they could do. We know for quite a while there was the fuse version of ZFS. mergerfs isn't 1/100th as complex but there was a reason they went with fuse. As does many other enterprise scale distributed filesystems.

The problem with mergerfs wrt performance is often latency. Not memory. Not CPU. Latency introduced by low end hardware or network filessystems. mergerfs doesn't do much with the CPU. Some basic resource management to keep track of nodes in the filesystem which would be just as needed in the kernel. The "CPU" people see is often really just dispatching requests to underlying filesystems. Something that would still need to happen in kernel. There would be fewer hops, maybe better scheduling (right now a problem with FUSE is thread scheduling), but that's why things are being done to address those. Pinning of threads to kernel awareness in FUSE of such things. Throw in passthrough and then mergerfs is doing little more than basic orchestration. Google wants it for Android so the CPU can go to sleep more. That's not a problem though on servers. It's more a theoretical throughput concern. And the reality is most people's systems storage is slower than mergerfs and their hardware couldn't handle more concurrency. Another reason I added thread pinning and scheduling options.

Subscribe to the fuse mailing list and fsdevel kernel list (or search them) for details.