Technical Behavior and Limitations

Do hardlinks work?

Yes. See also the option inodecalc for how inode values are calculated.

What mergerfs does not do is fake hard links across branches. Read the section "rename & link" for how it works.

Remember that hardlinks will NOT work across devices. That includes between the original filesystem and a mergerfs pool, between two separate pools of the same underlying filesystems, or bind mounts of paths within the mergerfs pool. The latter is common when using Docker or Podman. Multiple volumes (bind mounts) to the same underlying filesystem are considered different devices. There is no way to link between them. You should mount in the highest directory in the mergerfs pool that includes all the paths you need if you want links to work.

How does mergerfs handle moving and copying of files?

This is a very common mistaken assumption regarding how filesystems work. There is no such thing as "move" or "copy." These concepts are high level behaviors made up of numerous independent steps and not individual filesystem functions.

A "move" can include a "copy" so lets describe copy first.

When an application copies a file from source to destination it can do so in a number of ways but the basics are the following.

1. open the source file.
2. create the destination file.
3. read a chunk of data from source and write to destination. Continue till it runs out of data to copy.
4. Copy file metadata (stat) such as ownership (chown), permissions (chmod), timestamps (utimes), extended attributes (getxattr, setxattr), etc.
5. close source and destination files.

"move" is typically a rename(src,dst) and if that errors with EXDEV (meaning the source and destination are on different filesystems) the application will "copy" the file as described above and then it removes (unlink) the source.

The rename(src,dst), open(src), create(dst), data copying, metadata copying, unlink(src), etc. are entirely distinct and separate events. There is really no practical way to know that what is ultimately occurring is the "copying" of a file or what the source file would be. Since the source is not known there is no way to know how large a created file is destined to become. This is why it is impossible for mergerfs to choose the branch for a create based on file size. The only context provided when a file is created, besides the name, is the permissions, if it is to be read and/or written, and some low level settings for the operating system.

All of this means that mergerfs can not make decisions when a file is created based on file size or the source of the data. That information is simply not available. At best mergerfs could respond to files reaching a certain size when writing data or when a file is closed.

Related: if a user wished to have mergerfs perform certain activities based on the name of a file it is common and even best practice for a program to write to a temporary file first and then rename to its final destination. That temporary file name will typically be random and have no indication of the type of file being written.

Does FICLONE or FICLONERANGE work?

Unfortunately not. FUSE, the technology mergerfs is based on, does not support the clone_file_range feature needed for it to work. mergerfs won't even know such a request is made. The kernel will simply return an error back to the application making the request.

Should FUSE gain the ability mergerfs will be updated to support it.

Why do I get an "out of space" / "no space left on device" / ENOSPC error even though there appears to be lots of space available?

First make sure you've read the sections above about policies, path preservation, branch filtering, and the options minfreespace, moveonenospc, statfs, and statfs_ignore.

mergerfs is simply presenting a union of the content within multiple branches. The reported free space is an aggregate of space available within the pool (behavior modified by statfs and statfs_ignore). It does not represent a contiguous space. In the same way that read-only filesystems, those with quotas, or reserved space report the full theoretical space available.

Due to path preservation, branch tagging, read-only status, and minfreespace settings it is perfectly valid that ENOSPC / "out of space" / "no space left on device" be returned. It is doing what was asked of it: filtering possible branches due to those settings. Only one error can be returned and if one of the reasons for filtering a branch was minfreespace then it will be returned as such. moveonenospc is only relevant to writing a file which is too large for the filesystem it's currently on.

It is also possible that the filesystem selected has run out of inodes. Use df -i to list the total and available inodes per filesystem.

If you don't care about path preservation then simply change the create policy to one which isn't. mfs is probably what most are looking for. The reason it's not default is because it was originally set to epmfs and changing it now would change people's setup. Such a setting change will likely occur in mergerfs 3.

Why does the total available space in mergerfs not equal outside?

Are you using ext2/3/4? With reserve for root? mergerfs uses available space for statfs calculations. If you've reserved space for root then it won't show up.

You can remove the reserve by running: tune2fs -m 0 <device>

I notice massive slowdowns of writes when enabling cache.files.

When file caching is enabled in any form (cache.files!=off) it will issue getxattr requests for security.capability prior to every single write. This will usually result in performance degradation, especially when using a network filesystem (such as NFS or SMB.) Unfortunately at this moment, the kernel is not caching the response.

To work around this situation mergerfs offers a few solutions.

1. Set security_capability=false. It will short circuit any call and return ENOATTR. This still means though that mergerfs will receive the request before every write but at least it doesn't get passed through to the underlying filesystem.
2. Set xattr=noattr. Same as above but applies to all calls to getxattr. Not just security.capability. This will not be cached by the kernel either but mergerfs' runtime config system will still function.
3. Set xattr=nosys. Results in mergerfs returning ENOSYS which will be cached by the kernel. No future xattr calls will be forwarded to mergerfs. The downside is that also means the xattr based config and query functionality won't work either.
4. Disable file caching. If you aren't using applications which use mmap it's probably simpler to just disable it altogether. The kernel won't send the requests when caching is disabled.

Why can't I see my files / directories?

It's almost always a permissions issue. Unlike mhddfs and unionfs-fuse, which runs as root and attempts to access content as such, mergerfs always changes its credentials to that of the caller. This means that if the user does not have access to a file or directory than neither will mergerfs. However, because mergerfs is creating a union of paths it may be able to read some files and directories on one filesystem but not another resulting in an incomplete set.

Whenever you run into a split permission issue (seeing some but not all files) try using mergerfs.fsck tool to check for and fix the mismatch. If you aren't seeing anything at all be sure that the basic permissions are correct. The user and group values are correct and that directories have their executable bit set. A common mistake by users new to Linux is to chmod -R 644 when they should have chmod -R u=rwX,go=rX.

If using a network filesystem such as NFS or SMB (Samba) be sure to pay close attention to anything regarding permissioning and users. Root squashing and user translation for instance has bitten a few mergerfs users. Some of these also affect the use of mergerfs from container platforms such as Docker.