From 9670fb970420ba07c78f0b4e45e6acd11c86ad6e Mon Sep 17 00:00:00 2001 From: Antonio SJ Musumeci Date: Sun, 29 Jan 2023 23:24:09 -0500 Subject: [PATCH] Update README regarding requesting support and policy descriptions --- README.md | 68 +++--- man/mergerfs.1 | 557 ++++++++++++++++++++++++------------------------- 2 files changed, 316 insertions(+), 309 deletions(-) diff --git a/README.md b/README.md index 02a20668..d742c619 100644 --- a/README.md +++ b/README.md @@ -493,31 +493,30 @@ A policy's behavior differs, as mentioned above, based on the function it is use | Policy | Description | |------------------|------------------------------------------------------------| -| all | Search: Same as **epall**. Action: Same as **epall**. Create: for **mkdir**, **mknod**, and **symlink** it will apply to all branches. **create** works like **ff**. | -| epall (existing path, all) | Search: Same as **epff** (but more expensive because it doesn't stop after finding a valid branch). Action: apply to all found. Create: for **mkdir**, **mknod**, and **symlink** it will apply to all found. **create** works like **epff** (but more expensive because it doesn't stop after finding a valid branch). | +| all | Search: For **mkdir**, **mknod**, and **symlink** it will apply to all branches. **create** works like **ff**. | +| epall (existing path, all) | For **mkdir**, **mknod**, and **symlink** it will apply to all found. **create** works like **epff** (but more expensive because it doesn't stop after finding a valid branch). | | epff (existing path, first found) | Given the order of the branches, as defined at mount time or configured at runtime, act on the first one found where the relative path exists. | | eplfs (existing path, least free space) | Of all the branches on which the relative path exists choose the drive with the least free space. | | eplus (existing path, least used space) | Of all the branches on which the relative path exists choose the drive with the least used space. | | epmfs (existing path, most free space) | Of all the branches on which the relative path exists choose the drive with the most free space. | | eppfrd (existing path, percentage free random distribution) | Like **pfrd** but limited to existing paths. | | eprand (existing path, random) | Calls **epall** and then randomizes. Returns 1. | -| erofs | Exclusively return **-1** with **errno** set to **EROFS** (read-only filesystem). | -| ff (first found) | Search: Same as **epff**. Action: Same as **epff**. Create: Given the order of the drives, as defined at mount time or configured at runtime, act on the first one found. | -| lfs (least free space) | Search: Same as **eplfs**. Action: Same as **eplfs**. Create: Pick the drive with the least available free space. | -| lus (least used space) | Search: Same as **eplus**. Action: Same as **eplus**. Create: Pick the drive with the least used space. | -| mfs (most free space) | Search: Same as **epmfs**. Action: Same as **epmfs**. Create: Pick the drive with the most available free space. | -| msplfs (most shared path, least free space) | Search: Same as **eplfs**. Action: Same as **eplfs**. Create: like **eplfs** but walk back the path if it fails to find a branch at that level. | -| msplus (most shared path, least used space) | Search: Same as **eplus**. Action: Same as **eplus**. Create: like **eplus** but walk back the path if it fails to find a branch at that level. | -| mspmfs (most shared path, most free space) | Search: Same as **epmfs**. Action: Same as **epmfs**. Create: like **epmfs** but walk back the path if it fails to find a branch at that level. | -| msppfrd (most shared path, percentage free random distribution) | Search: Same as **eppfrd**. Action: Same as **eppfrd**. Create: Like **eppfrd** but will walk back the path if it fails to find a branch at that level. | +| ff (first found) | Given the order of the drives, as defined at mount time or configured at runtime, act on the first one found. | +| lfs (least free space) | Pick the drive with the least available free space. | +| lus (least used space) | Pick the drive with the least used space. | +| mfs (most free space) | Pick the drive with the most available free space. | +| msplfs (most shared path, least free space) | Like **eplfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. | +| msplus (most shared path, least used space) | Like **eplus** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. | +| mspmfs (most shared path, most free space) | Like **epmfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. | +| msppfrd (most shared path, percentage free random distribution) | Like **eppfrd** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. | | newest | Pick the file / directory with the largest mtime. | -| pfrd (percentage free random distribution) | Search: Same as **eppfrd**. Action: Same as **eppfrd**. Create: Chooses a branch at random with the likelihood of selection based on a branch's available space relative to the total. | -| rand (random) | Calls **all** and then randomizes. Returns 1. | +| pfrd (percentage free random distribution) | Chooses a branch at random with the likelihood of selection based on a branch's available space relative to the total. | +| rand (random) | Calls **all** and then randomizes. Returns 1 branch. | **NOTE:** If you are using an underlying filesystem that reserves blocks such as ext2, ext3, or ext4 be aware that mergerfs respects the reservation by using `f_bavail` (number of free blocks for unprivileged users) rather than `f_bfree` (number of free blocks) in policy calculations. **df** does NOT use `f_bavail`, it uses `f_bfree`, so direct comparisons between **df** output and mergerfs' policies is not appropriate. -#### Defaults #### +#### Defaults | Category | Policy | |----------|--------| @@ -1383,28 +1382,47 @@ For non-Linux systems mergerfs uses a read-write lock and changes credentials on # SUPPORT -Filesystems are complex and difficult to debug. mergerfs, while being just a proxy of sorts, is also very difficult to debug given the large number of possible settings it can have itself and the massive number of environments it can run in. When reporting on a suspected issue **please, please** include as much of the below information as possible otherwise it will be difficult or impossible to diagnose. Also please make sure to read all of the above documentation as it includes nearly every known system or user issue previously encountered. +Filesystems are complex and difficult to debug. mergerfs, while being +just a proxy of sorts, can be difficult to debug given the large +number of possible settings it can have itself and the number of +environments it can run in. When reporting on a suspected issue +**please** include as much of the below information as possible +otherwise it will be difficult or impossible to diagnose. Also please +read the above documentation as it provides details on many previously +encountered questions/issues. + + +**Please make sure you are using the [latest +release](https://github.com/trapexit/mergerfs/releases) or have tried +it in comparison. Old versions, which are often included in distros +like Debian and Ubuntu, are not ever going to be updated and your bug +may have been addressed already.** -**Please make sure you are using the [latest release](https://github.com/trapexit/mergerfs/releases) or have tried it in comparison. Old versions, which are often included in distros like Debian and Ubuntu, are not ever going to be updated and your bug may have been addressed already.** **For commercial support or feature requests please contact me directly.** #### Information to include in bug reports +* [Information about the broader problem along with any attempted + solutions.](https://xyproblem.info) +* Solution already ruled out and why. * Version of mergerfs: `mergerfs -V` -* mergerfs settings / arguments: from fstab, systemd unit, command line, etc. -* Version of the OS: `uname -a` +* mergerfs settings / arguments: from fstab, systemd unit, command + line, OMV plugin, etc. +* Version of the OS: `uname -a` and `lsb_release -a` * List of branches, their filesystem types, sizes (before and after issue): `df -h` -* **All** information about the relevant branches and paths: permissions, ownership, etc. +* **All** information about the relevant paths and files: permissions, ownership, etc. * **All** information about the client app making the requests: version, uid/gid -* Runtime environment: mostly are things running inside containers or not +* Runtime environment: + * Is mergerfs running within a container? + * Are the client apps using mergerfs running in a container? * A `strace` of the app having problems: * `strace -fvTtt -s 256 -o /tmp/app.strace.txt ` -* A `strace` of mergerfs while the program is trying to do whatever it's failing to do: +* A `strace` of mergerfs while the program is trying to do whatever it is failing to do: * `strace -fvTtt -s 256 -p -o /tmp/mergerfs.strace.txt` * **Precise** directions on replicating the issue. Do not leave **anything** out. -* Try to recreate the problem in the simplest way using standard programs: ln, mv, cp, ls, dd, etc. +* Try to recreate the problem in the simplest way using standard programs: `ln`, `mv`, `cp`, `ls`, `dd`, etc. #### Contact / Issue submission @@ -1418,7 +1436,10 @@ Filesystems are complex and difficult to debug. mergerfs, while being just a pro #### Support development -This software is released under the very liberal ISC license and is therefore free to use for personal or commercial uses. That said if you like this software and have the means please consider supporting its development. +This software is released under the very liberal ISC license and is +therefore free to use for personal or commercial uses. That said if +you like this software and have the means please consider supporting +its development. https://github.com/trapexit/support @@ -1431,4 +1452,3 @@ https://github.com/trapexit/support * https://github.com/trapexit/mergerfs-tools * https://github.com/trapexit/scorch * https://github.com/trapexit/bbf -* https://github.com/trapexit/backup-and-recovery-howtos diff --git a/man/mergerfs.1 b/man/mergerfs.1 index 9060aade..849b1a57 100644 --- a/man/mergerfs.1 +++ b/man/mergerfs.1 @@ -1,14 +1,14 @@ .\"t -.\" Automatically generated by Pandoc 2.5 +.\" Automatically generated by Pandoc 2.9.2.1 .\" -.TH "mergerfs" "1" "2022\-05\-23" "mergerfs user manual" "" +.TH "mergerfs" "1" "2023-01-26" "mergerfs user manual" "" .hy .SH NAME .PP -mergerfs \- a featureful union filesystem +mergerfs - a featureful union filesystem .SH SYNOPSIS .PP -mergerfs \-o +mergerfs -o .SH DESCRIPTION .PP \f[B]mergerfs\f[R] is a union filesystem geared towards simplifying @@ -34,11 +34,11 @@ Works with heterogeneous filesystem types .IP \[bu] 2 Moving of file when filesystem runs out of space while writing .IP \[bu] 2 -Ignore read\-only filesystems when creating files +Ignore read-only filesystems when creating files .IP \[bu] 2 -Turn read\-only files into symlinks to underlying file +Turn read-only files into symlinks to underlying file .IP \[bu] 2 -Hard link copy\-on\-write / CoW +Hard link copy-on-write / CoW .IP \[bu] 2 Support for POSIX ACLs .IP \[bu] 2 @@ -56,29 +56,29 @@ Read more about policies below. A + B = C /disk1 /disk2 /merged | | | -+\-\- /dir1 +\-\- /dir1 +\-\- /dir1 ++-- /dir1 +-- /dir1 +-- /dir1 | | | | | | -| +\-\- file1 | +\-\- file2 | +\-\- file1 -| | +\-\- file3 | +\-\- file2 -+\-\- /dir2 | | +\-\- file3 -| | +\-\- /dir3 | -| +\-\- file4 | +\-\- /dir2 -| +\-\- file5 | | -+\-\- file6 | +\-\- file4 +| +-- file1 | +-- file2 | +-- file1 +| | +-- file3 | +-- file2 ++-- /dir2 | | +-- file3 +| | +-- /dir3 | +| +-- file4 | +-- /dir2 +| +-- file5 | | ++-- file6 | +-- file4 | - +\-\- /dir3 + +-- /dir3 | | - | +\-\- file5 + | +-- file5 | - +\-\- file6 + +-- file6 \f[R] .fi .PP -mergerfs does \f[B]NOT\f[R] support the copy\-on\-write (CoW) or -whiteout behaviors found in \f[B]aufs\f[R] and \f[B]overlayfs\f[R]. -You can \f[B]not\f[R] mount a read\-only filesystem and write to it. -However, mergerfs will ignore read\-only drives when creating new files -so you can mix read\-write and read\-only drives. +mergerfs does \f[B]NOT\f[R] support the copy-on-write (CoW) or whiteout +behaviors found in \f[B]aufs\f[R] and \f[B]overlayfs\f[R]. +You can \f[B]not\f[R] mount a read-only filesystem and write to it. +However, mergerfs will ignore read-only drives when creating new files +so you can mix read-write and read-only drives. It also does \f[B]NOT\f[R] split data across drives. It is not RAID0 / striping. It is simply a union of other filesystems. @@ -106,19 +106,18 @@ If you don\[cq]t already know that you have a special use case then just start with one of the following option sets. .SS You need \f[C]mmap\f[R] (used by rtorrent and many sqlite3 base software) .PP -\f[C]allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=mfs\f[R] +\f[C]cache.files=partial,dropcacheonclose=true,category.create=mfs\f[R] .SS You don\[cq]t need \f[C]mmap\f[R] .PP -\f[C]allow_other,use_ino,cache.files=off,dropcacheonclose=true,category.create=mfs\f[R] +\f[C]cache.files=off,dropcacheonclose=true,category.create=mfs\f[R] .PP See the mergerfs wiki for real world deployments (https://github.com/trapexit/mergerfs/wiki/Real-World-Deployments) for comparisons / ideas. .SH OPTIONS .PP -These options are the same regardless you use them with the -\f[C]mergerfs\f[R] commandline program, used in fstab, or in a config -file. +These options are the same regardless of whether you use them with the +\f[C]mergerfs\f[R] commandline program, in fstab, or in a config file. .SS mount options .IP \[bu] 2 \f[B]config\f[R]: Path to a config file. @@ -126,10 +125,6 @@ Same arguments as below in key=val / ini style format. .IP \[bu] 2 \f[B]branches\f[R]: Colon delimited list of branches. .IP \[bu] 2 -\f[B]allow_other\f[R]: A libfuse option which allows users besides the -one which ran mergerfs to see the filesystem. -This is required for most use\-cases. -.IP \[bu] 2 \f[B]minfreespace=SIZE\f[R]: The minimum space value used for creation policies. Can be overridden by branch specific option. @@ -146,19 +141,14 @@ metadata possible) and if successful the original is unlinked and the write retried. (default: false, true = mfs) .IP \[bu] 2 -\f[B]use_ino\f[R]: Causes mergerfs to supply file/directory inodes -rather than libfuse. -While not a default it is recommended it be enabled so that linked files -share the same inode value. -.IP \[bu] 2 -\f[B]inodecalc=passthrough|path\-hash|devino\-hash|hybrid\-hash\f[R]: +\f[B]inodecalc=passthrough|path-hash|devino-hash|hybrid-hash\f[R]: Selects the inode calculation algorithm. -(default: hybrid\-hash) +(default: hybrid-hash) .IP \[bu] 2 \f[B]dropcacheonclose=BOOL\f[R]: When a file is requested to be closed call \f[C]posix_fadvise\f[R] on it first to instruct the kernel that we no longer need the data and it can drop its cache. -Recommended when \f[B]cache.files=partial|full|auto\-full\f[R] to limit +Recommended when \f[B]cache.files=partial|full|auto-full\f[R] to limit double caching. (default: false) .IP \[bu] 2 @@ -172,7 +162,7 @@ Please read more below before using. the \f[B]symlinkify\f[R] behavior. (default: 3600) .IP \[bu] 2 -\f[B]nullrw=BOOL\f[R]: Turns reads and writes into no\-ops. +\f[B]nullrw=BOOL\f[R]: Turns reads and writes into no-ops. The request will succeed but do nothing. Useful for benchmarking mergerfs. (default: false) @@ -180,7 +170,7 @@ Useful for benchmarking mergerfs. \f[B]ignorepponrename=BOOL\f[R]: Ignore path preserving on rename. Typically rename and link act differently depending on the policy of \f[C]create\f[R] (read below). -Enabling this will cause rename and link to always use the non\-path +Enabling this will cause rename and link to always use the non-path preserving behavior. This means files, when renamed or linked, will stay on the same drive. (default: false) @@ -199,8 +189,8 @@ disabled. \f[B]link_cow=BOOL\f[R]: When enabled if a regular file is opened which has a link count > 1 it will copy the file to a temporary file and rename over the original. -Breaking the link and providing a basic copy\-on\-write function similar -to cow\-shell. +Breaking the link and providing a basic copy-on-write function similar +to cow-shell. (default: false) .IP \[bu] 2 \f[B]statfs=base|full\f[R]: Controls how statfs works. @@ -210,25 +200,30 @@ path exists. (default: base) .IP \[bu] 2 \f[B]statfs_ignore=none|ro|nc\f[R]: `ro' will cause statfs calculations -to ignore available space for branches mounted or tagged as `read\-only' +to ignore available space for branches mounted or tagged as `read-only' or `no create'. `nc' will ignore available space for branches tagged as `no create'. (default: none) .IP \[bu] 2 \f[B]nfsopenhack=off|git|all\f[R]: A workaround for exporting mergerfs over NFS where there are issues with creating files for write while -setting the mode to read\-only. +setting the mode to read-only. (default: off) .IP \[bu] 2 -\f[B]follow\-symlinks=never|directory|regular|all\f[R]: Turns symlinks +\f[B]branches-mount-timeout=UINT\f[R]: Number of seconds to wait at +startup for branches to be a mount other than the mountpoint\[cq]s +filesystem. +(default: 0) +.IP \[bu] 2 +\f[B]follow-symlinks=never|directory|regular|all\f[R]: Turns symlinks into what they point to. (default: never) .IP \[bu] 2 -\f[B]link\-exdev=passthrough|rel\-symlink|abs\-base\-symlink|abs\-pool\-symlink\f[R]: +\f[B]link-exdev=passthrough|rel-symlink|abs-base-symlink|abs-pool-symlink\f[R]: When a link fails with EXDEV optionally create a symlink to the file instead. .IP \[bu] 2 -\f[B]rename\-exdev=passthrough|rel\-symlink|abs\-symlink\f[R]: When a +\f[B]rename-exdev=passthrough|rel-symlink|abs-symlink\f[R]: When a rename fails with EXDEV optionally move the file to a special directory and symlink to it. .IP \[bu] 2 @@ -247,19 +242,38 @@ message. Only available on Linux >= 4.20 and ignored otherwise. (min: 1; max: 256; default: 256) .IP \[bu] 2 -\f[B]threads=INT\f[R]: Number of threads to use in multithreaded mode. +\f[B]threads=INT\f[R]: Number of threads to use. +When used alone (\f[C]process-thread-count=-1\f[R]) it sets the number +of threads reading and processing FUSE messages. +When used together it sets the number of threads reading from FUSE. When set to zero it will attempt to discover and use the number of logical cores. -If the lookup fails it will fall back to using 4. If the thread count is set negative it will look up the number of cores then divide by the absolute value. ie. -threads=\-2 on an 8 core machine will result in 8 / 2 = 4 threads. +threads=-2 on an 8 core machine will result in 8 / 2 = 4 threads. There will always be at least 1 thread. +If set to -1 in combination with \f[C]process-thread-count\f[R] then it +will try to pick reasonable values based on CPU thread count. NOTE: higher number of threads increases parallelism but usually decreases throughput. (default: 0) .IP \[bu] 2 +\f[B]read-thread-count=INT\f[R]: Alias for \f[C]threads\f[R]. +.IP \[bu] 2 +\f[B]process-thread-count=INT\f[R]: Enables separate thread pool to +asynchronously process FUSE requests. +In this mode \f[C]read-thread-count\f[R] refers to the number of threads +reading FUSE messages which are dispatched to process threads. +-1 means disabled otherwise acts like \f[C]read-thread-count\f[R]. +(default: -1) +.IP \[bu] 2 +\f[B]scheduling-priority=INT\f[R]: Set mergerfs\[cq] scheduling +priority. +Valid values range from -20 to 19. +See \f[C]setpriority\f[R] man page for more details. +(default: -10) +.IP \[bu] 2 \f[B]fsname=STR\f[R]: Sets the name of the filesystem as seen in \f[B]mount\f[R], \f[B]df\f[R], etc. Defaults to a list of the source paths concatenated together with the @@ -286,7 +300,9 @@ the search category. (default: 0) .IP \[bu] 2 \f[B]cache.statfs=UINT\f[R]: `statfs' cache timeout in seconds. -(default: 0) +(default: +.RS 2 +.RE .IP \[bu] 2 \f[B]cache.attr=UINT\f[R]: File attribute cache timeout in seconds. (default: 1) @@ -298,7 +314,7 @@ the search category. timeout in seconds. (default: 0) .IP \[bu] 2 -\f[B]cache.files=libfuse|off|partial|full|auto\-full\f[R]: File page +\f[B]cache.files=libfuse|off|partial|full|auto-full\f[R]: File page caching mode (default: libfuse) .IP \[bu] 2 \f[B]cache.writeback=BOOL\f[R]: Enable kernel writeback caching @@ -310,24 +326,24 @@ caching mode (default: libfuse) \f[B]cache.readdir=BOOL\f[R]: Cache readdir (if supported by kernel) (default: false) .IP \[bu] 2 -\f[B]direct_io\f[R]: deprecated \- Bypass page cache. +\f[B]direct_io\f[R]: deprecated - Bypass page cache. Use \f[C]cache.files=off\f[R] instead. (default: false) .IP \[bu] 2 -\f[B]kernel_cache\f[R]: deprecated \- Do not invalidate data cache on +\f[B]kernel_cache\f[R]: deprecated - Do not invalidate data cache on file open. Use \f[C]cache.files=full\f[R] instead. (default: false) .IP \[bu] 2 -\f[B]auto_cache\f[R]: deprecated \- Invalidate data cache if file mtime +\f[B]auto_cache\f[R]: deprecated - Invalidate data cache if file mtime or size change. -Use \f[C]cache.files=auto\-full\f[R] instead. +Use \f[C]cache.files=auto-full\f[R] instead. (default: false) .IP \[bu] 2 -\f[B]async_read\f[R]: deprecated \- Perform reads asynchronously. +\f[B]async_read\f[R]: deprecated - Perform reads asynchronously. Use \f[C]async_read=true\f[R] instead. .IP \[bu] 2 -\f[B]sync_read\f[R]: deprecated \- Perform reads synchronously. +\f[B]sync_read\f[R]: deprecated - Perform reads synchronously. Use \f[C]async_read=false\f[R] instead. .PP \f[B]NOTE:\f[R] Options are evaluated in the order listed so if the @@ -370,15 +386,15 @@ designation and using commas as delimiters. Example: /mnt/drive=RW,1234 .SS branch type .IP \[bu] 2 -RW: (read/write) \- Default behavior. +RW: (read/write) - Default behavior. Will be eligible in all policy categories. .IP \[bu] 2 -RO: (read\-only) \- Will be excluded from \f[C]create\f[R] and +RO: (read-only) - Will be excluded from \f[C]create\f[R] and \f[C]action\f[R] policies. -Same as a read\-only mounted filesystem would be (though faster to +Same as a read-only mounted filesystem would be (though faster to process). .IP \[bu] 2 -NC: (no\-create) \- Will be excluded from \f[C]create\f[R] policies. +NC: (no-create) - Will be excluded from \f[C]create\f[R] policies. You can\[cq]t create on that branch but you can change or delete. .SS minfreespace .PP @@ -393,7 +409,7 @@ the shell itself will apply the glob itself.\f[R] .IP .nf \f[C] -# mergerfs \-o allow_other,use_ino /mnt/disk\[rs]*:/mnt/cdrom /media/drives +# mergerfs /mnt/disk\[rs]*:/mnt/cdrom /media/drives \f[R] .fi .PP @@ -406,7 +422,7 @@ tools use \f[B]/etc/fstab\f[R]. .nf \f[C] # -/mnt/disk*:/mnt/cdrom /mnt/pool fuse.mergerfs allow_other,use_ino 0 0 +/mnt/disk*:/mnt/cdrom /mnt/pool fuse.mergerfs minfreespace=16G 0 0 \f[R] .fi .PP @@ -453,7 +469,7 @@ Mostly intended for testing as using this does not address any of the problems mentioned above and could confuse file deduplication software as inodes from different filesystems can be the same. .IP \[bu] 2 -path\-hash: Hashes the relative path of the entry in question. +path-hash: Hashes the relative path of the entry in question. The underlying file\[cq]s values are completely ignored. This means the inode value will always be the same for that file path. This is useful when using NFS and you make changes out of band such as @@ -463,24 +479,24 @@ recognizable via inodes. That \f[B]does not\f[R] mean hard links don\[cq]t work. They will. .IP \[bu] 2 -path\-hash32: 32bit version of path\-hash. +path-hash32: 32bit version of path-hash. .IP \[bu] 2 -devino\-hash: Hashes the device id and inode of the underlying entry. +devino-hash: Hashes the device id and inode of the underlying entry. This won\[cq]t prevent issues with NFS should the policy pick a different file or files move out of band but will present the same inode for underlying files that do too. .IP \[bu] 2 -devino\-hash32: 32bit version of devino\-hash. +devino-hash32: 32bit version of devino-hash. .IP \[bu] 2 -hybrid\-hash: Performs \f[C]path\-hash\f[R] on directories and -\f[C]devino\-hash\f[R] on other file types. +hybrid-hash: Performs \f[C]path-hash\f[R] on directories and +\f[C]devino-hash\f[R] on other file types. Since directories can\[cq]t have hard links the static value won\[cq]t make a difference and the files will get values useful for finding duplicates. Probably the best to use if not using NFS. As such it is the default. .IP \[bu] 2 -hybrid\-hash32: 32bit version of hybrid\-hash. +hybrid-hash32: 32bit version of hybrid-hash. .PP 32bit versions are provided as there is some software which does not handle 64bit inodes well. @@ -512,11 +528,8 @@ and the kernel use internally (also called the \[dq]nodeid\[dq]). \f[R] .fi .PP -In the future the \f[C]use_ino\f[R] option will probably be removed as -this feature should replace the original libfuse inode calculation -strategy. -Currently you still need to use \f[C]use_ino\f[R] in order to enable -\f[C]inodecalc\f[R]. +As of version 2.35.0 the \f[C]use_ino\f[R] option has been removed. +mergerfs should always be managing inode values. .SS fuse_msg_size .PP FUSE applications communicate with the kernel over a special character @@ -550,13 +563,13 @@ On kernels before 4.20 the value has no effect. The reason the value is configurable is to enable experimentation and benchmarking. See the BENCHMARKING section for examples. -.SS follow\-symlinks +.SS follow-symlinks .PP This feature, when enabled, will cause symlinks to be interpreted by mergerfs as their target (depending on the mode). .PP When there is a getattr/stat request for a file mergerfs will check if -the file is a symlink and depending on the \f[C]follow\-symlinks\f[R] +the file is a symlink and depending on the \f[C]follow-symlinks\f[R] setting will replace the information about the symlink with that of that which it points to. .PP @@ -577,25 +590,25 @@ Symlinks which do not point to anything are left as is. WARNING: This feature works but there might be edge cases yet found. If you find any odd behaviors please file a ticket on github (https://github.com/trapexit/mergerfs/issues). -.SS link\-exdev +.SS link-exdev .PP If using path preservation and a \f[C]link\f[R] fails with EXDEV make a call to \f[C]symlink\f[R] where the \f[C]target\f[R] is the \f[C]oldlink\f[R] and the \f[C]linkpath\f[R] is the \f[C]newpath\f[R]. The \f[C]target\f[R] value is determined by the value of -\f[C]link\-exdev\f[R]. +\f[C]link-exdev\f[R]. .IP \[bu] 2 passthrough: Return EXDEV as normal. .IP \[bu] 2 -rel\-symlink: A relative path from the \f[C]newpath\f[R]. +rel-symlink: A relative path from the \f[C]newpath\f[R]. .IP \[bu] 2 -abs\-base\-symlink: A absolute value using the underlying branch. +abs-base-symlink: A absolute value using the underlying branch. .IP \[bu] 2 -abs\-pool\-symlink: A absolute value using the mergerfs mount point. +abs-pool-symlink: A absolute value using the mergerfs mount point. .PP NOTE: It is possible that some applications check the file they link. In those cases it is possible it will error or complain. -.SS rename\-exdev +.SS rename-exdev .PP If using path preservation and a \f[C]rename\f[R] fails with EXDEV: .IP "1." 3 @@ -605,26 +618,26 @@ Move file from \f[B]/branch/a/b/c\f[R] to symlink the rename\[cq]s \f[C]newpath\f[R] to the moved file. .PP The \f[C]target\f[R] value is determined by the value of -\f[C]rename\-exdev\f[R]. +\f[C]rename-exdev\f[R]. .IP \[bu] 2 passthrough: Return EXDEV as normal. .IP \[bu] 2 -rel\-symlink: A relative path from the \f[C]newpath\f[R]. +rel-symlink: A relative path from the \f[C]newpath\f[R]. .IP \[bu] 2 -abs\-symlink: A absolute value using the mergerfs mount point. +abs-symlink: A absolute value using the mergerfs mount point. .PP NOTE: It is possible that some applications check the file they rename. In those cases it is possible it will error or complain. .PP -NOTE: The reason \f[C]abs\-symlink\f[R] is not split into two like -\f[C]link\-exdev\f[R] is due to the complexities in managing absolute +NOTE: The reason \f[C]abs-symlink\f[R] is not split into two like +\f[C]link-exdev\f[R] is due to the complexities in managing absolute base symlinks when multiple \f[C]oldpaths\f[R] exist. .SS symlinkify .PP Due to the levels of indirection introduced by mergerfs and the underlying technology FUSE there can be varying levels of performance degradation. -This feature will turn non\-directories which are not writable into +This feature will turn non-directories which are not writable into symlinks to the original file found by the \f[C]readlink\f[R] policy after the mtime and ctime are older than the timeout. .PP @@ -652,7 +665,7 @@ By disabling the underlying I/O we can test the theoretical performance boundaries. .PP By enabling \f[C]nullrw\f[R] mergerfs will work as it always does -\f[B]except\f[R] that all reads and writes will be no\-ops. +\f[B]except\f[R] that all reads and writes will be no-ops. A write will succeed (the size of the write will be returned as if it were successful) but mergerfs does nothing with the data it was given. Similarly a read will return the size requested but won\[cq]t touch the @@ -692,12 +705,12 @@ When mergerfs (or any FUSE filesystem) is exported over NFS some of these issues come up due to how NFS and FUSE interact. .PP This hack addresses the issue where the creation of a file with a -read\-only mode but with a read/write or write only flag. +read-only mode but with a read/write or write only flag. Normally this is perfectly valid but NFS chops the one open call into multiple calls. Exactly how it is translated depends on the configuration and versions of the NFS server and clients but it results in a permission error -because a normal user is not allowed to open a read\-only file as +because a normal user is not allowed to open a read-only file as writable. .PP Even though it\[cq]s a more niche situation this hack breaks normal @@ -734,8 +747,8 @@ the base path provided is used. Not the full path to the file in question. Meaning that mounts in the branch won\[cq]t be considered in the space calculations. -The reason is that it doesn\[cq]t really work for non\-path preserving -policies and can lead to non\-obvious behaviors. +The reason is that it doesn\[cq]t really work for non-path preserving +policies and can lead to non-obvious behaviors. .PP NOTE: While any policy can be assigned to a function or category though some may not be very useful in practice. @@ -797,31 +810,31 @@ Policies basically search branches and create a list of files / paths for functions to work on. The policy is responsible for filtering and sorting the branches. Filters include \f[B]minfreespace\f[R], whether or not a branch is -mounted read\-only, and the branch tagging (RO,NC,RW). +mounted read-only, and the branch tagging (RO,NC,RW). These filters are applied across all policies unless otherwise noted. .IP \[bu] 2 No \f[B]search\f[R] function policies filter. .IP \[bu] 2 All \f[B]action\f[R] function policies filter out branches which are -mounted \f[B]read\-only\f[R] or tagged as \f[B]RO (read\-only)\f[R]. +mounted \f[B]read-only\f[R] or tagged as \f[B]RO (read-only)\f[R]. .IP \[bu] 2 All \f[B]create\f[R] function policies filter out branches which are -mounted \f[B]read\-only\f[R], tagged \f[B]RO (read\-only)\f[R] or -\f[B]NC (no create)\f[R], or has available space less than +mounted \f[B]read-only\f[R], tagged \f[B]RO (read-only)\f[R] or \f[B]NC +(no create)\f[R], or has available space less than \f[C]minfreespace\f[R]. .PP Policies may have their own additional filtering such as those that require existing paths to be present. .PP If all branches are filtered an error will be returned. -Typically \f[B]EROFS\f[R] (read\-only filesystem) or \f[B]ENOSPC\f[R] -(no space left on device) depending on the most recent reason for -filtering a branch. +Typically \f[B]EROFS\f[R] (read-only filesystem) or \f[B]ENOSPC\f[R] (no +space left on device) depending on the most recent reason for filtering +a branch. \f[B]ENOENT\f[R] will be returned if no eligible branch is found. .SS Path Preservation .PP Policies, as described below, are of two basic types. -\f[C]path preserving\f[R] and \f[C]non\-path preserving\f[R]. +\f[C]path preserving\f[R] and \f[C]non-path preserving\f[R]. .PP All policies which start with \f[C]ep\f[R] (\f[B]epff\f[R], \f[B]eplfs\f[R], \f[B]eplus\f[R], \f[B]epmfs\f[R], \f[B]eprand\f[R]) are @@ -831,7 +844,7 @@ All policies which start with \f[C]ep\f[R] (\f[B]epff\f[R], A path preserving policy will only consider drives where the relative path being accessed already exists. .PP -When using non\-path preserving policies paths will be cloned to target +When using non-path preserving policies paths will be cloned to target drives as necessary. .PP With the \f[C]msp\f[R] or \f[C]most shared path\f[R] policies they are @@ -861,20 +874,15 @@ _ T{ all T}@T{ -Search: Same as \f[B]epall\f[R]. -Action: Same as \f[B]epall\f[R]. -Create: for \f[B]mkdir\f[R], \f[B]mknod\f[R], and \f[B]symlink\f[R] it +Search: For \f[B]mkdir\f[R], \f[B]mknod\f[R], and \f[B]symlink\f[R] it will apply to all branches. \f[B]create\f[R] works like \f[B]ff\f[R]. T} T{ epall (existing path, all) T}@T{ -Search: Same as \f[B]epff\f[R] (but more expensive because it -doesn\[cq]t stop after finding a valid branch). -Action: apply to all found. -Create: for \f[B]mkdir\f[R], \f[B]mknod\f[R], and \f[B]symlink\f[R] it -will apply to all found. +For \f[B]mkdir\f[R], \f[B]mknod\f[R], and \f[B]symlink\f[R] it will +apply to all found. \f[B]create\f[R] works like \f[B]epff\f[R] (but more expensive because it doesn\[cq]t stop after finding a valid branch). T} @@ -914,71 +922,53 @@ Calls \f[B]epall\f[R] and then randomizes. Returns 1. T} T{ -erofs -T}@T{ -Exclusively return \f[B]\-1\f[R] with \f[B]errno\f[R] set to -\f[B]EROFS\f[R] (read\-only filesystem). -T} -T{ ff (first found) T}@T{ -Search: Same as \f[B]epff\f[R]. -Action: Same as \f[B]epff\f[R]. -Create: Given the order of the drives, as defined at mount time or -configured at runtime, act on the first one found. +Given the order of the drives, as defined at mount time or configured at +runtime, act on the first one found. T} T{ lfs (least free space) T}@T{ -Search: Same as \f[B]eplfs\f[R]. -Action: Same as \f[B]eplfs\f[R]. -Create: Pick the drive with the least available free space. +Pick the drive with the least available free space. T} T{ lus (least used space) T}@T{ -Search: Same as \f[B]eplus\f[R]. -Action: Same as \f[B]eplus\f[R]. -Create: Pick the drive with the least used space. +Pick the drive with the least used space. T} T{ mfs (most free space) T}@T{ -Search: Same as \f[B]epmfs\f[R]. -Action: Same as \f[B]epmfs\f[R]. -Create: Pick the drive with the most available free space. +Pick the drive with the most available free space. T} T{ msplfs (most shared path, least free space) T}@T{ -Search: Same as \f[B]eplfs\f[R]. -Action: Same as \f[B]eplfs\f[R]. -Create: like \f[B]eplfs\f[R] but walk back the path if it fails to find -a branch at that level. +Like \f[B]eplfs\f[R] but if it fails to find a branch it will try again +with the parent directory. +Continues this pattern till finding one. T} T{ msplus (most shared path, least used space) T}@T{ -Search: Same as \f[B]eplus\f[R]. -Action: Same as \f[B]eplus\f[R]. -Create: like \f[B]eplus\f[R] but walk back the path if it fails to find -a branch at that level. +Like \f[B]eplus\f[R] but if it fails to find a branch it will try again +with the parent directory. +Continues this pattern till finding one. T} T{ mspmfs (most shared path, most free space) T}@T{ -Search: Same as \f[B]epmfs\f[R]. -Action: Same as \f[B]epmfs\f[R]. -Create: like \f[B]epmfs\f[R] but walk back the path if it fails to find -a branch at that level. +Like \f[B]epmfs\f[R] but if it fails to find a branch it will try again +with the parent directory. +Continues this pattern till finding one. T} T{ msppfrd (most shared path, percentage free random distribution) T}@T{ -Search: Same as \f[B]eppfrd\f[R]. -Action: Same as \f[B]eppfrd\f[R]. -Create: Like \f[B]eppfrd\f[R] but will walk back the path if it fails to -find a branch at that level. +Like \f[B]eppfrd\f[R] but if it fails to find a branch it will try again +with the parent directory. +Continues this pattern till finding one. T} T{ newest @@ -988,16 +978,14 @@ T} T{ pfrd (percentage free random distribution) T}@T{ -Search: Same as \f[B]eppfrd\f[R]. -Action: Same as \f[B]eppfrd\f[R]. -Create: Chooses a branch at random with the likelihood of selection -based on a branch\[cq]s available space relative to the total. +Chooses a branch at random with the likelihood of selection based on a +branch\[cq]s available space relative to the total. T} T{ rand (random) T}@T{ Calls \f[B]all\f[R] and then randomizes. -Returns 1. +Returns 1 branch. T} .TE .PP @@ -1056,7 +1044,7 @@ filesystem. \f[C]rename\f[R] only works within a single filesystem or device. If a rename can\[cq]t be done atomically due to the source and destination paths existing on different mount points it will return -\f[B]\-1\f[R] with \f[B]errno = EXDEV\f[R] (cross device / improper +\f[B]-1\f[R] with \f[B]errno = EXDEV\f[R] (cross device / improper link). So if a \f[C]rename\f[R]\[cq]s source and target are on different drives within the pool it creates an issue. @@ -1067,7 +1055,7 @@ This made the code simple and was technically compliant with POSIX requirements. However, many applications fail to handle EXDEV at all and treat it as a normal error or otherwise handle it poorly. -Such apps include: gvfsd\-fuse v1.20.3 and prior, Finder / CIFS/SMB +Such apps include: gvfsd-fuse v1.20.3 and prior, Finder / CIFS/SMB client in Apple OSX 10.9+, NZBGet, Samba\[cq]s recycling bin feature. .PP As a result a compromise was made in order to get most software to work @@ -1089,7 +1077,7 @@ policy If create policy returns the same drive as currently evaluating then clone the path .IP \[bu] 2 -Re\-attempt rename +Re-attempt rename .RE .IP \[bu] 2 If \f[B]any\f[R] of the renames succeed the higher level rename is @@ -1229,7 +1217,7 @@ First get the code from github (https://github.com/trapexit/mergerfs). \f[C] $ git clone https://github.com/trapexit/mergerfs.git $ # or -$ wget https://github.com/trapexit/mergerfs/releases/download//mergerfs\-.tar.gz +$ wget https://github.com/trapexit/mergerfs/releases/download//mergerfs-.tar.gz \f[R] .fi .SS Debian / Ubuntu @@ -1237,20 +1225,20 @@ $ wget https://github.com/trapexit/mergerfs/releases/download//mergerfs\-_.deb +$ sudo dpkg -i ../mergerfs__.deb \f[R] .fi .SS RHEL / CentOS /Fedora .IP .nf \f[C] -$ su \- +$ su - # cd mergerfs -# tools/install\-build\-pkgs +# tools/install-build-pkgs # make rpm -# rpm \-i rpmbuild/RPMS//mergerfs\-..rpm +# rpm -i rpmbuild/RPMS//mergerfs-..rpm \f[R] .fi .SS Generically @@ -1271,9 +1259,9 @@ $ sudo make install $ make help usage: make -make USE_XATTR=0 \- build program without xattrs functionality -make STATIC=1 \- build static binary -make LTO=1 \- build with link time optimization +make USE_XATTR=0 - build program without xattrs functionality +make STATIC=1 - build static binary +make LTO=1 - build with link time optimization \f[R] .fi .SH UPGRADE @@ -1283,8 +1271,8 @@ instance. Simply install the new version of mergerfs and follow the instructions below. .PP -Add \f[C]nonempty\f[R] to your mergerfs option list and call mergerfs -again or if using \f[C]/etc/fstab\f[R] call for it to mount again. +Run mergerfs again or if using \f[C]/etc/fstab\f[R] call for it to mount +again. Existing open files and such will continue to work fine though they won\[cq]t see runtime changes since any such change would be the new mount. @@ -1304,7 +1292,7 @@ A problem with this approach is that the underlying instance will continue to run even if the software using it stop or are restarted. To work around this you can use a \[lq]lazy umount\[rq]. Before mounting over top the mount point with the new instance of -mergerfs issue: \f[C]umount \-l \f[R]. +mergerfs issue: \f[C]umount -l \f[R]. .SH RUNTIME CONFIG .SS .mergerfs pseudo file .IP @@ -1325,10 +1313,10 @@ If you wish for values to persist they must be included as options wherever you configure the mounting of mergerfs (/etc/fstab). .SS Keys .PP -Use \f[C]getfattr \-d /mountpoint/.mergerfs\f[R] or -\f[C]xattr \-l /mountpoint/.mergerfs\f[R] to see all supported keys. -Some are informational and therefore read\-only. -\f[C]setxattr\f[R] will return EINVAL (invalid argument) on read\-only +Use \f[C]getfattr -d /mountpoint/.mergerfs\f[R] or +\f[C]xattr -l /mountpoint/.mergerfs\f[R] to see all supported keys. +Some are informational and therefore read-only. +\f[C]setxattr\f[R] will return EINVAL (invalid argument) on read-only keys. .SS Values .PP @@ -1364,23 +1352,23 @@ T}@T{ append T} T{ -\-[list] +-[list] T}@T{ remove all values provided T} T{ -\-< +-< T}@T{ remove first in list T} T{ -\-> +-> T}@T{ remove last in list T} .TE .PP -\f[C]xattr \-w user.mergerfs.branches + userspace +Given the relatively high cost of FUSE due to the kernel <-> userspace round trips there are kernel side caches for file entries and attributes. The entry cache limits the \f[C]lookup\f[R] calls to mergerfs which ask @@ -1534,7 +1522,7 @@ The options for setting these are \f[C]cache.entry\f[R] and \f[C]cache.negative_entry\f[R] for the entry cache and \f[C]cache.attr\f[R] for the attributes cache. \f[C]cache.negative_entry\f[R] refers to the timeout for negative -responses to lookups (non\-existent files). +responses to lookups (non-existent files). .SS writeback caching .PP When \f[C]cache.files\f[R] is enabled the default is for it to perform @@ -1588,7 +1576,7 @@ read/write presumably to keep file handle usage low). Of the syscalls used by mergerfs in policies the \f[C]statfs\f[R] / \f[C]statvfs\f[R] call is perhaps the most expensive. It\[cq]s used to find out the available space of a drive and whether it -is mounted read\-only. +is mounted read-only. Depending on the setup and usage pattern these queries can be relatively costly. When \f[C]cache.statfs\f[R] is enabled all calls to \f[C]statfs\f[R] by @@ -1648,10 +1636,10 @@ RAID would probably be the better solution. If you\[cq]re going to use mergerfs there are other tactics that may help: spreading the data across drives (see the mergerfs.dup tool) and setting \f[C]func.open=rand\f[R], using \f[C]symlinkify\f[R], or using -dm\-cache or a similar technology to add tiered cache to the underlying +dm-cache or a similar technology to add tiered cache to the underlying device. .PP -With #2 one could use dm\-cache as well but there is another solution +With #2 one could use dm-cache as well but there is another solution which requires only mergerfs and a cronjob. .IP "1." 3 Create 2 mergerfs pools. @@ -1692,10 +1680,10 @@ frequency is appropriate for your workflow. .PP Move files from cache to backing pool based only on the last time the file was accessed. -Replace \f[C]\-atime\f[R] with \f[C]\-amin\f[R] if you want minutes -rather than days. -May want to use the \f[C]fadvise\f[R] / \f[C]\-\-drop\-cache\f[R] -version of rsync or run rsync with the tool \[lq]nocache\[rq]. +Replace \f[C]-atime\f[R] with \f[C]-amin\f[R] if you want minutes rather +than days. +May want to use the \f[C]fadvise\f[R] / \f[C]--drop-cache\f[R] version +of rsync or run rsync with the tool \[lq]nocache\[rq]. .PP \f[I]NOTE:\f[R] The arguments to these scripts include the cache \f[B]drive\f[R]. @@ -1707,7 +1695,7 @@ You could have data loss if the source is the cache pool. #!/bin/bash if [ $# != 3 ]; then - echo \[dq]usage: $0 \[dq] + echo \[dq]usage: $0 \[dq] exit 1 fi @@ -1715,8 +1703,8 @@ CACHE=\[dq]${1}\[dq] BACKING=\[dq]${2}\[dq] N=${3} -find \[dq]${CACHE}\[dq] \-type f \-atime +${N} \-printf \[aq]%P\[rs]n\[aq] | \[rs] - rsync \-\-files\-from=\- \-axqHAXWES \-\-preallocate \-\-remove\-source\-files \[dq]${CACHE}/\[dq] \[dq]${BACKING}/\[dq] +find \[dq]${CACHE}\[dq] -type f -atime +${N} -printf \[aq]%P\[rs]n\[aq] | \[rs] + rsync --files-from=- -axqHAXWES --preallocate --remove-source-files \[dq]${CACHE}/\[dq] \[dq]${BACKING}/\[dq] \f[R] .fi .SS percentage full expiring @@ -1734,7 +1722,7 @@ You could have data loss if the source is the cache pool. #!/bin/bash if [ $# != 3 ]; then - echo \[dq]usage: $0 \[dq] + echo \[dq]usage: $0 \[dq] exit 1 fi @@ -1742,15 +1730,15 @@ CACHE=\[dq]${1}\[dq] BACKING=\[dq]${2}\[dq] PERCENTAGE=${3} -set \-o errexit -while [ $(df \-\-output=pcent \[dq]${CACHE}\[dq] | grep \-v Use | cut \-d\[aq]%\[aq] \-f1) \-gt ${PERCENTAGE} ] +set -o errexit +while [ $(df --output=pcent \[dq]${CACHE}\[dq] | grep -v Use | cut -d\[aq]%\[aq] -f1) -gt ${PERCENTAGE} ] do - FILE=$(find \[dq]${CACHE}\[dq] \-type f \-printf \[aq]%A\[at] %P\[rs]n\[aq] | \[rs] + FILE=$(find \[dq]${CACHE}\[dq] -type f -printf \[aq]%A\[at] %P\[rs]n\[aq] | \[rs] sort | \[rs] - head \-n 1 | \[rs] - cut \-d\[aq] \[aq] \-f2\-) - test \-n \[dq]${FILE}\[dq] - rsync \-axqHAXWESR \-\-preallocate \-\-remove\-source\-files \[dq]${CACHE}/./${FILE}\[dq] \[dq]${BACKING}/\[dq] + head -n 1 | \[rs] + cut -d\[aq] \[aq] -f2-) + test -n \[dq]${FILE}\[dq] + rsync -axqHAXWESR --preallocate --remove-source-files \[dq]${CACHE}/./${FILE}\[dq] \[dq]${BACKING}/\[dq] done \f[R] .fi @@ -1803,17 +1791,17 @@ disable \f[C]async_read\f[R] test theoretical performance using \f[C]nullrw\f[R] or mounting a ram disk .IP \[bu] 2 -use \f[C]symlinkify\f[R] if your data is largely static and read\-only +use \f[C]symlinkify\f[R] if your data is largely static and read-only .IP \[bu] 2 use tiered cache drives .IP \[bu] 2 use LVM and LVM cache to place a SSD in front of your HDDs .IP \[bu] 2 if \f[C]cache.files\f[R] is enabled increase readahead: -\f[C]echo \[dq]1024\[dq] > /sys/class/bdi/0:$(stat \-c%d /MOUNT)/read_ahead_kb\f[R] +\f[C]echo \[dq]1024\[dq] > /sys/class/bdi/0:$(stat -c%d /MOUNT)/read_ahead_kb\f[R] .IP \[bu] 2 increase readahead on all devices: -\f[C]ls \-1 /sys/class/bdi/*/read_ahead_kb | xargs \-n1 \-I{} sh \-c \[dq]echo 1024 > {}\[dq]\f[R] +\f[C]ls -1 /sys/class/bdi/*/read_ahead_kb | xargs -n1 -I{} sh -c \[dq]echo 1024 > {}\[dq]\f[R] .PP If you come across a setting that significantly impacts performance please contact trapexit so he may investigate further. @@ -1844,7 +1832,7 @@ narrow down precisely which component is leading to the slowdown. Preferably test the following in the order listed (but not combined). .IP "1." 3 Enable \f[C]nullrw\f[R] mode with \f[C]nullrw=true\f[R]. -This will effectively make reads and writes no\-ops. +This will effectively make reads and writes no-ops. Removing the underlying device / filesystem from the equation. This will give us the top theoretical speeds. .IP "2." 3 @@ -1852,7 +1840,7 @@ Mount mergerfs over \f[C]tmpfs\f[R]. \f[C]tmpfs\f[R] is a RAM disk. Extremely high speed and very low latency. This is a more realistic best case scenario. -Example: \f[C]mount \-t tmpfs \-o size=2G tmpfs /tmp/tmpfs\f[R] +Example: \f[C]mount -t tmpfs -o size=2G tmpfs /tmp/tmpfs\f[R] .IP "3." 3 Mount mergerfs over a local drive. NVMe, SSD, HDD, etc. @@ -1930,12 +1918,9 @@ shouldn\[cq]t be used. Ensure you\[cq]re using the latest version. Few distros have the latest version. .IP \[bu] 2 -\f[B]use_ino\f[R] will only work when used with mergerfs 2.18.0 and -above. -.IP \[bu] 2 -Run mergerfs as \f[C]root\f[R] (with \f[B]allow_other\f[R]) unless -you\[cq]re merging paths which are owned exclusively and fully by the -same user otherwise strange permission issues may arise. +Run mergerfs as \f[C]root\f[R] unless you\[cq]re merging paths which are +owned exclusively and fully by the same user otherwise strange +permission issues may arise. mergerfs is designed and intended to be run as \f[C]root\f[R]. .IP \[bu] 2 If you don\[cq]t see some directories and files you expect, policies @@ -1947,14 +1932,14 @@ permissions. If you still have permission issues be sure you are using POSIX ACL compliant filesystems. mergerfs doesn\[cq]t generally make exceptions for FAT, NTFS, or other -non\-POSIX filesystem. +non-POSIX filesystem. .IP \[bu] 2 Do \f[B]not\f[R] use \f[C]cache.files=off\f[R] if you expect applications (such as rtorrent) to use mmap (http://linux.die.net/man/2/mmap) files. Shared mmap is not currently supported in FUSE w/ page caching disabled. Enabling \f[C]dropcacheonclose\f[R] is recommended when -\f[C]cache.files=partial|full|auto\-full\f[R]. +\f[C]cache.files=partial|full|auto-full\f[R]. .IP \[bu] 2 Kodi (http://kodi.tv), Plex (http://plex.tv), Subsonic (http://subsonic.org), etc. @@ -2009,8 +1994,8 @@ Run in verbose mode to better understand what\[cq]s happening: .IP .nf \f[C] -$ mv \-v /mnt/pool/foo /mnt/disk1/foo -copied \[aq]/mnt/pool/foo\[aq] \-> \[aq]/mnt/disk1/foo\[aq] +$ mv -v /mnt/pool/foo /mnt/disk1/foo +copied \[aq]/mnt/pool/foo\[aq] -> \[aq]/mnt/disk1/foo\[aq] removed \[aq]/mnt/pool/foo\[aq] $ ls /mnt/pool/foo ls: cannot access \[aq]/mnt/pool/foo\[aq]: No such file or directory @@ -2039,12 +2024,10 @@ Be sure to use the following options: .IP \[bu] 2 noforget .IP \[bu] 2 -use_ino -.IP \[bu] 2 -inodecalc=path\-hash +inodecalc=path-hash .SS rtorrent fails with ENODEV (No such device) .PP -Be sure to set \f[C]cache.files=partial|full|auto\-full\f[R] or turn off +Be sure to set \f[C]cache.files=partial|full|auto-full\f[R] or turn off \f[C]direct_io\f[R]. rtorrent and some other applications use mmap (http://linux.die.net/man/2/mmap) to read and write to files and @@ -2085,8 +2068,8 @@ indicate actual drive or OS errors. The error will only be returned by mergerfs if using a path preserving policy as described in the policy section above. If you do not care about path preservation simply change the mergerfs -policy to the non\-path preserving version. -For example: \f[C]\-o category.create=mfs\f[R] +policy to the non-path preserving version. +For example: \f[C]-o category.create=mfs\f[R] .PP Ideally the offending software would be fixed and it is recommended that if you run into this problem you contact the software\[cq]s author and @@ -2103,7 +2086,7 @@ Workaround: Copy the file/directory and then remove the original rather than move. .PP This isn\[cq]t an issue with Samba but some SMB clients. -GVFS\-fuse v1.20.3 and prior (found in Ubuntu 14.04 among others) failed +GVFS-fuse v1.20.3 and prior (found in Ubuntu 14.04 among others) failed to handle certain error codes correctly. Particularly \f[B]STATUS_NOT_SAME_DEVICE\f[R] which comes from the \f[B]EXDEV\f[R] which is returned by \f[B]rename\f[R] when the call is @@ -2113,19 +2096,19 @@ alternate action to accomplish its goal. In the case of \f[B]mv\f[R] or similar it tries \f[B]rename\f[R] and on \f[B]EXDEV\f[R] falls back to a manual copying of data between the two locations and unlinking the source. -In these older versions of GVFS\-fuse if it received \f[B]EXDEV\f[R] it +In these older versions of GVFS-fuse if it received \f[B]EXDEV\f[R] it would translate that into \f[B]EIO\f[R]. This would cause \f[B]mv\f[R] or most any application attempting to move files around on that SMB share to fail with a IO error. .PP -GVFS\-fuse v1.22.0 (https://bugzilla.gnome.org/show_bug.cgi?id=734568) +GVFS-fuse v1.22.0 (https://bugzilla.gnome.org/show_bug.cgi?id=734568) and above fixed this issue but a large number of systems use the older release. On Ubuntu the version can be checked by issuing -\f[C]apt\-cache showpkg gvfs\-fuse\f[R]. +\f[C]apt-cache showpkg gvfs-fuse\f[R]. Most distros released in 2015 seem to have the updated release and will work fine but older systems may not. -Upgrading gvfs\-fuse or the distro in general will address the problem. +Upgrading gvfs-fuse or the distro in general will address the problem. .PP In Apple\[cq]s MacOSX 10.9 they replaced Samba (client and server) with their own product. @@ -2140,18 +2123,15 @@ software doesn\[cq]t handle the situation well. This is unfortunately a common failure of software which moves files around. The standard indicates that an implementation \f[C]MAY\f[R] choose to -support non\-user home directory trashing of files (which is a +support non-user home directory trashing of files (which is a \f[C]MUST\f[R]). The implementation \f[C]MAY\f[R] also support \[lq]top directory trashes\[rq] which many probably do. .PP To create a \f[C]$topdir/.Trash\f[R] directory as defined in the standard use the -mergerfs\-tools (https://github.com/trapexit/mergerfs-tools) tool +mergerfs-tools (https://github.com/trapexit/mergerfs-tools) tool \f[C]mergerfs.mktrash\f[R]. -.SS tar: Directory renamed before its status could be extracted -.PP -Make sure to use the \f[C]use_ino\f[R] option. .SS Supplemental user groups .PP Due to the overhead of @@ -2205,7 +2185,7 @@ If using mergerfs below 2.22.0: If suddenly the mergerfs mount point disappears and \f[C]Transport endpoint is not connected\f[R] is returned when attempting to perform actions within the mount directory \f[B]and\f[R] -the version of libfuse (use \f[C]mergerfs \-v\f[R] to find the version) +the version of libfuse (use \f[C]mergerfs -v\f[R] to find the version) is older than \f[C]2.9.4\f[R] its likely due to a bug in libfuse. Affected versions of libfuse can be found in Debian Wheezy, Ubuntu Precise and others. @@ -2228,7 +2208,7 @@ in which an invalid message appears to be transmitted to libfuse (used by mergerfs) causing it to exit. No messages will be printed in any logs as it\[cq]s not a proper crash. Debugging of the issue is still ongoing and can be followed via the -fuse\-devel +fuse-devel thread (https://sourceforge.net/p/fuse/mailman/message/35662577). .SS rm: fts_read failed: No such file or directory .PP @@ -2316,7 +2296,7 @@ the documentation will be improved. .PP That said, for the average person, the following should be fine: .PP -\f[C]\-o use_ino,cache.files=off,dropcacheonclose=true,allow_other,category.create=mfs\f[R] +\f[C]-o cache.files=off,dropcacheonclose=true,category.create=mfs\f[R] .SS Why are all my files ending up on 1 drive?! .PP Did you start with empty drives? @@ -2347,9 +2327,8 @@ grouping together directory content by drive. .SS Do hardlinks work? .PP Yes. -You need to use \f[C]use_ino\f[R] to support proper reporting of inodes -but they work regardless. -See also the option \f[C]inodecalc\f[R]. +See also the option \f[C]inodecalc\f[R] for how inode values are +calculated. .PP What mergerfs does not do is fake hard links across branches. Read the section \[lq]rename & link\[rq] for how it works. @@ -2372,7 +2351,7 @@ They are completely unrelated pieces of software. .PP Yes. With Docker you\[cq]ll need to include -\f[C]\-\-cap\-add=SYS_ADMIN \-\-device=/dev/fuse \-\-security\-opt=apparmor:unconfined\f[R] +\f[C]--cap-add=SYS_ADMIN --device=/dev/fuse --security-opt=apparmor:unconfined\f[R] or similar with other container runtimes. You should also be running it as root or given sufficient caps to change user and group identity as well as have root like filesystem @@ -2389,25 +2368,25 @@ managing shared files. .PP Also, as mentioned by hotio (https://hotio.dev/containers/mergerfs), with Docker you should probably be mounting with -\f[C]bind\-propagation\f[R] set to \f[C]slave\f[R]. -.SS Does mergerfs support CoW / copy\-on\-write / writes to read\-only filesystems? +\f[C]bind-propagation\f[R] set to \f[C]slave\f[R]. +.SS Does mergerfs support CoW / copy-on-write / writes to read-only filesystems? .PP Not in the sense of a filesystem like BTRFS or ZFS nor in the overlayfs or aufs sense. It does offer a -cow\-shell (http://manpages.ubuntu.com/manpages/bionic/man1/cow-shell.1.html) +cow-shell (http://manpages.ubuntu.com/manpages/bionic/man1/cow-shell.1.html) like hard link breaking (copy to temp file then rename over original) which can be useful when wanting to save space by hardlinking duplicate files but wish to treat each name as if it were a unique and separate file. .PP -If you want to write to a read\-only filesystem you should look at +If you want to write to a read-only filesystem you should look at overlayfs. You can always include the overlayfs mount into a mergerfs pool. .SS Why can\[cq]t I see my files / directories? .PP It\[cq]s almost always a permissions issue. -Unlike mhddfs and unionfs\-fuse, which runs as root and attempts to +Unlike mhddfs and unionfs-fuse, which runs as root and attempts to access content as such, mergerfs always changes its credentials to that of the caller. This means that if the user does not have access to a file or directory @@ -2424,8 +2403,8 @@ If you aren\[cq]t seeing anything at all be sure that the basic permissions are correct. The user and group values are correct and that directories have their executable bit set. -A common mistake by users new to Linux is to \f[C]chmod \-R 644\f[R] -when they should have \f[C]chmod \-R u=rwX,go=rX\f[R]. +A common mistake by users new to Linux is to \f[C]chmod -R 644\f[R] when +they should have \f[C]chmod -R u=rwX,go=rX\f[R]. .PP If using a network filesystem such as NFS, SMB, CIFS (Samba) be sure to pay close attention to anything regarding permissioning and users. @@ -2506,8 +2485,8 @@ Longer term the plan is to rewrite mergerfs to use the low level API. .PP See above first. .PP -If/when mergerfs is rewritten to use the low\-level API then it\[cq]ll -be plausible to support system libfuse but till then it\[cq]s simply too +If/when mergerfs is rewritten to use the low-level API then it\[cq]ll be +plausible to support system libfuse but till then it\[cq]s simply too much work to manage the differences across the versions. .SS Why use mergerfs over mhddfs? .PP @@ -2518,16 +2497,16 @@ the same or maybe better performance. .PP Below is an example of mhddfs and mergerfs setup to work similarly. .PP -\f[C]mhddfs \-o mlimit=4G,allow_other /mnt/drive1,/mnt/drive2 /mnt/pool\f[R] +\f[C]mhddfs -o mlimit=4G,allow_other /mnt/drive1,/mnt/drive2 /mnt/pool\f[R] .PP -\f[C]mergerfs \-o minfreespace=4G,allow_other,category.create=ff /mnt/drive1:/mnt/drive2 /mnt/pool\f[R] +\f[C]mergerfs -o minfreespace=4G,category.create=ff /mnt/drive1:/mnt/drive2 /mnt/pool\f[R] .SS Why use mergerfs over aufs? .PP aufs is mostly abandoned and no longer available in many distros. .PP While aufs can offer better peak performance mergerfs provides more configurability and is generally easier to use. -mergerfs however does not offer the overlay / copy\-on\-write (CoW) +mergerfs however does not offer the overlay / copy-on-write (CoW) features which aufs and overlayfs have. .SS Why use mergerfs over unionfs? .PP @@ -2606,10 +2585,10 @@ The reported free space is an aggregate of space available within the pool (behavior modified by \f[B]statfs\f[R] and \f[B]statfs_ignore\f[R]). It does not represent a contiguous space. -In the same way that read\-only filesystems, those with quotas, or +In the same way that read-only filesystems, those with quotas, or reserved space report the full theoretical space available. .PP -Due to path preservation, branch tagging, read\-only status, and +Due to path preservation, branch tagging, read-only status, and \f[B]minfreespace\f[R] settings it is perfectly valid that \f[C]ENOSPC\f[R] / \[lq]out of space\[rq] / \[lq]no space left on device\[rq] be returned. @@ -2621,7 +2600,7 @@ branch was \f[B]minfreespace\f[R] then it will be returned as such. large for the drive its currently on. .PP It is also possible that the filesystem selected has run out of inodes. -Use \f[C]df \-i\f[R] to list the total and available inodes per +Use \f[C]df -i\f[R] to list the total and available inodes per filesystem. .PP If you don\[cq]t care about path preservation then simply change the @@ -2637,7 +2616,7 @@ With reserve for root? mergerfs uses available space for statfs calculations. If you\[cq]ve reserved space for root then it won\[cq]t show up. .PP -You can remove the reserve by running: \f[C]tune2fs \-m 0 \f[R] +You can remove the reserve by running: \f[C]tune2fs -m 0 \f[R] .SS Can mergerfs mounts be exported over NFS? .PP Yes, however if you do anything which may changes files out of band @@ -2648,9 +2627,7 @@ Be sure to use the following options: .IP \[bu] 2 noforget .IP \[bu] 2 -use_ino -.IP \[bu] 2 -inodecalc=path\-hash +inodecalc=path-hash .SS Can mergerfs mounts be exported over Samba / SMB? .PP Yes. @@ -2722,7 +2699,7 @@ that thread only. Jumping back to \f[B]root\f[R] as necessary should escalated privileges be needed (for instance: to clone paths between drives). .PP -For non\-Linux systems mergerfs uses a read\-write lock and changes +For non-Linux systems mergerfs uses a read-write lock and changes credentials only when necessary. If multiple threads are to be user X then only the first one will need to change the processes credentials. @@ -2737,14 +2714,14 @@ assuming there are few users. .SH SUPPORT .PP Filesystems are complex and difficult to debug. -mergerfs, while being just a proxy of sorts, is also very difficult to -debug given the large number of possible settings it can have itself and -the massive number of environments it can run in. -When reporting on a suspected issue \f[B]please, please\f[R] include as -much of the below information as possible otherwise it will be difficult -or impossible to diagnose. -Also please make sure to read all of the above documentation as it -includes nearly every known system or user issue previously encountered. +mergerfs, while being just a proxy of sorts, can be difficult to debug +given the large number of possible settings it can have itself and the +number of environments it can run in. +When reporting on a suspected issue \f[B]please\f[R] include as much of +the below information as possible otherwise it will be difficult or +impossible to diagnose. +Also please read the above documentation as it provides details on many +previously encountered questions/issues. .PP \f[B]Please make sure you are using the latest release (https://github.com/trapexit/mergerfs/releases) or have tried it @@ -2756,42 +2733,54 @@ have been addressed already.\f[R] directly.\f[R] .SS Information to include in bug reports .IP \[bu] 2 -Version of mergerfs: \f[C]mergerfs \-V\f[R] +Information about the broader problem along with any attempted +solutions. (https://xyproblem.info) +.IP \[bu] 2 +Solution already ruled out and why. +.IP \[bu] 2 +Version of mergerfs: \f[C]mergerfs -V\f[R] .IP \[bu] 2 mergerfs settings / arguments: from fstab, systemd unit, command line, -etc. +OMV plugin, etc. .IP \[bu] 2 -Version of the OS: \f[C]uname \-a\f[R] +Version of the OS: \f[C]uname -a\f[R] and \f[C]lsb_release -a\f[R] .IP \[bu] 2 List of branches, their filesystem types, sizes (before and after -issue): \f[C]df \-h\f[R] +issue): \f[C]df -h\f[R] .IP \[bu] 2 -\f[B]All\f[R] information about the relevant branches and paths: +\f[B]All\f[R] information about the relevant paths and files: permissions, ownership, etc. .IP \[bu] 2 \f[B]All\f[R] information about the client app making the requests: version, uid/gid .IP \[bu] 2 -Runtime environment: mostly are things running inside containers or not +Runtime environment: +.RS 2 +.IP \[bu] 2 +Is mergerfs running within a container? +.IP \[bu] 2 +Are the client apps using mergerfs running in a container? +.RE .IP \[bu] 2 A \f[C]strace\f[R] of the app having problems: .RS 2 .IP \[bu] 2 -\f[C]strace \-fvTtt \-s 256 \-o /tmp/app.strace.txt \f[R] +\f[C]strace -fvTtt -s 256 -o /tmp/app.strace.txt \f[R] .RE .IP \[bu] 2 A \f[C]strace\f[R] of mergerfs while the program is trying to do -whatever it\[cq]s failing to do: +whatever it is failing to do: .RS 2 .IP \[bu] 2 -\f[C]strace \-fvTtt \-s 256 \-p \-o /tmp/mergerfs.strace.txt\f[R] +\f[C]strace -fvTtt -s 256 -p -o /tmp/mergerfs.strace.txt\f[R] .RE .IP \[bu] 2 \f[B]Precise\f[R] directions on replicating the issue. Do not leave \f[B]anything\f[R] out. .IP \[bu] 2 Try to recreate the problem in the simplest way using standard programs: -ln, mv, cp, ls, dd, etc. +\f[C]ln\f[R], \f[C]mv\f[R], \f[C]cp\f[R], \f[C]ls\f[R], \f[C]dd\f[R], +etc. .SS Contact / Issue submission .IP \[bu] 2 github.com: https://github.com/trapexit/mergerfs/issues @@ -2819,12 +2808,10 @@ https://github.com/trapexit/mergerfs .IP \[bu] 2 https://github.com/trapexit/mergerfs/wiki .IP \[bu] 2 -https://github.com/trapexit/mergerfs\-tools +https://github.com/trapexit/mergerfs-tools .IP \[bu] 2 https://github.com/trapexit/scorch .IP \[bu] 2 https://github.com/trapexit/bbf -.IP \[bu] 2 -https://github.com/trapexit/backup\-and\-recovery\-howtos .SH AUTHORS Antonio SJ Musumeci .