mirror of https://github.com/trapexit/mergerfs.git
Browse Source
Deployed 04186919 with MkDocs version: 1.6.1
Deployed 04186919 with MkDocs version: 1.6.1
Antonio SJ Musumeci
3 days ago
68 changed files with 374 additions and 3214 deletions
-
23404.html
-
23benchmarking/index.html
-
23config/branches/index.html
-
23config/cache/index.html
-
23config/deprecated_options/index.html
-
11config/export-support.md~
-
26config/export-support/index.html
-
33config/flush-on-close/index.html
-
16config/follow-symlinks.md~
-
24config/follow-symlinks/index.html
-
23config/func_readdir/index.html
-
248config/functions_categories_and_policies/index.html
-
6config/fuse_msg_size.md~
-
27config/fuse_msg_size/index.html
-
47config/inodecalc.md~
-
23config/inodecalc/index.html
-
1590config/ioctl/index.html
-
11config/link-exdev.md~
-
24config/link-exdev/index.html
-
2config/link_cow.md~
-
34config/link_cow/index.html
-
20config/nfsopenhack.md~
-
30config/nfsopenhack/index.html
-
16config/nullrw.md~
-
27config/nullrw/index.html
-
23config/options/index.html
-
15config/pin-threads.md~
-
23config/pin-threads/index.html
-
2config/readahead.md~
-
54config/readahead/index.html
-
19config/rename-exdev.md~
-
24config/rename-exdev/index.html
-
23config/rename_and_link/index.html
-
104config/statfs/index.html
-
20config/symlinkify.md~
-
27config/symlinkify/index.html
-
23config/terminology/index.html
-
9config/threads.md~
-
63config/threads/index.html
-
7config/xattr.md~
-
24config/xattr/index.html
-
23error_handling/index.html
-
23faq/compatibility_and_integration/index.html
-
23faq/configuration_and_policies/index.html
-
87faq/limit_drive_spinup/index.html
-
23faq/recommendations_and_warnings/index.html
-
23faq/reliability_and_scalability/index.html
-
27faq/technical_behavior_and_limitations/index.html
-
23faq/usage_and_functionality/index.html
-
23index.html
-
23known_issues_bugs/index.html
-
23media_and_publicity/index.html
-
23performance/index.html
-
23project_comparisons/index.html
-
23quickstart/index.html
-
27related_projects/index.html
-
23remote_filesystems/index.html
-
23runtime_interfaces/index.html
-
2search/search_index.json
-
23setup/build/index.html
-
23setup/installation/index.html
-
23setup/upgrade/index.html
-
102sitemap.xml
-
BINsitemap.xml.gz
-
27sponsorship_and_donations/index.html
-
66support/index.html
-
23tips_notes/index.html
-
23tooling/index.html
@ -1,11 +0,0 @@ |
|||
# export-support |
|||
|
|||
In theory, this flag should not be exposed to the end user. It is a |
|||
low-level FUSE flag which indicates whether or not the kernel can send |
|||
certain kinds of messages to it for the purposes of using it with |
|||
NFS. mergerfs does support these messages but due to bugs and quirks |
|||
found in the kernel and mergerfs this option is provided just in case |
|||
it is needed for debugging. |
|||
|
|||
Given that this flag is set when the FUSE connection is first |
|||
initiated it is not possible to change during run time. |
|||
@ -1,16 +0,0 @@ |
|||
# follow-symlinks |
|||
|
|||
This feature, when enabled, will cause symlinks to be interpreted by mergerfs as their target (depending on the mode). |
|||
|
|||
When there is a getattr/stat request for a file mergerfs will check if the file is a symlink and depending on the follow-symlinks setting will replace the information about the symlink with that of that which it points to. |
|||
|
|||
When unlink'ing or rmdir'ing the followed symlink it will remove the symlink itself and not that which it points to. |
|||
|
|||
never: Behave as normal. Symlinks are treated as such. |
|||
directory: Resolve symlinks only which point to directories. |
|||
regular: Resolve symlinks only which point to regular files. |
|||
all: Resolve all symlinks to that which they point to. |
|||
Symlinks which do not point to anything are left as is. |
|||
|
|||
WARNING: This feature works but there might be edge cases yet found. If you find any odd behaviors please file a ticket on github. |
|||
|
|||
@ -1,6 +0,0 @@ |
|||
# fuse_msg_size |
|||
FUSE applications communicate with the kernel over a special character device: /dev/fuse. A large portion of the overhead associated with FUSE is the cost of going back and forth between user space and kernel space over that device. Generally speaking, the fewer trips needed the better the performance will be. Reducing the number of trips can be done a number of ways. Kernel level caching and increasing message sizes being two significant ones. When it comes to reads and writes if the message size is doubled the number of trips are approximately halved. |
|||
|
|||
In Linux 4.20 a new feature was added allowing the negotiation of the max message size. Since the size is in multiples of pages the feature is called max_pages. There is a maximum max_pages value of 256 (1MiB) and minimum of 1 (4KiB). The default used by Linux >=4.20, and hardcoded value used before 4.20, is 32 (128KiB). In mergerfs it's referred to as fuse_msg_size to make it clear what it impacts and provide some abstraction. |
|||
|
|||
Since there should be no downsides to increasing fuse_msg_size / max_pages, outside a minor bump in RAM usage due to larger message buffers, mergerfs defaults the value to 256. On kernels before 4.20 the value has no effect. The reason the value is configurable is to enable experimentation and benchmarking. See the BENCHMARKING section for examples. |
|||
@ -1,47 +0,0 @@ |
|||
# inodecalc |
|||
|
|||
Inodes (st_ino) are unique identifiers within a filesystem. Each |
|||
mounted filesystem has device ID (st_dev) as well and together they |
|||
can uniquely identify a file on the whole of the system. Entries on |
|||
the same device with the same inode are in fact references to the same |
|||
underlying file. It is a many to one relationship between names and an |
|||
inode. Directories, however, do not have multiple links on most |
|||
systems due to the complexity they add. |
|||
|
|||
FUSE allows the server (mergerfs) to set inode values but not device |
|||
IDs. Creating an inode value is somewhat complex in mergerfs' case as |
|||
files aren't really in its control. If a policy changes what directory |
|||
or file is to be selected or something changes out of band it becomes |
|||
unclear what value should be used. Most software does not to care what |
|||
the values are but those that do often break if a value changes |
|||
unexpectedly. The tool find will abort a directory walk if it sees a |
|||
directory inode change. NFS can return stale handle errors if the |
|||
inode changes out of band. File dedup tools will usually leverage |
|||
device ids and inodes as a shortcut in searching for duplicate files |
|||
and would resort to full file comparisons should it find different |
|||
inode values. |
|||
|
|||
mergerfs offers multiple ways to calculate the inode in hopes of |
|||
covering different usecases. |
|||
|
|||
* `passthrough`: Passes through the underlying inode value. Mostly |
|||
intended for testing as using this does not address any of the |
|||
problems mentioned above and could confuse file deduplication |
|||
software as inodes from different filesystems can be the same. |
|||
* `path-hash`: Hashes the relative path of the entry in question. The |
|||
underlying file's values are completely ignored. This means the |
|||
inode value will always be the same for that file path. This is |
|||
useful when using NFS and you make changes out of band such as copy |
|||
data between branches. This also means that entries that do point to |
|||
the same file will not be recognizable via inodes. That does not |
|||
mean hard links don't work. They will. |
|||
* `path-hash32`: 32bit version of path-hash. |
|||
* `devino-hash`: Hashes the device id and inode of the underlying |
|||
entry. This won't prevent issues with NFS should the policy pick a |
|||
different file or files move out of band but will present the same |
|||
inode for underlying files that do too. |
|||
* `devino-hash32`: 32bit version of devino-hash. |
|||
|
|||
hybrid-hash: Performs path-hash on directories and devino-hash on other file types. Since directories can't have hard links the static value won't make a difference and the files will get values useful for finding duplicates. Probably the best to use if not using NFS. As such it is the default. |
|||
|
|||
hybrid-hash32: 32bit version of hybrid-hash. |
|||
1590
config/ioctl/index.html
File diff suppressed because it is too large
View File
File diff suppressed because it is too large
View File
@ -1,11 +0,0 @@ |
|||
# link-exdev |
|||
|
|||
If using path preservation and a `link` fails with `EXDEV` make a call |
|||
to symlink where the target is the oldlink and the linkpath is the |
|||
newpath. The target value is determined by the value of link-exdev. |
|||
|
|||
passthrough: Return EXDEV as normal. |
|||
rel-symlink: A relative path from the newpath. |
|||
abs-base-symlink: An absolute value using the underlying branch. |
|||
abs-pool-symlink: An absolute value using the mergerfs mount point. |
|||
NOTE: It is possible that some applications check the file they link. In those cases, it is possible it will error or complain. |
|||
@ -0,0 +1,2 @@ |
|||
# link_cow |
|||
|
|||
@ -1,20 +0,0 @@ |
|||
# nfsopenhack |
|||
|
|||
NFS is not fully POSIX compliant and historically certain behaviors, |
|||
such as opening files with O_EXCL, are not or not well supported. When |
|||
mergerfs (or any FUSE filesystem) is exported over NFS some of these |
|||
issues come up due to how NFS and FUSE interact. |
|||
|
|||
This hack addresses the issue where the creation of a file with a |
|||
read-only mode but with a read/write or write only flag. Normally this |
|||
is perfectly valid but NFS chops the one open call into multiple |
|||
calls. Exactly how it is translated depends on the configuration and |
|||
versions of the NFS server and clients but it results in a permission |
|||
error because a normal user is not allowed to open a read-only file as |
|||
writable. |
|||
|
|||
Even though it's a more niche situation this hack breaks normal |
|||
security and behavior and as such is `off` by default. If set to `git` |
|||
it will only perform the hack when the path in question includes |
|||
`/.git/`. `all` will result in it applying anytime a read-only file |
|||
which is empty is opened for writing. |
|||
@ -1,16 +0,0 @@ |
|||
# nullrw |
|||
|
|||
Due to how FUSE works there is an overhead to all requests made to a |
|||
FUSE filesystem that wouldn't exist for an in kernel one. Meaning that |
|||
even a simple passthrough will have some slowdown. However, generally |
|||
the overhead is minimal in comparison to the cost of the underlying |
|||
I/O. By disabling the underlying I/O we can test the theoretical |
|||
performance boundaries. |
|||
|
|||
By enabling `nullrw` mergerfs will work as it always does **except** |
|||
that all reads and writes will be no-ops. A write will succeed (the |
|||
size of the write will be returned as if it were successful) but |
|||
mergerfs does nothing with the data it was given. Similarly a read |
|||
will return the size requested but won't touch the buffer. |
|||
|
|||
See the [benchmarking](../benchmarking.md) section for suggestions on how to test. |
|||
@ -1,15 +0,0 @@ |
|||
# pin-threads |
|||
|
|||
Simple strategies for pinning read and/or process threads. If process |
|||
threads are not enabled then the strategy simply works on the read |
|||
threads. Invalid values are ignored. |
|||
|
|||
* `pin-threads=R1L`: All read threads pinned to a single logical CPU. |
|||
* `pin-threads=R1P`: All read threads pinned to a single physical CPU. |
|||
* `pin-threads=RP1L`: All read and process threads pinned to a single logical CPU. |
|||
RP1P: All read and process threads pinned to a single physical CPU. |
|||
R1LP1L: All read threads pinned to a single logical CPU, all process threads pinned to a (if possible) different logical CPU. |
|||
R1PP1P: All read threads pinned to a single physical CPU, all process threads pinned to a (if possible) different logical CPU. |
|||
RPSL: All read and process threads are spread across all logical CPUs. |
|||
RPSP: All read and process threads are spread across all physical CPUs. |
|||
R1PPSP: All read threads are pinned to a single physical CPU while process threads are spread across all other physical CPUs. |
|||
@ -0,0 +1,2 @@ |
|||
# readahead |
|||
|
|||
@ -1,19 +0,0 @@ |
|||
# rename-exdev |
|||
|
|||
If using path preservation and a `rename` fails with `EXDEV`: |
|||
|
|||
1. Move file from **/branch/a/b/c** to **/branch/.mergerfs_rename_exdev/a/b/c**. |
|||
2. symlink the rename's `newpath` to the moved file. |
|||
|
|||
The `target` value is determined by the value of `rename-exdev`. |
|||
|
|||
- passthrough: Return EXDEV as normal. |
|||
- rel-symlink: A relative path from the `newpath`. |
|||
- abs-symlink: An absolute value using the mergerfs mount point. |
|||
|
|||
NOTE: It is possible that some applications check the file they |
|||
rename. In those cases it is possible it will error or complain. |
|||
|
|||
NOTE: The reason `abs-symlink` is not split into two like `link-exdev` |
|||
is due to the complexities in managing absolute base symlinks when |
|||
multiple `oldpaths` exist. |
|||
@ -0,0 +1,20 @@ |
|||
# symlinkify |
|||
|
|||
Due to the levels of indirection introduced by mergerfs and the |
|||
underlying technology FUSE there can be varying levels of performance |
|||
degradation. This feature will turn non-directories which are not |
|||
writable into symlinks to the original file found by the `readlink` |
|||
policy after the mtime and ctime are older than the timeout. |
|||
|
|||
**WARNING:** The current implementation has a known issue in which if |
|||
the file is open and being used when the file is converted to a |
|||
symlink then the application which has that file open will receive an |
|||
error when using it. This is unlikely to occur in practice but is |
|||
something to keep in mind. |
|||
|
|||
**WARNING:** Some backup solutions, such as CrashPlan, do not backup |
|||
the target of a symlink. If using this feature it will be necessary to |
|||
point any backup software to the original filesystems or configure the |
|||
software to follow symlinks if such an option is |
|||
available. Alternatively, create two mounts. One for backup and one |
|||
for general consumption. |
|||
@ -0,0 +1,9 @@ |
|||
# threads |
|||
|
|||
## read-thread-count |
|||
|
|||
|
|||
## process-thread-count |
|||
|
|||
|
|||
## process-thread-queue-depth |
|||
@ -1,7 +0,0 @@ |
|||
# xattr |
|||
|
|||
Runtime extended attribute support can be managed via the `xattr` option. By default it will passthrough any xattr calls. Given xattr support is rarely used and can have significant performance implications mergerfs allows it to be disabled at runtime. The performance problems mostly comes when file caching is enabled. The kernel will send a `getxattr` for `security.capability` _before every single write_. It doesn't cache the responses to any `getxattr`. This might be addressed in the future but for now mergerfs can really only offer the following workarounds. |
|||
|
|||
`noattr` will cause mergerfs to short circuit all xattr calls and return ENOATTR where appropriate. mergerfs still gets all the requests but they will not be forwarded on to the underlying filesystems. The runtime control will still function in this mode. |
|||
|
|||
`nosys` will cause mergerfs to return ENOSYS for any xattr call. The difference with `noattr` is that the kernel will cache this fact and itself short circuit future calls. This is more efficient than `noattr` but will cause mergerfs' runtime control via the hidden file to stop working. |
|||
2
search/search_index.json
File diff suppressed because it is too large
View File
File diff suppressed because it is too large
View File
Write
Preview
Loading…
Cancel
Save
Reference in new issue