From c769e959e8389cadda2e9fab2f1e7e0c55156f07 Mon Sep 17 00:00:00 2001 From: Antonio SJ Musumeci Date: Tue, 8 Apr 2025 23:19:21 -0500 Subject: [PATCH] Add doc pages for minfreespace and moveonenospc + other misc updates --- mkdocs/docs/config/branches.md | 7 ++- .../config/functions_categories_policies.md | 36 +++++++----- mkdocs/docs/config/minfreespace.md | 20 +++++++ mkdocs/docs/config/moveonenospc.md | 46 +++++++++++++++ mkdocs/docs/config/options.md | 17 +++--- mkdocs/docs/faq/have_you_considered.md | 22 ++++++++ .../faq/technical_behavior_and_limitations.md | 33 ----------- mkdocs/docs/faq/why_isnt_it_working.md | 56 +++++++++++++++---- mkdocs/mkdocs.yml | 3 + 9 files changed, 170 insertions(+), 70 deletions(-) create mode 100644 mkdocs/docs/config/minfreespace.md create mode 100644 mkdocs/docs/config/moveonenospc.md create mode 100644 mkdocs/docs/faq/have_you_considered.md diff --git a/mkdocs/docs/config/branches.md b/mkdocs/docs/config/branches.md index 74e584b3..3d8ea03f 100644 --- a/mkdocs/docs/config/branches.md +++ b/mkdocs/docs/config/branches.md @@ -14,6 +14,7 @@ and a individual minfreespace value. The values are set by prepending an `=` at the end of a branch designation and using commas as delimiters. Example: `/mnt/drive=RW,1234` + ### branch mode - RW: (read/write) - Default behavior. Will be eligible in all policy @@ -24,10 +25,12 @@ delimiters. Example: `/mnt/drive=RW,1234` - NC: (no-create) - Will be excluded from `create` policies. You can't create on that branch but you can change or delete. + ### minfreespace -Same purpose and syntax as the global option but specific to the -branch. If not set the global value is used. +Same purpose and syntax as the [global option](minfreespace.md) but +specific to the branch. If not set the global value is used. + ### globbing diff --git a/mkdocs/docs/config/functions_categories_policies.md b/mkdocs/docs/config/functions_categories_policies.md index 23aa950f..5a9459b6 100644 --- a/mkdocs/docs/config/functions_categories_policies.md +++ b/mkdocs/docs/config/functions_categories_policies.md @@ -23,6 +23,7 @@ some may not be very useful in practice. For instance: **rand** very odd behavior if used for `chmod` if there were more than one copy of the file. + ## Functions and their Category classifications | Category | Functions | @@ -35,15 +36,17 @@ of the file. In cases where something may be searched for (such as a path to clone) **getattr** will usually be used. + ## Policies -A policy is the algorithm used to choose a branch or branches for a -function to work on or generally how the function behaves. +A policy is an algorithm designed to select one or more branches for a +function to operate on. Any function in the `create` category will clone the relative path if needed. Some other functions (`rename`,`link`,`ioctl`) have special requirements or behaviors which you can read more about below. + ## Filtering Most policies basically search branches and create a list of files / paths @@ -73,6 +76,7 @@ to be read-only as such (IE will set the mode `RO`) and will rerun the policy and try again. This is mostly for `ext4` filesystems that can suddenly become read-only when it encounters an error. + ## Path Preservation Policies, as described below, are of two basic classifications. `path @@ -93,6 +97,7 @@ With the `msp` or `most shared path` policies they are defined as behaviors since `ignorepponrename` is available to disable that behavior. + ## Policy descriptions A policy's behavior differs, as mentioned above, based on the function @@ -102,25 +107,25 @@ but it makes things a bit more uniform. | Policy | Description | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| all | Search: For **mkdir**, **mknod**, and **symlink** it will apply to all branches. **create** works like **ff**. | -| epall (existing path, all) | For **mkdir**, **mknod**, and **symlink** it will apply to all found. **create** works like **epff** (but more expensive because it doesn't stop after finding a valid branch). | -| epff (existing path, first found) | Given the order of the branches, as defined at mount time or configured at runtime, act on the first one found where the relative path exists. | -| eplfs (existing path, least free space) | Of all the branches on which the relative path exists choose the branch with the least free space. | -| eplus (existing path, least used space) | Of all the branches on which the relative path exists choose the branch with the least used space. | -| epmfs (existing path, most free space) | Of all the branches on which the relative path exists choose the branch with the most free space. | -| eppfrd (existing path, percentage free random distribution) | Like **pfrd** but limited to existing paths. | -| eprand (existing path, random) | Calls **epall** and then randomizes. Returns 1. | +| pfrd (percentage free random distribution) | Selects a branch randomly, with the probability of selection proportional to its available space relative to the total available space across all branches. For instance, if Branch A has 100 GB free and Branch B has 50 GB free, Branch A is twice as likely to be chosen. | +| rand (random) | Calls **all** and then randomizes. Returns 1 branch. | +| mfs (most free space) | Pick the branch with the most available free space. | | ff (first found) | Given the order of the branches, as defined at mount time or configured at runtime, act on the first one found. | | lfs (least free space) | Pick the branch with the least available free space. | | lus (least used space) | Pick the branch with the least used space. | -| mfs (most free space) | Pick the branch with the most available free space. | +| all | Search: For **mkdir**, **mknod**, and **symlink** it will apply to all branches. **create** works like **ff**. | +| msppfrd (most shared path, percentage free random distribution) | Like **eppfrd** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. | +| mspmfs (most shared path, most free space) | Like **epmfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. | | msplfs (most shared path, least free space) | Like **eplfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. | | msplus (most shared path, least used space) | Like **eplus** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. | -| mspmfs (most shared path, most free space) | Like **epmfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. | -| msppfrd (most shared path, percentage free random distribution) | Like **eppfrd** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. | +| eppfrd (existing path, percentage free random distribution) | Like **pfrd** but limited to existing paths. | +| epmfs (existing path, most free space) | Of all the branches on which the relative path exists choose the branch with the most free space. | +| eprand (existing path, random) | Calls **epall** and then randomizes. Returns 1. | +| epff (existing path, first found) | Given the order of the branches, as defined at mount time or configured at runtime, act on the first one found where the relative path exists. | +| eplfs (existing path, least free space) | Of all the branches on which the relative path exists choose the branch with the least free space. | +| eplus (existing path, least used space) | Of all the branches on which the relative path exists choose the branch with the least used space. | +| epall (existing path, all) | For **mkdir**, **mknod**, and **symlink** it will apply to all found. **create** works like **epff** (but more expensive because it doesn't stop after finding a valid branch). | | newest | Pick the file / directory with the largest mtime. | -| pfrd (percentage free random distribution) | Chooses a branch at random with the likelihood of selection based on a branch's available space relative to the total. | -| rand (random) | Calls **all** and then randomizes. Returns 1 branch. | **NOTE:** If you are using an underlying filesystem that reserves blocks such as ext2, ext3, or ext4 be aware that mergerfs respects the @@ -130,6 +135,7 @@ policy calculations. **df** does NOT use `f_bavail`, it uses `f_bfree`, so direct comparisons between **df** output and mergerfs' policies is not appropriate. + ## Defaults | Category | Policy | diff --git a/mkdocs/docs/config/minfreespace.md b/mkdocs/docs/config/minfreespace.md new file mode 100644 index 00000000..70cf92d2 --- /dev/null +++ b/mkdocs/docs/config/minfreespace.md @@ -0,0 +1,20 @@ +# minfreespace + +default: `4G` + +`minfreespace` is used to set a minimum free space threshold when +running create [policies](functions_categories_policies.md). Any +branch with free space less than the `minfreespace` value will be +skipped. This is the global default but individual branches can [have +their own value.](branches.md#minfreespace). + +This is useful for a couple reasons. + +* Since mergerfs does not split files across branches it is best to + leave space to write a newly created file on a branch to minimize + the risk of running out of space and needing + [moveonenospc](moveonenospc.md). +* Some filesystems' performance degrades when filled. `minfreespace` + provides a buffer so they are less likely to be filled. +* Similar to `ext4`'s reserved space option it minimizes the problems + that come from running out of space. diff --git a/mkdocs/docs/config/moveonenospc.md b/mkdocs/docs/config/moveonenospc.md new file mode 100644 index 00000000..baff6bf7 --- /dev/null +++ b/mkdocs/docs/config/moveonenospc.md @@ -0,0 +1,46 @@ +# moveonenospc + +When writing to a file a [number of +errors](https://man7.org/linux/man-pages/man2/write.2.html#ERRORS) are +possible. The `ENOSPC` error indicates their is no room for the +data. That could be true due to the filesystem having actually no +available space for data, or because a secondary resource (such as +inodes) has been used up, or the filesystem might have a quota feature +that is limiting how much storage a particular user may use, or (as on +`ext4` there might be a feature which reserves space for privileged +processes. + +Since mergerfs [does not offer](../index.md#non-features) splitting of +files across filesystems there are situations where a file is opened +or created on a filesystem which is nearly full and eventually +receives a `ENOSPC` error despite the pool having capacity. The +`moveonenospc` feature allows the user to have some control over this +situation. + +When enabled and a `write` fails with `ENOSPC` or `EDQUOT` mergerfs +will: + +1. Run the policy defined by `moveonenospc` to find a target branch. +2. ["Move"](../faq/technical_behavior_and_limitations.md#how-does-mergerfs-handle-moving-and-copying-of-files) + the file from the source branch to the target branch. +3. Retry the `write` and continue on as normal. + +If at any point something fails the progress so far will be cleaned up +as appropriate and the original error returned. + +If `moveonenospc` is disabled the underlying error will be returned. + +NOTE: This feature has NO affect on policies. It ONLY applies to the +literal write function. If the create function returns `ENOSPC` or the +[policy returns `ENOSPC`](functions_categories_policies.md#filtering) +that error will be returned back to the application making the +`create` request. + + +## Additional Reading + +* [Functions, Categories, Policies](functions_categories_policies.md) +* [Why do I get an "out of space" / "no space left on device" / ENOSPC + error even though there appears to be lots of space + available?](../faq/why_isnt_it_working.md#why-do-i-get-an-out-of-space-no-space-left-on-device-enospc-error-even-though-there-appears-to-be-lots-of-space-available) + diff --git a/mkdocs/docs/config/options.md b/mkdocs/docs/config/options.md index e05ef5ac..32b33597 100644 --- a/mkdocs/docs/config/options.md +++ b/mkdocs/docs/config/options.md @@ -24,18 +24,19 @@ These options are the same regardless of whether you use them with the key=val / ini style format. - **[branches](branches.md)**: Colon delimited list of branches. Used primarily in config file. -- **minfreespace=SIZE**: The minimum available space of a branch - necessary to be considered for a create +- **[minfreespace](minfreespace.md)=SIZE**: The minimum available + space of a branch necessary to be considered for a create [policy](functions_categories_policies.md). This is a default value applied to all branches and can be overwritten when configuring [branches](branches.md). Understands 'K', 'M', and 'G' to represent kilobyte, megabyte, and gigabyte respectively. (default: 4G) -- **moveonenospc=BOOL|POLICY**: When enabled if a **write** fails with - **ENOSPC** (no space left on device) or **EDQUOT** (disk quota - exceeded) the policy selected will run to find a new location for - the file. An attempt to move the file to that branch will occur - (keeping all metadata possible) and if successful the original is - unlinked and the write retried. (default: pfrd) +- **[moveonenospc](moveonenospc.md)=BOOL|POLICY**: When enabled if a + **write** fails with **ENOSPC** (no space left on device) or + **EDQUOT** (disk quota exceeded) the policy selected will run to + find a new location for the file. An attempt to move the file to + that branch will occur (keeping all metadata possible) and if + successful the original is unlinked and the write retried. (default: + pfrd) - **[inodecalc](inodecalc.md)=passthrough|path-hash|devino-hash|hybrid-hash**: Selects the inode calculation algorithm. (default: hybrid-hash) - **dropcacheonclose=BOOL**: When a file is requested to be closed diff --git a/mkdocs/docs/faq/have_you_considered.md b/mkdocs/docs/faq/have_you_considered.md new file mode 100644 index 00000000..06344a18 --- /dev/null +++ b/mkdocs/docs/faq/have_you_considered.md @@ -0,0 +1,22 @@ +# "Have You Considered?" + +## Have you considered limiting drive spinup? + +Yes. See [Limiting Drive Spinup](limit_drive_spinup.md). + + +## Have you considered porting to MacOS? + +Yes, but the FUSE implementation on MacOS has gotten a bit complicated +and fragmented. I'm not opposed to supporting it but given the low +demand, lack of the author having a modern MacOS system, the fact +mergerfs does not use standard +[libfuse](https://github.com/libfuse/libfuse) API, and MacOS FUSE +situation it is unlikely. + + +## Have you considered porting mergerfs to Windows (WinFsp)? + +Similar to MacOS the demand is very low and mergerfs does not use the +[libfuse](https://github.com/libfuse/libfuse) library so porting to +WinFsp would first mean porting it to `libfuse`. diff --git a/mkdocs/docs/faq/technical_behavior_and_limitations.md b/mkdocs/docs/faq/technical_behavior_and_limitations.md index 39f32146..37b472f5 100644 --- a/mkdocs/docs/faq/technical_behavior_and_limitations.md +++ b/mkdocs/docs/faq/technical_behavior_and_limitations.md @@ -84,39 +84,6 @@ an error back to the application making the request. Should FUSE gain the ability mergerfs will be updated to support it. -## Why do I get an "out of space" / "no space left on device" / ENOSPC error even though there appears to be lots of space available? - -First make sure you've read the sections above about policies, path -preservation, branch filtering, and the options `minfreespace`, -`moveonenospc`, `statfs`, and `statfs_ignore`. - -mergerfs is simply presenting a union of the content within multiple -branches. The reported free space is an aggregate of space available -within the pool (behavior modified by `statfs` and -`statfs_ignore`). It does not represent a contiguous space. In the -same way that read-only filesystems, those with quotas, or reserved -space report the full theoretical space available. - -Due to path preservation, branch tagging, read-only status, and -`minfreespace` settings it is perfectly valid that `ENOSPC` / "out of -space" / "no space left on device" be returned. It is doing what was -asked of it: filtering possible branches due to those settings. Only -one error can be returned and if one of the reasons for filtering a -branch was `minfreespace` then it will be returned as -such. `moveonenospc` is only relevant to writing a file which is too -large for the filesystem it's currently on. - -It is also possible that the filesystem selected has run out of -inodes. Use `df -i` to list the total and available inodes per -filesystem. - -If you don't care about path preservation then simply change the -`create` policy to one which isn't. `mfs` is probably what most are -looking for. The reason it's not default is because it was originally -set to `epmfs` and changing it now would change people's setup. Such a -setting change will likely occur in mergerfs 3. - - ## Why does the total available space in mergerfs not equal outside? Are you using ext2/3/4? With reserve for root? mergerfs uses available diff --git a/mkdocs/docs/faq/why_isnt_it_working.md b/mkdocs/docs/faq/why_isnt_it_working.md index fb1afedb..581d3a9d 100644 --- a/mkdocs/docs/faq/why_isnt_it_working.md +++ b/mkdocs/docs/faq/why_isnt_it_working.md @@ -2,23 +2,22 @@ ## I modified mergerfs' config but it still behaves the same. -mergerfs, like most filesystems, are given their options/arguments -at mount time. Unlike most filesystems, mergerfs also has the ability -to modify certain options at [runtime](../runtime_interfaces.md). +mergerfs, like other filesystems, are given their options/arguments at +mount time. You can not simply modify the [source of the +configuration](../quickstart.md#usage) and have those settings applied +any more than you would for other filesystems. It is the user's +responsibility to [restart](../setup/upgrade.md) mergerfs to pick up +the changes or use the [runtime interface](../runtime_interfaces.md). -That said: mergerfs does not actively try to monitor typical methods -of configuration (nor should it.) As such if changes are made to -`/etc/fstab`, a systemd unit file, etc. it will have no knowledge of -those changes. It is the user's responsibility to -[restart](../setup/upgrade.md) mergerfs to pick up the changes or use -the [runtime interface](../runtime_interfaces.md). +NOTE: the [runtime interface](../runtime_interfaces.md) is **just** +for runtime changes. It does **NOT** save those changed values +anywhere. ## Why are all my files ending up on 1 filesystem?! -Did you start with empty filesystems? Did you explicitly configure a -`category.create` policy? Are you using an `existing path` / `path -preserving` policy? +Did you start with empty filesystems? Are you using an `existing path` +policy? The default create policy is `epmfs`. That is a path preserving algorithm. With such a policy for `mkdir` and `create` with a set of @@ -37,6 +36,39 @@ and wish your files to be spread across all your filesystems change restrictive [policy](../config/functions_categories_policies.md). +## Why do I get an "out of space" / "no space left on device" / ENOSPC error even though there appears to be lots of space available? + +First make sure you've read the sections above about [policies, path +preservation, branch +filtering,](../config/functions_categories_policies.md) and the +options `minfreespace`, [moveonenospc](../config/moveonenospc.md), +[statfs](../config/statfs.md), and +[statfs_ignore](../config/statfs.md#statfs_ignore). + +mergerfs is simply presenting a union of the content within multiple +branches. The reported free space is an aggregate of space available +within the pool (behavior modified by `statfs` and +`statfs_ignore`). It does not represent a contiguous space. In the +same way that read-only filesystems, those with quotas, or reserved +space report the full theoretical space available. Not the practical +usable space. + +Due to using an `existing path` based policy, setting a [branch's +mode](../config/branches.md#branch-mode) to `NC` or `RO`, a +filesystems read-only status, and/or `minfreespace` setting it is +[perfectly +valid](../config/functions_categories_policies.md#filtering) that +`ENOSPC` / "out of space" / "no space left on device" be returned when +attempting to create a file despite there being actual space available +somewhere in the pool. It is doing what was asked of it: filtering +possible branches due to those settings. If that is not the behavior +you want you need to modify the settings accordingly. + +It is also possible that the filesystem selected has run out of +inodes. Use `df -i` to list the total and available inodes per +filesystem. + + ## Why isn't the create policy working? It probably is. The policies rather straight forward and well tested. diff --git a/mkdocs/mkdocs.yml b/mkdocs/mkdocs.yml index f3f6d6d6..f9a996e0 100644 --- a/mkdocs/mkdocs.yml +++ b/mkdocs/mkdocs.yml @@ -63,8 +63,10 @@ nav: - config/deprecated_options.md - config/branches.md - config/functions_categories_policies.md + - config/minfreespace.md - config/func_readdir.md - config/rename_and_link.md + - config/moveonenospc.md - config/cache.md - config/readahead.md - config/inodecalc.md @@ -100,6 +102,7 @@ nav: - faq/compatibility_and_integration.md - faq/recommendations_and_warnings.md - faq/technical_behavior_and_limitations.md + - faq/have_you_considered.md - faq/limit_drive_spinup.md - related_projects.md - media_and_publicity.md