diff --git a/mkdocs/docs/faq/recommendations_and_warnings.md b/mkdocs/docs/faq/recommendations_and_warnings.md index c8ea83b5..def5c479 100644 --- a/mkdocs/docs/faq/recommendations_and_warnings.md +++ b/mkdocs/docs/faq/recommendations_and_warnings.md @@ -9,9 +9,9 @@ * VM images: For the same reasons as databases. VM images are accessed very aggressively and mergerfs will introduce a lot of extra latency. * As replacement for RAID: mergerfs is just for pooling branches. If - you need that kind of device performance aggregation or high - availability you should stick with RAID. However, it is fine to put - a filesystem which is on a RAID setup in mergerfs. + you need device performance aggregation or high availability you + should stick with RAID. However, it is fine to put a filesystem + which is on a RAID setup in mergerfs. **However, if using [passthrough](../config/passthrough.md) the above situations are less likely to be a concern. Best to do testing for diff --git a/mkdocs/docs/faq/why_isnt_it_working.md b/mkdocs/docs/faq/why_isnt_it_working.md index d17788ca..0bf5f819 100644 --- a/mkdocs/docs/faq/why_isnt_it_working.md +++ b/mkdocs/docs/faq/why_isnt_it_working.md @@ -230,13 +230,29 @@ the software's author and request proper handling of `EXDEV` errors. * [Moving files and directories fails with Samba](../known_issues_bugs.md#moving-files-and-directories-fails-with-samba) -## Why is rtorrent failing with ENODEV (No such device)? +## Why is software needing mmap failing? -Be sure to set -[cache.files=partial|full|auto-full|per-process](../config/cache.md) -or use Linux kernel v6.6 or above. rtorrent and some other -applications use [mmap](http://linux.die.net/man/2/mmap) to read and -write to files and offer no fallback to traditional methods. +See [Known Issues and Bugs#3rd Party +Software](../known_issues_bugs.md#3rd-party-software) and +[QuickStart](../quickstart.md). + + +## Why don't quick scans fail to pick up new media? + +Media consumption software like Plex, Jellyfin, Emby, Airsonic, +Navidrome, etc. will use timestamps of directories to see if it should +be scanned. Be sure to set `func.getattr=newest` to catch that. + +A full scan would still pick up all media and for those concerned +about device activity it is recommened to disable quick scans and +leave periodic full scans so the activity is managed. Or to configure +your software which manages downloads (such as *Arr family of +software) to trigger specific scans when files are added to the pool. + +Additional reading: + +* [Does inotify and fanotify +work?](compatibility_and_integration.md#does-inotify-and-fanotify-work) ## Why don't I see mergerfs options in mount command or /proc/mounts? diff --git a/mkdocs/docs/index.md b/mkdocs/docs/index.md index c9de6602..1012cf43 100644 --- a/mkdocs/docs/index.md +++ b/mkdocs/docs/index.md @@ -5,7 +5,10 @@ [union filesystem](https://en.wikipedia.org/wiki/Union_mount) geared towards simplifying storage and management of files across numerous commodity storage devices. It is similar to [**mhddfs**, **unionfs**, -**aufs**, **DrivePool**, etc.](project_comparisons.md) +**aufs**, **DrivePool**, etc.](project_comparisons.md) Allowing for +the aggregation of storage without the more rigid requirements and +failure conditions of [RAID and other +technologies](project_comparisons.md). ## Features @@ -35,7 +38,8 @@ commodity storage devices. It is similar to [**mhddfs**, **unionfs**, * Read/write overlay on top of read-only filesystem like OverlayFS * File whiteout -* RAID like parity calculation (see [SnapRAID](https://www.snapraid.it)) +* RAID like redundancy (see [SnapRAID](https://www.snapraid.it) and + [NonRAID](https://github.com/qvr/nonraid)) * Redundancy * Splitting of files across branches diff --git a/mkdocs/docs/known_issues_bugs.md b/mkdocs/docs/known_issues_bugs.md index 927a3ee9..fc683da5 100644 --- a/mkdocs/docs/known_issues_bugs.md +++ b/mkdocs/docs/known_issues_bugs.md @@ -91,36 +91,41 @@ on [page caching](config/cache.md). ## 3rd Party Software -### SQLite3, Plex, Jellyfin do not work with mergerfs - -It does. If you're trying to put the software's config / metadata / -database on mergerfs you can't set [cache.files=off](config/cache.md) -(unless you use Linux v6.6 or above and -[direct-io-allow-mmap](config/options.md) is enabled) because they are -using **sqlite3** with **mmap** enabled and have failed to properly -handle the situation where **mmap** may not be available. +### Software using mmap + +* software using sqlite3 w/ mmap + * Plex (for config) + * Jellyfin (for config) + * Emby (for config) + * Radarr + * Sonarr + * Lidarr +* other + * rtorrent + * qbittorrent (with 'Disk IO type = Memory mapped files') + +[mmap](https://en.wikipedia.org/wiki/Mmap) is a filesystem and +operating system feature allowing files to be interacted with as if it +was normal memory. Support for `mmap` is **required** by a number of +projects. FUSE (and therefore mergerfs), in order to support `mmap`, +requires [page caching](config/cache.md) to be +enabled. IE `cache.files` must not be set to `off` (if using Linux before +v6.6 or mergerfs before v2.41.0.) + +In newer versions of mergerfs and Linux `off` will work as it can now +auto enable page caching (and therefore `mmap`) as needed. That said it is recommended that config and runtime files be stored on SSDs on a regular filesystem for performance reasons. See [What should mergerfs NOT be used for?](faq/recommendations_and_warnings.md#what-should-mergerfs-not-be-used-for). -Other software that leverages **sqlite3** which require **mmap** -includes Radarr, Sonarr, and Lidarr. However, many programs use -**sqlite3** and do not require **mmap**. - -It is recommended that you reach out to the developers of the software -you are having troubles with and ask them to add a fallback to regular -file IO when **mmap** is unavailable. It is not only more compatible -but also can be more performant in certain situations. - -If the issue is that quick scans do not seem to pick up media then be -sure to set `func.getattr=newest`. That said a full scan will pick up -all media and it will put less load on the host to use time based -library scans or to configure downloading software to trigger a scan -when files are added to the pool. See [Does inotify and fanotify -work?](faq/compatibility_and_integration.md#does-inotify-and-fanotify-work) -for more details. +Technically **sqlite3** does not require `mmap`, and in fact the +default is not to use it, but many programs enable the feature and +provide no error handling when not supported. Ideally the software +would catch the error and use traditional IO instead. Not only is it +more compatible but could also be more performant in certain +situations. ### backup software diff --git a/mkdocs/docs/project_comparisons.md b/mkdocs/docs/project_comparisons.md index 5d22972d..8c8ea333 100644 --- a/mkdocs/docs/project_comparisons.md +++ b/mkdocs/docs/project_comparisons.md @@ -1,5 +1,8 @@ # Project Comparisons +There are no solutions, only trade-offs. + + ## mhddfs * [https://romanrm.net/mhddfs](https://romanrm.net/mhddfs) diff --git a/mkdocs/docs/quickstart.md b/mkdocs/docs/quickstart.md index 50e2738c..48d47fba 100644 --- a/mkdocs/docs/quickstart.md +++ b/mkdocs/docs/quickstart.md @@ -25,24 +25,22 @@ cases. * func.getattr=newest * dropcacheonclose=false -In previous versions of Linux it was unable to support `mmap` if [page -caching](config/cache.md) was disabled (ie: -`cache.files=off`). However, it now will enable page caching if needed -for a particular file if `mmap` is requested. +See [Known Issues and Bugs#Software Using +mmap](known_issues_bugs.md#software-using-mmap) for more details why +the Linux and mergerfs version matter and its relation to `mmap`. -[mmap](https://en.wikipedia.org/wiki/Mmap) is needed by certain -software to read and write to a file. However, many software could -work without it and fail to have proper error handling for when it is -unavailable. Many programs that use **sqlite3** will require `mmap` -despite [sqlite3 working perfectly -fine](known_issues_bugs.md#sqlite3-plex-jellyfin-do-not-work-with-mergerfs) -without it (and in some cases can be more performant with regular file -IO.) +**NOTE:** This "auto mmap" feature is only supported in mergerfs +v2.41.0 and above. For older versions see below or the docs from that +release. ### You use Linux v6.5 or below -#### You need `mmap` (used by rtorrent and many sqlite3 base software) +#### You need `mmap` + +To keep the details in a single location: see [Known Issues and +Bugs#Software Using mmap](known_issues_bugs.md#software-using-mmap) +for more details. * cache.files=auto-full * category.create=pfrd diff --git a/mkdocs/docs/tips_notes.md b/mkdocs/docs/tips_notes.md index 3a1ed380..01b30513 100644 --- a/mkdocs/docs/tips_notes.md +++ b/mkdocs/docs/tips_notes.md @@ -19,9 +19,10 @@ compliant filesystems. mergerfs doesn't generally make exceptions for FAT, NTFS, or other non-POSIX filesystem. * Unless using Linux v6.6 or above do **not** use `cache.files=off` if - you expect applications (such as rtorrent) to use + you expect applications to use [mmap](http://linux.die.net/man/2/mmap). Enabling `dropcacheonclose` - is recommended when `cache.files=auto-full`. + is recommended when `cache.files=auto-full`. See + [quickstart](quickstart.md). * [Kodi](http://kodi.tv), [Plex](http://plex.tv), [Subsonic](http://subsonic.org), etc. can use directory [mtime](http://linux.die.net/man/2/stat) to more efficiently