You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

2434 lines
100 KiB

11 years ago
2 years ago
2 years ago
2 years ago
4 years ago
5 years ago
5 years ago
5 years ago
3 years ago
5 years ago
  1. % mergerfs(1) mergerfs user manual
  2. # NAME
  3. mergerfs - a featureful union filesystem
  4. # SYNOPSIS
  5. mergerfs -o<options> <branches> <mountpoint>
  6. # DESCRIPTION
  7. **mergerfs** is a union filesystem geared towards simplifying storage
  8. and management of files across numerous commodity storage devices. It
  9. is similar to **mhddfs**, **unionfs**, and **aufs**.
  10. # FEATURES
  11. * Configurable behaviors / file placement
  12. * Ability to add or remove filesystems at will
  13. * Resistance to individual filesystem failure
  14. * Support for extended attributes (xattrs)
  15. * Support for file attributes (chattr)
  16. * Runtime configurable (via xattrs)
  17. * Works with heterogeneous filesystem types
  18. * Moving of file when filesystem runs out of space while writing
  19. * Ignore read-only filesystems when creating files
  20. * Turn read-only files into symlinks to underlying file
  21. * Hard link copy-on-write / CoW
  22. * Support for POSIX ACLs
  23. * Misc other things
  24. # HOW IT WORKS
  25. mergerfs logically merges multiple paths together. Think a union of
  26. sets. The file/s or directory/s acted on or presented through mergerfs
  27. are based on the policy chosen for that particular action. Read more
  28. about policies below.
  29. ```
  30. A + B = C
  31. /disk1 /disk2 /merged
  32. | | |
  33. +-- /dir1 +-- /dir1 +-- /dir1
  34. | | | | | |
  35. | +-- file1 | +-- file2 | +-- file1
  36. | | +-- file3 | +-- file2
  37. +-- /dir2 | | +-- file3
  38. | | +-- /dir3 |
  39. | +-- file4 | +-- /dir2
  40. | +-- file5 | |
  41. +-- file6 | +-- file4
  42. |
  43. +-- /dir3
  44. | |
  45. | +-- file5
  46. |
  47. +-- file6
  48. ```
  49. mergerfs does **not** support the copy-on-write (CoW) or whiteout
  50. behaviors found in **aufs** and **overlayfs**. You can **not** mount a
  51. read-only filesystem and write to it. However, mergerfs will ignore
  52. read-only drives when creating new files so you can mix read-write and
  53. read-only drives. It also does **not** split data across drives. It is
  54. not RAID0 / striping. It is simply a union of other filesystems.
  55. # TERMINOLOGY
  56. * branch: A base path used in the pool.
  57. * pool: The mergerfs mount. The union of the branches.
  58. * relative path: The path in the pool relative to the branch and mount.
  59. * function: A filesystem call (open, unlink, create, getattr, rmdir, etc.)
  60. * category: A collection of functions based on basic behavior (action, create, search).
  61. * policy: The algorithm used to select a file when performing a function.
  62. * path preservation: Aspect of some policies which includes checking the path for which a file would be created.
  63. # BASIC SETUP
  64. If you don't already know that you have a special use case then just
  65. start with one of the following option sets.
  66. #### You need `mmap` (used by rtorrent and many sqlite3 base software)
  67. `cache.files=partial,dropcacheonclose=true,category.create=mfs`
  68. #### You don't need `mmap`
  69. `cache.files=off,dropcacheonclose=true,category.create=mfs`
  70. ### Command Line
  71. `mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs /mnt/hdd0:/mnt/hdd1 /media`
  72. ### /etc/fstab
  73. `/mnt/hdd0:/mnt/hdd1 /media fuse.mergerfs cache.files=partial,dropcacheonclose=true,category.create=mfs 0 0`
  74. ### systemd mount
  75. https://github.com/trapexit/mergerfs/wiki/systemd
  76. ```
  77. [Unit]
  78. Description=mergerfs service
  79. [Service]
  80. Type=simple
  81. KillMode=none
  82. ExecStart=/usr/bin/mergerfs \
  83. -f \
  84. -o cache.files=partial \
  85. -o dropcacheonclose=true \
  86. -o category.create=mfs \
  87. /mnt/hdd0:/mnt/hdd1 \
  88. /media
  89. ExecStop=/bin/fusermount -uz /media
  90. Restart=on-failure
  91. [Install]
  92. WantedBy=default.target
  93. ```
  94. See the mergerfs [wiki for real world
  95. deployments](https://github.com/trapexit/mergerfs/wiki/Real-World-Deployments)
  96. for comparisons / ideas.
  97. # OPTIONS
  98. These options are the same regardless of whether you use them with the
  99. `mergerfs` commandline program, in fstab, or in a config file.
  100. ### mount options
  101. * **config**: Path to a config file. Same arguments as below in
  102. key=val / ini style format.
  103. * **branches**: Colon delimited list of branches.
  104. * **minfreespace=SIZE**: The minimum space value used for creation
  105. policies. Can be overridden by branch specific option. Understands
  106. 'K', 'M', and 'G' to represent kilobyte, megabyte, and gigabyte
  107. respectively. (default: 4G)
  108. * **moveonenospc=BOOL|POLICY**: When enabled if a **write** fails with
  109. **ENOSPC** (no space left on device) or **EDQUOT** (disk quota
  110. exceeded) the policy selected will run to find a new location for
  111. the file. An attempt to move the file to that branch will occur
  112. (keeping all metadata possible) and if successful the original is
  113. unlinked and the write retried. (default: false, true = mfs)
  114. * **inodecalc=passthrough|path-hash|devino-hash|hybrid-hash**: Selects
  115. the inode calculation algorithm. (default: hybrid-hash)
  116. * **dropcacheonclose=BOOL**: When a file is requested to be closed
  117. call `posix_fadvise` on it first to instruct the kernel that we no
  118. longer need the data and it can drop its cache. Recommended when
  119. **cache.files=partial|full|auto-full|per-process** to limit double
  120. caching. (default: false)
  121. * **symlinkify=BOOL**: When enabled and a file is not writable and its
  122. mtime or ctime is older than **symlinkify_timeout** files will be
  123. reported as symlinks to the original files. Please read more below
  124. before using. (default: false)
  125. * **symlinkify_timeout=UINT**: Time to wait, in seconds, to activate
  126. the **symlinkify** behavior. (default: 3600)
  127. * **nullrw=BOOL**: Turns reads and writes into no-ops. The request
  128. will succeed but do nothing. Useful for benchmarking
  129. mergerfs. (default: false)
  130. * **lazy-umount-mountpoint=BOOL**: mergerfs will attempt to "lazy
  131. umount" the mountpoint before mounting itself. Useful when
  132. performing live upgrades of mergerfs. (default: false)
  133. * **ignorepponrename=BOOL**: Ignore path preserving on
  134. rename. Typically rename and link act differently depending on the
  135. policy of `create` (read below). Enabling this will cause rename and
  136. link to always use the non-path preserving behavior. This means
  137. files, when renamed or linked, will stay on the same
  138. drive. (default: false)
  139. * **security_capability=BOOL**: If false return ENOATTR when xattr
  140. security.capability is queried. (default: true)
  141. * **xattr=passthrough|noattr|nosys**: Runtime control of
  142. xattrs. Default is to passthrough xattr requests. 'noattr' will
  143. short circuit as if nothing exists. 'nosys' will respond with ENOSYS
  144. as if xattrs are not supported or disabled. (default: passthrough)
  145. * **link_cow=BOOL**: When enabled if a regular file is opened which
  146. has a link count > 1 it will copy the file to a temporary file and
  147. rename over the original. Breaking the link and providing a basic
  148. copy-on-write function similar to cow-shell. (default: false)
  149. * **statfs=base|full**: Controls how statfs works. 'base' means it
  150. will always use all branches in statfs calculations. 'full' is in
  151. effect path preserving and only includes drives where the path
  152. exists. (default: base)
  153. * **statfs_ignore=none|ro|nc**: 'ro' will cause statfs calculations to
  154. ignore available space for branches mounted or tagged as 'read-only'
  155. or 'no create'. 'nc' will ignore available space for branches tagged
  156. as 'no create'. (default: none)
  157. * **nfsopenhack=off|git|all**: A workaround for exporting mergerfs
  158. over NFS where there are issues with creating files for write while
  159. setting the mode to read-only. (default: off)
  160. * **branches-mount-timeout=UINT**: Number of seconds to wait at
  161. startup for branches to be a mount other than the mountpoint's
  162. filesystem. (default: 0)
  163. * **follow-symlinks=never|directory|regular|all**: Turns symlinks into
  164. what they point to. (default: never)
  165. * **link-exdev=passthrough|rel-symlink|abs-base-symlink|abs-pool-symlink**:
  166. When a link fails with EXDEV optionally create a symlink to the file
  167. instead.
  168. * **rename-exdev=passthrough|rel-symlink|abs-symlink**: When a rename
  169. fails with EXDEV optionally move the file to a special directory and
  170. symlink to it.
  171. * **readahead=UINT**: Set readahead (in kilobytes) for mergerfs and
  172. branches if greater than 0. (default: 0)
  173. * **posix_acl=BOOL**: Enable POSIX ACL support (if supported by kernel
  174. and underlying filesystem). (default: false)
  175. * **async_read=BOOL**: Perform reads asynchronously. If disabled or
  176. unavailable the kernel will ensure there is at most one pending read
  177. request per file handle and will attempt to order requests by
  178. offset. (default: true)
  179. * **fuse_msg_size=UINT**: Set the max number of pages per FUSE
  180. message. Only available on Linux >= 4.20 and ignored
  181. otherwise. (min: 1; max: 256; default: 256)
  182. * **threads=INT**: Number of threads to use. When used alone
  183. (`process-thread-count=-1`) it sets the number of threads reading
  184. and processing FUSE messages. When used together it sets the number
  185. of threads reading from FUSE. When set to zero it will attempt to
  186. discover and use the number of logical cores. If the thread count is
  187. set negative it will look up the number of cores then divide by the
  188. absolute value. ie. threads=-2 on an 8 core machine will result in 8
  189. / 2 = 4 threads. There will always be at least 1 thread. If set to
  190. -1 in combination with `process-thread-count` then it will try to
  191. pick reasonable values based on CPU thread count. NOTE: higher
  192. number of threads increases parallelism but usually decreases
  193. throughput. (default: 0)
  194. * **read-thread-count=INT**: Alias for `threads`.
  195. * **process-thread-count=INT**: Enables separate thread pool to
  196. asynchronously process FUSE requests. In this mode
  197. `read-thread-count` refers to the number of threads reading FUSE
  198. messages which are dispatched to process threads. -1 means disabled
  199. otherwise acts like `read-thread-count`. (default: -1)
  200. * **pin-threads=STR**: Selects a strategy to pin threads to CPUs
  201. (default: unset)
  202. * **scheduling-priority=INT**: Set mergerfs' scheduling
  203. priority. Valid values range from -20 to 19. See `setpriority` man
  204. page for more details. (default: -10)
  205. * **fsname=STR**: Sets the name of the filesystem as seen in
  206. **mount**, **df**, etc. Defaults to a list of the source paths
  207. concatenated together with the longest common prefix removed.
  208. * **func.FUNC=POLICY**: Sets the specific FUSE function's policy. See
  209. below for the list of value types. Example: **func.getattr=newest**
  210. * **category.action=POLICY**: Sets policy of all FUSE functions in the
  211. action category. (default: epall)
  212. * **category.create=POLICY**: Sets policy of all FUSE functions in the
  213. create category. (default: epmfs)
  214. * **category.search=POLICY**: Sets policy of all FUSE functions in the
  215. search category. (default: ff)
  216. * **cache.open=UINT**: 'open' policy cache timeout in
  217. seconds. (default: 0)
  218. * **cache.statfs=UINT**: 'statfs' cache timeout in seconds. (default:
  219. 0)
  220. * **cache.attr=UINT**: File attribute cache timeout in
  221. seconds. (default: 1)
  222. * **cache.entry=UINT**: File name lookup cache timeout in
  223. seconds. (default: 1)
  224. * **cache.negative_entry=UINT**: Negative file name lookup cache
  225. timeout in seconds. (default: 0)
  226. * **cache.files=libfuse|off|partial|full|auto-full|per-process**: File
  227. page caching mode (default: libfuse)
  228. * **cache.files.process-names=LIST**: A pipe | delimited list of
  229. process [comm](https://man7.org/linux/man-pages/man5/proc.5.html)
  230. names to enable page caching for when
  231. `cache.files=per-process`. (default: "rtorrent|qbittorrent-nox")
  232. * **cache.writeback=BOOL**: Enable kernel writeback caching (default:
  233. false)
  234. * **cache.symlinks=BOOL**: Cache symlinks (if supported by kernel)
  235. (default: false)
  236. * **cache.readdir=BOOL**: Cache readdir (if supported by kernel)
  237. (default: false)
  238. * **direct_io**: deprecated - Bypass page cache. Use `cache.files=off`
  239. instead. (default: false)
  240. * **kernel_cache**: deprecated - Do not invalidate data cache on file
  241. open. Use `cache.files=full` instead. (default: false)
  242. * **auto_cache**: deprecated - Invalidate data cache if file mtime or
  243. size change. Use `cache.files=auto-full` instead. (default: false)
  244. * **async_read**: deprecated - Perform reads asynchronously. Use
  245. `async_read=true` instead.
  246. * **sync_read**: deprecated - Perform reads synchronously. Use
  247. `async_read=false` instead.
  248. * **use_ino**: deprecated - Always enabled.
  249. * **allow_other**: deprecated - Always enabled.
  250. * **splice_read**: deprecated - Does nothing.
  251. * **splice_write**: deprecated - Does nothing.
  252. * **splice_move**: deprecated - Does nothing.
  253. * **allow_other**: deprecated - mergerfs always sets this FUSE option
  254. as normal permissions can be used to limit access.
  255. * **use_ino**: deprecated - mergerfs should always control inode
  256. calculation so this is enabled all the time.
  257. **NOTE:** Options are evaluated in the order listed so if the options
  258. are **func.rmdir=rand,category.action=ff** the **action** category
  259. setting will override the **rmdir** setting.
  260. **NOTE:** Always look at the documentation for the version of mergerfs
  261. you're using. Not all features are available in older releases. Use
  262. `man mergerfs` or find the docs as linked in the release.
  263. #### Value Types
  264. * BOOL = 'true' | 'false'
  265. * INT = [MIN_INT,MAX_INT]
  266. * UINT = [0,MAX_INT]
  267. * SIZE = 'NNM'; NN = INT, M = 'K' | 'M' | 'G' | 'T'
  268. * STR = string (may refer to an enumerated value, see details of
  269. argument)
  270. * FUNC = filesystem function
  271. * CATEGORY = function category
  272. * POLICY = mergerfs function policy
  273. ### branches
  274. The 'branches' argument is a colon (':') delimited list of paths to be
  275. pooled together. It does not matter if the paths are on the same or
  276. different drives nor does it matter the filesystem (within
  277. reason). Used and available space will not be duplicated for paths on
  278. the same device and any features which aren't supported by the
  279. underlying filesystem (such as file attributes or extended attributes)
  280. will return the appropriate errors.
  281. Branches currently have two options which can be set. A type which
  282. impacts whether or not the branch is included in a policy calculation
  283. and a individual minfreespace value. The values are set by prepending
  284. an `=` at the end of a branch designation and using commas as
  285. delimiters. Example: /mnt/drive=RW,1234
  286. #### branch mode
  287. * RW: (read/write) - Default behavior. Will be eligible in all policy
  288. categories.
  289. * RO: (read-only) - Will be excluded from `create` and `action`
  290. policies. Same as a read-only mounted filesystem would be (though
  291. faster to process).
  292. * NC: (no-create) - Will be excluded from `create` policies. You can't
  293. create on that branch but you can change or delete.
  294. #### minfreespace
  295. Same purpose and syntax as the global option but specific to the
  296. branch. If not set the global value is used.
  297. #### globbing
  298. To make it easier to include multiple branches mergerfs supports
  299. [globbing](http://linux.die.net/man/7/glob). **The globbing tokens
  300. MUST be escaped when using via the shell else the shell itself will
  301. apply the glob itself.**
  302. ```
  303. # mergerfs /mnt/hdd\*:/mnt/ssd /media
  304. ```
  305. The above line will use all mount points in /mnt prefixed with **hdd** and **ssd**.
  306. To have the pool mounted at boot or otherwise accessible from related tools use **/etc/fstab**.
  307. ```
  308. # <file system> <mount point> <type> <options> <dump> <pass>
  309. /mnt/hdd*:/mnt/ssd /media fuse.mergerfs minfreespace=16G 0 0
  310. ```
  311. **NOTE:** the globbing is done at mount or when updated using the
  312. runtime API. If a new directory is added matching the glob after the
  313. fact it will not be automatically included.
  314. **NOTE:** for mounting via **fstab** to work you must have
  315. **mount.fuse** installed. For Ubuntu/Debian it is included in the
  316. **fuse** package.
  317. ### inodecalc
  318. Inodes (st_ino) are unique identifiers within a filesystem. Each
  319. mounted filesystem has device ID (st_dev) as well and together they
  320. can uniquely identify a file on the whole of the system. Entries on
  321. the same device with the same inode are in fact references to the same
  322. underlying file. It is a many to one relationship between names and an
  323. inode. Directories, however, do not have multiple links on most
  324. systems due to the complexity they add.
  325. FUSE allows the server (mergerfs) to set inode values but not device
  326. IDs. Creating an inode value is somewhat complex in mergerfs' case as
  327. files aren't really in its control. If a policy changes what directory
  328. or file is to be selected or something changes out of band it becomes
  329. unclear what value should be used. Most software does not to care what
  330. the values are but those that do often break if a value changes
  331. unexpectedly. The tool `find` will abort a directory walk if it sees a
  332. directory inode change. NFS will return stale handle errors if the
  333. inode changes out of band. File dedup tools will usually leverage
  334. device ids and inodes as a shortcut in searching for duplicate files
  335. and would resort to full file comparisons should it find different
  336. inode values.
  337. mergerfs offers multiple ways to calculate the inode in hopes of
  338. covering different usecases.
  339. * passthrough: Passes through the underlying inode value. Mostly
  340. intended for testing as using this does not address any of the
  341. problems mentioned above and could confuse file deduplication
  342. software as inodes from different filesystems can be the same.
  343. * path-hash: Hashes the relative path of the entry in question. The
  344. underlying file's values are completely ignored. This means the
  345. inode value will always be the same for that file path. This is
  346. useful when using NFS and you make changes out of band such as copy
  347. data between branches. This also means that entries that do point to
  348. the same file will not be recognizable via inodes. That **does not**
  349. mean hard links don't work. They will.
  350. * path-hash32: 32bit version of path-hash.
  351. * devino-hash: Hashes the device id and inode of the underlying
  352. entry. This won't prevent issues with NFS should the policy pick a
  353. different file or files move out of band but will present the same
  354. inode for underlying files that do too.
  355. * devino-hash32: 32bit version of devino-hash.
  356. * hybrid-hash: Performs `path-hash` on directories and `devino-hash`
  357. on other file types. Since directories can't have hard links the
  358. static value won't make a difference and the files will get values
  359. useful for finding duplicates. Probably the best to use if not using
  360. NFS. As such it is the default.
  361. * hybrid-hash32: 32bit version of hybrid-hash.
  362. 32bit versions are provided as there is some software which does not
  363. handle 64bit inodes well.
  364. While there is a risk of hash collision in tests of a couple million
  365. entries there were zero collisions. Unlike a typical filesystem FUSE
  366. filesystems can reuse inodes and not refer to the same entry. The
  367. internal identifier used to reference a file in FUSE is different from
  368. the inode value presented. The former is the `nodeid` and is actually
  369. a tuple of 2 64bit values: `nodeid` and `generation`. This tuple is
  370. not client facing. The inode that is presented to the client is passed
  371. through the kernel uninterpreted.
  372. From FUSE docs for `use_ino`:
  373. ```
  374. Honor the st_ino field in the functions getattr() and
  375. fill_dir(). This value is used to fill in the st_ino field
  376. in the stat(2), lstat(2), fstat(2) functions and the d_ino
  377. field in the readdir(2) function. The filesystem does not
  378. have to guarantee uniqueness, however some applications
  379. rely on this value being unique for the whole filesystem.
  380. Note that this does *not* affect the inode that libfuse
  381. and the kernel use internally (also called the "nodeid").
  382. ```
  383. As of version 2.35.0 the `use_ino` option has been removed. mergerfs
  384. should always be managing inode values.
  385. ### pin-threads
  386. Simple strategies for pinning read and/or process threads. If process
  387. threads are not enabled than the strategy simply works on the read
  388. threads. Invalid values are ignored.
  389. * R1L: All read threads pinned to a single logical CPU.
  390. * R1P: All read threads pinned to a single physical CPU.
  391. * RP1L: All read and process threads pinned to a single logical CPU.
  392. * RP1P: All read and process threads pinned to a single physical CPU.
  393. * R1LP1L: All read threads pinned to a single logical CPU, all process
  394. threads pinned to a (if possible) different logical CPU.
  395. * R1PP1P: All read threads pinned to a single physical CPU, all
  396. process threads pinned to a (if possible) different logical CPU.
  397. * RPSL: All read and process threads are spread across all logical CPUs.
  398. * RPSP: All read and process threads are spread across all physical CPUs.
  399. * R1PPSP: All read threads are pinned to a single physical CPU while
  400. process threads are spread across all other phsycial CPUs.
  401. ### fuse_msg_size
  402. FUSE applications communicate with the kernel over a special character
  403. device: `/dev/fuse`. A large portion of the overhead associated with
  404. FUSE is the cost of going back and forth from user space and kernel
  405. space over that device. Generally speaking the fewer trips needed the
  406. better the performance will be. Reducing the number of trips can be
  407. done a number of ways. Kernel level caching and increasing message
  408. sizes being two significant ones. When it comes to reads and writes if
  409. the message size is doubled the number of trips are approximately
  410. halved.
  411. In Linux 4.20 a new feature was added allowing the negotiation of the
  412. max message size. Since the size is in multiples of
  413. [pages](https://en.wikipedia.org/wiki/Page_(computer_memory)) the
  414. feature is called `max_pages`. There is a maximum `max_pages` value of
  415. 256 (1MiB) and minimum of 1 (4KiB). The default used by Linux >=4.20,
  416. and hardcoded value used before 4.20, is 32 (128KiB). In mergerfs its
  417. referred to as `fuse_msg_size` to make it clear what it impacts and
  418. provide some abstraction.
  419. Since there should be no downsides to increasing `fuse_msg_size` /
  420. `max_pages`, outside a minor bump in RAM usage due to larger message
  421. buffers, mergerfs defaults the value to 256. On kernels before 4.20
  422. the value has no effect. The reason the value is configurable is to
  423. enable experimentation and benchmarking. See the BENCHMARKING section
  424. for examples.
  425. ### follow-symlinks
  426. This feature, when enabled, will cause symlinks to be interpreted by
  427. mergerfs as their target (depending on the mode).
  428. When there is a getattr/stat request for a file mergerfs will check if
  429. the file is a symlink and depending on the `follow-symlinks` setting
  430. will replace the information about the symlink with that of that which
  431. it points to.
  432. When unlink'ing or rmdir'ing the followed symlink it will remove the
  433. symlink itself and not that which it points to.
  434. * never: Behave as normal. Symlinks are treated as such.
  435. * directory: Resolve symlinks only which point to directories.
  436. * regular: Resolve symlinks only which point to regular files.
  437. * all: Resolve all symlinks to that which they point to.
  438. Symlinks which do not point to anything are left as is.
  439. WARNING: This feature works but there might be edge cases yet
  440. found. If you find any odd behaviors please file a ticket on
  441. [github](https://github.com/trapexit/mergerfs/issues).
  442. ### link-exdev
  443. If using path preservation and a `link` fails with EXDEV make a call
  444. to `symlink` where the `target` is the `oldlink` and the `linkpath` is
  445. the `newpath`. The `target` value is determined by the value of
  446. `link-exdev`.
  447. * passthrough: Return EXDEV as normal.
  448. * rel-symlink: A relative path from the `newpath`.
  449. * abs-base-symlink: A absolute value using the underlying branch.
  450. * abs-pool-symlink: A absolute value using the mergerfs mount point.
  451. NOTE: It is possible that some applications check the file they
  452. link. In those cases it is possible it will error or complain.
  453. ### rename-exdev
  454. If using path preservation and a `rename` fails with EXDEV:
  455. 1. Move file from **/branch/a/b/c** to **/branch/.mergerfs_rename_exdev/a/b/c**.
  456. 2. symlink the rename's `newpath` to the moved file.
  457. The `target` value is determined by the value of `rename-exdev`.
  458. * passthrough: Return EXDEV as normal.
  459. * rel-symlink: A relative path from the `newpath`.
  460. * abs-symlink: A absolute value using the mergerfs mount point.
  461. NOTE: It is possible that some applications check the file they
  462. rename. In those cases it is possible it will error or complain.
  463. NOTE: The reason `abs-symlink` is not split into two like `link-exdev`
  464. is due to the complexities in managing absolute base symlinks when
  465. multiple `oldpaths` exist.
  466. ### symlinkify
  467. Due to the levels of indirection introduced by mergerfs and the
  468. underlying technology FUSE there can be varying levels of performance
  469. degradation. This feature will turn non-directories which are not
  470. writable into symlinks to the original file found by the `readlink`
  471. policy after the mtime and ctime are older than the timeout.
  472. **WARNING:** The current implementation has a known issue in which if
  473. the file is open and being used when the file is converted to a
  474. symlink then the application which has that file open will receive an
  475. error when using it. This is unlikely to occur in practice but is
  476. something to keep in mind.
  477. **WARNING:** Some backup solutions, such as CrashPlan, do not backup
  478. the target of a symlink. If using this feature it will be necessary to
  479. point any backup software to the original drives or configure the
  480. software to follow symlinks if such an option is
  481. available. Alternatively create two mounts. One for backup and one for
  482. general consumption.
  483. ### nullrw
  484. Due to how FUSE works there is an overhead to all requests made to a
  485. FUSE filesystem that wouldn't exist for an in kernel one. Meaning that
  486. even a simple passthrough will have some slowdown. However, generally
  487. the overhead is minimal in comparison to the cost of the underlying
  488. I/O. By disabling the underlying I/O we can test the theoretical
  489. performance boundaries.
  490. By enabling `nullrw` mergerfs will work as it always does **except**
  491. that all reads and writes will be no-ops. A write will succeed (the
  492. size of the write will be returned as if it were successful) but
  493. mergerfs does nothing with the data it was given. Similarly a read
  494. will return the size requested but won't touch the buffer.
  495. See the BENCHMARKING section for suggestions on how to test.
  496. ### xattr
  497. Runtime extended attribute support can be managed via the `xattr`
  498. option. By default it will passthrough any xattr calls. Given xattr
  499. support is rarely used and can have significant performance
  500. implications mergerfs allows it to be disabled at runtime. The
  501. performance problems mostly comes when file caching is enabled. The
  502. kernel will send a `getxattr` for `security.capability` *before every
  503. single write*. It doesn't cache the responses to any `getxattr`. This
  504. might be addressed in the future but for now mergerfs can really only
  505. offer the following workarounds.
  506. `noattr` will cause mergerfs to short circuit all xattr calls and
  507. return ENOATTR where appropriate. mergerfs still gets all the requests
  508. but they will not be forwarded on to the underlying filesystems. The
  509. runtime control will still function in this mode.
  510. `nosys` will cause mergerfs to return ENOSYS for any xattr call. The
  511. difference with `noattr` is that the kernel will cache this fact and
  512. itself short circuit future calls. This is more efficient than
  513. `noattr` but will cause mergerfs' runtime control via the hidden file
  514. to stop working.
  515. ### nfsopenhack
  516. NFS is not fully POSIX compliant and historically certain behaviors,
  517. such as opening files with O_EXCL, are not or not well supported. When
  518. mergerfs (or any FUSE filesystem) is exported over NFS some of these
  519. issues come up due to how NFS and FUSE interact.
  520. This hack addresses the issue where the creation of a file with a
  521. read-only mode but with a read/write or write only flag. Normally this
  522. is perfectly valid but NFS chops the one open call into multiple
  523. calls. Exactly how it is translated depends on the configuration and
  524. versions of the NFS server and clients but it results in a permission
  525. error because a normal user is not allowed to open a read-only file as
  526. writable.
  527. Even though it's a more niche situation this hack breaks normal
  528. security and behavior and as such is `off` by default. If set to `git`
  529. it will only perform the hack when the path in question includes
  530. `/.git/`. `all` will result it applying anytime a readonly file which
  531. is empty is opened for writing.
  532. # FUNCTIONS, CATEGORIES and POLICIES
  533. The POSIX filesystem API is made up of a number of
  534. functions. **creat**, **stat**, **chown**, etc. For ease of
  535. configuration in mergerfs most of the core functions are grouped into
  536. 3 categories: **action**, **create**, and **search**. These functions
  537. and categories can be assigned a policy which dictates which branch is
  538. chosen when performing that function.
  539. Some functions, listed in the category `N/A` below, can not be
  540. assigned the normal policies. These functions work with file handles,
  541. rather than file paths, which were created by `open` or `create`. That
  542. said many times the current FUSE kernel driver will not always provide
  543. the file handle when a client calls `fgetattr`, `fchown`, `fchmod`,
  544. `futimens`, `ftruncate`, etc. This means it will call the regular,
  545. path based, versions. `readdir` has no real need for a policy given
  546. the purpose is merely to return a list of entries in a
  547. directory. `statfs`'s behavior can be modified via other options.
  548. When using policies which are based on a branch's available space the
  549. base path provided is used. Not the full path to the file in
  550. question. Meaning that mounts in the branch won't be considered in the
  551. space calculations. The reason is that it doesn't really work for
  552. non-path preserving policies and can lead to non-obvious behaviors.
  553. NOTE: While any policy can be assigned to a function or category
  554. though some may not be very useful in practice. For instance: **rand**
  555. (random) may be useful for file creation (create) but could lead to
  556. very odd behavior if used for `chmod` if there were more than one copy
  557. of the file.
  558. ### Functions and their Category classifications
  559. | Category | FUSE Functions |
  560. |----------|-------------------------------------------------------------------------------------|
  561. | action | chmod, chown, link, removexattr, rename, rmdir, setxattr, truncate, unlink, utimens |
  562. | create | create, mkdir, mknod, symlink |
  563. | search | access, getattr, getxattr, ioctl (directories), listxattr, open, readlink |
  564. | N/A | fchmod, fchown, futimens, ftruncate, fallocate, fgetattr, fsync, ioctl (files), read, readdir, release, statfs, write, copy_file_range |
  565. In cases where something may be searched for (such as a path to clone)
  566. **getattr** will usually be used.
  567. ### Policies
  568. A policy is the algorithm used to choose a branch or branches for a
  569. function to work on. Think of them as ways to filter and sort
  570. branches.
  571. Any function in the `create` category will clone the relative path if
  572. needed. Some other functions (`rename`,`link`,`ioctl`) have special
  573. requirements or behaviors which you can read more about below.
  574. #### Filtering
  575. Policies basically search branches and create a list of files / paths
  576. for functions to work on. The policy is responsible for filtering and
  577. sorting the branches. Filters include **minfreespace**, whether or not
  578. a branch is mounted read-only, and the branch tagging
  579. (RO,NC,RW). These filters are applied across all policies unless
  580. otherwise noted.
  581. * No **search** function policies filter.
  582. * All **action** function policies filter out branches which are
  583. mounted **read-only** or tagged as **RO (read-only)**.
  584. * All **create** function policies filter out branches which are
  585. mounted **read-only**, tagged **RO (read-only)** or **NC (no
  586. create)**, or has available space less than `minfreespace`.
  587. Policies may have their own additional filtering such as those that
  588. require existing paths to be present.
  589. If all branches are filtered an error will be returned. Typically
  590. **EROFS** (read-only filesystem) or **ENOSPC** (no space left on
  591. device) depending on the most recent reason for filtering a
  592. branch. **ENOENT** will be returned if no eligible branch is found.
  593. #### Path Preservation
  594. Policies, as described below, are of two basic classifications. `path
  595. preserving` and `non-path preserving`.
  596. All policies which start with `ep` (**epff**, **eplfs**, **eplus**,
  597. **epmfs**, **eprand**) are `path preserving`. `ep` stands for
  598. `existing path`.
  599. A path preserving policy will only consider drives where the relative
  600. path being accessed already exists.
  601. When using non-path preserving policies paths will be cloned to target
  602. drives as necessary.
  603. With the `msp` or `most shared path` policies they are defined as
  604. `path preserving` for the purpose of controlling `link` and `rename`'s
  605. behaviors since `ignorepponrename` is available to disable that
  606. behavior.
  607. #### Policy descriptions
  608. A policy's behavior differs, as mentioned above, based on the function
  609. it is used with. Sometimes it really might not make sense to even
  610. offer certain policies because they are literally the same as others
  611. but it makes things a bit more uniform.
  612. | Policy | Description |
  613. |------------------|------------------------------------------------------------|
  614. | all | Search: For **mkdir**, **mknod**, and **symlink** it will apply to all branches. **create** works like **ff**. |
  615. | epall (existing path, all) | For **mkdir**, **mknod**, and **symlink** it will apply to all found. **create** works like **epff** (but more expensive because it doesn't stop after finding a valid branch). |
  616. | epff (existing path, first found) | Given the order of the branches, as defined at mount time or configured at runtime, act on the first one found where the relative path exists. |
  617. | eplfs (existing path, least free space) | Of all the branches on which the relative path exists choose the drive with the least free space. |
  618. | eplus (existing path, least used space) | Of all the branches on which the relative path exists choose the drive with the least used space. |
  619. | epmfs (existing path, most free space) | Of all the branches on which the relative path exists choose the drive with the most free space. |
  620. | eppfrd (existing path, percentage free random distribution) | Like **pfrd** but limited to existing paths. |
  621. | eprand (existing path, random) | Calls **epall** and then randomizes. Returns 1. |
  622. | ff (first found) | Given the order of the drives, as defined at mount time or configured at runtime, act on the first one found. |
  623. | lfs (least free space) | Pick the drive with the least available free space. |
  624. | lus (least used space) | Pick the drive with the least used space. |
  625. | mfs (most free space) | Pick the drive with the most available free space. |
  626. | msplfs (most shared path, least free space) | Like **eplfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
  627. | msplus (most shared path, least used space) | Like **eplus** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
  628. | mspmfs (most shared path, most free space) | Like **epmfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
  629. | msppfrd (most shared path, percentage free random distribution) | Like **eppfrd** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
  630. | newest | Pick the file / directory with the largest mtime. |
  631. | pfrd (percentage free random distribution) | Chooses a branch at random with the likelihood of selection based on a branch's available space relative to the total. |
  632. | rand (random) | Calls **all** and then randomizes. Returns 1 branch. |
  633. **NOTE:** If you are using an underlying filesystem that reserves
  634. blocks such as ext2, ext3, or ext4 be aware that mergerfs respects the
  635. reservation by using `f_bavail` (number of free blocks for
  636. unprivileged users) rather than `f_bfree` (number of free blocks) in
  637. policy calculations. **df** does NOT use `f_bavail`, it uses
  638. `f_bfree`, so direct comparisons between **df** output and mergerfs'
  639. policies is not appropriate.
  640. #### Defaults
  641. | Category | Policy |
  642. |----------|--------|
  643. | action | epall |
  644. | create | epmfs |
  645. | search | ff |
  646. #### ioctl
  647. When `ioctl` is used with an open file then it will use the file
  648. handle which was created at the original `open` call. However, when
  649. using `ioctl` with a directory mergerfs will use the `open` policy to
  650. find the directory to act on.
  651. #### rename & link ####
  652. **NOTE:** If you're receiving errors from software when files are
  653. moved / renamed / linked then you should consider changing the create
  654. policy to one which is **not** path preserving, enabling
  655. `ignorepponrename`, or contacting the author of the offending software
  656. and requesting that `EXDEV` (cross device / improper link) be properly
  657. handled.
  658. `rename` and `link` are tricky functions in a union
  659. filesystem. `rename` only works within a single filesystem or
  660. device. If a rename can't be done atomically due to the source and
  661. destination paths existing on different mount points it will return
  662. **-1** with **errno = EXDEV** (cross device / improper link). So if a
  663. `rename`'s source and target are on different drives within the pool
  664. it creates an issue.
  665. Originally mergerfs would return EXDEV whenever a rename was requested
  666. which was cross directory in any way. This made the code simple and
  667. was technically compliant with POSIX requirements. However, many
  668. applications fail to handle EXDEV at all and treat it as a normal
  669. error or otherwise handle it poorly. Such apps include: gvfsd-fuse
  670. v1.20.3 and prior, Finder / CIFS/SMB client in Apple OSX 10.9+,
  671. NZBGet, Samba's recycling bin feature.
  672. As a result a compromise was made in order to get most software to
  673. work while still obeying mergerfs' policies. Below is the basic logic.
  674. * If using a **create** policy which tries to preserve directory paths (epff,eplfs,eplus,epmfs)
  675. * Using the **rename** policy get the list of files to rename
  676. * For each file attempt rename:
  677. * If failure with ENOENT (no such file or directory) run **create** policy
  678. * If create policy returns the same drive as currently evaluating then clone the path
  679. * Re-attempt rename
  680. * If **any** of the renames succeed the higher level rename is considered a success
  681. * If **no** renames succeed the first error encountered will be returned
  682. * On success:
  683. * Remove the target from all drives with no source file
  684. * Remove the source from all drives which failed to rename
  685. * If using a **create** policy which does **not** try to preserve directory paths
  686. * Using the **rename** policy get the list of files to rename
  687. * Using the **getattr** policy get the target path
  688. * For each file attempt rename:
  689. * If the source drive != target drive:
  690. * Clone target path from target drive to source drive
  691. * Rename
  692. * If **any** of the renames succeed the higher level rename is considered a success
  693. * If **no** renames succeed the first error encountered will be returned
  694. * On success:
  695. * Remove the target from all drives with no source file
  696. * Remove the source from all drives which failed to rename
  697. The the removals are subject to normal entitlement checks.
  698. The above behavior will help minimize the likelihood of EXDEV being
  699. returned but it will still be possible.
  700. **link** uses the same strategy but without the removals.
  701. #### readdir ####
  702. [readdir](http://linux.die.net/man/3/readdir) is different from all
  703. other filesystem functions. While it could have its own set of
  704. policies to tweak its behavior at this time it provides a simple union
  705. of files and directories found. Remember that any action or
  706. information queried about these files and directories come from the
  707. respective function. For instance: an **ls** is a **readdir** and for
  708. each file/directory returned **getattr** is called. Meaning the policy
  709. of **getattr** is responsible for choosing the file/directory which is
  710. the source of the metadata you see in an **ls**.
  711. #### statfs / statvfs ####
  712. [statvfs](http://linux.die.net/man/2/statvfs) normalizes the source
  713. drives based on the fragment size and sums the number of adjusted
  714. blocks and inodes. This means you will see the combined space of all
  715. sources. Total, used, and free. The sources however are dedupped based
  716. on the drive so multiple sources on the same drive will not result in
  717. double counting its space. Filesystems mounted further down the tree
  718. of the branch will not be included when checking the mount's stats.
  719. The options `statfs` and `statfs_ignore` can be used to modify
  720. `statfs` behavior.
  721. # ERROR HANDLING
  722. POSIX filesystem functions offer a single return code meaning that
  723. there is some complication regarding the handling of multiple branches
  724. as mergerfs does. It tries to handle errors in a way that would
  725. generally return meaningful values for that particular function.
  726. ### chmod, chown, removexattr, setxattr, truncate, utimens
  727. 1) if no error: return 0 (success)
  728. 2) if no successes: return first error
  729. 3) if one of the files acted on was the same as the related search function: return its value
  730. 4) return 0 (success)
  731. While doing this increases the complexity and cost of error handling,
  732. particularly step 3, this provides probably the most reasonable return
  733. value.
  734. ### unlink, rmdir
  735. 1) if no errors: return 0 (success)
  736. 2) return first error
  737. Older version of mergerfs would return success if any success occurred
  738. but for unlink and rmdir there are downstream assumptions that, while
  739. not impossible to occur, can confuse some software.
  740. ### others
  741. For search functions there is always a single thing acted on and as
  742. such whatever return value that comes from the single function call is
  743. returned.
  744. For create functions `mkdir`, `mknod`, and `symlink` which don't
  745. return a file descriptor and therefore can have `all` or `epall`
  746. policies it will return success if any of the calls succeed and an
  747. error otherwise.
  748. # INSTALL
  749. https://github.com/trapexit/mergerfs/releases
  750. If your distribution's package manager includes mergerfs check if the
  751. version is up to date. If out of date it is recommended to use
  752. the latest release found on the release page. Details for common
  753. distros are below.
  754. #### Debian
  755. Most Debian installs are of a stable branch and therefore do not have
  756. the most up to date software. While mergerfs is available via `apt` it
  757. is suggested that uses install the most recent version available from
  758. the [releases page](https://github.com/trapexit/mergerfs/releases).
  759. #### prebuilt deb
  760. ```
  761. wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs_<ver>.debian-<rel>_<arch>.deb
  762. dpkg -i mergerfs_<ver>.debian-<rel>_<arch>.deb
  763. ```
  764. #### apt
  765. ```
  766. sudo apt install -y mergerfs
  767. ```
  768. #### Ubuntu
  769. Most Ubuntu installs are of a stable branch and therefore do not have
  770. the most up to date software. While mergerfs is available via `apt` it
  771. is suggested that uses install the most recent version available from
  772. the [releases page](https://github.com/trapexit/mergerfs/releases).
  773. #### prebuilt deb
  774. ```
  775. wget https://github.com/trapexit/mergerfs/releases/download/<version>/mergerfs_<ver>.ubuntu-<rel>_<arch>.deb
  776. dpkg -i mergerfs_<ver>.ubuntu-<rel>_<arch>.deb
  777. ```
  778. #### apt
  779. ```
  780. sudo apt install -y mergerfs
  781. ```
  782. #### Raspberry Pi OS
  783. Effectively the same as Debian or Ubuntu.
  784. #### Fedora
  785. ```
  786. wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-<ver>.fc<rel>.<arch>.rpm
  787. sudo rpm -i mergerfs-<ver>.fc<rel>.<arch>.rpm
  788. ```
  789. #### CentOS / Rocky
  790. ```
  791. wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-<ver>.el<rel>.<arch>.rpm
  792. sudo rpm -i mergerfs-<ver>.el<rel>.<arch>.rpm
  793. ```
  794. #### ArchLinux
  795. 1. Setup AUR
  796. 2. Install `mergerfs`
  797. #### Other
  798. Static binaries are provided for situations where native packages are
  799. unavailable.
  800. ```
  801. wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-static-linux_<arch>.tar.gz
  802. sudo tar xvf mergerfs-static-linux_<arch>.tar.gz -C /
  803. ```
  804. # BUILD
  805. **NOTE:** Prebuilt packages can be found at and recommended for most
  806. users: https://github.com/trapexit/mergerfs/releases
  807. **NOTE:** Only tagged releases are supported. `master` and other
  808. branches should be considered works in progress.
  809. First get the code from [github](https://github.com/trapexit/mergerfs).
  810. ```
  811. $ git clone https://github.com/trapexit/mergerfs.git
  812. $ # or
  813. $ wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-<ver>.tar.gz
  814. ```
  815. #### Debian / Ubuntu
  816. ```
  817. $ cd mergerfs
  818. $ sudo tools/install-build-pkgs
  819. $ make deb
  820. $ sudo dpkg -i ../mergerfs_<version>_<arch>.deb
  821. ```
  822. #### RHEL / CentOS / Rocky / Fedora
  823. ```
  824. $ su -
  825. # cd mergerfs
  826. # tools/install-build-pkgs
  827. # make rpm
  828. # rpm -i rpmbuild/RPMS/<arch>/mergerfs-<version>.<arch>.rpm
  829. ```
  830. #### Generic
  831. Have git, g++, make, python installed.
  832. ```
  833. $ cd mergerfs
  834. $ make
  835. $ sudo make install
  836. ```
  837. #### Build options
  838. ```
  839. $ make help
  840. usage: make
  841. make USE_XATTR=0 - build program without xattrs functionality
  842. make STATIC=1 - build static binary
  843. make LTO=1 - build with link time optimization
  844. ```
  845. # UPGRADE
  846. mergerfs can be upgraded live by mounting on top of the previous
  847. instance. Simply install the new version of mergerfs and follow the
  848. instructions below.
  849. Run mergerfs again or if using `/etc/fstab` call for it to mount
  850. again. Existing open files and such will continue to work fine though
  851. they won't see runtime changes since any such change would be the new
  852. mount. If you plan on changing settings with the new mount you should
  853. / could apply those before mounting the new version.
  854. ```
  855. $ sudo mount /mnt/mergerfs
  856. $ mount | grep mergerfs
  857. media on /mnt/mergerfs type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
  858. media on /mnt/mergerfs type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
  859. ```
  860. A problem with this approach is that the underlying instance will
  861. continue to run even if the software using it stop or are
  862. restarted. To work around this you can use a "lazy umount". Before
  863. mounting over top the mount point with the new instance of mergerfs
  864. issue: `umount -l <mergerfs_mountpoint>`. Or you can let mergerfs do
  865. it by setting the option `lazy-umount-mountpoint=true`.
  866. # RUNTIME CONFIG
  867. #### .mergerfs pseudo file ####
  868. ```
  869. <mountpoint>/.mergerfs
  870. ```
  871. There is a pseudo file available at the mount point which allows for
  872. the runtime modification of certain **mergerfs** options. The file
  873. will not show up in **readdir** but can be **stat**'ed and manipulated
  874. via [{list,get,set}xattrs](http://linux.die.net/man/2/listxattr)
  875. calls.
  876. Any changes made at runtime are **not** persisted. If you wish for
  877. values to persist they must be included as options wherever you
  878. configure the mounting of mergerfs (/etc/fstab).
  879. ##### Keys #####
  880. Use `getfattr -d /mountpoint/.mergerfs` or `xattr -l
  881. /mountpoint/.mergerfs` to see all supported keys. Some are
  882. informational and therefore read-only. `setxattr` will return EINVAL
  883. (invalid argument) on read-only keys.
  884. ##### Values #####
  885. Same as the command line.
  886. ###### user.mergerfs.branches ######
  887. Used to query or modify the list of branches. When modifying there are
  888. several shortcuts to easy manipulation of the list.
  889. | Value | Description |
  890. |--------------|-------------|
  891. | [list] | set |
  892. | +<[list] | prepend |
  893. | +>[list] | append |
  894. | -[list] | remove all values provided |
  895. | -< | remove first in list |
  896. | -> | remove last in list |
  897. `xattr -w user.mergerfs.branches +</mnt/drive3 /mnt/pool/.mergerfs`
  898. The `=NC`, `=RO`, `=RW` syntax works just as on the command line.
  899. ##### Example #####
  900. ```
  901. [trapexit:/mnt/mergerfs] $ getfattr -d .mergerfs
  902. user.mergerfs.branches="/mnt/a=RW:/mnt/b=RW"
  903. user.mergerfs.minfreespace="4294967295"
  904. user.mergerfs.moveonenospc="false"
  905. ...
  906. [trapexit:/mnt/mergerfs] $ getfattr -n user.mergerfs.category.search .mergerfs
  907. user.mergerfs.category.search="ff"
  908. [trapexit:/mnt/mergerfs] $ setfattr -n user.mergerfs.category.search -v newest .mergerfs
  909. [trapexit:/mnt/mergerfs] $ getfattr -n user.mergerfs.category.search .mergerfs
  910. user.mergerfs.category.search="newest"
  911. ```
  912. #### file / directory xattrs ####
  913. While they won't show up when using `getfattr` **mergerfs** offers a
  914. number of special xattrs to query information about the files
  915. served. To access the values you will need to issue a
  916. [getxattr](http://linux.die.net/man/2/getxattr) for one of the
  917. following:
  918. * **user.mergerfs.basepath**: the base mount point for the file given the current getattr policy
  919. * **user.mergerfs.relpath**: the relative path of the file from the perspective of the mount point
  920. * **user.mergerfs.fullpath**: the full path of the original file given the getattr policy
  921. * **user.mergerfs.allpaths**: a NUL ('\0') separated list of full paths to all files found
  922. # TOOLING
  923. * https://github.com/trapexit/mergerfs-tools
  924. * mergerfs.ctl: A tool to make it easier to query and configure mergerfs at runtime
  925. * mergerfs.fsck: Provides permissions and ownership auditing and the ability to fix them
  926. * mergerfs.dedup: Will help identify and optionally remove duplicate files
  927. * mergerfs.dup: Ensure there are at least N copies of a file across the pool
  928. * mergerfs.balance: Rebalance files across drives by moving them from the most filled to the least filled
  929. * mergerfs.consolidate: move files within a single mergerfs directory to the drive with most free space
  930. * https://github.com/trapexit/scorch
  931. * scorch: A tool to help discover silent corruption of files and keep track of files
  932. * https://github.com/trapexit/bbf
  933. * bbf (bad block finder): a tool to scan for and 'fix' hard drive bad blocks and find the files using those blocks
  934. # CACHING
  935. #### page caching
  936. https://en.wikipedia.org/wiki/Page_cache
  937. * cache.files=off: Disables page caching. Underlying files cached,
  938. mergerfs files are not.
  939. * cache.files=partial: Enables page caching. Underlying files cached,
  940. mergerfs files cached while open.
  941. * cache.files=full: Enables page caching. Underlying files cached,
  942. mergerfs files cached across opens.
  943. * cache.files=auto-full: Enables page caching. Underlying files
  944. cached, mergerfs files cached across opens if mtime and size are
  945. unchanged since previous open.
  946. * cache.files=libfuse: follow traditional libfuse `direct_io`,
  947. `kernel_cache`, and `auto_cache` arguments.
  948. * cache.files=per-process: Enable page caching only for processes
  949. which 'comm' name matches one of the values defined in
  950. `cache.files.process-names`.
  951. FUSE, which mergerfs uses, offers a number of page caching
  952. modes. mergerfs tries to simplify their use via the `cache.files`
  953. option. It can and should replace usage of `direct_io`,
  954. `kernel_cache`, and `auto_cache`.
  955. Due to mergerfs using FUSE and therefore being a userland process
  956. proxying existing filesystems the kernel will double cache the content
  957. being read and written through mergerfs. Once from the underlying
  958. filesystem and once from mergerfs (it sees them as two separate
  959. entities). Using `cache.files=off` will keep the double caching from
  960. happening by disabling caching of mergerfs but this has the side
  961. effect that *all* read and write calls will be passed to mergerfs
  962. which may be slower than enabling caching, you lose shared `mmap`
  963. support which can affect apps such as rtorrent, and no read-ahead will
  964. take place. The kernel will still cache the underlying filesystem data
  965. but that only helps so much given mergerfs will still process all
  966. requests.
  967. If you do enable file page caching,
  968. `cache.files=partial|full|auto-full`, you should also enable
  969. `dropcacheonclose` which will cause mergerfs to instruct the kernel to
  970. flush the underlying file's page cache when the file is closed. This
  971. behavior is the same as the rsync fadvise / drop cache patch and Feh's
  972. nocache project.
  973. If most files are read once through and closed (like media) it is best
  974. to enable `dropcacheonclose` regardless of caching mode in order to
  975. minimize buffer bloat.
  976. It is difficult to balance memory usage, cache bloat & duplication,
  977. and performance. Ideally mergerfs would be able to disable caching for
  978. the files it reads/writes but allow page caching for itself. That
  979. would limit the FUSE overhead. However, there isn't a good way to
  980. achieve this. It would need to open all files with O_DIRECT which
  981. places limitations on the what underlying filesystems would be
  982. supported and complicates the code.
  983. kernel documentation: https://www.kernel.org/doc/Documentation/filesystems/fuse-io.txt
  984. #### entry & attribute caching
  985. Given the relatively high cost of FUSE due to the kernel <-> userspace
  986. round trips there are kernel side caches for file entries and
  987. attributes. The entry cache limits the `lookup` calls to mergerfs
  988. which ask if a file exists. The attribute cache limits the need to
  989. make `getattr` calls to mergerfs which provide file attributes (mode,
  990. size, type, etc.). As with the page cache these should not be used if
  991. the underlying filesystems are being manipulated at the same time as
  992. it could lead to odd behavior or data corruption. The options for
  993. setting these are `cache.entry` and `cache.negative_entry` for the
  994. entry cache and `cache.attr` for the attributes
  995. cache. `cache.negative_entry` refers to the timeout for negative
  996. responses to lookups (non-existent files).
  997. #### writeback caching
  998. When `cache.files` is enabled the default is for it to perform
  999. writethrough caching. This behavior won't help improve performance as
  1000. each write still goes one for one through the filesystem. By enabling
  1001. the FUSE writeback cache small writes may be aggregated by the kernel
  1002. and then sent to mergerfs as one larger request. This can greatly
  1003. improve the throughput for apps which write to files
  1004. inefficiently. The amount the kernel can aggregate is limited by the
  1005. size of a FUSE message. Read the `fuse_msg_size` section for more
  1006. details.
  1007. There is a small side effect as a result of enabling writeback
  1008. caching. Underlying files won't ever be opened with O_APPEND or
  1009. O_WRONLY. The former because the kernel then manages append mode and
  1010. the latter because the kernel may request file data from mergerfs to
  1011. populate the write cache. The O_APPEND change means that if a file is
  1012. changed outside of mergerfs it could lead to corruption as the kernel
  1013. won't know the end of the file has changed. That said any time you use
  1014. caching you should keep from using the same file outside of mergerfs
  1015. at the same time.
  1016. Note that if an application is properly sizing writes then writeback
  1017. caching will have little or no effect. It will only help with writes
  1018. of sizes below the FUSE message size (128K on older kernels, 1M on
  1019. newer).
  1020. #### policy caching
  1021. Policies are run every time a function (with a policy as mentioned
  1022. above) is called. These policies can be expensive depending on
  1023. mergerfs' setup and client usage patterns. Generally we wouldn't want
  1024. to cache policy results because it may result in stale responses if
  1025. the underlying drives are used directly.
  1026. The `open` policy cache will cache the result of an `open` policy for
  1027. a particular input for `cache.open` seconds or until the file is
  1028. unlinked. Each file close (release) will randomly chose to clean up
  1029. the cache of expired entries.
  1030. This cache is really only useful in cases where you have a large
  1031. number of branches and `open` is called on the same files repeatedly
  1032. (like **Transmission** which opens and closes a file on every
  1033. read/write presumably to keep file handle usage low).
  1034. #### statfs caching
  1035. Of the syscalls used by mergerfs in policies the `statfs` / `statvfs`
  1036. call is perhaps the most expensive. It's used to find out the
  1037. available space of a drive and whether it is mounted
  1038. read-only. Depending on the setup and usage pattern these queries can
  1039. be relatively costly. When `cache.statfs` is enabled all calls to
  1040. `statfs` by a policy will be cached for the number of seconds its set
  1041. to.
  1042. Example: If the create policy is `mfs` and the timeout is 60 then for
  1043. that 60 seconds the same drive will be returned as the target for
  1044. creates because the available space won't be updated for that time.
  1045. #### symlink caching
  1046. As of version 4.20 Linux supports symlink caching. Significant
  1047. performance increases can be had in workloads which use a lot of
  1048. symlinks. Setting `cache.symlinks=true` will result in requesting
  1049. symlink caching from the kernel only if supported. As a result its
  1050. safe to enable it on systems prior to 4.20. That said it is disabled
  1051. by default for now. You can see if caching is enabled by querying the
  1052. xattr `user.mergerfs.cache.symlinks` but given it must be requested at
  1053. startup you can not change it at runtime.
  1054. #### readdir caching
  1055. As of version 4.20 Linux supports readdir caching. This can have a
  1056. significant impact on directory traversal. Especially when combined
  1057. with entry (`cache.entry`) and attribute (`cache.attr`)
  1058. caching. Setting `cache.readdir=true` will result in requesting
  1059. readdir caching from the kernel on each `opendir`. If the kernel
  1060. doesn't support readdir caching setting the option to `true` has no
  1061. effect. This option is configurable at runtime via xattr
  1062. `user.mergerfs.cache.readdir`.
  1063. #### tiered caching
  1064. Some storage technologies support what some call "tiered" caching. The
  1065. placing of usually smaller, faster storage as a transparent cache to
  1066. larger, slower storage. NVMe, SSD, Optane in front of traditional HDDs
  1067. for instance.
  1068. MergerFS does not natively support any sort of tiered caching. Most
  1069. users have no use for such a feature and its inclusion would
  1070. complicate the code. However, there are a few situations where a cache
  1071. drive could help with a typical mergerfs setup.
  1072. 1. Fast network, slow drives, many readers: You've a 10+Gbps network
  1073. with many readers and your regular drives can't keep up.
  1074. 2. Fast network, slow drives, small'ish bursty writes: You have a
  1075. 10+Gbps network and wish to transfer amounts of data less than your
  1076. cache drive but wish to do so quickly.
  1077. With #1 it's arguable if you should be using mergerfs at all. RAID
  1078. would probably be the better solution. If you're going to use mergerfs
  1079. there are other tactics that may help: spreading the data across
  1080. drives (see the mergerfs.dup tool) and setting `func.open=rand`, using
  1081. `symlinkify`, or using dm-cache or a similar technology to add tiered
  1082. cache to the underlying device.
  1083. With #2 one could use dm-cache as well but there is another solution
  1084. which requires only mergerfs and a cronjob.
  1085. 1. Create 2 mergerfs pools. One which includes just the slow drives
  1086. and one which has both the fast drives (SSD,NVME,etc.) and slow
  1087. drives.
  1088. 2. The 'cache' pool should have the cache drives listed first.
  1089. 3. The best `create` policies to use for the 'cache' pool would
  1090. probably be `ff`, `epff`, `lfs`, or `eplfs`. The latter two under
  1091. the assumption that the cache drive(s) are far smaller than the
  1092. backing drives. If using path preserving policies remember that
  1093. you'll need to manually create the core directories of those paths
  1094. you wish to be cached. Be sure the permissions are in sync. Use
  1095. `mergerfs.fsck` to check / correct them. You could also tag the
  1096. slow drives as `=NC` though that'd mean if the cache drives fill
  1097. you'd get "out of space" errors.
  1098. 4. Enable `moveonenospc` and set `minfreespace` appropriately. To make
  1099. sure there is enough room on the "slow" pool you might want to set
  1100. `minfreespace` to at least as large as the size of the largest
  1101. cache drive if not larger. This way in the worst case the whole of
  1102. the cache drive(s) can be moved to the other drives.
  1103. 5. Set your programs to use the cache pool.
  1104. 6. Save one of the below scripts or create you're own.
  1105. 7. Use `cron` (as root) to schedule the command at whatever frequency
  1106. is appropriate for your workflow.
  1107. ##### time based expiring
  1108. Move files from cache to backing pool based only on the last time the
  1109. file was accessed. Replace `-atime` with `-amin` if you want minutes
  1110. rather than days. May want to use the `fadvise` / `--drop-cache`
  1111. version of rsync or run rsync with the tool "nocache".
  1112. *NOTE:* The arguments to these scripts include the cache
  1113. **drive**. Not the pool with the cache drive. You could have data loss
  1114. if the source is the cache pool.
  1115. ```
  1116. #!/bin/bash
  1117. if [ $# != 3 ]; then
  1118. echo "usage: $0 <cache-drive> <backing-pool> <days-old>"
  1119. exit 1
  1120. fi
  1121. CACHE="${1}"
  1122. BACKING="${2}"
  1123. N=${3}
  1124. find "${CACHE}" -type f -atime +${N} -printf '%P\n' | \
  1125. rsync --files-from=- -axqHAXWES --preallocate --remove-source-files "${CACHE}/" "${BACKING}/"
  1126. ```
  1127. ##### percentage full expiring
  1128. Move the oldest file from the cache to the backing pool. Continue till
  1129. below percentage threshold.
  1130. *NOTE:* The arguments to these scripts include the cache
  1131. **drive**. Not the pool with the cache drive. You could have data loss
  1132. if the source is the cache pool.
  1133. ```
  1134. #!/bin/bash
  1135. if [ $# != 3 ]; then
  1136. echo "usage: $0 <cache-drive> <backing-pool> <percentage>"
  1137. exit 1
  1138. fi
  1139. CACHE="${1}"
  1140. BACKING="${2}"
  1141. PERCENTAGE=${3}
  1142. set -o errexit
  1143. while [ $(df --output=pcent "${CACHE}" | grep -v Use | cut -d'%' -f1) -gt ${PERCENTAGE} ]
  1144. do
  1145. FILE=$(find "${CACHE}" -type f -printf '%A@ %P\n' | \
  1146. sort | \
  1147. head -n 1 | \
  1148. cut -d' ' -f2-)
  1149. test -n "${FILE}"
  1150. rsync -axqHAXWESR --preallocate --remove-source-files "${CACHE}/./${FILE}" "${BACKING}/"
  1151. done
  1152. ```
  1153. # PERFORMANCE
  1154. mergerfs is at its core just a proxy and therefore its theoretical max
  1155. performance is that of the underlying devices. However, given it is a
  1156. FUSE filesystem working from userspace there is an increase in
  1157. overhead relative to kernel based solutions. That said the performance
  1158. can match the theoretical max but it depends greatly on the system's
  1159. configuration. Especially when adding network filesystems into the mix
  1160. there are many variables which can impact performance. Drive speeds
  1161. and latency, network speeds and latency, general concurrency,
  1162. read/write sizes, etc. Unfortunately, given the number of variables it
  1163. has been difficult to find a single set of settings which provide
  1164. optimal performance. If you're having performance issues please look
  1165. over the suggestions below (including the benchmarking section.)
  1166. NOTE: be sure to read about these features before changing them to
  1167. understand what behaviors it may impact
  1168. * disable `security_capability` and/or `xattr`
  1169. * increase cache timeouts `cache.attr`, `cache.entry`, `cache.negative_entry`
  1170. * enable (or disable) page caching (`cache.files`)
  1171. * enable `cache.writeback`
  1172. * enable `cache.statfs`
  1173. * enable `cache.symlinks`
  1174. * enable `cache.readdir`
  1175. * change the number of worker threads
  1176. * disable `posix_acl`
  1177. * disable `async_read`
  1178. * test theoretical performance using `nullrw` or mounting a ram disk
  1179. * use `symlinkify` if your data is largely static and read-only
  1180. * use tiered cache drives
  1181. * use LVM and LVM cache to place a SSD in front of your HDDs
  1182. * increase readahead: `readahead=1024`
  1183. If you come across a setting that significantly impacts performance
  1184. please contact trapexit so he may investigate further. Please test
  1185. both against your normal setup, a singular branch, and with
  1186. `nullrw=true`
  1187. # BENCHMARKING
  1188. Filesystems are complicated. They do many things and many of those are
  1189. interconnected. Additionally, the OS, drivers, hardware, etc. all can
  1190. impact performance. Therefore, when benchmarking, it is **necessary**
  1191. that the test focus as narrowly as possible.
  1192. For most throughput is the key benchmark. To test throughput `dd` is
  1193. useful but **must** be used with the correct settings in order to
  1194. ensure the filesystem or device is actually being tested. The OS can
  1195. and will cache data. Without forcing synchronous reads and writes
  1196. and/or disabling caching the values returned will not be
  1197. representative of the device's true performance.
  1198. When benchmarking through mergerfs ensure you only use 1 branch to
  1199. remove any possibility of the policies complicating the
  1200. situation. Benchmark the underlying filesystem first and then mount
  1201. mergerfs over it and test again. If you're experience speeds below
  1202. your expectation you will need to narrow down precisely which
  1203. component is leading to the slowdown. Preferably test the following in
  1204. the order listed (but not combined).
  1205. 1. Enable `nullrw` mode with `nullrw=true`. This will effectively make
  1206. reads and writes no-ops. Removing the underlying device /
  1207. filesystem from the equation. This will give us the top theoretical
  1208. speeds.
  1209. 2. Mount mergerfs over `tmpfs`. `tmpfs` is a RAM disk. Extremely high
  1210. speed and very low latency. This is a more realistic best case
  1211. scenario. Example: `mount -t tmpfs -o size=2G tmpfs /tmp/tmpfs`
  1212. 3. Mount mergerfs over a local drive. NVMe, SSD, HDD, etc. If you have
  1213. more than one I'd suggest testing each of them as drives and/or
  1214. controllers (their drivers) could impact performance.
  1215. 4. Finally, if you intend to use mergerfs with a network filesystem,
  1216. either as the source of data or to combine with another through
  1217. mergerfs, test each of those alone as above.
  1218. Once you find the component which has the performance issue you can do
  1219. further testing with different options to see if they impact
  1220. performance. For reads and writes the most relevant would be:
  1221. `cache.files`, `async_read`. Less likely but relevant when using NFS
  1222. or with certain filesystems would be `security_capability`, `xattr`,
  1223. and `posix_acl`. If you find a specific system, drive, filesystem,
  1224. controller, etc. that performs poorly contact trapexit so he may
  1225. investigate further.
  1226. Sometimes the problem is really the application accessing or writing
  1227. data through mergerfs. Some software use small buffer sizes which can
  1228. lead to more requests and therefore greater overhead. You can test
  1229. this out yourself by replace `bs=1M` in the examples below with `ibs`
  1230. or `obs` and using a size of `512` instead of `1M`. In one example
  1231. test using `nullrw` the write speed dropped from 4.9GB/s to 69.7MB/s
  1232. when moving from `1M` to `512`. Similar results were had when testing
  1233. reads. Small writes overhead may be improved by leveraging a write
  1234. cache but in casual tests little gain was found. More tests will need
  1235. to be done before this feature would become available. If you have an
  1236. app that appears slow with mergerfs it could be due to this. Contact
  1237. trapexit so he may investigate further.
  1238. ### write benchmark
  1239. ```
  1240. $ dd if=/dev/zero of=/mnt/mergerfs/1GB.file bs=1M count=1024 oflag=nocache conv=fdatasync status=progress
  1241. ```
  1242. ### read benchmark
  1243. ```
  1244. $ dd if=/mnt/mergerfs/1GB.file of=/dev/null bs=1M count=1024 iflag=nocache conv=fdatasync status=progress
  1245. ```
  1246. ### other benchmarks
  1247. If you are attempting to benchmark other behaviors you must ensure you
  1248. clear kernel caches before runs. In fact it would be a good deal to
  1249. run before the read and write benchmarks as well just in case.
  1250. ```
  1251. sync
  1252. echo 3 | sudo tee /proc/sys/vm/drop_caches
  1253. ```
  1254. # TIPS / NOTES
  1255. * This document is literal and thorough. If a suspected feature isn't
  1256. mentioned it doesn't exist. If certain libfuse arguments aren't
  1257. listed they probably shouldn't be used.
  1258. * Ensure you're using the latest version.
  1259. * Run mergerfs as `root`. mergerfs is designed and intended to be run
  1260. as `root` and may exibit incorrect behavior if run otherwise..
  1261. * If you don't see some directories and files you expect, policies
  1262. seem to skip branches, you get strange permission errors, etc. be
  1263. sure the underlying filesystems' permissions are all the same. Use
  1264. `mergerfs.fsck` to audit the drive for out of sync permissions.
  1265. * If you still have permission issues be sure you are using POSIX ACL
  1266. compliant filesystems. mergerfs doesn't generally make exceptions
  1267. for FAT, NTFS, or other non-POSIX filesystem.
  1268. * Do **not** use `cache.files=off` if you expect applications (such as
  1269. rtorrent) to use [mmap](http://linux.die.net/man/2/mmap)
  1270. files. Shared mmap is not currently supported in FUSE w/ page
  1271. caching disabled. Enabling `dropcacheonclose` is recommended when
  1272. `cache.files=partial|full|auto-full`.
  1273. * [Kodi](http://kodi.tv), [Plex](http://plex.tv),
  1274. [Subsonic](http://subsonic.org), etc. can use directory
  1275. [mtime](http://linux.die.net/man/2/stat) to more efficiently
  1276. determine whether to scan for new content rather than simply
  1277. performing a full scan. If using the default **getattr** policy of
  1278. **ff** it's possible those programs will miss an update on account
  1279. of it returning the first directory found's **stat** info and it's a
  1280. later directory on another mount which had the **mtime** recently
  1281. updated. To fix this you will want to set
  1282. **func.getattr=newest**. Remember though that this is just
  1283. **stat**. If the file is later **open**'ed or **unlink**'ed and the
  1284. policy is different for those then a completely different file or
  1285. directory could be acted on.
  1286. * Some policies mixed with some functions may result in strange
  1287. behaviors. Not that some of these behaviors and race conditions
  1288. couldn't happen outside **mergerfs** but that they are far more
  1289. likely to occur on account of the attempt to merge together multiple
  1290. sources of data which could be out of sync due to the different
  1291. policies.
  1292. * For consistency its generally best to set **category** wide policies
  1293. rather than individual **func**'s. This will help limit the
  1294. confusion of tools such as
  1295. [rsync](http://linux.die.net/man/1/rsync). However, the flexibility
  1296. is there if needed.
  1297. # KNOWN ISSUES / BUGS
  1298. #### kernel issues & bugs
  1299. [https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs](https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs)
  1300. #### directory mtime is not being updated
  1301. Remember that the default policy for `getattr` is `ff`. The
  1302. information for the first directory found will be returned. If it
  1303. wasn't the directory which had been updated then it will appear
  1304. outdated.
  1305. The reason this is the default is because any other policy would be
  1306. more expensive and for many applications it is unnecessary. To always
  1307. return the directory with the most recent mtime or a faked value based
  1308. on all found would require a scan of all drives.
  1309. If you always want the directory information from the one with the
  1310. most recent mtime then use the `newest` policy for `getattr`.
  1311. #### 'mv /mnt/pool/foo /mnt/disk1/foo' removes 'foo'
  1312. This is not a bug.
  1313. Run in verbose mode to better understand what's happening:
  1314. ```
  1315. $ mv -v /mnt/pool/foo /mnt/disk1/foo
  1316. copied '/mnt/pool/foo' -> '/mnt/disk1/foo'
  1317. removed '/mnt/pool/foo'
  1318. $ ls /mnt/pool/foo
  1319. ls: cannot access '/mnt/pool/foo': No such file or directory
  1320. ```
  1321. `mv`, when working across devices, is copying the source to target and
  1322. then removing the source. Since the source **is** the target in this
  1323. case, depending on the unlink policy, it will remove the just copied
  1324. file and other files across the branches.
  1325. If you want to move files to one drive just copy them there and use
  1326. mergerfs.dedup to clean up the old paths or manually remove them from
  1327. the branches directly.
  1328. #### cached memory appears greater than it should be
  1329. Use `cache.files=off` and/or `dropcacheonclose=true`. See the section
  1330. on page caching.
  1331. #### NFS clients returning ESTALE / Stale file handle
  1332. NFS does not like out of band changes. That is especially true of
  1333. inode values.
  1334. Be sure to use the following options:
  1335. * noforget
  1336. * inodecalc=path-hash
  1337. #### rtorrent fails with ENODEV (No such device)
  1338. Be sure to set `cache.files=partial|full|auto-full|per-processe` or
  1339. turn off `direct_io`. rtorrent and some other applications use
  1340. [mmap](http://linux.die.net/man/2/mmap) to read and write to files and
  1341. offer no fallback to traditional methods. FUSE does not currently
  1342. support mmap while using `direct_io`. There may be a performance
  1343. penalty on writes with `direct_io` off as well as the problem of
  1344. double caching but it's the only way to get such applications to
  1345. work. If the performance loss is too high for other apps you can mount
  1346. mergerfs twice. Once with `direct_io` enabled and one without it. Be
  1347. sure to set `dropcacheonclose=true` if not using `direct_io`.
  1348. #### Plex doesn't work with mergerfs
  1349. It does. If you're trying to put Plex's config / metadata / database
  1350. on mergerfs you can't set `cache.files=off` because Plex is using
  1351. sqlite3 with mmap enabled. Shared mmap is not supported by Linux's
  1352. FUSE implementation when page caching is disabled. To fix this place
  1353. the data elsewhere (preferable) or enable `cache.files` (with
  1354. `dropcacheonclose=true`). Sqlite3 does not need mmap but the developer
  1355. needs to fall back to standard IO if mmap fails.
  1356. This applies to other software: Radarr, Sonarr, Lidarr, Jellyfin, etc.
  1357. I would recommend reaching out to the developers of the software
  1358. you're having troubles with and asking them to add a fallback to
  1359. regular file IO when mmap is unavailable.
  1360. If the issue is that scanning doesn't seem to pick up media then be
  1361. sure to set `func.getattr=newest` though generally a full scan will
  1362. pick up all media anyway.
  1363. #### When a program tries to move or rename a file it fails
  1364. Please read the section above regarding [rename & link](#rename--link).
  1365. The problem is that many applications do not properly handle `EXDEV`
  1366. errors which `rename` and `link` may return even though they are
  1367. perfectly valid situations which do not indicate actual drive or OS
  1368. errors. The error will only be returned by mergerfs if using a path
  1369. preserving policy as described in the policy section above. If you do
  1370. not care about path preservation simply change the mergerfs policy to
  1371. the non-path preserving version. For example: `-o category.create=mfs`
  1372. Ideally the offending software would be fixed and it is recommended
  1373. that if you run into this problem you contact the software's author
  1374. and request proper handling of `EXDEV` errors.
  1375. #### my 32bit software has problems
  1376. Some software have problems with 64bit inode values. The symptoms can
  1377. include EOVERFLOW errors when trying to list files. You can address
  1378. this by setting `inodecalc` to one of the 32bit based algos as
  1379. described in the relevant section.
  1380. #### Samba: Moving files / directories fails
  1381. Workaround: Copy the file/directory and then remove the original
  1382. rather than move.
  1383. This isn't an issue with Samba but some SMB clients. GVFS-fuse v1.20.3
  1384. and prior (found in Ubuntu 14.04 among others) failed to handle
  1385. certain error codes correctly. Particularly **STATUS_NOT_SAME_DEVICE**
  1386. which comes from the **EXDEV** which is returned by **rename** when
  1387. the call is crossing mount points. When a program gets an **EXDEV** it
  1388. needs to explicitly take an alternate action to accomplish its
  1389. goal. In the case of **mv** or similar it tries **rename** and on
  1390. **EXDEV** falls back to a manual copying of data between the two
  1391. locations and unlinking the source. In these older versions of
  1392. GVFS-fuse if it received **EXDEV** it would translate that into
  1393. **EIO**. This would cause **mv** or most any application attempting to
  1394. move files around on that SMB share to fail with a IO error.
  1395. [GVFS-fuse v1.22.0](https://bugzilla.gnome.org/show_bug.cgi?id=734568)
  1396. and above fixed this issue but a large number of systems use the older
  1397. release. On Ubuntu the version can be checked by issuing `apt-cache
  1398. showpkg gvfs-fuse`. Most distros released in 2015 seem to have the
  1399. updated release and will work fine but older systems may
  1400. not. Upgrading gvfs-fuse or the distro in general will address the
  1401. problem.
  1402. In Apple's MacOSX 10.9 they replaced Samba (client and server) with
  1403. their own product. It appears their new client does not handle
  1404. **EXDEV** either and responds similar to older release of gvfs on
  1405. Linux.
  1406. #### Trashing files occasionally fails
  1407. This is the same issue as with Samba. `rename` returns `EXDEV` (in our
  1408. case that will really only happen with path preserving policies like
  1409. `epmfs`) and the software doesn't handle the situation well. This is
  1410. unfortunately a common failure of software which moves files
  1411. around. The standard indicates that an implementation `MAY` choose to
  1412. support non-user home directory trashing of files (which is a
  1413. `MUST`). The implementation `MAY` also support "top directory trashes"
  1414. which many probably do.
  1415. To create a `$topdir/.Trash` directory as defined in the standard use
  1416. the [mergerfs-tools](https://github.com/trapexit/mergerfs-tools) tool
  1417. `mergerfs.mktrash`.
  1418. #### Supplemental user groups
  1419. Due to the overhead of
  1420. [getgroups/setgroups](http://linux.die.net/man/2/setgroups) mergerfs
  1421. utilizes a cache. This cache is opportunistic and per thread. Each
  1422. thread will query the supplemental groups for a user when that
  1423. particular thread needs to change credentials and will keep that data
  1424. for the lifetime of the thread. This means that if a user is added to
  1425. a group it may not be picked up without the restart of
  1426. mergerfs. However, since the high level FUSE API's (at least the
  1427. standard version) thread pool dynamically grows and shrinks it's
  1428. possible that over time a thread will be killed and later a new thread
  1429. with no cache will start and query the new data.
  1430. The gid cache uses fixed storage to simplify the design and be
  1431. compatible with older systems which may not have C++11
  1432. compilers. There is enough storage for 256 users' supplemental
  1433. groups. Each user is allowed up to 32 supplemental groups. Linux >=
  1434. 2.6.3 allows up to 65535 groups per user but most other *nixs allow
  1435. far less. NFS allowing only 16. The system does handle overflow
  1436. gracefully. If the user has more than 32 supplemental groups only the
  1437. first 32 will be used. If more than 256 users are using the system
  1438. when an uncached user is found it will evict an existing user's cache
  1439. at random. So long as there aren't more than 256 active users this
  1440. should be fine. If either value is too low for your needs you will
  1441. have to modify `gidcache.hpp` to increase the values. Note that doing
  1442. so will increase the memory needed by each thread.
  1443. While not a bug some users have found when using containers that
  1444. supplemental groups defined inside the container don't work properly
  1445. with regard to permissions. This is expected as mergerfs lives outside
  1446. the container and therefore is querying the host's group
  1447. database. There might be a hack to work around this (make mergerfs
  1448. read the /etc/group file in the container) but it is not yet
  1449. implemented and would be limited to Linux and the /etc/group
  1450. DB. Preferably users would mount in the host group file into the
  1451. containers or use a standard shared user & groups technology like NIS
  1452. or LDAP.
  1453. # FAQ
  1454. #### How well does mergerfs scale? Is it "production ready?"
  1455. Users have reported running mergerfs on everything from a Raspberry Pi
  1456. to dual socket Xeon systems with >20 cores. I'm aware of at least a
  1457. few companies which use mergerfs in production. [Open Media
  1458. Vault](https://www.openmediavault.org) includes mergerfs as its sole
  1459. solution for pooling drives. The author of mergerfs had it running for
  1460. over 300 days managing 16+ drives with reasonably heavy 24/7 read and
  1461. write usage. Stopping only after the machine's power supply died.
  1462. Most serious issues (crashes or data corruption) have been due to
  1463. [kernel
  1464. bugs](https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs). All
  1465. of which are fixed in stable releases.
  1466. #### Can mergerfs be used with drives which already have data / are in use?
  1467. Yes. MergerFS is a proxy and does **NOT** interfere with the normal
  1468. form or function of the drives / mounts / paths it manages.
  1469. MergerFS is **not** a traditional filesystem. MergerFS is **not**
  1470. RAID. It does **not** manipulate the data that passes through it. It
  1471. does **not** shard data across drives. It merely shards some
  1472. **behavior** and aggregates others.
  1473. #### Can mergerfs be removed without affecting the data?
  1474. See the previous question's answer.
  1475. #### What policies should I use?
  1476. Unless you're doing something more niche the average user is probably
  1477. best off using `mfs` for `category.create`. It will spread files out
  1478. across your branches based on available space. Use `mspmfs` if you
  1479. want to try to colocate the data a bit more. You may want to use `lus`
  1480. if you prefer a slightly different distribution of data if you have a
  1481. mix of smaller and larger drives. Generally though `mfs`, `lus`, or
  1482. even `rand` are good for the general use case. If you are starting
  1483. with an imbalanced pool you can use the tool **mergerfs.balance** to
  1484. redistribute files across the pool.
  1485. If you really wish to try to colocate files based on directory you can
  1486. set `func.create` to `epmfs` or similar and `func.mkdir` to `rand` or
  1487. `eprand` depending on if you just want to colocate generally or on
  1488. specific branches. Either way the *need* to colocate is rare. For
  1489. instance: if you wish to remove the drive regularly and want the data
  1490. to predictably be on that drive or if you don't use backup at all and
  1491. don't wish to replace that data piecemeal. In which case using path
  1492. preservation can help but will require some manual
  1493. attention. Colocating after the fact can be accomplished using the
  1494. **mergerfs.consolidate** tool. If you don't need strict colocation
  1495. which the `ep` policies provide then you can use the `msp` based
  1496. policies which will walk back the path till finding a branch that
  1497. works.
  1498. Ultimately there is no correct answer. It is a preference or based on
  1499. some particular need. mergerfs is very easy to test and experiment
  1500. with. I suggest creating a test setup and experimenting to get a sense
  1501. of what you want.
  1502. `epmfs` is the default `category.create` policy because `ep` policies
  1503. are not going to change the general layout of the branches. It won't
  1504. place files/dirs on branches that don't already have the relative
  1505. branch. So it keeps the system in a known state. It's much easier to
  1506. stop using `epmfs` or redistribute files around the filesystem than it
  1507. is to consolidate them back.
  1508. #### What settings should I use?
  1509. Depends on what features you want. Generally speaking there are no
  1510. "wrong" settings. All settings are performance or feature related. The
  1511. best bet is to read over the available options and choose what fits
  1512. your situation. If something isn't clear from the documentation please
  1513. reach out and the documentation will be improved.
  1514. That said, for the average person, the following should be fine:
  1515. `cache.files=off,dropcacheonclose=true,category.create=mfs`
  1516. #### Why are all my files ending up on 1 drive?!
  1517. Did you start with empty drives? Did you explicitly configure a
  1518. `category.create` policy? Are you using an `existing path` / `path
  1519. preserving` policy?
  1520. The default create policy is `epmfs`. That is a path preserving
  1521. algorithm. With such a policy for `mkdir` and `create` with a set of
  1522. empty drives it will select only 1 drive when the first directory is
  1523. created. Anything, files or directories, created in that first
  1524. directory will be placed on the same branch because it is preserving
  1525. paths.
  1526. This catches a lot of new users off guard but changing the default
  1527. would break the setup for many existing users. If you do not care
  1528. about path preservation and wish your files to be spread across all
  1529. your drives change to `mfs` or similar policy as described above. If
  1530. you do want path preservation you'll need to perform the manual act of
  1531. creating paths on the drives you want the data to land on before
  1532. transferring your data. Setting `func.mkdir=epall` can simplify
  1533. managing path preservation for `create`. Or use `func.mkdir=rand` if
  1534. you're interested in just grouping together directory content by
  1535. drive.
  1536. #### Do hardlinks work?
  1537. Yes. See also the option `inodecalc` for how inode values are
  1538. calculated.
  1539. What mergerfs does not do is fake hard links across branches. Read
  1540. the section "rename & link" for how it works.
  1541. Remember that hardlinks will NOT work across devices. That includes
  1542. between the original filesystem and a mergerfs pool, between two
  1543. separate pools of the same underlying filesystems, or bind mounts of
  1544. paths within the mergerfs pool. The latter is common when using Docker
  1545. or Podman. Multiple volumes (bind mounts) to the same underlying
  1546. filesystem are considered different devices. There is no way to link
  1547. between them. You should mount in the highest directory in the
  1548. mergerfs pool that includes all the paths you need if you want links
  1549. to work.
  1550. #### Can I use mergerfs without SnapRAID? SnapRAID without mergerfs?
  1551. Yes. They are completely unrelated pieces of software.
  1552. #### Can mergerfs run via Docker, Podman, Kubernetes, etc.
  1553. Yes. With Docker you'll need to include `--cap-add=SYS_ADMIN
  1554. --device=/dev/fuse --security-opt=apparmor:unconfined` or similar with
  1555. other container runtimes. You should also be running it as root or
  1556. given sufficient caps to change user and group identity as well as
  1557. have root like filesystem permissions.
  1558. Keep in mind that you **MUST** consider identity when using
  1559. containers. For example: supplemental groups will be picked up from
  1560. the container unless you properly manage users and groups by sharing
  1561. relevant /etc files or by using some other means to share identity
  1562. across containers. Similarly if you use "rootless" containers and user
  1563. namespaces to do uid/gid translations you **MUST** consider that while
  1564. managing shared files.
  1565. Also, as mentioned by [hotio](https://hotio.dev/containers/mergerfs),
  1566. with Docker you should probably be mounting with `bind-propagation`
  1567. set to `slave`.
  1568. #### Does mergerfs support CoW / copy-on-write / writes to read-only filesystems?
  1569. Not in the sense of a filesystem like BTRFS or ZFS nor in the
  1570. overlayfs or aufs sense. It does offer a
  1571. [cow-shell](http://manpages.ubuntu.com/manpages/bionic/man1/cow-shell.1.html)
  1572. like hard link breaking (copy to temp file then rename over original)
  1573. which can be useful when wanting to save space by hardlinking
  1574. duplicate files but wish to treat each name as if it were a unique and
  1575. separate file.
  1576. If you want to write to a read-only filesystem you should look at
  1577. overlayfs. You can always include the overlayfs mount into a mergerfs
  1578. pool.
  1579. #### Why can't I see my files / directories?
  1580. It's almost always a permissions issue. Unlike mhddfs and
  1581. unionfs-fuse, which runs as root and attempts to access content as
  1582. such, mergerfs always changes its credentials to that of the
  1583. caller. This means that if the user does not have access to a file or
  1584. directory than neither will mergerfs. However, because mergerfs is
  1585. creating a union of paths it may be able to read some files and
  1586. directories on one drive but not another resulting in an incomplete
  1587. set.
  1588. Whenever you run into a split permission issue (seeing some but not
  1589. all files) try using
  1590. [mergerfs.fsck](https://github.com/trapexit/mergerfs-tools) tool to
  1591. check for and fix the mismatch. If you aren't seeing anything at all
  1592. be sure that the basic permissions are correct. The user and group
  1593. values are correct and that directories have their executable bit
  1594. set. A common mistake by users new to Linux is to `chmod -R 644` when
  1595. they should have `chmod -R u=rwX,go=rX`.
  1596. If using a network filesystem such as NFS, SMB, CIFS (Samba) be sure
  1597. to pay close attention to anything regarding permissioning and
  1598. users. Root squashing and user translation for instance has bitten a
  1599. few mergerfs users. Some of these also affect the use of mergerfs from
  1600. container platforms such as Docker.
  1601. #### Why use FUSE? Why not a kernel based solution?
  1602. As with any solutions to a problem there are advantages and
  1603. disadvantages to each one.
  1604. A FUSE based solution has all the downsides of FUSE:
  1605. * Higher IO latency due to the trips in and out of kernel space
  1606. * Higher general overhead due to trips in and out of kernel space
  1607. * Double caching when using page caching
  1608. * Misc limitations due to FUSE's design
  1609. But FUSE also has a lot of upsides:
  1610. * Easier to offer a cross platform solution
  1611. * Easier forward and backward compatibility
  1612. * Easier updates for users
  1613. * Easier and faster release cadence
  1614. * Allows more flexibility in design and features
  1615. * Overall easier to write, secure, and maintain
  1616. * Much lower barrier to entry (getting code into the kernel takes a
  1617. lot of time and effort initially)
  1618. FUSE was chosen because of all the advantages listed above. The
  1619. negatives of FUSE do not outweigh the positives.
  1620. #### Is my OS's libfuse needed for mergerfs to work?
  1621. No. Normally `mount.fuse` is needed to get mergerfs (or any FUSE
  1622. filesystem to mount using the `mount` command but in vendoring the
  1623. libfuse library the `mount.fuse` app has been renamed to
  1624. `mount.mergerfs` meaning the filesystem type in `fstab` can simply be
  1625. `mergerfs`. That said there should be no harm in having it installed
  1626. and continuing to using `fuse.mergerfs` as the type in `/etc/fstab`.
  1627. If `mergerfs` doesn't work as a type it could be due to how the
  1628. `mount.mergerfs` tool was installed. Must be in `/sbin/` with proper
  1629. permissions.
  1630. #### Why was splice support removed?
  1631. After a lot of testing over the years splicing always appeared to be
  1632. at best provide equivalent performance and in cases worse
  1633. performance. Splice is not supported on other platforms forcing a
  1634. traditional read/write fallback to be provided. The splice code was
  1635. removed to simplify the codebase.
  1636. #### Why use mergerfs over mhddfs?
  1637. mhddfs is no longer maintained and has some known stability and
  1638. security issues (see below). MergerFS provides a superset of mhddfs'
  1639. features and should offer the same or maybe better performance.
  1640. Below is an example of mhddfs and mergerfs setup to work similarly.
  1641. `mhddfs -o mlimit=4G,allow_other /mnt/drive1,/mnt/drive2 /mnt/pool`
  1642. `mergerfs -o minfreespace=4G,category.create=ff /mnt/drive1:/mnt/drive2 /mnt/pool`
  1643. #### Why use mergerfs over aufs?
  1644. aufs is mostly abandoned and no longer available in many distros.
  1645. While aufs can offer better peak performance mergerfs provides more
  1646. configurability and is generally easier to use. mergerfs however does
  1647. not offer the overlay / copy-on-write (CoW) features which aufs and
  1648. overlayfs have.
  1649. #### Why use mergerfs over unionfs?
  1650. UnionFS is more like aufs than mergerfs in that it offers overlay /
  1651. CoW features. If you're just looking to create a union of drives and
  1652. want flexibility in file/directory placement then mergerfs offers that
  1653. whereas unionfs is more for overlaying RW filesystems over RO ones.
  1654. #### Why use mergerfs over overlayfs?
  1655. Same reasons as with unionfs.
  1656. #### Why use mergerfs over LVM/ZFS/BTRFS/RAID0 drive concatenation / striping?
  1657. With simple JBOD / drive concatenation / stripping / RAID0 a single
  1658. drive failure will result in full pool failure. mergerfs performs a
  1659. similar function without the possibility of catastrophic failure and
  1660. the difficulties in recovery. Drives may fail, however, all other data
  1661. will continue to be accessible.
  1662. When combined with something like [SnapRaid](http://www.snapraid.it)
  1663. and/or an offsite backup solution you can have the flexibility of JBOD
  1664. without the single point of failure.
  1665. #### Why use mergerfs over ZFS?
  1666. MergerFS is not intended to be a replacement for ZFS. MergerFS is
  1667. intended to provide flexible pooling of arbitrary drives (local or
  1668. remote), of arbitrary sizes, and arbitrary filesystems. For `write
  1669. once, read many` usecases such as bulk media storage. Where data
  1670. integrity and backup is managed in other ways. In that situation ZFS
  1671. can introduce a number of costs and limitations as described
  1672. [here](http://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html),
  1673. [here](https://markmcb.com/2020/01/07/five-years-of-btrfs/), and
  1674. [here](https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSWhyNoRealReshaping).
  1675. #### Why use mergerfs over UnRAID?
  1676. UnRAID is a full OS and its storage layer, as I understand, is
  1677. proprietary and closed source. Users who have experience with both
  1678. have said they prefer the flexibility offered by mergerfs and for some
  1679. the fact it is free and open source is important.
  1680. There are a number of UnRAID users who use mergerfs as well though I'm
  1681. not entirely familiar with the use case.
  1682. #### What should mergerfs NOT be used for?
  1683. * databases: Even if the database stored data in separate files
  1684. (mergerfs wouldn't offer much otherwise) the higher latency of the
  1685. indirection will kill performance. If it is a lightly used SQLITE
  1686. database then it may be fine but you'll need to test.
  1687. * VM images: For the same reasons as databases. VM images are accessed
  1688. very aggressively and mergerfs will introduce too much latency (if
  1689. it works at all).
  1690. * As replacement for RAID: mergerfs is just for pooling branches. If
  1691. you need that kind of device performance aggregation or high
  1692. availability you should stick with RAID.
  1693. #### Can drives be written to directly? Outside of mergerfs while pooled?
  1694. Yes, however it's not recommended to use the same file from within the
  1695. pool and from without at the same time (particularly
  1696. writing). Especially if using caching of any kind (cache.files,
  1697. cache.entry, cache.attr, cache.negative_entry, cache.symlinks,
  1698. cache.readdir, etc.) as there could be a conflict between cached data
  1699. and not.
  1700. #### Why do I get an "out of space" / "no space left on device" / ENOSPC error even though there appears to be lots of space available?
  1701. First make sure you've read the sections above about policies, path
  1702. preservation, branch filtering, and the options **minfreespace**,
  1703. **moveonenospc**, **statfs**, and **statfs_ignore**.
  1704. mergerfs is simply presenting a union of the content within multiple
  1705. branches. The reported free space is an aggregate of space available
  1706. within the pool (behavior modified by **statfs** and
  1707. **statfs_ignore**). It does not represent a contiguous space. In the
  1708. same way that read-only filesystems, those with quotas, or reserved
  1709. space report the full theoretical space available.
  1710. Due to path preservation, branch tagging, read-only status, and
  1711. **minfreespace** settings it is perfectly valid that `ENOSPC` / "out
  1712. of space" / "no space left on device" be returned. It is doing what
  1713. was asked of it: filtering possible branches due to those
  1714. settings. Only one error can be returned and if one of the reasons for
  1715. filtering a branch was **minfreespace** then it will be returned as
  1716. such. **moveonenospc** is only relevant to writing a file which is too
  1717. large for the drive its currently on.
  1718. It is also possible that the filesystem selected has run out of
  1719. inodes. Use `df -i` to list the total and available inodes per
  1720. filesystem.
  1721. If you don't care about path preservation then simply change the
  1722. `create` policy to one which isn't. `mfs` is probably what most are
  1723. looking for. The reason it's not default is because it was originally
  1724. set to `epmfs` and changing it now would change people's setup. Such a
  1725. setting change will likely occur in mergerfs 3.
  1726. #### Why does the total available space in mergerfs not equal outside?
  1727. Are you using ext2/3/4? With reserve for root? mergerfs uses available
  1728. space for statfs calculations. If you've reserved space for root then
  1729. it won't show up.
  1730. You can remove the reserve by running: `tune2fs -m 0 <device>`
  1731. #### Can mergerfs mounts be exported over NFS?
  1732. Yes, however if you do anything which may changes files out of band
  1733. (including for example using the `newest` policy) it will result in
  1734. "stale file handle" errors unless properly setup.
  1735. Be sure to use the following options:
  1736. * noforget
  1737. * inodecalc=path-hash
  1738. #### Can mergerfs mounts be exported over Samba / SMB?
  1739. Yes. While some users have reported problems it appears to always be
  1740. related to how Samba is setup in relation to permissions.
  1741. #### Can mergerfs mounts be used over SSHFS?
  1742. Yes.
  1743. #### I notice massive slowdowns of writes when enabling cache.files.
  1744. When file caching is enabled in any form (`cache.files!=off` or
  1745. `direct_io=false`) it will issue `getxattr` requests for
  1746. `security.capability` prior to *every single write*. This will usually
  1747. result in a performance degradation, especially when using a network
  1748. filesystem (such as NFS or CIFS/SMB/Samba.) Unfortunately at this
  1749. moment the kernel is not caching the response.
  1750. To work around this situation mergerfs offers a few solutions.
  1751. 1. Set `security_capability=false`. It will short circuit any call and
  1752. return `ENOATTR`. This still means though that mergerfs will
  1753. receive the request before every write but at least it doesn't get
  1754. passed through to the underlying filesystem.
  1755. 2. Set `xattr=noattr`. Same as above but applies to *all* calls to
  1756. getxattr. Not just `security.capability`. This will not be cached
  1757. by the kernel either but mergerfs' runtime config system will still
  1758. function.
  1759. 3. Set `xattr=nosys`. Results in mergerfs returning `ENOSYS` which
  1760. *will* be cached by the kernel. No future xattr calls will be
  1761. forwarded to mergerfs. The downside is that also means the xattr
  1762. based config and query functionality won't work either.
  1763. 4. Disable file caching. If you aren't using applications which use
  1764. `mmap` it's probably simpler to just disable it all together. The
  1765. kernel won't send the requests when caching is disabled.
  1766. #### It's mentioned that there are some security issues with mhddfs. What are they? How does mergerfs address them?
  1767. [mhddfs](https://github.com/trapexit/mhddfs) manages running as
  1768. **root** by calling
  1769. [getuid()](https://github.com/trapexit/mhddfs/blob/cae96e6251dd91e2bdc24800b4a18a74044f6672/src/main.c#L319)
  1770. and if it returns **0** then it will
  1771. [chown](http://linux.die.net/man/1/chown) the file. Not only is that a
  1772. race condition but it doesn't handle other situations. Rather than
  1773. attempting to simulate POSIX ACL behavior the proper way to manage
  1774. this is to use [seteuid](http://linux.die.net/man/2/seteuid) and
  1775. [setegid](http://linux.die.net/man/2/setegid), in effect becoming the
  1776. user making the original call, and perform the action as them. This is
  1777. what mergerfs does and why mergerfs should always run as root.
  1778. In Linux setreuid syscalls apply only to the thread. GLIBC hides this
  1779. away by using realtime signals to inform all threads to change
  1780. credentials. Taking after **Samba**, mergerfs uses
  1781. **syscall(SYS_setreuid,...)** to set the callers credentials for that
  1782. thread only. Jumping back to **root** as necessary should escalated
  1783. privileges be needed (for instance: to clone paths between drives).
  1784. For non-Linux systems mergerfs uses a read-write lock and changes
  1785. credentials only when necessary. If multiple threads are to be user X
  1786. then only the first one will need to change the processes
  1787. credentials. So long as the other threads need to be user X they will
  1788. take a readlock allowing multiple threads to share the
  1789. credentials. Once a request comes in to run as user Y that thread will
  1790. attempt a write lock and change to Y's credentials when it can. If the
  1791. ability to give writers priority is supported then that flag will be
  1792. used so threads trying to change credentials don't starve. This isn't
  1793. the best solution but should work reasonably well assuming there are
  1794. few users.
  1795. # SUPPORT
  1796. Filesystems are complex and difficult to debug. mergerfs, while being
  1797. just a proxy of sorts, can be difficult to debug given the large
  1798. number of possible settings it can have itself and the number of
  1799. environments it can run in. When reporting on a suspected issue
  1800. **please** include as much of the below information as possible
  1801. otherwise it will be difficult or impossible to diagnose. Also please
  1802. read the above documentation as it provides details on many previously
  1803. encountered questions/issues.
  1804. **Please make sure you are using the [latest
  1805. release](https://github.com/trapexit/mergerfs/releases) or have tried
  1806. it in comparison. Old versions, which are often included in distros
  1807. like Debian and Ubuntu, are not ever going to be updated and the issue
  1808. you are encountering may have been addressed already.**
  1809. **For commercial support or feature requests please [contact me
  1810. directly.](mailto:support@spawn.link)**
  1811. #### Information to include in bug reports
  1812. * [Information about the broader problem along with any attempted
  1813. solutions.](https://xyproblem.info)
  1814. * Solution already ruled out and why.
  1815. * Version of mergerfs: `mergerfs --version`
  1816. * mergerfs settings / arguments: from fstab, systemd unit, command
  1817. line, OMV plugin, etc.
  1818. * Version of the OS: `uname -a` and `lsb_release -a`
  1819. * List of branches, their filesystem types, sizes (before and after issue): `df -h`
  1820. * **All** information about the relevant paths and files: permissions, ownership, etc.
  1821. * **All** information about the client app making the requests: version, uid/gid
  1822. * Runtime environment:
  1823. * Is mergerfs running within a container?
  1824. * Are the client apps using mergerfs running in a container?
  1825. * A `strace` of the app having problems:
  1826. * `strace -fvTtt -s 256 -o /tmp/app.strace.txt <cmd>`
  1827. * A `strace` of mergerfs while the program is trying to do whatever it is failing to do:
  1828. * `strace -fvTtt -s 256 -p <mergerfsPID> -o /tmp/mergerfs.strace.txt`
  1829. * **Precise** directions on replicating the issue. Do not leave **anything** out.
  1830. * Try to recreate the problem in the simplest way using standard programs: `ln`, `mv`, `cp`, `ls`, `dd`, etc.
  1831. #### Contact / Issue submission
  1832. * github.com: https://github.com/trapexit/mergerfs/issues
  1833. * discord: https://discord.gg/MpAr69V
  1834. * reddit: https://www.reddit.com/r/mergerfs
  1835. #### Donations
  1836. https://github.com/trapexit/support
  1837. Development and support of a project like mergerfs requires a
  1838. significant amount of time and effort. The software is released under
  1839. the very liberal ISC license and is therefore free to use for personal
  1840. or commercial uses.
  1841. If you are a personal user and find mergerfs and its support valuable
  1842. and would like to support the project financially it would be very
  1843. much appreciated.
  1844. If you are using mergerfs commercially please consider sponsoring the
  1845. project to ensure it continues to be maintained and receive
  1846. updates. If custom features are needed feel free to [contact me
  1847. directly](mailto:support@spawn.link).
  1848. # LINKS
  1849. * https://spawn.link
  1850. * https://github.com/trapexit/mergerfs
  1851. * https://github.com/trapexit/mergerfs/wiki
  1852. * https://github.com/trapexit/mergerfs-tools
  1853. * https://github.com/trapexit/scorch
  1854. * https://github.com/trapexit/bbf