You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

2403 lines
100 KiB

11 years ago
2 years ago
4 years ago
5 years ago
5 years ago
5 years ago
3 years ago
5 years ago
  1. % mergerfs(1) mergerfs user manual
  2. # NAME
  3. mergerfs - a featureful union filesystem
  4. # SYNOPSIS
  5. mergerfs -o<options> <branches> <mountpoint>
  6. # DESCRIPTION
  7. **mergerfs** is a union filesystem geared towards simplifying storage
  8. and management of files across numerous commodity storage devices. It
  9. is similar to **mhddfs**, **unionfs**, and **aufs**.
  10. # FEATURES
  11. * Configurable behaviors / file placement
  12. * Ability to add or remove filesystems at will
  13. * Resistance to individual filesystem failure
  14. * Support for extended attributes (xattrs)
  15. * Support for file attributes (chattr)
  16. * Runtime configurable (via xattrs)
  17. * Works with heterogeneous filesystem types
  18. * Moving of file when filesystem runs out of space while writing
  19. * Ignore read-only filesystems when creating files
  20. * Turn read-only files into symlinks to underlying file
  21. * Hard link copy-on-write / CoW
  22. * Support for POSIX ACLs
  23. * Misc other things
  24. # HOW IT WORKS
  25. mergerfs logically merges multiple paths together. Think a union of
  26. sets. The file/s or directory/s acted on or presented through mergerfs
  27. are based on the policy chosen for that particular action. Read more
  28. about policies below.
  29. ```
  30. A + B = C
  31. /disk1 /disk2 /merged
  32. | | |
  33. +-- /dir1 +-- /dir1 +-- /dir1
  34. | | | | | |
  35. | +-- file1 | +-- file2 | +-- file1
  36. | | +-- file3 | +-- file2
  37. +-- /dir2 | | +-- file3
  38. | | +-- /dir3 |
  39. | +-- file4 | +-- /dir2
  40. | +-- file5 | |
  41. +-- file6 | +-- file4
  42. |
  43. +-- /dir3
  44. | |
  45. | +-- file5
  46. |
  47. +-- file6
  48. ```
  49. mergerfs does **not** support the copy-on-write (CoW) or whiteout
  50. behaviors found in **aufs** and **overlayfs**. You can **not** mount a
  51. read-only filesystem and write to it. However, mergerfs will ignore
  52. read-only filesystems when creating new files so you can mix
  53. read-write and read-only filesystems. It also does **not** split data
  54. across filesystems. It is not RAID0 / striping. It is simply a union of
  55. other filesystems.
  56. # TERMINOLOGY
  57. * branch: A base path used in the pool.
  58. * pool: The mergerfs mount. The union of the branches.
  59. * relative path: The path in the pool relative to the branch and mount.
  60. * function: A filesystem call (open, unlink, create, getattr, rmdir, etc.)
  61. * category: A collection of functions based on basic behavior (action, create, search).
  62. * policy: The algorithm used to select a file when performing a function.
  63. * path preservation: Aspect of some policies which includes checking the path for which a file would be created.
  64. # BASIC SETUP
  65. If you don't already know that you have a special use case then just
  66. start with one of the following option sets.
  67. #### You need `mmap` (used by rtorrent and many sqlite3 base software)
  68. `cache.files=partial,dropcacheonclose=true,category.create=mfs`
  69. #### You don't need `mmap`
  70. `cache.files=off,dropcacheonclose=true,category.create=mfs`
  71. ### Command Line
  72. `mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs /mnt/hdd\*:/mnt/otherhdd:/mnt/rohdd=ro /media`
  73. ### /etc/fstab
  74. `/mnt/hdd*:/mnt/otherhdd:/mnt/rohdd=ro /media fuse.mergerfs cache.files=partial,dropcacheonclose=true,category.create=mfs 0 0`
  75. ### systemd mount
  76. https://github.com/trapexit/mergerfs/wiki/systemd
  77. ```
  78. [Unit]
  79. Description=mergerfs service
  80. [Service]
  81. Type=simple
  82. KillMode=none
  83. ExecStart=/usr/bin/mergerfs \
  84. -f \
  85. -o cache.files=partial \
  86. -o dropcacheonclose=true \
  87. -o category.create=mfs \
  88. /mnt/hdd0:/mnt/hdd1 \
  89. /media
  90. ExecStop=/bin/fusermount -uz /media
  91. Restart=on-failure
  92. [Install]
  93. WantedBy=default.target
  94. ```
  95. See the mergerfs [wiki for real world
  96. deployments](https://github.com/trapexit/mergerfs/wiki/Real-World-Deployments)
  97. for comparisons / ideas.
  98. # OPTIONS
  99. These options are the same regardless of whether you use them with the
  100. `mergerfs` commandline program, in fstab, or in a config file.
  101. ### mount options
  102. * **config**: Path to a config file. Same arguments as below in
  103. key=val / ini style format.
  104. * **branches**: Colon delimited list of branches.
  105. * **minfreespace=SIZE**: The minimum space value used for creation
  106. policies. Can be overridden by branch specific option. Understands
  107. 'K', 'M', and 'G' to represent kilobyte, megabyte, and gigabyte
  108. respectively. (default: 4G)
  109. * **moveonenospc=BOOL|POLICY**: When enabled if a **write** fails with
  110. **ENOSPC** (no space left on device) or **EDQUOT** (disk quota
  111. exceeded) the policy selected will run to find a new location for
  112. the file. An attempt to move the file to that branch will occur
  113. (keeping all metadata possible) and if successful the original is
  114. unlinked and the write retried. (default: false, true = mfs)
  115. * **inodecalc=passthrough|path-hash|devino-hash|hybrid-hash**: Selects
  116. the inode calculation algorithm. (default: hybrid-hash)
  117. * **dropcacheonclose=BOOL**: When a file is requested to be closed
  118. call `posix_fadvise` on it first to instruct the kernel that we no
  119. longer need the data and it can drop its cache. Recommended when
  120. **cache.files=partial|full|auto-full|per-process** to limit double
  121. caching. (default: false)
  122. * **symlinkify=BOOL**: When enabled and a file is not writable and its
  123. mtime or ctime is older than **symlinkify_timeout** files will be
  124. reported as symlinks to the original files. Please read more below
  125. before using. (default: false)
  126. * **symlinkify_timeout=UINT**: Time to wait, in seconds, to activate
  127. the **symlinkify** behavior. (default: 3600)
  128. * **nullrw=BOOL**: Turns reads and writes into no-ops. The request
  129. will succeed but do nothing. Useful for benchmarking
  130. mergerfs. (default: false)
  131. * **lazy-umount-mountpoint=BOOL**: mergerfs will attempt to "lazy
  132. umount" the mountpoint before mounting itself. Useful when
  133. performing live upgrades of mergerfs. (default: false)
  134. * **ignorepponrename=BOOL**: Ignore path preserving on
  135. rename. Typically rename and link act differently depending on the
  136. policy of `create` (read below). Enabling this will cause rename and
  137. link to always use the non-path preserving behavior. This means
  138. files, when renamed or linked, will stay on the same
  139. filesystem. (default: false)
  140. * **security_capability=BOOL**: If false return ENOATTR when xattr
  141. security.capability is queried. (default: true)
  142. * **xattr=passthrough|noattr|nosys**: Runtime control of
  143. xattrs. Default is to passthrough xattr requests. 'noattr' will
  144. short circuit as if nothing exists. 'nosys' will respond with ENOSYS
  145. as if xattrs are not supported or disabled. (default: passthrough)
  146. * **link_cow=BOOL**: When enabled if a regular file is opened which
  147. has a link count > 1 it will copy the file to a temporary file and
  148. rename over the original. Breaking the link and providing a basic
  149. copy-on-write function similar to cow-shell. (default: false)
  150. * **statfs=base|full**: Controls how statfs works. 'base' means it
  151. will always use all branches in statfs calculations. 'full' is in
  152. effect path preserving and only includes branches where the path
  153. exists. (default: base)
  154. * **statfs_ignore=none|ro|nc**: 'ro' will cause statfs calculations to
  155. ignore available space for branches mounted or tagged as 'read-only'
  156. or 'no create'. 'nc' will ignore available space for branches tagged
  157. as 'no create'. (default: none)
  158. * **nfsopenhack=off|git|all**: A workaround for exporting mergerfs
  159. over NFS where there are issues with creating files for write while
  160. setting the mode to read-only. (default: off)
  161. * **branches-mount-timeout=UINT**: Number of seconds to wait at
  162. startup for branches to be a mount other than the mountpoint's
  163. filesystem. (default: 0)
  164. * **follow-symlinks=never|directory|regular|all**: Turns symlinks into
  165. what they point to. (default: never)
  166. * **link-exdev=passthrough|rel-symlink|abs-base-symlink|abs-pool-symlink**:
  167. When a link fails with EXDEV optionally create a symlink to the file
  168. instead.
  169. * **rename-exdev=passthrough|rel-symlink|abs-symlink**: When a rename
  170. fails with EXDEV optionally move the file to a special directory and
  171. symlink to it.
  172. * **readahead=UINT**: Set readahead (in kilobytes) for mergerfs and
  173. branches if greater than 0. (default: 0)
  174. * **posix_acl=BOOL**: Enable POSIX ACL support (if supported by kernel
  175. and underlying filesystem). (default: false)
  176. * **async_read=BOOL**: Perform reads asynchronously. If disabled or
  177. unavailable the kernel will ensure there is at most one pending read
  178. request per file handle and will attempt to order requests by
  179. offset. (default: true)
  180. * **fuse_msg_size=UINT**: Set the max number of pages per FUSE
  181. message. Only available on Linux >= 4.20 and ignored
  182. otherwise. (min: 1; max: 256; default: 256)
  183. * **threads=INT**: Number of threads to use. When used alone
  184. (`process-thread-count=-1`) it sets the number of threads reading
  185. and processing FUSE messages. When used together it sets the number
  186. of threads reading from FUSE. When set to zero it will attempt to
  187. discover and use the number of logical cores. If the thread count is
  188. set negative it will look up the number of cores then divide by the
  189. absolute value. ie. threads=-2 on an 8 core machine will result in 8
  190. / 2 = 4 threads. There will always be at least 1 thread. If set to
  191. -1 in combination with `process-thread-count` then it will try to
  192. pick reasonable values based on CPU thread count. NOTE: higher
  193. number of threads increases parallelism but usually decreases
  194. throughput. (default: 0)
  195. * **read-thread-count=INT**: Alias for `threads`.
  196. * **process-thread-count=INT**: Enables separate thread pool to
  197. asynchronously process FUSE requests. In this mode
  198. `read-thread-count` refers to the number of threads reading FUSE
  199. messages which are dispatched to process threads. -1 means disabled
  200. otherwise acts like `read-thread-count`. (default: -1)
  201. * **pin-threads=STR**: Selects a strategy to pin threads to CPUs
  202. (default: unset)
  203. * **scheduling-priority=INT**: Set mergerfs' scheduling
  204. priority. Valid values range from -20 to 19. See `setpriority` man
  205. page for more details. (default: -10)
  206. * **fsname=STR**: Sets the name of the filesystem as seen in
  207. **mount**, **df**, etc. Defaults to a list of the source paths
  208. concatenated together with the longest common prefix removed.
  209. * **func.FUNC=POLICY**: Sets the specific FUSE function's policy. See
  210. below for the list of value types. Example: **func.getattr=newest**
  211. * **category.action=POLICY**: Sets policy of all FUSE functions in the
  212. action category. (default: epall)
  213. * **category.create=POLICY**: Sets policy of all FUSE functions in the
  214. create category. (default: epmfs)
  215. * **category.search=POLICY**: Sets policy of all FUSE functions in the
  216. search category. (default: ff)
  217. * **cache.open=UINT**: 'open' policy cache timeout in
  218. seconds. (default: 0)
  219. * **cache.statfs=UINT**: 'statfs' cache timeout in seconds. (default:
  220. 0)
  221. * **cache.attr=UINT**: File attribute cache timeout in
  222. seconds. (default: 1)
  223. * **cache.entry=UINT**: File name lookup cache timeout in
  224. seconds. (default: 1)
  225. * **cache.negative_entry=UINT**: Negative file name lookup cache
  226. timeout in seconds. (default: 0)
  227. * **cache.files=libfuse|off|partial|full|auto-full|per-process**: File
  228. page caching mode (default: libfuse)
  229. * **cache.files.process-names=LIST**: A pipe | delimited list of
  230. process [comm](https://man7.org/linux/man-pages/man5/proc.5.html)
  231. names to enable page caching for when
  232. `cache.files=per-process`. (default: "rtorrent|qbittorrent-nox")
  233. * **cache.writeback=BOOL**: Enable kernel writeback caching (default:
  234. false)
  235. * **cache.symlinks=BOOL**: Cache symlinks (if supported by kernel)
  236. (default: false)
  237. * **cache.readdir=BOOL**: Cache readdir (if supported by kernel)
  238. (default: false)
  239. * **direct_io**: deprecated - Bypass page cache. Use `cache.files=off`
  240. instead. (default: false)
  241. * **kernel_cache**: deprecated - Do not invalidate data cache on file
  242. open. Use `cache.files=full` instead. (default: false)
  243. * **auto_cache**: deprecated - Invalidate data cache if file mtime or
  244. size change. Use `cache.files=auto-full` instead. (default: false)
  245. * **async_read**: deprecated - Perform reads asynchronously. Use
  246. `async_read=true` instead.
  247. * **sync_read**: deprecated - Perform reads synchronously. Use
  248. `async_read=false` instead.
  249. * **use_ino**: deprecated - Always enabled.
  250. * **allow_other**: deprecated - Always enabled.
  251. * **splice_read**: deprecated - Does nothing.
  252. * **splice_write**: deprecated - Does nothing.
  253. * **splice_move**: deprecated - Does nothing.
  254. * **allow_other**: deprecated - mergerfs always sets this FUSE option
  255. as normal permissions can be used to limit access.
  256. * **use_ino**: deprecated - mergerfs should always control inode
  257. calculation so this is enabled all the time.
  258. **NOTE:** Options are evaluated in the order listed so if the options
  259. are **func.rmdir=rand,category.action=ff** the **action** category
  260. setting will override the **rmdir** setting.
  261. **NOTE:** Always look at the documentation for the version of mergerfs
  262. you're using. Not all features are available in older releases. Use
  263. `man mergerfs` or find the docs as linked in the release.
  264. #### Value Types
  265. * BOOL = 'true' | 'false'
  266. * INT = [MIN_INT,MAX_INT]
  267. * UINT = [0,MAX_INT]
  268. * SIZE = 'NNM'; NN = INT, M = 'K' | 'M' | 'G' | 'T'
  269. * STR = string (may refer to an enumerated value, see details of
  270. argument)
  271. * FUNC = filesystem function
  272. * CATEGORY = function category
  273. * POLICY = mergerfs function policy
  274. ### branches
  275. The 'branches' argument is a colon (':') delimited list of paths to be
  276. pooled together. It does not matter if the paths are on the same or
  277. different filesystems nor does it matter the filesystem type (within
  278. reason). Used and available space will not be duplicated for paths on
  279. the same filesystem and any features which aren't supported by the
  280. underlying filesystem (such as file attributes or extended attributes)
  281. will return the appropriate errors.
  282. Branches currently have two options which can be set. A type which
  283. impacts whether or not the branch is included in a policy calculation
  284. and a individual minfreespace value. The values are set by prepending
  285. an `=` at the end of a branch designation and using commas as
  286. delimiters. Example: `/mnt/drive=RW,1234`
  287. #### branch mode
  288. * RW: (read/write) - Default behavior. Will be eligible in all policy
  289. categories.
  290. * RO: (read-only) - Will be excluded from `create` and `action`
  291. policies. Same as a read-only mounted filesystem would be (though
  292. faster to process).
  293. * NC: (no-create) - Will be excluded from `create` policies. You can't
  294. create on that branch but you can change or delete.
  295. #### minfreespace
  296. Same purpose and syntax as the global option but specific to the
  297. branch. If not set the global value is used.
  298. #### globbing
  299. To make it easier to include multiple branches mergerfs supports
  300. [globbing](http://linux.die.net/man/7/glob). **The globbing tokens
  301. MUST be escaped when using via the shell else the shell itself will
  302. apply the glob itself.**
  303. ```
  304. # mergerfs /mnt/hdd\*:/mnt/ssd /media
  305. ```
  306. The above line will use all mount points in /mnt prefixed with **hdd** and **ssd**.
  307. To have the pool mounted at boot or otherwise accessible from related tools use **/etc/fstab**.
  308. ```
  309. # <file system> <mount point> <type> <options> <dump> <pass>
  310. /mnt/hdd*:/mnt/ssd /media fuse.mergerfs minfreespace=16G 0 0
  311. ```
  312. **NOTE:** the globbing is done at mount or when updated using the
  313. runtime API. If a new directory is added matching the glob after the
  314. fact it will not be automatically included.
  315. **NOTE:** for mounting via **fstab** to work you must have
  316. **mount.fuse** installed. For Ubuntu/Debian it is included in the
  317. **fuse** package.
  318. ### inodecalc
  319. Inodes (st_ino) are unique identifiers within a filesystem. Each
  320. mounted filesystem has device ID (st_dev) as well and together they
  321. can uniquely identify a file on the whole of the system. Entries on
  322. the same device with the same inode are in fact references to the same
  323. underlying file. It is a many to one relationship between names and an
  324. inode. Directories, however, do not have multiple links on most
  325. systems due to the complexity they add.
  326. FUSE allows the server (mergerfs) to set inode values but not device
  327. IDs. Creating an inode value is somewhat complex in mergerfs' case as
  328. files aren't really in its control. If a policy changes what directory
  329. or file is to be selected or something changes out of band it becomes
  330. unclear what value should be used. Most software does not to care what
  331. the values are but those that do often break if a value changes
  332. unexpectedly. The tool `find` will abort a directory walk if it sees a
  333. directory inode change. NFS will return stale handle errors if the
  334. inode changes out of band. File dedup tools will usually leverage
  335. device ids and inodes as a shortcut in searching for duplicate files
  336. and would resort to full file comparisons should it find different
  337. inode values.
  338. mergerfs offers multiple ways to calculate the inode in hopes of
  339. covering different usecases.
  340. * passthrough: Passes through the underlying inode value. Mostly
  341. intended for testing as using this does not address any of the
  342. problems mentioned above and could confuse file deduplication
  343. software as inodes from different filesystems can be the same.
  344. * path-hash: Hashes the relative path of the entry in question. The
  345. underlying file's values are completely ignored. This means the
  346. inode value will always be the same for that file path. This is
  347. useful when using NFS and you make changes out of band such as copy
  348. data between branches. This also means that entries that do point to
  349. the same file will not be recognizable via inodes. That **does not**
  350. mean hard links don't work. They will.
  351. * path-hash32: 32bit version of path-hash.
  352. * devino-hash: Hashes the device id and inode of the underlying
  353. entry. This won't prevent issues with NFS should the policy pick a
  354. different file or files move out of band but will present the same
  355. inode for underlying files that do too.
  356. * devino-hash32: 32bit version of devino-hash.
  357. * hybrid-hash: Performs `path-hash` on directories and `devino-hash`
  358. on other file types. Since directories can't have hard links the
  359. static value won't make a difference and the files will get values
  360. useful for finding duplicates. Probably the best to use if not using
  361. NFS. As such it is the default.
  362. * hybrid-hash32: 32bit version of hybrid-hash.
  363. 32bit versions are provided as there is some software which does not
  364. handle 64bit inodes well.
  365. While there is a risk of hash collision in tests of a couple million
  366. entries there were zero collisions. Unlike a typical filesystem FUSE
  367. filesystems can reuse inodes and not refer to the same entry. The
  368. internal identifier used to reference a file in FUSE is different from
  369. the inode value presented. The former is the `nodeid` and is actually
  370. a tuple of 2 64bit values: `nodeid` and `generation`. This tuple is
  371. not client facing. The inode that is presented to the client is passed
  372. through the kernel uninterpreted.
  373. From FUSE docs for `use_ino`:
  374. ```
  375. Honor the st_ino field in the functions getattr() and
  376. fill_dir(). This value is used to fill in the st_ino field
  377. in the stat(2), lstat(2), fstat(2) functions and the d_ino
  378. field in the readdir(2) function. The filesystem does not
  379. have to guarantee uniqueness, however some applications
  380. rely on this value being unique for the whole filesystem.
  381. Note that this does *not* affect the inode that libfuse
  382. and the kernel use internally (also called the "nodeid").
  383. ```
  384. As of version 2.35.0 the `use_ino` option has been removed. mergerfs
  385. should always be managing inode values.
  386. ### pin-threads
  387. Simple strategies for pinning read and/or process threads. If process
  388. threads are not enabled than the strategy simply works on the read
  389. threads. Invalid values are ignored.
  390. * R1L: All read threads pinned to a single logical CPU.
  391. * R1P: All read threads pinned to a single physical CPU.
  392. * RP1L: All read and process threads pinned to a single logical CPU.
  393. * RP1P: All read and process threads pinned to a single physical CPU.
  394. * R1LP1L: All read threads pinned to a single logical CPU, all process
  395. threads pinned to a (if possible) different logical CPU.
  396. * R1PP1P: All read threads pinned to a single physical CPU, all
  397. process threads pinned to a (if possible) different logical CPU.
  398. * RPSL: All read and process threads are spread across all logical CPUs.
  399. * RPSP: All read and process threads are spread across all physical CPUs.
  400. * R1PPSP: All read threads are pinned to a single physical CPU while
  401. process threads are spread across all other phsycial CPUs.
  402. ### fuse_msg_size
  403. FUSE applications communicate with the kernel over a special character
  404. device: `/dev/fuse`. A large portion of the overhead associated with
  405. FUSE is the cost of going back and forth from user space and kernel
  406. space over that device. Generally speaking the fewer trips needed the
  407. better the performance will be. Reducing the number of trips can be
  408. done a number of ways. Kernel level caching and increasing message
  409. sizes being two significant ones. When it comes to reads and writes if
  410. the message size is doubled the number of trips are approximately
  411. halved.
  412. In Linux 4.20 a new feature was added allowing the negotiation of the
  413. max message size. Since the size is in multiples of
  414. [pages](https://en.wikipedia.org/wiki/Page_(computer_memory)) the
  415. feature is called `max_pages`. There is a maximum `max_pages` value of
  416. 256 (1MiB) and minimum of 1 (4KiB). The default used by Linux >=4.20,
  417. and hardcoded value used before 4.20, is 32 (128KiB). In mergerfs its
  418. referred to as `fuse_msg_size` to make it clear what it impacts and
  419. provide some abstraction.
  420. Since there should be no downsides to increasing `fuse_msg_size` /
  421. `max_pages`, outside a minor bump in RAM usage due to larger message
  422. buffers, mergerfs defaults the value to 256. On kernels before 4.20
  423. the value has no effect. The reason the value is configurable is to
  424. enable experimentation and benchmarking. See the BENCHMARKING section
  425. for examples.
  426. ### follow-symlinks
  427. This feature, when enabled, will cause symlinks to be interpreted by
  428. mergerfs as their target (depending on the mode).
  429. When there is a getattr/stat request for a file mergerfs will check if
  430. the file is a symlink and depending on the `follow-symlinks` setting
  431. will replace the information about the symlink with that of that which
  432. it points to.
  433. When unlink'ing or rmdir'ing the followed symlink it will remove the
  434. symlink itself and not that which it points to.
  435. * never: Behave as normal. Symlinks are treated as such.
  436. * directory: Resolve symlinks only which point to directories.
  437. * regular: Resolve symlinks only which point to regular files.
  438. * all: Resolve all symlinks to that which they point to.
  439. Symlinks which do not point to anything are left as is.
  440. WARNING: This feature works but there might be edge cases yet
  441. found. If you find any odd behaviors please file a ticket on
  442. [github](https://github.com/trapexit/mergerfs/issues).
  443. ### link-exdev
  444. If using path preservation and a `link` fails with EXDEV make a call
  445. to `symlink` where the `target` is the `oldlink` and the `linkpath` is
  446. the `newpath`. The `target` value is determined by the value of
  447. `link-exdev`.
  448. * passthrough: Return EXDEV as normal.
  449. * rel-symlink: A relative path from the `newpath`.
  450. * abs-base-symlink: A absolute value using the underlying branch.
  451. * abs-pool-symlink: A absolute value using the mergerfs mount point.
  452. NOTE: It is possible that some applications check the file they
  453. link. In those cases it is possible it will error or complain.
  454. ### rename-exdev
  455. If using path preservation and a `rename` fails with EXDEV:
  456. 1. Move file from **/branch/a/b/c** to **/branch/.mergerfs_rename_exdev/a/b/c**.
  457. 2. symlink the rename's `newpath` to the moved file.
  458. The `target` value is determined by the value of `rename-exdev`.
  459. * passthrough: Return EXDEV as normal.
  460. * rel-symlink: A relative path from the `newpath`.
  461. * abs-symlink: A absolute value using the mergerfs mount point.
  462. NOTE: It is possible that some applications check the file they
  463. rename. In those cases it is possible it will error or complain.
  464. NOTE: The reason `abs-symlink` is not split into two like `link-exdev`
  465. is due to the complexities in managing absolute base symlinks when
  466. multiple `oldpaths` exist.
  467. ### symlinkify
  468. Due to the levels of indirection introduced by mergerfs and the
  469. underlying technology FUSE there can be varying levels of performance
  470. degradation. This feature will turn non-directories which are not
  471. writable into symlinks to the original file found by the `readlink`
  472. policy after the mtime and ctime are older than the timeout.
  473. **WARNING:** The current implementation has a known issue in which if
  474. the file is open and being used when the file is converted to a
  475. symlink then the application which has that file open will receive an
  476. error when using it. This is unlikely to occur in practice but is
  477. something to keep in mind.
  478. **WARNING:** Some backup solutions, such as CrashPlan, do not backup
  479. the target of a symlink. If using this feature it will be necessary to
  480. point any backup software to the original filesystems or configure the
  481. software to follow symlinks if such an option is available.
  482. Alternatively create two mounts. One for backup and one for general
  483. consumption.
  484. ### nullrw
  485. Due to how FUSE works there is an overhead to all requests made to a
  486. FUSE filesystem that wouldn't exist for an in kernel one. Meaning that
  487. even a simple passthrough will have some slowdown. However, generally
  488. the overhead is minimal in comparison to the cost of the underlying
  489. I/O. By disabling the underlying I/O we can test the theoretical
  490. performance boundaries.
  491. By enabling `nullrw` mergerfs will work as it always does **except**
  492. that all reads and writes will be no-ops. A write will succeed (the
  493. size of the write will be returned as if it were successful) but
  494. mergerfs does nothing with the data it was given. Similarly a read
  495. will return the size requested but won't touch the buffer.
  496. See the BENCHMARKING section for suggestions on how to test.
  497. ### xattr
  498. Runtime extended attribute support can be managed via the `xattr`
  499. option. By default it will passthrough any xattr calls. Given xattr
  500. support is rarely used and can have significant performance
  501. implications mergerfs allows it to be disabled at runtime. The
  502. performance problems mostly comes when file caching is enabled. The
  503. kernel will send a `getxattr` for `security.capability` *before every
  504. single write*. It doesn't cache the responses to any `getxattr`. This
  505. might be addressed in the future but for now mergerfs can really only
  506. offer the following workarounds.
  507. `noattr` will cause mergerfs to short circuit all xattr calls and
  508. return ENOATTR where appropriate. mergerfs still gets all the requests
  509. but they will not be forwarded on to the underlying filesystems. The
  510. runtime control will still function in this mode.
  511. `nosys` will cause mergerfs to return ENOSYS for any xattr call. The
  512. difference with `noattr` is that the kernel will cache this fact and
  513. itself short circuit future calls. This is more efficient than
  514. `noattr` but will cause mergerfs' runtime control via the hidden file
  515. to stop working.
  516. ### nfsopenhack
  517. NFS is not fully POSIX compliant and historically certain behaviors,
  518. such as opening files with O_EXCL, are not or not well supported. When
  519. mergerfs (or any FUSE filesystem) is exported over NFS some of these
  520. issues come up due to how NFS and FUSE interact.
  521. This hack addresses the issue where the creation of a file with a
  522. read-only mode but with a read/write or write only flag. Normally this
  523. is perfectly valid but NFS chops the one open call into multiple
  524. calls. Exactly how it is translated depends on the configuration and
  525. versions of the NFS server and clients but it results in a permission
  526. error because a normal user is not allowed to open a read-only file as
  527. writable.
  528. Even though it's a more niche situation this hack breaks normal
  529. security and behavior and as such is `off` by default. If set to `git`
  530. it will only perform the hack when the path in question includes
  531. `/.git/`. `all` will result it applying anytime a readonly file which
  532. is empty is opened for writing.
  533. # FUNCTIONS, CATEGORIES and POLICIES
  534. The POSIX filesystem API is made up of a number of
  535. functions. **creat**, **stat**, **chown**, etc. For ease of
  536. configuration in mergerfs most of the core functions are grouped into
  537. 3 categories: **action**, **create**, and **search**. These functions
  538. and categories can be assigned a policy which dictates which branch is
  539. chosen when performing that function.
  540. Some functions, listed in the category `N/A` below, can not be
  541. assigned the normal policies. These functions work with file handles,
  542. rather than file paths, which were created by `open` or `create`. That
  543. said many times the current FUSE kernel driver will not always provide
  544. the file handle when a client calls `fgetattr`, `fchown`, `fchmod`,
  545. `futimens`, `ftruncate`, etc. This means it will call the regular,
  546. path based, versions. `readdir` has no real need for a policy given
  547. the purpose is merely to return a list of entries in a
  548. directory. `statfs`'s behavior can be modified via other options.
  549. When using policies which are based on a branch's available space the
  550. base path provided is used. Not the full path to the file in
  551. question. Meaning that mounts in the branch won't be considered in the
  552. space calculations. The reason is that it doesn't really work for
  553. non-path preserving policies and can lead to non-obvious behaviors.
  554. NOTE: While any policy can be assigned to a function or category
  555. though some may not be very useful in practice. For instance: **rand**
  556. (random) may be useful for file creation (create) but could lead to
  557. very odd behavior if used for `chmod` if there were more than one copy
  558. of the file.
  559. ### Functions and their Category classifications
  560. | Category | FUSE Functions |
  561. |----------|-------------------------------------------------------------------------------------|
  562. | action | chmod, chown, link, removexattr, rename, rmdir, setxattr, truncate, unlink, utimens |
  563. | create | create, mkdir, mknod, symlink |
  564. | search | access, getattr, getxattr, ioctl (directories), listxattr, open, readlink |
  565. | N/A | fchmod, fchown, futimens, ftruncate, fallocate, fgetattr, fsync, ioctl (files), read, readdir, release, statfs, write, copy_file_range |
  566. In cases where something may be searched for (such as a path to clone)
  567. **getattr** will usually be used.
  568. ### Policies
  569. A policy is the algorithm used to choose a branch or branches for a
  570. function to work on. Think of them as ways to filter and sort
  571. branches.
  572. Any function in the `create` category will clone the relative path if
  573. needed. Some other functions (`rename`,`link`,`ioctl`) have special
  574. requirements or behaviors which you can read more about below.
  575. #### Filtering
  576. Policies basically search branches and create a list of files / paths
  577. for functions to work on. The policy is responsible for filtering and
  578. sorting the branches. Filters include **minfreespace**, whether or not
  579. a branch is mounted read-only, and the branch tagging
  580. (RO,NC,RW). These filters are applied across all policies unless
  581. otherwise noted.
  582. * No **search** function policies filter.
  583. * All **action** function policies filter out branches which are
  584. mounted **read-only** or tagged as **RO (read-only)**.
  585. * All **create** function policies filter out branches which are
  586. mounted **read-only**, tagged **RO (read-only)** or **NC (no
  587. create)**, or has available space less than `minfreespace`.
  588. Policies may have their own additional filtering such as those that
  589. require existing paths to be present.
  590. If all branches are filtered an error will be returned. Typically
  591. **EROFS** (read-only filesystem) or **ENOSPC** (no space left on
  592. device) depending on the most recent reason for filtering a
  593. branch. **ENOENT** will be returned if no eligible branch is found.
  594. #### Path Preservation
  595. Policies, as described below, are of two basic classifications. `path
  596. preserving` and `non-path preserving`.
  597. All policies which start with `ep` (**epff**, **eplfs**, **eplus**,
  598. **epmfs**, **eprand**) are `path preserving`. `ep` stands for
  599. `existing path`.
  600. A path preserving policy will only consider branches where the relative
  601. path being accessed already exists.
  602. When using non-path preserving policies paths will be cloned to target
  603. branches as necessary.
  604. With the `msp` or `most shared path` policies they are defined as
  605. `path preserving` for the purpose of controlling `link` and `rename`'s
  606. behaviors since `ignorepponrename` is available to disable that
  607. behavior.
  608. #### Policy descriptions
  609. A policy's behavior differs, as mentioned above, based on the function
  610. it is used with. Sometimes it really might not make sense to even
  611. offer certain policies because they are literally the same as others
  612. but it makes things a bit more uniform.
  613. | Policy | Description |
  614. |------------------|------------------------------------------------------------|
  615. | all | Search: For **mkdir**, **mknod**, and **symlink** it will apply to all branches. **create** works like **ff**. |
  616. | epall (existing path, all) | For **mkdir**, **mknod**, and **symlink** it will apply to all found. **create** works like **epff** (but more expensive because it doesn't stop after finding a valid branch). |
  617. | epff (existing path, first found) | Given the order of the branches, as defined at mount time or configured at runtime, act on the first one found where the relative path exists. |
  618. | eplfs (existing path, least free space) | Of all the branches on which the relative path exists choose the branch with the least free space. |
  619. | eplus (existing path, least used space) | Of all the branches on which the relative path exists choose the branch with the least used space. |
  620. | epmfs (existing path, most free space) | Of all the branches on which the relative path exists choose the branch with the most free space. |
  621. | eppfrd (existing path, percentage free random distribution) | Like **pfrd** but limited to existing paths. |
  622. | eprand (existing path, random) | Calls **epall** and then randomizes. Returns 1. |
  623. | ff (first found) | Given the order of the branches, as defined at mount time or configured at runtime, act on the first one found. |
  624. | lfs (least free space) | Pick the branch with the least available free space. |
  625. | lus (least used space) | Pick the branch with the least used space. |
  626. | mfs (most free space) | Pick the branch with the most available free space. |
  627. | msplfs (most shared path, least free space) | Like **eplfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
  628. | msplus (most shared path, least used space) | Like **eplus** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
  629. | mspmfs (most shared path, most free space) | Like **epmfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
  630. | msppfrd (most shared path, percentage free random distribution) | Like **eppfrd** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
  631. | newest | Pick the file / directory with the largest mtime. |
  632. | pfrd (percentage free random distribution) | Chooses a branch at random with the likelihood of selection based on a branch's available space relative to the total. |
  633. | rand (random) | Calls **all** and then randomizes. Returns 1 branch. |
  634. **NOTE:** If you are using an underlying filesystem that reserves
  635. blocks such as ext2, ext3, or ext4 be aware that mergerfs respects the
  636. reservation by using `f_bavail` (number of free blocks for
  637. unprivileged users) rather than `f_bfree` (number of free blocks) in
  638. policy calculations. **df** does NOT use `f_bavail`, it uses
  639. `f_bfree`, so direct comparisons between **df** output and mergerfs'
  640. policies is not appropriate.
  641. #### Defaults
  642. | Category | Policy |
  643. |----------|--------|
  644. | action | epall |
  645. | create | epmfs |
  646. | search | ff |
  647. #### ioctl
  648. When `ioctl` is used with an open file then it will use the file
  649. handle which was created at the original `open` call. However, when
  650. using `ioctl` with a directory mergerfs will use the `open` policy to
  651. find the directory to act on.
  652. #### rename & link ####
  653. **NOTE:** If you're receiving errors from software when files are
  654. moved / renamed / linked then you should consider changing the create
  655. policy to one which is **not** path preserving, enabling
  656. `ignorepponrename`, or contacting the author of the offending software
  657. and requesting that `EXDEV` (cross device / improper link) be properly
  658. handled.
  659. `rename` and `link` are tricky functions in a union
  660. filesystem. `rename` only works within a single filesystem or
  661. device. If a rename can't be done atomically due to the source and
  662. destination paths existing on different mount points it will return
  663. **-1** with **errno = EXDEV** (cross device / improper link). So if a
  664. `rename`'s source and target are on different filesystems within the pool
  665. it creates an issue.
  666. Originally mergerfs would return EXDEV whenever a rename was requested
  667. which was cross directory in any way. This made the code simple and
  668. was technically compliant with POSIX requirements. However, many
  669. applications fail to handle EXDEV at all and treat it as a normal
  670. error or otherwise handle it poorly. Such apps include: gvfsd-fuse
  671. v1.20.3 and prior, Finder / CIFS/SMB client in Apple OSX 10.9+,
  672. NZBGet, Samba's recycling bin feature.
  673. As a result a compromise was made in order to get most software to
  674. work while still obeying mergerfs' policies. Below is the basic logic.
  675. * If using a **create** policy which tries to preserve directory paths (epff,eplfs,eplus,epmfs)
  676. * Using the **rename** policy get the list of files to rename
  677. * For each file attempt rename:
  678. * If failure with ENOENT (no such file or directory) run **create** policy
  679. * If create policy returns the same branch as currently evaluating then clone the path
  680. * Re-attempt rename
  681. * If **any** of the renames succeed the higher level rename is considered a success
  682. * If **no** renames succeed the first error encountered will be returned
  683. * On success:
  684. * Remove the target from all branches with no source file
  685. * Remove the source from all branches which failed to rename
  686. * If using a **create** policy which does **not** try to preserve directory paths
  687. * Using the **rename** policy get the list of files to rename
  688. * Using the **getattr** policy get the target path
  689. * For each file attempt rename:
  690. * If the source branch != target branch:
  691. * Clone target path from target branch to source branch
  692. * Rename
  693. * If **any** of the renames succeed the higher level rename is considered a success
  694. * If **no** renames succeed the first error encountered will be returned
  695. * On success:
  696. * Remove the target from all branches with no source file
  697. * Remove the source from all branches which failed to rename
  698. The the removals are subject to normal entitlement checks.
  699. The above behavior will help minimize the likelihood of EXDEV being
  700. returned but it will still be possible.
  701. **link** uses the same strategy but without the removals.
  702. #### readdir ####
  703. [readdir](http://linux.die.net/man/3/readdir) is different from all
  704. other filesystem functions. While it could have its own set of
  705. policies to tweak its behavior at this time it provides a simple union
  706. of files and directories found. Remember that any action or
  707. information queried about these files and directories come from the
  708. respective function. For instance: an **ls** is a **readdir** and for
  709. each file/directory returned **getattr** is called. Meaning the policy
  710. of **getattr** is responsible for choosing the file/directory which is
  711. the source of the metadata you see in an **ls**.
  712. #### statfs / statvfs ####
  713. [statvfs](http://linux.die.net/man/2/statvfs) normalizes the source
  714. filesystems based on the fragment size and sums the number of adjusted
  715. blocks and inodes. This means you will see the combined space of all
  716. sources. Total, used, and free. The sources however are dedupped based
  717. on the filesystem so multiple sources on the same drive will not result in
  718. double counting its space. Other filesystems mounted further down the tree
  719. of the branch will not be included when checking the mount's stats.
  720. The options `statfs` and `statfs_ignore` can be used to modify
  721. `statfs` behavior.
  722. # ERROR HANDLING
  723. POSIX filesystem functions offer a single return code meaning that
  724. there is some complication regarding the handling of multiple branches
  725. as mergerfs does. It tries to handle errors in a way that would
  726. generally return meaningful values for that particular function.
  727. ### chmod, chown, removexattr, setxattr, truncate, utimens
  728. 1) if no error: return 0 (success)
  729. 2) if no successes: return first error
  730. 3) if one of the files acted on was the same as the related search function: return its value
  731. 4) return 0 (success)
  732. While doing this increases the complexity and cost of error handling,
  733. particularly step 3, this provides probably the most reasonable return
  734. value.
  735. ### unlink, rmdir
  736. 1) if no errors: return 0 (success)
  737. 2) return first error
  738. Older version of mergerfs would return success if any success occurred
  739. but for unlink and rmdir there are downstream assumptions that, while
  740. not impossible to occur, can confuse some software.
  741. ### others
  742. For search functions there is always a single thing acted on and as
  743. such whatever return value that comes from the single function call is
  744. returned.
  745. For create functions `mkdir`, `mknod`, and `symlink` which don't
  746. return a file descriptor and therefore can have `all` or `epall`
  747. policies it will return success if any of the calls succeed and an
  748. error otherwise.
  749. # INSTALL
  750. https://github.com/trapexit/mergerfs/releases
  751. If your distribution's package manager includes mergerfs check if the
  752. version is up to date. If out of date it is recommended to use
  753. the latest release found on the release page. Details for common
  754. distros are below.
  755. #### Debian
  756. Most Debian installs are of a stable branch and therefore do not have
  757. the most up to date software. While mergerfs is available via `apt` it
  758. is suggested that uses install the most recent version available from
  759. the [releases page](https://github.com/trapexit/mergerfs/releases).
  760. #### prebuilt deb
  761. ```
  762. wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs_<ver>.debian-<rel>_<arch>.deb
  763. dpkg -i mergerfs_<ver>.debian-<rel>_<arch>.deb
  764. ```
  765. #### apt
  766. ```
  767. sudo apt install -y mergerfs
  768. ```
  769. #### Ubuntu
  770. Most Ubuntu installs are of a stable branch and therefore do not have
  771. the most up to date software. While mergerfs is available via `apt` it
  772. is suggested that uses install the most recent version available from
  773. the [releases page](https://github.com/trapexit/mergerfs/releases).
  774. #### prebuilt deb
  775. ```
  776. wget https://github.com/trapexit/mergerfs/releases/download/<version>/mergerfs_<ver>.ubuntu-<rel>_<arch>.deb
  777. dpkg -i mergerfs_<ver>.ubuntu-<rel>_<arch>.deb
  778. ```
  779. #### apt
  780. ```
  781. sudo apt install -y mergerfs
  782. ```
  783. #### Raspberry Pi OS
  784. Effectively the same as Debian or Ubuntu.
  785. #### Fedora
  786. ```
  787. wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-<ver>.fc<rel>.<arch>.rpm
  788. sudo rpm -i mergerfs-<ver>.fc<rel>.<arch>.rpm
  789. ```
  790. #### CentOS / Rocky
  791. ```
  792. wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-<ver>.el<rel>.<arch>.rpm
  793. sudo rpm -i mergerfs-<ver>.el<rel>.<arch>.rpm
  794. ```
  795. #### ArchLinux
  796. 1. Setup AUR
  797. 2. Install `mergerfs`
  798. #### Other
  799. Static binaries are provided for situations where native packages are
  800. unavailable.
  801. ```
  802. wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-static-linux_<arch>.tar.gz
  803. sudo tar xvf mergerfs-static-linux_<arch>.tar.gz -C /
  804. ```
  805. # BUILD
  806. **NOTE:** Prebuilt packages can be found at and recommended for most
  807. users: https://github.com/trapexit/mergerfs/releases
  808. **NOTE:** Only tagged releases are supported. `master` and other
  809. branches should be considered works in progress.
  810. First get the code from [github](https://github.com/trapexit/mergerfs).
  811. ```
  812. $ git clone https://github.com/trapexit/mergerfs.git
  813. $ # or
  814. $ wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-<ver>.tar.gz
  815. ```
  816. #### Debian / Ubuntu
  817. ```
  818. $ cd mergerfs
  819. $ sudo tools/install-build-pkgs
  820. $ make deb
  821. $ sudo dpkg -i ../mergerfs_<version>_<arch>.deb
  822. ```
  823. #### RHEL / CentOS / Rocky / Fedora
  824. ```
  825. $ su -
  826. # cd mergerfs
  827. # tools/install-build-pkgs
  828. # make rpm
  829. # rpm -i rpmbuild/RPMS/<arch>/mergerfs-<version>.<arch>.rpm
  830. ```
  831. #### Generic
  832. Have git, g++, make, python installed.
  833. ```
  834. $ cd mergerfs
  835. $ make
  836. $ sudo make install
  837. ```
  838. #### Build options
  839. ```
  840. $ make help
  841. usage: make
  842. make USE_XATTR=0 - build program without xattrs functionality
  843. make STATIC=1 - build static binary
  844. make LTO=1 - build with link time optimization
  845. ```
  846. # UPGRADE
  847. mergerfs can be upgraded live by mounting on top of the previous
  848. instance. Simply install the new version of mergerfs and follow the
  849. instructions below.
  850. Run mergerfs again or if using `/etc/fstab` call for it to mount
  851. again. Existing open files and such will continue to work fine though
  852. they won't see runtime changes since any such change would be the new
  853. mount. If you plan on changing settings with the new mount you should
  854. / could apply those before mounting the new version.
  855. ```
  856. $ sudo mount /mnt/mergerfs
  857. $ mount | grep mergerfs
  858. media on /mnt/mergerfs type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
  859. media on /mnt/mergerfs type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
  860. ```
  861. A problem with this approach is that the underlying instance will
  862. continue to run even if the software using it stop or are
  863. restarted. To work around this you can use a "lazy umount". Before
  864. mounting over top the mount point with the new instance of mergerfs
  865. issue: `umount -l <mergerfs_mountpoint>`. Or you can let mergerfs do
  866. it by setting the option `lazy-umount-mountpoint=true`.
  867. # RUNTIME CONFIG
  868. #### .mergerfs pseudo file ####
  869. ```
  870. <mountpoint>/.mergerfs
  871. ```
  872. There is a pseudo file available at the mount point which allows for
  873. the runtime modification of certain **mergerfs** options. The file
  874. will not show up in **readdir** but can be **stat**'ed and manipulated
  875. via [{list,get,set}xattrs](http://linux.die.net/man/2/listxattr)
  876. calls.
  877. Any changes made at runtime are **not** persisted. If you wish for
  878. values to persist they must be included as options wherever you
  879. configure the mounting of mergerfs (/etc/fstab).
  880. ##### Keys #####
  881. Use `getfattr -d /mountpoint/.mergerfs` or `xattr -l
  882. /mountpoint/.mergerfs` to see all supported keys. Some are
  883. informational and therefore read-only. `setxattr` will return EINVAL
  884. (invalid argument) on read-only keys.
  885. ##### Values #####
  886. Same as the command line.
  887. ###### user.mergerfs.branches ######
  888. Used to query or modify the list of branches. When modifying there are
  889. several shortcuts to easy manipulation of the list.
  890. | Value | Description |
  891. |--------------|-------------|
  892. | [list] | set |
  893. | +<[list] | prepend |
  894. | +>[list] | append |
  895. | -[list] | remove all values provided |
  896. | -< | remove first in list |
  897. | -> | remove last in list |
  898. `xattr -w user.mergerfs.branches +</mnt/drive3 /mnt/pool/.mergerfs`
  899. The `=NC`, `=RO`, `=RW` syntax works just as on the command line.
  900. ##### Example #####
  901. ```
  902. [trapexit:/mnt/mergerfs] $ getfattr -d .mergerfs
  903. user.mergerfs.branches="/mnt/a=RW:/mnt/b=RW"
  904. user.mergerfs.minfreespace="4294967295"
  905. user.mergerfs.moveonenospc="false"
  906. ...
  907. [trapexit:/mnt/mergerfs] $ getfattr -n user.mergerfs.category.search .mergerfs
  908. user.mergerfs.category.search="ff"
  909. [trapexit:/mnt/mergerfs] $ setfattr -n user.mergerfs.category.search -v newest .mergerfs
  910. [trapexit:/mnt/mergerfs] $ getfattr -n user.mergerfs.category.search .mergerfs
  911. user.mergerfs.category.search="newest"
  912. ```
  913. #### file / directory xattrs ####
  914. While they won't show up when using `getfattr` **mergerfs** offers a
  915. number of special xattrs to query information about the files
  916. served. To access the values you will need to issue a
  917. [getxattr](http://linux.die.net/man/2/getxattr) for one of the
  918. following:
  919. * **user.mergerfs.basepath**: the base mount point for the file given the current getattr policy
  920. * **user.mergerfs.relpath**: the relative path of the file from the perspective of the mount point
  921. * **user.mergerfs.fullpath**: the full path of the original file given the getattr policy
  922. * **user.mergerfs.allpaths**: a NUL ('\0') separated list of full paths to all files found
  923. # TOOLING
  924. * https://github.com/trapexit/mergerfs-tools
  925. * mergerfs.ctl: A tool to make it easier to query and configure mergerfs at runtime
  926. * mergerfs.fsck: Provides permissions and ownership auditing and the ability to fix them
  927. * mergerfs.dedup: Will help identify and optionally remove duplicate files
  928. * mergerfs.dup: Ensure there are at least N copies of a file across the pool
  929. * mergerfs.balance: Rebalance files across filesystems by moving them from the most filled to the least filled
  930. * mergerfs.consolidate: move files within a single mergerfs directory to the filesystem with most free space
  931. * https://github.com/trapexit/scorch
  932. * scorch: A tool to help discover silent corruption of files and keep track of files
  933. * https://github.com/trapexit/bbf
  934. * bbf (bad block finder): a tool to scan for and 'fix' hard drive bad blocks and find the files using those blocks
  935. # CACHING
  936. #### page caching
  937. https://en.wikipedia.org/wiki/Page_cache
  938. * cache.files=off: Disables page caching. Underlying files cached,
  939. mergerfs files are not.
  940. * cache.files=partial: Enables page caching. Underlying files cached,
  941. mergerfs files cached while open.
  942. * cache.files=full: Enables page caching. Underlying files cached,
  943. mergerfs files cached across opens.
  944. * cache.files=auto-full: Enables page caching. Underlying files
  945. cached, mergerfs files cached across opens if mtime and size are
  946. unchanged since previous open.
  947. * cache.files=libfuse: follow traditional libfuse `direct_io`,
  948. `kernel_cache`, and `auto_cache` arguments.
  949. * cache.files=per-process: Enable page caching only for processes
  950. which 'comm' name matches one of the values defined in
  951. `cache.files.process-names`.
  952. FUSE, which mergerfs uses, offers a number of page caching
  953. modes. mergerfs tries to simplify their use via the `cache.files`
  954. option. It can and should replace usage of `direct_io`,
  955. `kernel_cache`, and `auto_cache`.
  956. Due to mergerfs using FUSE and therefore being a userland process
  957. proxying existing filesystems the kernel will double cache the content
  958. being read and written through mergerfs. Once from the underlying
  959. filesystem and once from mergerfs (it sees them as two separate
  960. entities). Using `cache.files=off` will keep the double caching from
  961. happening by disabling caching of mergerfs but this has the side
  962. effect that *all* read and write calls will be passed to mergerfs
  963. which may be slower than enabling caching, you lose shared `mmap`
  964. support which can affect apps such as rtorrent, and no read-ahead will
  965. take place. The kernel will still cache the underlying filesystem data
  966. but that only helps so much given mergerfs will still process all
  967. requests.
  968. If you do enable file page caching,
  969. `cache.files=partial|full|auto-full`, you should also enable
  970. `dropcacheonclose` which will cause mergerfs to instruct the kernel to
  971. flush the underlying file's page cache when the file is closed. This
  972. behavior is the same as the rsync fadvise / drop cache patch and Feh's
  973. nocache project.
  974. If most files are read once through and closed (like media) it is best
  975. to enable `dropcacheonclose` regardless of caching mode in order to
  976. minimize buffer bloat.
  977. It is difficult to balance memory usage, cache bloat & duplication,
  978. and performance. Ideally mergerfs would be able to disable caching for
  979. the files it reads/writes but allow page caching for itself. That
  980. would limit the FUSE overhead. However, there isn't a good way to
  981. achieve this. It would need to open all files with O_DIRECT which
  982. places limitations on the what underlying filesystems would be
  983. supported and complicates the code.
  984. kernel documentation: https://www.kernel.org/doc/Documentation/filesystems/fuse-io.txt
  985. #### entry & attribute caching
  986. Given the relatively high cost of FUSE due to the kernel <-> userspace
  987. round trips there are kernel side caches for file entries and
  988. attributes. The entry cache limits the `lookup` calls to mergerfs
  989. which ask if a file exists. The attribute cache limits the need to
  990. make `getattr` calls to mergerfs which provide file attributes (mode,
  991. size, type, etc.). As with the page cache these should not be used if
  992. the underlying filesystems are being manipulated at the same time as
  993. it could lead to odd behavior or data corruption. The options for
  994. setting these are `cache.entry` and `cache.negative_entry` for the
  995. entry cache and `cache.attr` for the attributes
  996. cache. `cache.negative_entry` refers to the timeout for negative
  997. responses to lookups (non-existent files).
  998. #### writeback caching
  999. When `cache.files` is enabled the default is for it to perform
  1000. writethrough caching. This behavior won't help improve performance as
  1001. each write still goes one for one through the filesystem. By enabling
  1002. the FUSE writeback cache small writes may be aggregated by the kernel
  1003. and then sent to mergerfs as one larger request. This can greatly
  1004. improve the throughput for apps which write to files
  1005. inefficiently. The amount the kernel can aggregate is limited by the
  1006. size of a FUSE message. Read the `fuse_msg_size` section for more
  1007. details.
  1008. There is a small side effect as a result of enabling writeback
  1009. caching. Underlying files won't ever be opened with O_APPEND or
  1010. O_WRONLY. The former because the kernel then manages append mode and
  1011. the latter because the kernel may request file data from mergerfs to
  1012. populate the write cache. The O_APPEND change means that if a file is
  1013. changed outside of mergerfs it could lead to corruption as the kernel
  1014. won't know the end of the file has changed. That said any time you use
  1015. caching you should keep from using the same file outside of mergerfs
  1016. at the same time.
  1017. Note that if an application is properly sizing writes then writeback
  1018. caching will have little or no effect. It will only help with writes
  1019. of sizes below the FUSE message size (128K on older kernels, 1M on
  1020. newer).
  1021. #### statfs caching
  1022. Of the syscalls used by mergerfs in policies the `statfs` / `statvfs`
  1023. call is perhaps the most expensive. It's used to find out the
  1024. available space of a filesystem and whether it is mounted
  1025. read-only. Depending on the setup and usage pattern these queries can
  1026. be relatively costly. When `cache.statfs` is enabled all calls to
  1027. `statfs` by a policy will be cached for the number of seconds its set
  1028. to.
  1029. Example: If the create policy is `mfs` and the timeout is 60 then for
  1030. that 60 seconds the same filesystem will be returned as the target for
  1031. creates because the available space won't be updated for that time.
  1032. #### symlink caching
  1033. As of version 4.20 Linux supports symlink caching. Significant
  1034. performance increases can be had in workloads which use a lot of
  1035. symlinks. Setting `cache.symlinks=true` will result in requesting
  1036. symlink caching from the kernel only if supported. As a result its
  1037. safe to enable it on systems prior to 4.20. That said it is disabled
  1038. by default for now. You can see if caching is enabled by querying the
  1039. xattr `user.mergerfs.cache.symlinks` but given it must be requested at
  1040. startup you can not change it at runtime.
  1041. #### readdir caching
  1042. As of version 4.20 Linux supports readdir caching. This can have a
  1043. significant impact on directory traversal. Especially when combined
  1044. with entry (`cache.entry`) and attribute (`cache.attr`)
  1045. caching. Setting `cache.readdir=true` will result in requesting
  1046. readdir caching from the kernel on each `opendir`. If the kernel
  1047. doesn't support readdir caching setting the option to `true` has no
  1048. effect. This option is configurable at runtime via xattr
  1049. `user.mergerfs.cache.readdir`.
  1050. #### tiered caching
  1051. Some storage technologies support what some call "tiered" caching. The
  1052. placing of usually smaller, faster storage as a transparent cache to
  1053. larger, slower storage. NVMe, SSD, Optane in front of traditional HDDs
  1054. for instance.
  1055. MergerFS does not natively support any sort of tiered caching. Most
  1056. users have no use for such a feature and its inclusion would
  1057. complicate the code. However, there are a few situations where a cache
  1058. filesystem could help with a typical mergerfs setup.
  1059. 1. Fast network, slow filesystems, many readers: You've a 10+Gbps network
  1060. with many readers and your regular filesystems can't keep up.
  1061. 2. Fast network, slow filesystems, small'ish bursty writes: You have a
  1062. 10+Gbps network and wish to transfer amounts of data less than your
  1063. cache filesystem but wish to do so quickly.
  1064. With #1 it's arguable if you should be using mergerfs at all. RAID
  1065. would probably be the better solution. If you're going to use mergerfs
  1066. there are other tactics that may help: spreading the data across
  1067. filesystems (see the mergerfs.dup tool) and setting `func.open=rand`,
  1068. using `symlinkify`, or using dm-cache or a similar technology to add
  1069. tiered cache to the underlying device.
  1070. With #2 one could use dm-cache as well but there is another solution
  1071. which requires only mergerfs and a cronjob.
  1072. 1. Create 2 mergerfs pools. One which includes just the slow devices
  1073. and one which has both the fast devices (SSD,NVME,etc.) and slow
  1074. devices.
  1075. 2. The 'cache' pool should have the cache filesystems listed first.
  1076. 3. The best `create` policies to use for the 'cache' pool would
  1077. probably be `ff`, `epff`, `lfs`, or `eplfs`. The latter two under
  1078. the assumption that the cache filesystem(s) are far smaller than the
  1079. backing filesystems. If using path preserving policies remember that
  1080. you'll need to manually create the core directories of those paths
  1081. you wish to be cached. Be sure the permissions are in sync. Use
  1082. `mergerfs.fsck` to check / correct them. You could also set the
  1083. slow filesystems mode to `NC` though that'd mean if the cache
  1084. filesystems fill you'd get "out of space" errors.
  1085. 4. Enable `moveonenospc` and set `minfreespace` appropriately. To make
  1086. sure there is enough room on the "slow" pool you might want to set
  1087. `minfreespace` to at least as large as the size of the largest
  1088. cache filesystem if not larger. This way in the worst case the
  1089. whole of the cache filesystem(s) can be moved to the other drives.
  1090. 5. Set your programs to use the cache pool.
  1091. 6. Save one of the below scripts or create you're own.
  1092. 7. Use `cron` (as root) to schedule the command at whatever frequency
  1093. is appropriate for your workflow.
  1094. ##### time based expiring
  1095. Move files from cache to backing pool based only on the last time the
  1096. file was accessed. Replace `-atime` with `-amin` if you want minutes
  1097. rather than days. May want to use the `fadvise` / `--drop-cache`
  1098. version of rsync or run rsync with the tool "nocache".
  1099. *NOTE:* The arguments to these scripts include the cache
  1100. **filesystem** itself. Not the pool with the cache filesystem. You
  1101. could have data loss if the source is the cache pool.
  1102. [mergerfs.time-based-mover](tools/mergerfs.time-based-mover?raw=1)
  1103. ##### percentage full expiring
  1104. Move the oldest file from the cache to the backing pool. Continue till
  1105. below percentage threshold.
  1106. *NOTE:* The arguments to these scripts include the cache
  1107. **filesystem** itself. Not the pool with the cache filesystem. You
  1108. could have data loss if the source is the cache pool.
  1109. [mergerfs.percent-full-mover](tools/mergerfs.percent-full-mover?raw=1)
  1110. # PERFORMANCE
  1111. mergerfs is at its core just a proxy and therefore its theoretical max
  1112. performance is that of the underlying devices. However, given it is a
  1113. FUSE filesystem working from userspace there is an increase in
  1114. overhead relative to kernel based solutions. That said the performance
  1115. can match the theoretical max but it depends greatly on the system's
  1116. configuration. Especially when adding network filesystems into the mix
  1117. there are many variables which can impact performance. Device speeds
  1118. and latency, network speeds and latency, general concurrency,
  1119. read/write sizes, etc. Unfortunately, given the number of variables it
  1120. has been difficult to find a single set of settings which provide
  1121. optimal performance. If you're having performance issues please look
  1122. over the suggestions below (including the benchmarking section.)
  1123. NOTE: be sure to read about these features before changing them to
  1124. understand what behaviors it may impact
  1125. * disable `security_capability` and/or `xattr`
  1126. * increase cache timeouts `cache.attr`, `cache.entry`, `cache.negative_entry`
  1127. * enable (or disable) page caching (`cache.files`)
  1128. * enable `cache.writeback`
  1129. * enable `cache.statfs`
  1130. * enable `cache.symlinks`
  1131. * enable `cache.readdir`
  1132. * change the number of worker threads
  1133. * disable `posix_acl`
  1134. * disable `async_read`
  1135. * test theoretical performance using `nullrw` or mounting a ram disk
  1136. * use `symlinkify` if your data is largely static and read-only
  1137. * use tiered cache devices
  1138. * use LVM and LVM cache to place a SSD in front of your HDDs
  1139. * increase readahead: `readahead=1024`
  1140. If you come across a setting that significantly impacts performance
  1141. please contact trapexit so he may investigate further. Please test
  1142. both against your normal setup, a singular branch, and with
  1143. `nullrw=true`
  1144. # BENCHMARKING
  1145. Filesystems are complicated. They do many things and many of those are
  1146. interconnected. Additionally, the OS, drivers, hardware, etc. all can
  1147. impact performance. Therefore, when benchmarking, it is **necessary**
  1148. that the test focus as narrowly as possible.
  1149. For most throughput is the key benchmark. To test throughput `dd` is
  1150. useful but **must** be used with the correct settings in order to
  1151. ensure the filesystem or device is actually being tested. The OS can
  1152. and will cache data. Without forcing synchronous reads and writes
  1153. and/or disabling caching the values returned will not be
  1154. representative of the device's true performance.
  1155. When benchmarking through mergerfs ensure you only use 1 branch to
  1156. remove any possibility of the policies complicating the
  1157. situation. Benchmark the underlying filesystem first and then mount
  1158. mergerfs over it and test again. If you're experience speeds below
  1159. your expectation you will need to narrow down precisely which
  1160. component is leading to the slowdown. Preferably test the following in
  1161. the order listed (but not combined).
  1162. 1. Enable `nullrw` mode with `nullrw=true`. This will effectively make
  1163. reads and writes no-ops. Removing the underlying device /
  1164. filesystem from the equation. This will give us the top theoretical
  1165. speeds.
  1166. 2. Mount mergerfs over `tmpfs`. `tmpfs` is a RAM disk. Extremely high
  1167. speed and very low latency. This is a more realistic best case
  1168. scenario. Example: `mount -t tmpfs -o size=2G tmpfs /tmp/tmpfs`
  1169. 3. Mount mergerfs over a local device. NVMe, SSD, HDD, etc. If you
  1170. have more than one I'd suggest testing each of them as drives
  1171. and/or controllers (their drivers) could impact performance.
  1172. 4. Finally, if you intend to use mergerfs with a network filesystem,
  1173. either as the source of data or to combine with another through
  1174. mergerfs, test each of those alone as above.
  1175. Once you find the component which has the performance issue you can do
  1176. further testing with different options to see if they impact
  1177. performance. For reads and writes the most relevant would be:
  1178. `cache.files`, `async_read`. Less likely but relevant when using NFS
  1179. or with certain filesystems would be `security_capability`, `xattr`,
  1180. and `posix_acl`. If you find a specific system, device, filesystem,
  1181. controller, etc. that performs poorly contact trapexit so he may
  1182. investigate further.
  1183. Sometimes the problem is really the application accessing or writing
  1184. data through mergerfs. Some software use small buffer sizes which can
  1185. lead to more requests and therefore greater overhead. You can test
  1186. this out yourself by replace `bs=1M` in the examples below with `ibs`
  1187. or `obs` and using a size of `512` instead of `1M`. In one example
  1188. test using `nullrw` the write speed dropped from 4.9GB/s to 69.7MB/s
  1189. when moving from `1M` to `512`. Similar results were had when testing
  1190. reads. Small writes overhead may be improved by leveraging a write
  1191. cache but in casual tests little gain was found. More tests will need
  1192. to be done before this feature would become available. If you have an
  1193. app that appears slow with mergerfs it could be due to this. Contact
  1194. trapexit so he may investigate further.
  1195. ### write benchmark
  1196. ```
  1197. $ dd if=/dev/zero of=/mnt/mergerfs/1GB.file bs=1M count=1024 oflag=nocache conv=fdatasync status=progress
  1198. ```
  1199. ### read benchmark
  1200. ```
  1201. $ dd if=/mnt/mergerfs/1GB.file of=/dev/null bs=1M count=1024 iflag=nocache conv=fdatasync status=progress
  1202. ```
  1203. ### other benchmarks
  1204. If you are attempting to benchmark other behaviors you must ensure you
  1205. clear kernel caches before runs. In fact it would be a good deal to
  1206. run before the read and write benchmarks as well just in case.
  1207. ```
  1208. sync
  1209. echo 3 | sudo tee /proc/sys/vm/drop_caches
  1210. ```
  1211. # TIPS / NOTES
  1212. * This document is literal and thorough. If a suspected feature isn't
  1213. mentioned it doesn't exist. If certain libfuse arguments aren't
  1214. listed they probably shouldn't be used.
  1215. * Ensure you're using the latest version.
  1216. * Run mergerfs as `root`. mergerfs is designed and intended to be run
  1217. as `root` and may exibit incorrect behavior if run otherwise..
  1218. * If you don't see some directories and files you expect, policies
  1219. seem to skip branches, you get strange permission errors, etc. be
  1220. sure the underlying filesystems' permissions are all the same. Use
  1221. `mergerfs.fsck` to audit the filesystem for out of sync permissions.
  1222. * If you still have permission issues be sure you are using POSIX ACL
  1223. compliant filesystems. mergerfs doesn't generally make exceptions
  1224. for FAT, NTFS, or other non-POSIX filesystem.
  1225. * Do **not** use `cache.files=off` if you expect applications (such as
  1226. rtorrent) to use [mmap](http://linux.die.net/man/2/mmap)
  1227. files. Shared mmap is not currently supported in FUSE w/ page
  1228. caching disabled. Enabling `dropcacheonclose` is recommended when
  1229. `cache.files=partial|full|auto-full`.
  1230. * [Kodi](http://kodi.tv), [Plex](http://plex.tv),
  1231. [Subsonic](http://subsonic.org), etc. can use directory
  1232. [mtime](http://linux.die.net/man/2/stat) to more efficiently
  1233. determine whether to scan for new content rather than simply
  1234. performing a full scan. If using the default **getattr** policy of
  1235. **ff** it's possible those programs will miss an update on account
  1236. of it returning the first directory found's **stat** info and it's a
  1237. later directory on another mount which had the **mtime** recently
  1238. updated. To fix this you will want to set
  1239. **func.getattr=newest**. Remember though that this is just
  1240. **stat**. If the file is later **open**'ed or **unlink**'ed and the
  1241. policy is different for those then a completely different file or
  1242. directory could be acted on.
  1243. * Some policies mixed with some functions may result in strange
  1244. behaviors. Not that some of these behaviors and race conditions
  1245. couldn't happen outside **mergerfs** but that they are far more
  1246. likely to occur on account of the attempt to merge together multiple
  1247. sources of data which could be out of sync due to the different
  1248. policies.
  1249. * For consistency its generally best to set **category** wide policies
  1250. rather than individual **func**'s. This will help limit the
  1251. confusion of tools such as
  1252. [rsync](http://linux.die.net/man/1/rsync). However, the flexibility
  1253. is there if needed.
  1254. # KNOWN ISSUES / BUGS
  1255. #### kernel issues & bugs
  1256. [https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs](https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs)
  1257. #### directory mtime is not being updated
  1258. Remember that the default policy for `getattr` is `ff`. The
  1259. information for the first directory found will be returned. If it
  1260. wasn't the directory which had been updated then it will appear
  1261. outdated.
  1262. The reason this is the default is because any other policy would be
  1263. more expensive and for many applications it is unnecessary. To always
  1264. return the directory with the most recent mtime or a faked value based
  1265. on all found would require a scan of all filesystems.
  1266. If you always want the directory information from the one with the
  1267. most recent mtime then use the `newest` policy for `getattr`.
  1268. #### 'mv /mnt/pool/foo /mnt/disk1/foo' removes 'foo'
  1269. This is not a bug.
  1270. Run in verbose mode to better understand what's happening:
  1271. ```
  1272. $ mv -v /mnt/pool/foo /mnt/disk1/foo
  1273. copied '/mnt/pool/foo' -> '/mnt/disk1/foo'
  1274. removed '/mnt/pool/foo'
  1275. $ ls /mnt/pool/foo
  1276. ls: cannot access '/mnt/pool/foo': No such file or directory
  1277. ```
  1278. `mv`, when working across devices, is copying the source to target and
  1279. then removing the source. Since the source **is** the target in this
  1280. case, depending on the unlink policy, it will remove the just copied
  1281. file and other files across the branches.
  1282. If you want to move files to one filesystem just copy them there and
  1283. use mergerfs.dedup to clean up the old paths or manually remove them
  1284. from the branches directly.
  1285. #### cached memory appears greater than it should be
  1286. Use `cache.files=off` and/or `dropcacheonclose=true`. See the section
  1287. on page caching.
  1288. #### NFS clients returning ESTALE / Stale file handle
  1289. NFS does not like out of band changes. That is especially true of
  1290. inode values.
  1291. Be sure to use the following options:
  1292. * noforget
  1293. * inodecalc=path-hash
  1294. #### rtorrent fails with ENODEV (No such device)
  1295. Be sure to set `cache.files=partial|full|auto-full|per-processe` or
  1296. turn off `direct_io`. rtorrent and some other applications use
  1297. [mmap](http://linux.die.net/man/2/mmap) to read and write to files and
  1298. offer no fallback to traditional methods. FUSE does not currently
  1299. support mmap while using `direct_io`. There may be a performance
  1300. penalty on writes with `direct_io` off as well as the problem of
  1301. double caching but it's the only way to get such applications to
  1302. work. If the performance loss is too high for other apps you can mount
  1303. mergerfs twice. Once with `direct_io` enabled and one without it. Be
  1304. sure to set `dropcacheonclose=true` if not using `direct_io`.
  1305. #### Plex doesn't work with mergerfs
  1306. It does. If you're trying to put Plex's config / metadata / database
  1307. on mergerfs you can't set `cache.files=off` because Plex is using
  1308. sqlite3 with mmap enabled. Shared mmap is not supported by Linux's
  1309. FUSE implementation when page caching is disabled. To fix this place
  1310. the data elsewhere (preferable) or enable `cache.files` (with
  1311. `dropcacheonclose=true`). Sqlite3 does not need mmap but the developer
  1312. needs to fall back to standard IO if mmap fails.
  1313. This applies to other software: Radarr, Sonarr, Lidarr, Jellyfin, etc.
  1314. I would recommend reaching out to the developers of the software
  1315. you're having troubles with and asking them to add a fallback to
  1316. regular file IO when mmap is unavailable.
  1317. If the issue is that scanning doesn't seem to pick up media then be
  1318. sure to set `func.getattr=newest` though generally a full scan will
  1319. pick up all media anyway.
  1320. #### When a program tries to move or rename a file it fails
  1321. Please read the section above regarding [rename & link](#rename--link).
  1322. The problem is that many applications do not properly handle `EXDEV`
  1323. errors which `rename` and `link` may return even though they are
  1324. perfectly valid situations which do not indicate actual device,
  1325. filesystem, or OS errors. The error will only be returned by mergerfs
  1326. if using a path preserving policy as described in the policy section
  1327. above. If you do not care about path preservation simply change the
  1328. mergerfs policy to the non-path preserving version. For example: `-o
  1329. category.create=mfs` Ideally the offending software would be fixed and
  1330. it is recommended that if you run into this problem you contact the
  1331. software's author and request proper handling of `EXDEV` errors.
  1332. #### my 32bit software has problems
  1333. Some software have problems with 64bit inode values. The symptoms can
  1334. include EOVERFLOW errors when trying to list files. You can address
  1335. this by setting `inodecalc` to one of the 32bit based algos as
  1336. described in the relevant section.
  1337. #### Samba: Moving files / directories fails
  1338. Workaround: Copy the file/directory and then remove the original
  1339. rather than move.
  1340. This isn't an issue with Samba but some SMB clients. GVFS-fuse v1.20.3
  1341. and prior (found in Ubuntu 14.04 among others) failed to handle
  1342. certain error codes correctly. Particularly **STATUS_NOT_SAME_DEVICE**
  1343. which comes from the **EXDEV** which is returned by **rename** when
  1344. the call is crossing mount points. When a program gets an **EXDEV** it
  1345. needs to explicitly take an alternate action to accomplish its
  1346. goal. In the case of **mv** or similar it tries **rename** and on
  1347. **EXDEV** falls back to a manual copying of data between the two
  1348. locations and unlinking the source. In these older versions of
  1349. GVFS-fuse if it received **EXDEV** it would translate that into
  1350. **EIO**. This would cause **mv** or most any application attempting to
  1351. move files around on that SMB share to fail with a IO error.
  1352. [GVFS-fuse v1.22.0](https://bugzilla.gnome.org/show_bug.cgi?id=734568)
  1353. and above fixed this issue but a large number of systems use the older
  1354. release. On Ubuntu the version can be checked by issuing `apt-cache
  1355. showpkg gvfs-fuse`. Most distros released in 2015 seem to have the
  1356. updated release and will work fine but older systems may
  1357. not. Upgrading gvfs-fuse or the distro in general will address the
  1358. problem.
  1359. In Apple's MacOSX 10.9 they replaced Samba (client and server) with
  1360. their own product. It appears their new client does not handle
  1361. **EXDEV** either and responds similar to older release of gvfs on
  1362. Linux.
  1363. #### Trashing files occasionally fails
  1364. This is the same issue as with Samba. `rename` returns `EXDEV` (in our
  1365. case that will really only happen with path preserving policies like
  1366. `epmfs`) and the software doesn't handle the situation well. This is
  1367. unfortunately a common failure of software which moves files
  1368. around. The standard indicates that an implementation `MAY` choose to
  1369. support non-user home directory trashing of files (which is a
  1370. `MUST`). The implementation `MAY` also support "top directory trashes"
  1371. which many probably do.
  1372. To create a `$topdir/.Trash` directory as defined in the standard use
  1373. the [mergerfs-tools](https://github.com/trapexit/mergerfs-tools) tool
  1374. `mergerfs.mktrash`.
  1375. #### Supplemental user groups
  1376. Due to the overhead of
  1377. [getgroups/setgroups](http://linux.die.net/man/2/setgroups) mergerfs
  1378. utilizes a cache. This cache is opportunistic and per thread. Each
  1379. thread will query the supplemental groups for a user when that
  1380. particular thread needs to change credentials and will keep that data
  1381. for the lifetime of the thread. This means that if a user is added to
  1382. a group it may not be picked up without the restart of
  1383. mergerfs. However, since the high level FUSE API's (at least the
  1384. standard version) thread pool dynamically grows and shrinks it's
  1385. possible that over time a thread will be killed and later a new thread
  1386. with no cache will start and query the new data.
  1387. The gid cache uses fixed storage to simplify the design and be
  1388. compatible with older systems which may not have C++11
  1389. compilers. There is enough storage for 256 users' supplemental
  1390. groups. Each user is allowed up to 32 supplemental groups. Linux >=
  1391. 2.6.3 allows up to 65535 groups per user but most other *nixs allow
  1392. far less. NFS allowing only 16. The system does handle overflow
  1393. gracefully. If the user has more than 32 supplemental groups only the
  1394. first 32 will be used. If more than 256 users are using the system
  1395. when an uncached user is found it will evict an existing user's cache
  1396. at random. So long as there aren't more than 256 active users this
  1397. should be fine. If either value is too low for your needs you will
  1398. have to modify `gidcache.hpp` to increase the values. Note that doing
  1399. so will increase the memory needed by each thread.
  1400. While not a bug some users have found when using containers that
  1401. supplemental groups defined inside the container don't work properly
  1402. with regard to permissions. This is expected as mergerfs lives outside
  1403. the container and therefore is querying the host's group
  1404. database. There might be a hack to work around this (make mergerfs
  1405. read the /etc/group file in the container) but it is not yet
  1406. implemented and would be limited to Linux and the /etc/group
  1407. DB. Preferably users would mount in the host group file into the
  1408. containers or use a standard shared user & groups technology like NIS
  1409. or LDAP.
  1410. # FAQ
  1411. #### How well does mergerfs scale? Is it "production ready?"
  1412. Users have reported running mergerfs on everything from a Raspberry Pi
  1413. to dual socket Xeon systems with >20 cores. I'm aware of at least a
  1414. few companies which use mergerfs in production. [Open Media
  1415. Vault](https://www.openmediavault.org) includes mergerfs as its sole
  1416. solution for pooling filesystems. The author of mergerfs had it
  1417. running for over 300 days managing 16+ devices with reasonably heavy
  1418. 24/7 read and write usage. Stopping only after the machine's power
  1419. supply died.
  1420. Most serious issues (crashes or data corruption) have been due to
  1421. [kernel
  1422. bugs](https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs). All
  1423. of which are fixed in stable releases.
  1424. #### Can mergerfs be used with filesystems which already have data / are in use?
  1425. Yes. MergerFS is a proxy and does **NOT** interfere with the normal
  1426. form or function of the filesystems / mounts / paths it manages.
  1427. MergerFS is **not** a traditional filesystem. MergerFS is **not**
  1428. RAID. It does **not** manipulate the data that passes through it. It
  1429. does **not** shard data across filesystems. It merely shards some
  1430. **behavior** and aggregates others.
  1431. #### Can mergerfs be removed without affecting the data?
  1432. See the previous question's answer.
  1433. #### What policies should I use?
  1434. Unless you're doing something more niche the average user is probably
  1435. best off using `mfs` for `category.create`. It will spread files out
  1436. across your branches based on available space. Use `mspmfs` if you
  1437. want to try to colocate the data a bit more. You may want to use `lus`
  1438. if you prefer a slightly different distribution of data if you have a
  1439. mix of smaller and larger filesystems. Generally though `mfs`, `lus`,
  1440. or even `rand` are good for the general use case. If you are starting
  1441. with an imbalanced pool you can use the tool **mergerfs.balance** to
  1442. redistribute files across the pool.
  1443. If you really wish to try to colocate files based on directory you can
  1444. set `func.create` to `epmfs` or similar and `func.mkdir` to `rand` or
  1445. `eprand` depending on if you just want to colocate generally or on
  1446. specific branches. Either way the *need* to colocate is rare. For
  1447. instance: if you wish to remove the device regularly and want the data
  1448. to predictably be on that device or if you don't use backup at all and
  1449. don't wish to replace that data piecemeal. In which case using path
  1450. preservation can help but will require some manual
  1451. attention. Colocating after the fact can be accomplished using the
  1452. **mergerfs.consolidate** tool. If you don't need strict colocation
  1453. which the `ep` policies provide then you can use the `msp` based
  1454. policies which will walk back the path till finding a branch that
  1455. works.
  1456. Ultimately there is no correct answer. It is a preference or based on
  1457. some particular need. mergerfs is very easy to test and experiment
  1458. with. I suggest creating a test setup and experimenting to get a sense
  1459. of what you want.
  1460. `epmfs` is the default `category.create` policy because `ep` policies
  1461. are not going to change the general layout of the branches. It won't
  1462. place files/dirs on branches that don't already have the relative
  1463. branch. So it keeps the system in a known state. It's much easier to
  1464. stop using `epmfs` or redistribute files around the filesystem than it
  1465. is to consolidate them back.
  1466. #### What settings should I use?
  1467. Depends on what features you want. Generally speaking there are no
  1468. "wrong" settings. All settings are performance or feature related. The
  1469. best bet is to read over the available options and choose what fits
  1470. your situation. If something isn't clear from the documentation please
  1471. reach out and the documentation will be improved.
  1472. That said, for the average person, the following should be fine:
  1473. `cache.files=off,dropcacheonclose=true,category.create=mfs`
  1474. #### Why are all my files ending up on 1 filesystem?!
  1475. Did you start with empty filesystems? Did you explicitly configure a
  1476. `category.create` policy? Are you using an `existing path` / `path
  1477. preserving` policy?
  1478. The default create policy is `epmfs`. That is a path preserving
  1479. algorithm. With such a policy for `mkdir` and `create` with a set of
  1480. empty filesystems it will select only 1 filesystem when the first
  1481. directory is created. Anything, files or directories, created in that
  1482. first directory will be placed on the same branch because it is
  1483. preserving paths.
  1484. This catches a lot of new users off guard but changing the default
  1485. would break the setup for many existing users. If you do not care
  1486. about path preservation and wish your files to be spread across all
  1487. your filesystems change to `mfs` or similar policy as described
  1488. above. If you do want path preservation you'll need to perform the
  1489. manual act of creating paths on the filesystems you want the data to
  1490. land on before transferring your data. Setting `func.mkdir=epall` can
  1491. simplify managing path preservation for `create`. Or use
  1492. `func.mkdir=rand` if you're interested in just grouping together
  1493. directory content by filesystem.
  1494. #### Do hardlinks work?
  1495. Yes. See also the option `inodecalc` for how inode values are
  1496. calculated.
  1497. What mergerfs does not do is fake hard links across branches. Read
  1498. the section "rename & link" for how it works.
  1499. Remember that hardlinks will NOT work across devices. That includes
  1500. between the original filesystem and a mergerfs pool, between two
  1501. separate pools of the same underlying filesystems, or bind mounts of
  1502. paths within the mergerfs pool. The latter is common when using Docker
  1503. or Podman. Multiple volumes (bind mounts) to the same underlying
  1504. filesystem are considered different devices. There is no way to link
  1505. between them. You should mount in the highest directory in the
  1506. mergerfs pool that includes all the paths you need if you want links
  1507. to work.
  1508. #### Can I use mergerfs without SnapRAID? SnapRAID without mergerfs?
  1509. Yes. They are completely unrelated pieces of software.
  1510. #### Can mergerfs run via Docker, Podman, Kubernetes, etc.
  1511. Yes. With Docker you'll need to include `--cap-add=SYS_ADMIN
  1512. --device=/dev/fuse --security-opt=apparmor:unconfined` or similar with
  1513. other container runtimes. You should also be running it as root or
  1514. given sufficient caps to change user and group identity as well as
  1515. have root like filesystem permissions.
  1516. Keep in mind that you **MUST** consider identity when using
  1517. containers. For example: supplemental groups will be picked up from
  1518. the container unless you properly manage users and groups by sharing
  1519. relevant /etc files or by using some other means to share identity
  1520. across containers. Similarly if you use "rootless" containers and user
  1521. namespaces to do uid/gid translations you **MUST** consider that while
  1522. managing shared files.
  1523. Also, as mentioned by [hotio](https://hotio.dev/containers/mergerfs),
  1524. with Docker you should probably be mounting with `bind-propagation`
  1525. set to `slave`.
  1526. #### Does mergerfs support CoW / copy-on-write / writes to read-only filesystems?
  1527. Not in the sense of a filesystem like BTRFS or ZFS nor in the
  1528. overlayfs or aufs sense. It does offer a
  1529. [cow-shell](http://manpages.ubuntu.com/manpages/bionic/man1/cow-shell.1.html)
  1530. like hard link breaking (copy to temp file then rename over original)
  1531. which can be useful when wanting to save space by hardlinking
  1532. duplicate files but wish to treat each name as if it were a unique and
  1533. separate file.
  1534. If you want to write to a read-only filesystem you should look at
  1535. overlayfs. You can always include the overlayfs mount into a mergerfs
  1536. pool.
  1537. #### Why can't I see my files / directories?
  1538. It's almost always a permissions issue. Unlike mhddfs and
  1539. unionfs-fuse, which runs as root and attempts to access content as
  1540. such, mergerfs always changes its credentials to that of the
  1541. caller. This means that if the user does not have access to a file or
  1542. directory than neither will mergerfs. However, because mergerfs is
  1543. creating a union of paths it may be able to read some files and
  1544. directories on one filesystem but not another resulting in an
  1545. incomplete set.
  1546. Whenever you run into a split permission issue (seeing some but not
  1547. all files) try using
  1548. [mergerfs.fsck](https://github.com/trapexit/mergerfs-tools) tool to
  1549. check for and fix the mismatch. If you aren't seeing anything at all
  1550. be sure that the basic permissions are correct. The user and group
  1551. values are correct and that directories have their executable bit
  1552. set. A common mistake by users new to Linux is to `chmod -R 644` when
  1553. they should have `chmod -R u=rwX,go=rX`.
  1554. If using a network filesystem such as NFS, SMB, CIFS (Samba) be sure
  1555. to pay close attention to anything regarding permissioning and
  1556. users. Root squashing and user translation for instance has bitten a
  1557. few mergerfs users. Some of these also affect the use of mergerfs from
  1558. container platforms such as Docker.
  1559. #### Why use FUSE? Why not a kernel based solution?
  1560. As with any solutions to a problem there are advantages and
  1561. disadvantages to each one.
  1562. A FUSE based solution has all the downsides of FUSE:
  1563. * Higher IO latency due to the trips in and out of kernel space
  1564. * Higher general overhead due to trips in and out of kernel space
  1565. * Double caching when using page caching
  1566. * Misc limitations due to FUSE's design
  1567. But FUSE also has a lot of upsides:
  1568. * Easier to offer a cross platform solution
  1569. * Easier forward and backward compatibility
  1570. * Easier updates for users
  1571. * Easier and faster release cadence
  1572. * Allows more flexibility in design and features
  1573. * Overall easier to write, secure, and maintain
  1574. * Much lower barrier to entry (getting code into the kernel takes a
  1575. lot of time and effort initially)
  1576. FUSE was chosen because of all the advantages listed above. The
  1577. negatives of FUSE do not outweigh the positives.
  1578. #### Is my OS's libfuse needed for mergerfs to work?
  1579. No. Normally `mount.fuse` is needed to get mergerfs (or any FUSE
  1580. filesystem to mount using the `mount` command but in vendoring the
  1581. libfuse library the `mount.fuse` app has been renamed to
  1582. `mount.mergerfs` meaning the filesystem type in `fstab` can simply be
  1583. `mergerfs`. That said there should be no harm in having it installed
  1584. and continuing to using `fuse.mergerfs` as the type in `/etc/fstab`.
  1585. If `mergerfs` doesn't work as a type it could be due to how the
  1586. `mount.mergerfs` tool was installed. Must be in `/sbin/` with proper
  1587. permissions.
  1588. #### Why was splice support removed?
  1589. After a lot of testing over the years splicing always appeared to be
  1590. at best provide equivalent performance and in cases worse
  1591. performance. Splice is not supported on other platforms forcing a
  1592. traditional read/write fallback to be provided. The splice code was
  1593. removed to simplify the codebase.
  1594. #### Why use mergerfs over mhddfs?
  1595. mhddfs is no longer maintained and has some known stability and
  1596. security issues (see below). MergerFS provides a superset of mhddfs'
  1597. features and should offer the same or maybe better performance.
  1598. Below is an example of mhddfs and mergerfs setup to work similarly.
  1599. `mhddfs -o mlimit=4G,allow_other /mnt/drive1,/mnt/drive2 /mnt/pool`
  1600. `mergerfs -o minfreespace=4G,category.create=ff /mnt/drive1:/mnt/drive2 /mnt/pool`
  1601. #### Why use mergerfs over aufs?
  1602. aufs is mostly abandoned and no longer available in many distros.
  1603. While aufs can offer better peak performance mergerfs provides more
  1604. configurability and is generally easier to use. mergerfs however does
  1605. not offer the overlay / copy-on-write (CoW) features which aufs and
  1606. overlayfs have.
  1607. #### Why use mergerfs over unionfs?
  1608. UnionFS is more like aufs than mergerfs in that it offers overlay /
  1609. CoW features. If you're just looking to create a union of filesystems
  1610. and want flexibility in file/directory placement then mergerfs offers
  1611. that whereas unionfs is more for overlaying RW filesystems over RO
  1612. ones.
  1613. #### Why use mergerfs over overlayfs?
  1614. Same reasons as with unionfs.
  1615. #### Why use mergerfs over LVM/ZFS/BTRFS/RAID0 drive concatenation / striping?
  1616. With simple JBOD / drive concatenation / stripping / RAID0 a single
  1617. drive failure will result in full pool failure. mergerfs performs a
  1618. similar function without the possibility of catastrophic failure and
  1619. the difficulties in recovery. Drives may fail, however, all other data
  1620. will continue to be accessible.
  1621. When combined with something like [SnapRaid](http://www.snapraid.it)
  1622. and/or an offsite backup solution you can have the flexibility of JBOD
  1623. without the single point of failure.
  1624. #### Why use mergerfs over ZFS?
  1625. MergerFS is not intended to be a replacement for ZFS. MergerFS is
  1626. intended to provide flexible pooling of arbitrary filesystems (local
  1627. or remote), of arbitrary sizes, and arbitrary filesystems. For `write
  1628. once, read many` usecases such as bulk media storage. Where data
  1629. integrity and backup is managed in other ways. In that situation ZFS
  1630. can introduce a number of costs and limitations as described
  1631. [here](http://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html),
  1632. [here](https://markmcb.com/2020/01/07/five-years-of-btrfs/), and
  1633. [here](https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSWhyNoRealReshaping).
  1634. #### Why use mergerfs over UnRAID?
  1635. UnRAID is a full OS and its storage layer, as I understand, is
  1636. proprietary and closed source. Users who have experience with both
  1637. have said they prefer the flexibility offered by mergerfs and for some
  1638. the fact it is free and open source is important.
  1639. There are a number of UnRAID users who use mergerfs as well though I'm
  1640. not entirely familiar with the use case.
  1641. #### Why use mergerfs over StableBit's DrivePool?
  1642. DrivePool works only on Windows so not as common an alternative as
  1643. other Linux solutions. If you want to use Windows then DrivePool is a
  1644. good option. Functionally the two projects work a bit
  1645. differently. DrivePool always writes to the filesystem with the most
  1646. free space and later rebalances. mergerfs does not offer rebalance but
  1647. chooses a branch at file/directory create time. DrivePool's
  1648. rebalancing can be done differently in any directory and has file
  1649. pattern matching to further customize the behavior. mergerfs, not
  1650. having rebalancing does not have these features, but similar features
  1651. are planned for mergerfs v3. DrivePool has builtin file duplication
  1652. which mergerfs does not natively support (but can be done via an
  1653. external script.)
  1654. There are a lot of misc differences between the two projects but most
  1655. features in DrivePool can be replicated with external tools in
  1656. combination with mergerfs.
  1657. Additionally DrivePool is a closed source commercial product vs
  1658. mergerfs a ISC licensed OSS project.
  1659. #### What should mergerfs NOT be used for?
  1660. * databases: Even if the database stored data in separate files
  1661. (mergerfs wouldn't offer much otherwise) the higher latency of the
  1662. indirection will kill performance. If it is a lightly used SQLITE
  1663. database then it may be fine but you'll need to test.
  1664. * VM images: For the same reasons as databases. VM images are accessed
  1665. very aggressively and mergerfs will introduce too much latency (if
  1666. it works at all).
  1667. * As replacement for RAID: mergerfs is just for pooling branches. If
  1668. you need that kind of device performance aggregation or high
  1669. availability you should stick with RAID.
  1670. #### Can filesystems be written to directly? Outside of mergerfs while pooled?
  1671. Yes, however it's not recommended to use the same file from within the
  1672. pool and from without at the same time (particularly
  1673. writing). Especially if using caching of any kind (cache.files,
  1674. cache.entry, cache.attr, cache.negative_entry, cache.symlinks,
  1675. cache.readdir, etc.) as there could be a conflict between cached data
  1676. and not.
  1677. #### Why do I get an "out of space" / "no space left on device" / ENOSPC error even though there appears to be lots of space available?
  1678. First make sure you've read the sections above about policies, path
  1679. preservation, branch filtering, and the options **minfreespace**,
  1680. **moveonenospc**, **statfs**, and **statfs_ignore**.
  1681. mergerfs is simply presenting a union of the content within multiple
  1682. branches. The reported free space is an aggregate of space available
  1683. within the pool (behavior modified by **statfs** and
  1684. **statfs_ignore**). It does not represent a contiguous space. In the
  1685. same way that read-only filesystems, those with quotas, or reserved
  1686. space report the full theoretical space available.
  1687. Due to path preservation, branch tagging, read-only status, and
  1688. **minfreespace** settings it is perfectly valid that `ENOSPC` / "out
  1689. of space" / "no space left on device" be returned. It is doing what
  1690. was asked of it: filtering possible branches due to those
  1691. settings. Only one error can be returned and if one of the reasons for
  1692. filtering a branch was **minfreespace** then it will be returned as
  1693. such. **moveonenospc** is only relevant to writing a file which is too
  1694. large for the filesystem it's currently on.
  1695. It is also possible that the filesystem selected has run out of
  1696. inodes. Use `df -i` to list the total and available inodes per
  1697. filesystem.
  1698. If you don't care about path preservation then simply change the
  1699. `create` policy to one which isn't. `mfs` is probably what most are
  1700. looking for. The reason it's not default is because it was originally
  1701. set to `epmfs` and changing it now would change people's setup. Such a
  1702. setting change will likely occur in mergerfs 3.
  1703. #### Why does the total available space in mergerfs not equal outside?
  1704. Are you using ext2/3/4? With reserve for root? mergerfs uses available
  1705. space for statfs calculations. If you've reserved space for root then
  1706. it won't show up.
  1707. You can remove the reserve by running: `tune2fs -m 0 <device>`
  1708. #### Can mergerfs mounts be exported over NFS?
  1709. Yes, however if you do anything which may changes files out of band
  1710. (including for example using the `newest` policy) it will result in
  1711. "stale file handle" errors unless properly setup.
  1712. Be sure to use the following options:
  1713. * noforget
  1714. * inodecalc=path-hash
  1715. #### Can mergerfs mounts be exported over Samba / SMB?
  1716. Yes. While some users have reported problems it appears to always be
  1717. related to how Samba is setup in relation to permissions.
  1718. #### Can mergerfs mounts be used over SSHFS?
  1719. Yes.
  1720. #### I notice massive slowdowns of writes when enabling cache.files.
  1721. When file caching is enabled in any form (`cache.files!=off` or
  1722. `direct_io=false`) it will issue `getxattr` requests for
  1723. `security.capability` prior to *every single write*. This will usually
  1724. result in a performance degradation, especially when using a network
  1725. filesystem (such as NFS or CIFS/SMB/Samba.) Unfortunately at this
  1726. moment the kernel is not caching the response.
  1727. To work around this situation mergerfs offers a few solutions.
  1728. 1. Set `security_capability=false`. It will short circuit any call and
  1729. return `ENOATTR`. This still means though that mergerfs will
  1730. receive the request before every write but at least it doesn't get
  1731. passed through to the underlying filesystem.
  1732. 2. Set `xattr=noattr`. Same as above but applies to *all* calls to
  1733. getxattr. Not just `security.capability`. This will not be cached
  1734. by the kernel either but mergerfs' runtime config system will still
  1735. function.
  1736. 3. Set `xattr=nosys`. Results in mergerfs returning `ENOSYS` which
  1737. *will* be cached by the kernel. No future xattr calls will be
  1738. forwarded to mergerfs. The downside is that also means the xattr
  1739. based config and query functionality won't work either.
  1740. 4. Disable file caching. If you aren't using applications which use
  1741. `mmap` it's probably simpler to just disable it all together. The
  1742. kernel won't send the requests when caching is disabled.
  1743. #### It's mentioned that there are some security issues with mhddfs. What are they? How does mergerfs address them?
  1744. [mhddfs](https://github.com/trapexit/mhddfs) manages running as
  1745. **root** by calling
  1746. [getuid()](https://github.com/trapexit/mhddfs/blob/cae96e6251dd91e2bdc24800b4a18a74044f6672/src/main.c#L319)
  1747. and if it returns **0** then it will
  1748. [chown](http://linux.die.net/man/1/chown) the file. Not only is that a
  1749. race condition but it doesn't handle other situations. Rather than
  1750. attempting to simulate POSIX ACL behavior the proper way to manage
  1751. this is to use [seteuid](http://linux.die.net/man/2/seteuid) and
  1752. [setegid](http://linux.die.net/man/2/setegid), in effect becoming the
  1753. user making the original call, and perform the action as them. This is
  1754. what mergerfs does and why mergerfs should always run as root.
  1755. In Linux setreuid syscalls apply only to the thread. GLIBC hides this
  1756. away by using realtime signals to inform all threads to change
  1757. credentials. Taking after **Samba**, mergerfs uses
  1758. **syscall(SYS_setreuid,...)** to set the callers credentials for that
  1759. thread only. Jumping back to **root** as necessary should escalated
  1760. privileges be needed (for instance: to clone paths between
  1761. filesystems).
  1762. For non-Linux systems mergerfs uses a read-write lock and changes
  1763. credentials only when necessary. If multiple threads are to be user X
  1764. then only the first one will need to change the processes
  1765. credentials. So long as the other threads need to be user X they will
  1766. take a readlock allowing multiple threads to share the
  1767. credentials. Once a request comes in to run as user Y that thread will
  1768. attempt a write lock and change to Y's credentials when it can. If the
  1769. ability to give writers priority is supported then that flag will be
  1770. used so threads trying to change credentials don't starve. This isn't
  1771. the best solution but should work reasonably well assuming there are
  1772. few users.
  1773. # SUPPORT
  1774. Filesystems are complex and difficult to debug. mergerfs, while being
  1775. just a proxy of sorts, can be difficult to debug given the large
  1776. number of possible settings it can have itself and the number of
  1777. environments it can run in. When reporting on a suspected issue
  1778. **please** include as much of the below information as possible
  1779. otherwise it will be difficult or impossible to diagnose. Also please
  1780. read the above documentation as it provides details on many previously
  1781. encountered questions/issues.
  1782. **Please make sure you are using the [latest
  1783. release](https://github.com/trapexit/mergerfs/releases) or have tried
  1784. it in comparison. Old versions, which are often included in distros
  1785. like Debian and Ubuntu, are not ever going to be updated and the issue
  1786. you are encountering may have been addressed already.**
  1787. **For commercial support or feature requests please [contact me
  1788. directly.](mailto:support@spawn.link)**
  1789. #### Information to include in bug reports
  1790. * [Information about the broader problem along with any attempted
  1791. solutions.](https://xyproblem.info)
  1792. * Solution already ruled out and why.
  1793. * Version of mergerfs: `mergerfs --version`
  1794. * mergerfs settings / arguments: from fstab, systemd unit, command
  1795. line, OMV plugin, etc.
  1796. * Version of the OS: `uname -a` and `lsb_release -a`
  1797. * List of branches, their filesystem types, sizes (before and after issue): `df -h`
  1798. * **All** information about the relevant paths and files: permissions, ownership, etc.
  1799. * **All** information about the client app making the requests: version, uid/gid
  1800. * Runtime environment:
  1801. * Is mergerfs running within a container?
  1802. * Are the client apps using mergerfs running in a container?
  1803. * A `strace` of the app having problems:
  1804. * `strace -fvTtt -s 256 -o /tmp/app.strace.txt <cmd>`
  1805. * A `strace` of mergerfs while the program is trying to do whatever it is failing to do:
  1806. * `strace -fvTtt -s 256 -p <mergerfsPID> -o /tmp/mergerfs.strace.txt`
  1807. * **Precise** directions on replicating the issue. Do not leave **anything** out.
  1808. * Try to recreate the problem in the simplest way using standard programs: `ln`, `mv`, `cp`, `ls`, `dd`, etc.
  1809. #### Contact / Issue submission
  1810. * github.com: https://github.com/trapexit/mergerfs/issues
  1811. * discord: https://discord.gg/MpAr69V
  1812. * reddit: https://www.reddit.com/r/mergerfs
  1813. #### Donations
  1814. https://github.com/trapexit/support
  1815. Development and support of a project like mergerfs requires a
  1816. significant amount of time and effort. The software is released under
  1817. the very liberal ISC license and is therefore free to use for personal
  1818. or commercial uses.
  1819. If you are a personal user and find mergerfs and its support valuable
  1820. and would like to support the project financially it would be very
  1821. much appreciated.
  1822. If you are using mergerfs commercially please consider sponsoring the
  1823. project to ensure it continues to be maintained and receive
  1824. updates. If custom features are needed feel free to [contact me
  1825. directly](mailto:support@spawn.link).
  1826. # LINKS
  1827. * https://spawn.link
  1828. * https://github.com/trapexit/mergerfs
  1829. * https://github.com/trapexit/mergerfs/wiki
  1830. * https://github.com/trapexit/mergerfs-tools
  1831. * https://github.com/trapexit/scorch
  1832. * https://github.com/trapexit/bbf