You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

2784 lines
116 KiB

11 years ago
9 months ago
9 months ago
9 months ago
4 years ago
9 months ago
9 months ago
5 years ago
5 years ago
5 years ago
3 years ago
5 years ago
  1. % mergerfs(1) mergerfs user manual
  2. # NAME
  3. mergerfs - a featureful union filesystem
  4. # SYNOPSIS
  5. mergerfs -o<options> <branches> <mountpoint>
  6. # DESCRIPTION
  7. **mergerfs** is a union filesystem geared towards simplifying storage
  8. and management of files across numerous commodity storage devices. It
  9. is similar to **mhddfs**, **unionfs**, and **aufs**.
  10. # FEATURES
  11. * Configurable behaviors / file placement
  12. * Ability to add or remove filesystems at will
  13. * Resistance to individual filesystem failure
  14. * Support for extended attributes (xattrs)
  15. * Support for file attributes (chattr)
  16. * Runtime configurable (via xattrs)
  17. * Works with heterogeneous filesystem types
  18. * Moving of file when filesystem runs out of space while writing
  19. * Ignore read-only filesystems when creating files
  20. * Turn read-only files into symlinks to underlying file
  21. * Hard link copy-on-write / CoW
  22. * Support for POSIX ACLs
  23. * Misc other things
  24. # HOW IT WORKS
  25. mergerfs logically merges multiple paths together. Think a union of
  26. sets. The file/s or directory/s acted on or presented through mergerfs
  27. are based on the policy chosen for that particular action. Read more
  28. about policies below.
  29. ```
  30. A + B = C
  31. /disk1 /disk2 /merged
  32. | | |
  33. +-- /dir1 +-- /dir1 +-- /dir1
  34. | | | | | |
  35. | +-- file1 | +-- file2 | +-- file1
  36. | | +-- file3 | +-- file2
  37. +-- /dir2 | | +-- file3
  38. | | +-- /dir3 |
  39. | +-- file4 | +-- /dir2
  40. | +-- file5 | |
  41. +-- file6 | +-- file4
  42. |
  43. +-- /dir3
  44. | |
  45. | +-- file5
  46. |
  47. +-- file6
  48. ```
  49. mergerfs does **not** support the copy-on-write (CoW) or whiteout
  50. behaviors found in **aufs** and **overlayfs**. You can **not** mount a
  51. read-only filesystem and write to it. However, mergerfs will ignore
  52. read-only filesystems when creating new files so you can mix
  53. read-write and read-only filesystems. It also does **not** split data
  54. across filesystems. It is not RAID0 / striping. It is simply a union of
  55. other filesystems.
  56. # TERMINOLOGY
  57. * branch: A base path used in the pool.
  58. * pool: The mergerfs mount. The union of the branches.
  59. * relative path: The path in the pool relative to the branch and mount.
  60. * function: A filesystem call (open, unlink, create, getattr, rmdir, etc.)
  61. * category: A collection of functions based on basic behavior (action, create, search).
  62. * policy: The algorithm used to select a file when performing a function.
  63. * path preservation: Aspect of some policies which includes checking the path for which a file would be created.
  64. # BASIC SETUP
  65. If you don't already know that you have a special use case then just
  66. start with one of the following option sets.
  67. #### You need `mmap` (used by rtorrent and many sqlite3 base software)
  68. `cache.files=auto-full,dropcacheonclose=true,category.create=mfs`
  69. or if you are on a Linux kernel >= 6.6.x mergerfs will enable a mode
  70. that allows shared mmap when `cache.files=off`. To be sure of the best
  71. performance between `cache.files=off` and `cache.files=auto-full`
  72. you'll need to do your own benchmarking but often `off` is faster.
  73. #### You don't need `mmap`
  74. `cache.files=off,dropcacheonclose=true,category.create=mfs`
  75. ### Command Line
  76. `mergerfs -o cache.files=auto-full,dropcacheonclose=true,category.create=mfs /mnt/hdd0:/mnt/hdd1 /media`
  77. ### /etc/fstab
  78. `/mnt/hdd0:/mnt/hdd1 /media mergerfs cache.files=auto-full,dropcacheonclose=true,category.create=mfs 0 0`
  79. ### systemd mount
  80. https://github.com/trapexit/mergerfs/wiki/systemd
  81. ```
  82. [Unit]
  83. Description=mergerfs service
  84. [Service]
  85. Type=simple
  86. KillMode=none
  87. ExecStart=/usr/bin/mergerfs \
  88. -f \
  89. -o cache.files=auto-full \
  90. -o dropcacheonclose=true \
  91. -o category.create=mfs \
  92. /mnt/hdd0:/mnt/hdd1 \
  93. /media
  94. ExecStop=/bin/fusermount -uz /media
  95. Restart=on-failure
  96. [Install]
  97. WantedBy=default.target
  98. ```
  99. See the mergerfs [wiki for real world
  100. deployments](https://github.com/trapexit/mergerfs/wiki/Real-World-Deployments)
  101. for comparisons / ideas.
  102. # OPTIONS
  103. These options are the same regardless of whether you use them with the
  104. `mergerfs` commandline program, in fstab, or in a config file.
  105. ### mount options
  106. * **config**: Path to a config file. Same arguments as below in
  107. key=val / ini style format.
  108. * **branches**: Colon delimited list of branches.
  109. * **minfreespace=SIZE**: The minimum space value used for creation
  110. policies. Can be overridden by branch specific option. Understands
  111. 'K', 'M', and 'G' to represent kilobyte, megabyte, and gigabyte
  112. respectively. (default: 4G)
  113. * **moveonenospc=BOOL|POLICY**: When enabled if a **write** fails with
  114. **ENOSPC** (no space left on device) or **EDQUOT** (disk quota
  115. exceeded) the policy selected will run to find a new location for
  116. the file. An attempt to move the file to that branch will occur
  117. (keeping all metadata possible) and if successful the original is
  118. unlinked and the write retried. (default: false, true = mfs)
  119. * **inodecalc=passthrough|path-hash|devino-hash|hybrid-hash**: Selects
  120. the inode calculation algorithm. (default: hybrid-hash)
  121. * **dropcacheonclose=BOOL**: When a file is requested to be closed
  122. call `posix_fadvise` on it first to instruct the kernel that we no
  123. longer need the data and it can drop its cache. Recommended when
  124. **cache.files=partial|full|auto-full|per-process** to limit double
  125. caching. (default: false)
  126. * **passthrough=off|ro|wo|rw**: On Linux 6.9 and above passthrough allows for
  127. near native IO performance. See below for more details. (default:
  128. off)
  129. * **direct-io-allow-mmap=BOOL**: On Linux 6.6 and above it is
  130. possible to disable file page caching while still allowing for
  131. shared mmap support. mergerfs will enable this feature if available
  132. but an option is provided to turn it off for testing and debugging
  133. purposes. (default: true)
  134. * **symlinkify=BOOL**: When enabled and a file is not writable and its
  135. mtime or ctime is older than **symlinkify_timeout** files will be
  136. reported as symlinks to the original files. Please read more below
  137. before using. (default: false)
  138. * **symlinkify_timeout=UINT**: Time to wait, in seconds, to activate
  139. the **symlinkify** behavior. (default: 3600)
  140. * **nullrw=BOOL**: Turns reads and writes into no-ops. The request
  141. will succeed but do nothing. Useful for benchmarking
  142. mergerfs. (default: false)
  143. * **lazy-umount-mountpoint=BOOL**: mergerfs will attempt to "lazy
  144. umount" the mountpoint before mounting itself. Useful when
  145. performing live upgrades of mergerfs. (default: false)
  146. * **ignorepponrename=BOOL**: Ignore path preserving on
  147. rename. Typically rename and link act differently depending on the
  148. policy of `create` (read below). Enabling this will cause rename and
  149. link to always use the non-path preserving behavior. This means
  150. files, when renamed or linked, will stay on the same
  151. filesystem. (default: false)
  152. * **export-support=BOOL**: Sets a low-level FUSE feature intended to
  153. indicate the filesystem can support being exported via
  154. NFS. (default: true)
  155. * **security_capability=BOOL**: If false return ENOATTR when xattr
  156. security.capability is queried. (default: true)
  157. * **xattr=passthrough|noattr|nosys**: Runtime control of
  158. xattrs. Default is to passthrough xattr requests. 'noattr' will
  159. short circuit as if nothing exists. 'nosys' will respond with ENOSYS
  160. as if xattrs are not supported or disabled. (default: passthrough)
  161. * **link_cow=BOOL**: When enabled if a regular file is opened which
  162. has a link count > 1 it will copy the file to a temporary file and
  163. rename over the original. Breaking the link and providing a basic
  164. copy-on-write function similar to cow-shell. (default: false)
  165. * **statfs=base|full**: Controls how statfs works. 'base' means it
  166. will always use all branches in statfs calculations. 'full' is in
  167. effect path preserving and only includes branches where the path
  168. exists. (default: base)
  169. * **statfs_ignore=none|ro|nc**: 'ro' will cause statfs calculations to
  170. ignore available space for branches mounted or tagged as 'read-only'
  171. or 'no create'. 'nc' will ignore available space for branches tagged
  172. as 'no create'. (default: none)
  173. * **nfsopenhack=off|git|all**: A workaround for exporting mergerfs
  174. over NFS where there are issues with creating files for write while
  175. setting the mode to read-only. (default: off)
  176. * **branches-mount-timeout=UINT**: Number of seconds to wait at
  177. startup for branches to be a mount other than the mountpoint's
  178. filesystem. (default: 0)
  179. * **follow-symlinks=never|directory|regular|all**: Turns symlinks into
  180. what they point to. (default: never)
  181. * **link-exdev=passthrough|rel-symlink|abs-base-symlink|abs-pool-symlink**:
  182. When a link fails with EXDEV optionally create a symlink to the file
  183. instead.
  184. * **rename-exdev=passthrough|rel-symlink|abs-symlink**: When a rename
  185. fails with EXDEV optionally move the file to a special directory and
  186. symlink to it.
  187. * **readahead=UINT**: Set readahead (in kilobytes) for mergerfs and
  188. branches if greater than 0. (default: 0)
  189. * **posix_acl=BOOL**: Enable POSIX ACL support (if supported by kernel
  190. and underlying filesystem). (default: false)
  191. * **async_read=BOOL**: Perform reads asynchronously. If disabled or
  192. unavailable the kernel will ensure there is at most one pending read
  193. request per file handle and will attempt to order requests by
  194. offset. (default: true)
  195. * **fuse_msg_size=UINT**: Set the max number of pages per FUSE
  196. message. Only available on Linux >= 4.20 and ignored
  197. otherwise. (min: 1; max: 256; default: 256)
  198. * **threads=INT**: Number of threads to use. When used alone
  199. (`process-thread-count=-1`) it sets the number of threads reading
  200. and processing FUSE messages. When used together it sets the number
  201. of threads reading from FUSE. When set to zero it will attempt to
  202. discover and use the number of logical cores. If the thread count is
  203. set negative it will look up the number of cores then divide by the
  204. absolute value. ie. threads=-2 on an 8 core machine will result in 8
  205. / 2 = 4 threads. There will always be at least 1 thread. If set to
  206. -1 in combination with `process-thread-count` then it will try to
  207. pick reasonable values based on CPU thread count. NOTE: higher
  208. number of threads increases parallelism but usually decreases
  209. throughput. (default: 0)
  210. * **read-thread-count=INT**: Alias for `threads`.
  211. * **process-thread-count=INT**: Enables separate thread pool to
  212. asynchronously process FUSE requests. In this mode
  213. `read-thread-count` refers to the number of threads reading FUSE
  214. messages which are dispatched to process threads. -1 means disabled
  215. otherwise acts like `read-thread-count`. (default: -1)
  216. * **process-thread-queue-depth=UINT**: Sets the number of requests any
  217. single process thread can have queued up at one time. Meaning the
  218. total memory usage of the queues is queue depth multiplied by the
  219. number of process threads plus read thread count. 0 sets the depth
  220. to the same as the process thread count. (default: 0)
  221. * **pin-threads=STR**: Selects a strategy to pin threads to CPUs
  222. (default: unset)
  223. * **flush-on-close=never|always|opened-for-write**: Flush data cache
  224. on file close. Mostly for when writeback is enabled or merging
  225. network filesystems. (default: opened-for-write)
  226. * **scheduling-priority=INT**: Set mergerfs' scheduling
  227. priority. Valid values range from -20 to 19. See `setpriority` man
  228. page for more details. (default: -10)
  229. * **fsname=STR**: Sets the name of the filesystem as seen in
  230. **mount**, **df**, etc. Defaults to a list of the source paths
  231. concatenated together with the longest common prefix removed.
  232. * **func.FUNC=POLICY**: Sets the specific FUSE function's policy. See
  233. below for the list of value types. Example: **func.getattr=newest**
  234. * **func.readdir=seq|cosr|cor|cosr:INT|cor:INT**: Sets `readdir`
  235. policy. INT value sets the number of threads to use for
  236. concurrency. (default: seq)
  237. * **category.action=POLICY**: Sets policy of all FUSE functions in the
  238. action category. (default: epall)
  239. * **category.create=POLICY**: Sets policy of all FUSE functions in the
  240. create category. (default: epmfs)
  241. * **category.search=POLICY**: Sets policy of all FUSE functions in the
  242. search category. (default: ff)
  243. * **cache.open=UINT**: 'open' policy cache timeout in
  244. seconds. (default: 0)
  245. * **cache.statfs=UINT**: 'statfs' cache timeout in seconds. (default:
  246. 0)
  247. * **cache.attr=UINT**: File attribute cache timeout in
  248. seconds. (default: 1)
  249. * **cache.entry=UINT**: File name lookup cache timeout in
  250. seconds. (default: 1)
  251. * **cache.negative_entry=UINT**: Negative file name lookup cache
  252. timeout in seconds. (default: 0)
  253. * **cache.files=libfuse|off|partial|full|auto-full|per-process**: File
  254. page caching mode (default: libfuse)
  255. * **cache.files.process-names=LIST**: A pipe | delimited list of
  256. process [comm](https://man7.org/linux/man-pages/man5/proc.5.html)
  257. names to enable page caching for when
  258. `cache.files=per-process`. (default: "rtorrent|qbittorrent-nox")
  259. * **cache.writeback=BOOL**: Enable kernel writeback caching (default:
  260. false)
  261. * **cache.symlinks=BOOL**: Cache symlinks (if supported by kernel)
  262. (default: false)
  263. * **cache.readdir=BOOL**: Cache readdir (if supported by kernel)
  264. (default: false)
  265. * **parallel-direct-writes=BOOL**: Allow the kernel to dispatch
  266. multiple, parallel (non-extending) write requests for files opened
  267. with `cache.files=per-process` (if the process is not in `process-names`)
  268. or `cache.files=off`. (This requires kernel support, and was added in v6.2)
  269. * **direct_io**: deprecated - Bypass page cache. Use `cache.files=off`
  270. instead. (default: false)
  271. * **kernel_cache**: deprecated - Do not invalidate data cache on file
  272. open. Use `cache.files=full` instead. (default: false)
  273. * **auto_cache**: deprecated - Invalidate data cache if file mtime or
  274. size change. Use `cache.files=auto-full` instead. (default: false)
  275. * **async_read**: deprecated - Perform reads asynchronously. Use
  276. `async_read=true` instead.
  277. * **sync_read**: deprecated - Perform reads synchronously. Use
  278. `async_read=false` instead.
  279. * **splice_read**: deprecated - Does nothing.
  280. * **splice_write**: deprecated - Does nothing.
  281. * **splice_move**: deprecated - Does nothing.
  282. * **allow_other**: deprecated - mergerfs v2.35.0 and newer sets this FUSE option
  283. automatically if running as root.
  284. * **use_ino**: deprecated - mergerfs should always control inode
  285. calculation so this is enabled all the time.
  286. **NOTE:** Options are evaluated in the order listed so if the options
  287. are **func.rmdir=rand,category.action=ff** the **action** category
  288. setting will override the **rmdir** setting.
  289. **NOTE:** Always look at the documentation for the version of mergerfs
  290. you're using. Not all features are available in older releases. Use
  291. `man mergerfs` or find the docs as linked in the release.
  292. #### Value Types
  293. * BOOL = 'true' | 'false'
  294. * INT = [MIN_INT,MAX_INT]
  295. * UINT = [0,MAX_INT]
  296. * SIZE = 'NNM'; NN = INT, M = 'K' | 'M' | 'G' | 'T'
  297. * STR = string (may refer to an enumerated value, see details of
  298. argument)
  299. * FUNC = filesystem function
  300. * CATEGORY = function category
  301. * POLICY = mergerfs function policy
  302. ### branches
  303. The 'branches' argument is a colon (':') delimited list of paths to be
  304. pooled together. It does not matter if the paths are on the same or
  305. different filesystems nor does it matter the filesystem type (within
  306. reason). Used and available space will not be duplicated for paths on
  307. the same filesystem and any features which aren't supported by the
  308. underlying filesystem (such as file attributes or extended attributes)
  309. will return the appropriate errors.
  310. Branches currently have two options which can be set. A type which
  311. impacts whether or not the branch is included in a policy calculation
  312. and a individual minfreespace value. The values are set by prepending
  313. an `=` at the end of a branch designation and using commas as
  314. delimiters. Example: `/mnt/drive=RW,1234`
  315. #### branch mode
  316. * RW: (read/write) - Default behavior. Will be eligible in all policy
  317. categories.
  318. * RO: (read-only) - Will be excluded from `create` and `action`
  319. policies. Same as a read-only mounted filesystem would be (though
  320. faster to process).
  321. * NC: (no-create) - Will be excluded from `create` policies. You can't
  322. create on that branch but you can change or delete.
  323. #### minfreespace
  324. Same purpose and syntax as the global option but specific to the
  325. branch. If not set the global value is used.
  326. #### globbing
  327. To make it easier to include multiple branches mergerfs supports
  328. [globbing](http://linux.die.net/man/7/glob). **The globbing tokens
  329. MUST be escaped when using via the shell else the shell itself will
  330. apply the glob itself.**
  331. ```
  332. # mergerfs /mnt/hdd\*:/mnt/ssd /media
  333. ```
  334. The above line will use all mount points in /mnt prefixed with **hdd** and **ssd**.
  335. To have the pool mounted at boot or otherwise accessible from related tools use **/etc/fstab**.
  336. ```
  337. # <file system> <mount point> <type> <options> <dump> <pass>
  338. /mnt/hdd*:/mnt/ssd /media mergerfs minfreespace=16G 0 0
  339. ```
  340. **NOTE:** the globbing is done at mount or when updated using the
  341. runtime API. If a new directory is added matching the glob after the
  342. fact it will not be automatically included.
  343. **NOTE:** for mounting via **fstab** to work you must have
  344. **mount.fuse** installed. For Ubuntu/Debian it is included in the
  345. **fuse** package.
  346. ### passthrough
  347. With Linux 6.9 and above there is a feature in FUSE called
  348. "passthrough" which can improve performance by allowing a union or
  349. overlay filesystem (like mergerfs) to pass the file descriptor of a
  350. file opened on an underlying filesystem to the kernel to work on
  351. directly rather than calls always having to go to the FUSE server
  352. (mergerfs). As of Linux 6.9 the functions that can be passthroughed
  353. are regular file read/write IO and mmap. Other functions may be added
  354. in the future.
  355. Passthrough requires `cache.files` to be enabled. `cache.files=off`
  356. (which enables FUSE direct-io feature) will override `passthrough` and
  357. read/write requests will still be sent to mergerfs.
  358. If `cache.files=off` and `direct-io-allow-mmap=true` (the default)
  359. then regular read/write IO will be handled by mergerfs but mmap will
  360. be passthroughed.
  361. NOTE: `moveonenospc` will not work when using `passthrough` as it
  362. removes mergerfs entirely from the read/write process. If an ENOSPC
  363. error occurs it will be returned to the caller app immediately. Since
  364. mergerfs won't be handling the IO it therefore won't be able to handle
  365. any error returned by said IO.
  366. ### inodecalc
  367. Inodes (st_ino) are unique identifiers within a filesystem. Each
  368. mounted filesystem has device ID (st_dev) as well and together they
  369. can uniquely identify a file on the whole of the system. Entries on
  370. the same device with the same inode are in fact references to the same
  371. underlying file. It is a many to one relationship between names and an
  372. inode. Directories, however, do not have multiple links on most
  373. systems due to the complexity they add.
  374. FUSE allows the server (mergerfs) to set inode values but not device
  375. IDs. Creating an inode value is somewhat complex in mergerfs' case as
  376. files aren't really in its control. If a policy changes what directory
  377. or file is to be selected or something changes out of band it becomes
  378. unclear what value should be used. Most software does not to care what
  379. the values are but those that do often break if a value changes
  380. unexpectedly. The tool `find` will abort a directory walk if it sees a
  381. directory inode change. NFS can return stale handle errors if the
  382. inode changes out of band. File dedup tools will usually leverage
  383. device ids and inodes as a shortcut in searching for duplicate files
  384. and would resort to full file comparisons should it find different
  385. inode values.
  386. mergerfs offers multiple ways to calculate the inode in hopes of
  387. covering different usecases.
  388. * passthrough: Passes through the underlying inode value. Mostly
  389. intended for testing as using this does not address any of the
  390. problems mentioned above and could confuse file deduplication
  391. software as inodes from different filesystems can be the same.
  392. * path-hash: Hashes the relative path of the entry in question. The
  393. underlying file's values are completely ignored. This means the
  394. inode value will always be the same for that file path. This is
  395. useful when using NFS and you make changes out of band such as copy
  396. data between branches. This also means that entries that do point to
  397. the same file will not be recognizable via inodes. That **does not**
  398. mean hard links don't work. They will.
  399. * path-hash32: 32bit version of path-hash.
  400. * devino-hash: Hashes the device id and inode of the underlying
  401. entry. This won't prevent issues with NFS should the policy pick a
  402. different file or files move out of band but will present the same
  403. inode for underlying files that do too.
  404. * devino-hash32: 32bit version of devino-hash.
  405. * hybrid-hash: Performs `path-hash` on directories and `devino-hash`
  406. on other file types. Since directories can't have hard links the
  407. static value won't make a difference and the files will get values
  408. useful for finding duplicates. Probably the best to use if not using
  409. NFS. As such it is the default.
  410. * hybrid-hash32: 32bit version of hybrid-hash.
  411. 32bit versions are provided as there is some software which does not
  412. handle 64bit inodes well.
  413. While there is a risk of hash collision in tests of a couple million
  414. entries there were zero collisions. Unlike a typical filesystem FUSE
  415. filesystems can reuse inodes and not refer to the same entry. The
  416. internal identifier used to reference a file in FUSE is different from
  417. the inode value presented. The former is the `nodeid` and is actually
  418. a tuple of 2 64bit values: `nodeid` and `generation`. This tuple is
  419. not client facing. The inode that is presented to the client is passed
  420. through the kernel uninterpreted.
  421. From FUSE docs for `use_ino`:
  422. ```
  423. Honor the st_ino field in the functions getattr() and
  424. fill_dir(). This value is used to fill in the st_ino field
  425. in the stat(2), lstat(2), fstat(2) functions and the d_ino
  426. field in the readdir(2) function. The filesystem does not
  427. have to guarantee uniqueness, however some applications
  428. rely on this value being unique for the whole filesystem.
  429. Note that this does *not* affect the inode that libfuse
  430. and the kernel use internally (also called the "nodeid").
  431. ```
  432. As of version 2.35.0 the `use_ino` option has been removed. mergerfs
  433. should always be managing inode values.
  434. ### pin-threads
  435. Simple strategies for pinning read and/or process threads. If process
  436. threads are not enabled than the strategy simply works on the read
  437. threads. Invalid values are ignored.
  438. * R1L: All read threads pinned to a single logical CPU.
  439. * R1P: All read threads pinned to a single physical CPU.
  440. * RP1L: All read and process threads pinned to a single logical CPU.
  441. * RP1P: All read and process threads pinned to a single physical CPU.
  442. * R1LP1L: All read threads pinned to a single logical CPU, all process
  443. threads pinned to a (if possible) different logical CPU.
  444. * R1PP1P: All read threads pinned to a single physical CPU, all
  445. process threads pinned to a (if possible) different logical CPU.
  446. * RPSL: All read and process threads are spread across all logical CPUs.
  447. * RPSP: All read and process threads are spread across all physical CPUs.
  448. * R1PPSP: All read threads are pinned to a single physical CPU while
  449. process threads are spread across all other phsycial CPUs.
  450. ### fuse_msg_size
  451. FUSE applications communicate with the kernel over a special character
  452. device: `/dev/fuse`. A large portion of the overhead associated with
  453. FUSE is the cost of going back and forth from user space and kernel
  454. space over that device. Generally speaking the fewer trips needed the
  455. better the performance will be. Reducing the number of trips can be
  456. done a number of ways. Kernel level caching and increasing message
  457. sizes being two significant ones. When it comes to reads and writes if
  458. the message size is doubled the number of trips are approximately
  459. halved.
  460. In Linux 4.20 a new feature was added allowing the negotiation of the
  461. max message size. Since the size is in multiples of
  462. [pages](https://en.wikipedia.org/wiki/Page_(computer_memory)) the
  463. feature is called `max_pages`. There is a maximum `max_pages` value of
  464. 256 (1MiB) and minimum of 1 (4KiB). The default used by Linux >=4.20,
  465. and hardcoded value used before 4.20, is 32 (128KiB). In mergerfs its
  466. referred to as `fuse_msg_size` to make it clear what it impacts and
  467. provide some abstraction.
  468. Since there should be no downsides to increasing `fuse_msg_size` /
  469. `max_pages`, outside a minor bump in RAM usage due to larger message
  470. buffers, mergerfs defaults the value to 256. On kernels before 4.20
  471. the value has no effect. The reason the value is configurable is to
  472. enable experimentation and benchmarking. See the BENCHMARKING section
  473. for examples.
  474. ### follow-symlinks
  475. This feature, when enabled, will cause symlinks to be interpreted by
  476. mergerfs as their target (depending on the mode).
  477. When there is a getattr/stat request for a file mergerfs will check if
  478. the file is a symlink and depending on the `follow-symlinks` setting
  479. will replace the information about the symlink with that of that which
  480. it points to.
  481. When unlink'ing or rmdir'ing the followed symlink it will remove the
  482. symlink itself and not that which it points to.
  483. * never: Behave as normal. Symlinks are treated as such.
  484. * directory: Resolve symlinks only which point to directories.
  485. * regular: Resolve symlinks only which point to regular files.
  486. * all: Resolve all symlinks to that which they point to.
  487. Symlinks which do not point to anything are left as is.
  488. WARNING: This feature works but there might be edge cases yet
  489. found. If you find any odd behaviors please file a ticket on
  490. [github](https://github.com/trapexit/mergerfs/issues).
  491. ### link-exdev
  492. If using path preservation and a `link` fails with EXDEV make a call
  493. to `symlink` where the `target` is the `oldlink` and the `linkpath` is
  494. the `newpath`. The `target` value is determined by the value of
  495. `link-exdev`.
  496. * passthrough: Return EXDEV as normal.
  497. * rel-symlink: A relative path from the `newpath`.
  498. * abs-base-symlink: A absolute value using the underlying branch.
  499. * abs-pool-symlink: A absolute value using the mergerfs mount point.
  500. NOTE: It is possible that some applications check the file they
  501. link. In those cases it is possible it will error or complain.
  502. ### rename-exdev
  503. If using path preservation and a `rename` fails with EXDEV:
  504. 1. Move file from **/branch/a/b/c** to **/branch/.mergerfs_rename_exdev/a/b/c**.
  505. 2. symlink the rename's `newpath` to the moved file.
  506. The `target` value is determined by the value of `rename-exdev`.
  507. * passthrough: Return EXDEV as normal.
  508. * rel-symlink: A relative path from the `newpath`.
  509. * abs-symlink: A absolute value using the mergerfs mount point.
  510. NOTE: It is possible that some applications check the file they
  511. rename. In those cases it is possible it will error or complain.
  512. NOTE: The reason `abs-symlink` is not split into two like `link-exdev`
  513. is due to the complexities in managing absolute base symlinks when
  514. multiple `oldpaths` exist.
  515. ### symlinkify
  516. Due to the levels of indirection introduced by mergerfs and the
  517. underlying technology FUSE there can be varying levels of performance
  518. degradation. This feature will turn non-directories which are not
  519. writable into symlinks to the original file found by the `readlink`
  520. policy after the mtime and ctime are older than the timeout.
  521. **WARNING:** The current implementation has a known issue in which if
  522. the file is open and being used when the file is converted to a
  523. symlink then the application which has that file open will receive an
  524. error when using it. This is unlikely to occur in practice but is
  525. something to keep in mind.
  526. **WARNING:** Some backup solutions, such as CrashPlan, do not backup
  527. the target of a symlink. If using this feature it will be necessary to
  528. point any backup software to the original filesystems or configure the
  529. software to follow symlinks if such an option is available.
  530. Alternatively create two mounts. One for backup and one for general
  531. consumption.
  532. ### nullrw
  533. Due to how FUSE works there is an overhead to all requests made to a
  534. FUSE filesystem that wouldn't exist for an in kernel one. Meaning that
  535. even a simple passthrough will have some slowdown. However, generally
  536. the overhead is minimal in comparison to the cost of the underlying
  537. I/O. By disabling the underlying I/O we can test the theoretical
  538. performance boundaries.
  539. By enabling `nullrw` mergerfs will work as it always does **except**
  540. that all reads and writes will be no-ops. A write will succeed (the
  541. size of the write will be returned as if it were successful) but
  542. mergerfs does nothing with the data it was given. Similarly a read
  543. will return the size requested but won't touch the buffer.
  544. See the BENCHMARKING section for suggestions on how to test.
  545. ### xattr
  546. Runtime extended attribute support can be managed via the `xattr`
  547. option. By default it will passthrough any xattr calls. Given xattr
  548. support is rarely used and can have significant performance
  549. implications mergerfs allows it to be disabled at runtime. The
  550. performance problems mostly comes when file caching is enabled. The
  551. kernel will send a `getxattr` for `security.capability` *before every
  552. single write*. It doesn't cache the responses to any `getxattr`. This
  553. might be addressed in the future but for now mergerfs can really only
  554. offer the following workarounds.
  555. `noattr` will cause mergerfs to short circuit all xattr calls and
  556. return ENOATTR where appropriate. mergerfs still gets all the requests
  557. but they will not be forwarded on to the underlying filesystems. The
  558. runtime control will still function in this mode.
  559. `nosys` will cause mergerfs to return ENOSYS for any xattr call. The
  560. difference with `noattr` is that the kernel will cache this fact and
  561. itself short circuit future calls. This is more efficient than
  562. `noattr` but will cause mergerfs' runtime control via the hidden file
  563. to stop working.
  564. ### nfsopenhack
  565. NFS is not fully POSIX compliant and historically certain behaviors,
  566. such as opening files with O_EXCL, are not or not well supported. When
  567. mergerfs (or any FUSE filesystem) is exported over NFS some of these
  568. issues come up due to how NFS and FUSE interact.
  569. This hack addresses the issue where the creation of a file with a
  570. read-only mode but with a read/write or write only flag. Normally this
  571. is perfectly valid but NFS chops the one open call into multiple
  572. calls. Exactly how it is translated depends on the configuration and
  573. versions of the NFS server and clients but it results in a permission
  574. error because a normal user is not allowed to open a read-only file as
  575. writable.
  576. Even though it's a more niche situation this hack breaks normal
  577. security and behavior and as such is `off` by default. If set to `git`
  578. it will only perform the hack when the path in question includes
  579. `/.git/`. `all` will result it applying anytime a read-only file which
  580. is empty is opened for writing.
  581. ### export-support
  582. In theory this flag should not be exposed to the end user. It is a
  583. low-level FUSE flag which indicates whether or not the kernel can send
  584. certain kinds of messages to it for the purposes of using with
  585. NFS. mergerfs does support these messages but due bugs and quirks
  586. found in the kernel and mergerfs this option is provided just in case
  587. it is needed for debugging.
  588. Given that this flag is set when the FUSE connection is first
  589. initiated it is not possible to change during run time.
  590. # FUNCTIONS, CATEGORIES and POLICIES
  591. The POSIX filesystem API is made up of a number of
  592. functions. **creat**, **stat**, **chown**, etc. For ease of
  593. configuration in mergerfs most of the core functions are grouped into
  594. 3 categories: **action**, **create**, and **search**. These functions
  595. and categories can be assigned a policy which dictates which branch is
  596. chosen when performing that function.
  597. Some functions, listed in the category `N/A` below, can not be
  598. assigned the normal policies. These functions work with file handles,
  599. rather than file paths, which were created by `open` or `create`. That
  600. said many times the current FUSE kernel driver will not always provide
  601. the file handle when a client calls `fgetattr`, `fchown`, `fchmod`,
  602. `futimens`, `ftruncate`, etc. This means it will call the regular,
  603. path based, versions. `statfs`'s behavior can be modified via other
  604. options.
  605. When using policies which are based on a branch's available space the
  606. base path provided is used. Not the full path to the file in
  607. question. Meaning that mounts in the branch won't be considered in the
  608. space calculations. The reason is that it doesn't really work for
  609. non-path preserving policies and can lead to non-obvious behaviors.
  610. NOTE: While any policy can be assigned to a function or category,
  611. some may not be very useful in practice. For instance: **rand**
  612. (random) may be useful for file creation (create) but could lead to
  613. very odd behavior if used for `chmod` if there were more than one copy
  614. of the file.
  615. ### Functions and their Category classifications
  616. | Category | FUSE Functions |
  617. |----------|-------------------------------------------------------------------------------------|
  618. | action | chmod, chown, link, removexattr, rename, rmdir, setxattr, truncate, unlink, utimens |
  619. | create | create, mkdir, mknod, symlink |
  620. | search | access, getattr, getxattr, ioctl (directories), listxattr, open, readlink |
  621. | N/A | fchmod, fchown, futimens, ftruncate, fallocate, fgetattr, fsync, ioctl (files), read, readdir, release, statfs, write, copy_file_range |
  622. In cases where something may be searched for (such as a path to clone)
  623. **getattr** will usually be used.
  624. ### Policies
  625. A policy is the algorithm used to choose a branch or branches for a
  626. function to work on or generally how the function behaves.
  627. Any function in the `create` category will clone the relative path if
  628. needed. Some other functions (`rename`,`link`,`ioctl`) have special
  629. requirements or behaviors which you can read more about below.
  630. #### Filtering
  631. Most policies basically search branches and create a list of files / paths
  632. for functions to work on. The policy is responsible for filtering and
  633. sorting the branches. Filters include **minfreespace**, whether or not
  634. a branch is mounted read-only, and the branch tagging
  635. (RO,NC,RW). These filters are applied across most policies.
  636. * No **search** function policies filter.
  637. * All **action** function policies filter out branches which are
  638. mounted **read-only** or tagged as **RO (read-only)**.
  639. * All **create** function policies filter out branches which are
  640. mounted **read-only**, tagged **RO (read-only)** or **NC (no
  641. create)**, or has available space less than `minfreespace`.
  642. Policies may have their own additional filtering such as those that
  643. require existing paths to be present.
  644. If all branches are filtered an error will be returned. Typically
  645. **EROFS** (read-only filesystem) or **ENOSPC** (no space left on
  646. device) depending on the most recent reason for filtering a
  647. branch. **ENOENT** will be returned if no eligible branch is found.
  648. If **create**, **mkdir**, **mknod**, or **symlink** fail with `EROFS`
  649. or other fundimental errors then mergerfs will mark any branch found
  650. to be read-only as such (IE will set the mode `RO`) and will rerun the
  651. policy and try again. This is mostly for `ext4` filesystems that can
  652. suddenly become read-only when it encounters an error.
  653. #### Path Preservation
  654. Policies, as described below, are of two basic classifications. `path
  655. preserving` and `non-path preserving`.
  656. All policies which start with `ep` (**epff**, **eplfs**, **eplus**,
  657. **epmfs**, **eprand**) are `path preserving`. `ep` stands for
  658. `existing path`.
  659. A path preserving policy will only consider branches where the relative
  660. path being accessed already exists.
  661. When using non-path preserving policies paths will be cloned to target
  662. branches as necessary.
  663. With the `msp` or `most shared path` policies they are defined as
  664. `path preserving` for the purpose of controlling `link` and `rename`'s
  665. behaviors since `ignorepponrename` is available to disable that
  666. behavior.
  667. #### Policy descriptions
  668. A policy's behavior differs, as mentioned above, based on the function
  669. it is used with. Sometimes it really might not make sense to even
  670. offer certain policies because they are literally the same as others
  671. but it makes things a bit more uniform.
  672. | Policy | Description |
  673. |------------------|------------------------------------------------------------|
  674. | all | Search: For **mkdir**, **mknod**, and **symlink** it will apply to all branches. **create** works like **ff**. |
  675. | epall (existing path, all) | For **mkdir**, **mknod**, and **symlink** it will apply to all found. **create** works like **epff** (but more expensive because it doesn't stop after finding a valid branch). |
  676. | epff (existing path, first found) | Given the order of the branches, as defined at mount time or configured at runtime, act on the first one found where the relative path exists. |
  677. | eplfs (existing path, least free space) | Of all the branches on which the relative path exists choose the branch with the least free space. |
  678. | eplus (existing path, least used space) | Of all the branches on which the relative path exists choose the branch with the least used space. |
  679. | epmfs (existing path, most free space) | Of all the branches on which the relative path exists choose the branch with the most free space. |
  680. | eppfrd (existing path, percentage free random distribution) | Like **pfrd** but limited to existing paths. |
  681. | eprand (existing path, random) | Calls **epall** and then randomizes. Returns 1. |
  682. | ff (first found) | Given the order of the branches, as defined at mount time or configured at runtime, act on the first one found. |
  683. | lfs (least free space) | Pick the branch with the least available free space. |
  684. | lus (least used space) | Pick the branch with the least used space. |
  685. | mfs (most free space) | Pick the branch with the most available free space. |
  686. | msplfs (most shared path, least free space) | Like **eplfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
  687. | msplus (most shared path, least used space) | Like **eplus** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
  688. | mspmfs (most shared path, most free space) | Like **epmfs** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
  689. | msppfrd (most shared path, percentage free random distribution) | Like **eppfrd** but if it fails to find a branch it will try again with the parent directory. Continues this pattern till finding one. |
  690. | newest | Pick the file / directory with the largest mtime. |
  691. | pfrd (percentage free random distribution) | Chooses a branch at random with the likelihood of selection based on a branch's available space relative to the total. |
  692. | rand (random) | Calls **all** and then randomizes. Returns 1 branch. |
  693. **NOTE:** If you are using an underlying filesystem that reserves
  694. blocks such as ext2, ext3, or ext4 be aware that mergerfs respects the
  695. reservation by using `f_bavail` (number of free blocks for
  696. unprivileged users) rather than `f_bfree` (number of free blocks) in
  697. policy calculations. **df** does NOT use `f_bavail`, it uses
  698. `f_bfree`, so direct comparisons between **df** output and mergerfs'
  699. policies is not appropriate.
  700. #### Defaults
  701. | Category | Policy |
  702. |----------|--------|
  703. | action | epall |
  704. | create | epmfs |
  705. | search | ff |
  706. #### func.readdir
  707. examples: `func.readdir=seq`, `func.readdir=cor:4`
  708. `readdir` has policies to control how it manages reading directory
  709. content.
  710. | Policy | Description |
  711. |--------|-------------|
  712. | seq | "sequential" : Iterate over branches in the order defined. This is the default and traditional behavior found prior to the readdir policy introduction. |
  713. | cosr | "concurrent open, sequential read" : Concurrently open branch directories using a thread pool and process them in order of definition. This keeps memory and CPU usage low while also reducing the time spent waiting on branches to respond. Number of threads defaults to the number of logical cores. Can be overwritten via the syntax `func.readdir=cosr:N` where `N` is the number of threads. |
  714. | cor | "concurrent open and read" : Concurrently open branch directories and immediately start reading their contents using a thread pool. This will result in slightly higher memory and CPU usage but reduced latency. Particularly when using higher latency / slower speed network filesystem branches. Unlike `seq` and `cosr` the order of files could change due the async nature of the thread pool. Number of threads defaults to the number of logical cores. Can be overwritten via the syntax `func.readdir=cor:N` where `N` is the number of threads.
  715. Keep in mind that `readdir` mostly just provides a list of file names
  716. in a directory and possibly some basic metadata about said files. To
  717. know details about the files, as one would see from commands like
  718. `find` or `ls`, it is required to call `stat` on the file which is
  719. controlled by `fuse.getattr`.
  720. #### ioctl
  721. When `ioctl` is used with an open file then it will use the file
  722. handle which was created at the original `open` call. However, when
  723. using `ioctl` with a directory mergerfs will use the `open` policy to
  724. find the directory to act on.
  725. #### rename & link ####
  726. **NOTE:** If you're receiving errors from software when files are
  727. moved / renamed / linked then you should consider changing the create
  728. policy to one which is **not** path preserving, enabling
  729. `ignorepponrename`, or contacting the author of the offending software
  730. and requesting that `EXDEV` (cross device / improper link) be properly
  731. handled.
  732. `rename` and `link` are tricky functions in a union
  733. filesystem. `rename` only works within a single filesystem or
  734. device. If a rename can't be done atomically due to the source and
  735. destination paths existing on different mount points it will return
  736. **-1** with **errno = EXDEV** (cross device / improper link). So if a
  737. `rename`'s source and target are on different filesystems within the pool
  738. it creates an issue.
  739. Originally mergerfs would return EXDEV whenever a rename was requested
  740. which was cross directory in any way. This made the code simple and
  741. was technically compliant with POSIX requirements. However, many
  742. applications fail to handle EXDEV at all and treat it as a normal
  743. error or otherwise handle it poorly. Such apps include: gvfsd-fuse
  744. v1.20.3 and prior, Finder / CIFS/SMB client in Apple OSX 10.9+,
  745. NZBGet, Samba's recycling bin feature.
  746. As a result a compromise was made in order to get most software to
  747. work while still obeying mergerfs' policies. Below is the basic logic.
  748. * If using a **create** policy which tries to preserve directory paths (epff,eplfs,eplus,epmfs)
  749. * Using the **rename** policy get the list of files to rename
  750. * For each file attempt rename:
  751. * If failure with ENOENT (no such file or directory) run **create** policy
  752. * If create policy returns the same branch as currently evaluating then clone the path
  753. * Re-attempt rename
  754. * If **any** of the renames succeed the higher level rename is considered a success
  755. * If **no** renames succeed the first error encountered will be returned
  756. * On success:
  757. * Remove the target from all branches with no source file
  758. * Remove the source from all branches which failed to rename
  759. * If using a **create** policy which does **not** try to preserve directory paths
  760. * Using the **rename** policy get the list of files to rename
  761. * Using the **getattr** policy get the target path
  762. * For each file attempt rename:
  763. * If the source branch != target branch:
  764. * Clone target path from target branch to source branch
  765. * Rename
  766. * If **any** of the renames succeed the higher level rename is considered a success
  767. * If **no** renames succeed the first error encountered will be returned
  768. * On success:
  769. * Remove the target from all branches with no source file
  770. * Remove the source from all branches which failed to rename
  771. The the removals are subject to normal entitlement checks.
  772. The above behavior will help minimize the likelihood of EXDEV being
  773. returned but it will still be possible.
  774. **link** uses the same strategy but without the removals.
  775. #### statfs / statvfs ####
  776. [statvfs](http://linux.die.net/man/2/statvfs) normalizes the source
  777. filesystems based on the fragment size and sums the number of adjusted
  778. blocks and inodes. This means you will see the combined space of all
  779. sources. Total, used, and free. The sources however are dedupped based
  780. on the filesystem so multiple sources on the same drive will not result in
  781. double counting its space. Other filesystems mounted further down the tree
  782. of the branch will not be included when checking the mount's stats.
  783. The options `statfs` and `statfs_ignore` can be used to modify
  784. `statfs` behavior.
  785. #### flush-on-close
  786. https://lkml.kernel.org/linux-fsdevel/20211024132607.1636952-1-amir73il@gmail.com/T/
  787. By default FUSE would issue a flush before the release of a file
  788. descriptor. This was considered a bit aggressive and a feature added
  789. to give the FUSE server the ability to choose when that happens.
  790. Options:
  791. * always
  792. * never
  793. * opened-for-write
  794. For now it defaults to "opened-for-write" which is less aggressive
  795. than the behavior before this feature was added. It should not be a
  796. problem because the flush is really only relevant when a file is
  797. written to. Given flush is irrelevant for many filesystems in the
  798. future a branch specific flag may be added so only files opened on a
  799. specific branch would be flushed on close.
  800. # ERROR HANDLING
  801. POSIX filesystem functions offer a single return code meaning that
  802. there is some complication regarding the handling of multiple branches
  803. as mergerfs does. It tries to handle errors in a way that would
  804. generally return meaningful values for that particular function.
  805. ### chmod, chown, removexattr, setxattr, truncate, utimens
  806. 1) if no error: return 0 (success)
  807. 2) if no successes: return first error
  808. 3) if one of the files acted on was the same as the related search function: return its value
  809. 4) return 0 (success)
  810. While doing this increases the complexity and cost of error handling,
  811. particularly step 3, this provides probably the most reasonable return
  812. value.
  813. ### unlink, rmdir
  814. 1) if no errors: return 0 (success)
  815. 2) return first error
  816. Older version of mergerfs would return success if any success occurred
  817. but for unlink and rmdir there are downstream assumptions that, while
  818. not impossible to occur, can confuse some software.
  819. ### others
  820. For search functions there is always a single thing acted on and as
  821. such whatever return value that comes from the single function call is
  822. returned.
  823. For create functions `mkdir`, `mknod`, and `symlink` which don't
  824. return a file descriptor and therefore can have `all` or `epall`
  825. policies it will return success if any of the calls succeed and an
  826. error otherwise.
  827. # INSTALL
  828. https://github.com/trapexit/mergerfs/releases
  829. If your distribution's package manager includes mergerfs check if the
  830. version is up to date. If out of date it is recommended to use
  831. the latest release found on the release page. Details for common
  832. distros are below.
  833. #### Debian
  834. Most Debian installs are of a stable branch and therefore do not have
  835. the most up to date software. While mergerfs is available via `apt` it
  836. is suggested that uses install the most recent version available from
  837. the [releases page](https://github.com/trapexit/mergerfs/releases).
  838. #### prebuilt deb
  839. ```
  840. wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs_<ver>.debian-<rel>_<arch>.deb
  841. dpkg -i mergerfs_<ver>.debian-<rel>_<arch>.deb
  842. ```
  843. #### apt
  844. ```
  845. sudo apt install -y mergerfs
  846. ```
  847. #### Ubuntu
  848. Most Ubuntu installs are of a stable branch and therefore do not have
  849. the most up to date software. While mergerfs is available via `apt` it
  850. is suggested that uses install the most recent version available from
  851. the [releases page](https://github.com/trapexit/mergerfs/releases).
  852. #### prebuilt deb
  853. ```
  854. wget https://github.com/trapexit/mergerfs/releases/download/<version>/mergerfs_<ver>.ubuntu-<rel>_<arch>.deb
  855. dpkg -i mergerfs_<ver>.ubuntu-<rel>_<arch>.deb
  856. ```
  857. #### apt
  858. ```
  859. sudo apt install -y mergerfs
  860. ```
  861. #### Raspberry Pi OS
  862. Effectively the same as Debian or Ubuntu.
  863. #### Fedora
  864. ```
  865. wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-<ver>.fc<rel>.<arch>.rpm
  866. sudo rpm -i mergerfs-<ver>.fc<rel>.<arch>.rpm
  867. ```
  868. #### CentOS / Rocky
  869. ```
  870. wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-<ver>.el<rel>.<arch>.rpm
  871. sudo rpm -i mergerfs-<ver>.el<rel>.<arch>.rpm
  872. ```
  873. #### ArchLinux
  874. 1. Setup AUR
  875. 2. Install `mergerfs`
  876. #### Other
  877. Static binaries are provided for situations where native packages are
  878. unavailable.
  879. ```
  880. wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-static-linux_<arch>.tar.gz
  881. sudo tar xvf mergerfs-static-linux_<arch>.tar.gz -C /
  882. ```
  883. # BUILD
  884. **NOTE:** Prebuilt packages can be found at and recommended for most
  885. users: https://github.com/trapexit/mergerfs/releases
  886. **NOTE:** Only tagged releases are supported. `master` and other
  887. branches should be considered works in progress.
  888. First get the code from [github](https://github.com/trapexit/mergerfs).
  889. ```
  890. $ git clone https://github.com/trapexit/mergerfs.git
  891. $ # or
  892. $ wget https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs-<ver>.tar.gz
  893. ```
  894. #### Debian / Ubuntu
  895. ```
  896. $ cd mergerfs
  897. $ sudo tools/install-build-pkgs
  898. $ make deb
  899. $ sudo dpkg -i ../mergerfs_<version>_<arch>.deb
  900. ```
  901. #### RHEL / CentOS / Rocky / Fedora
  902. ```
  903. $ su -
  904. # cd mergerfs
  905. # tools/install-build-pkgs
  906. # make rpm
  907. # rpm -i rpmbuild/RPMS/<arch>/mergerfs-<version>.<arch>.rpm
  908. ```
  909. #### Generic
  910. Have git, g++, make, python installed.
  911. ```
  912. $ cd mergerfs
  913. $ make
  914. $ sudo make install
  915. ```
  916. #### Build options
  917. ```
  918. $ make help
  919. usage: make
  920. make USE_XATTR=0 - build program without xattrs functionality
  921. make STATIC=1 - build static binary
  922. make LTO=1 - build with link time optimization
  923. ```
  924. # UPGRADE
  925. mergerfs can be upgraded live by mounting on top of the previous
  926. instance. Simply install the new version of mergerfs and follow the
  927. instructions below.
  928. Run mergerfs again or if using `/etc/fstab` call for it to mount
  929. again. Existing open files and such will continue to work fine though
  930. they won't see runtime changes since any such change would be the new
  931. mount. If you plan on changing settings with the new mount you should
  932. / could apply those before mounting the new version.
  933. ```
  934. $ sudo mount /mnt/mergerfs
  935. $ mount | grep mergerfs
  936. media on /mnt/mergerfs type mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
  937. media on /mnt/mergerfs type mergerfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
  938. ```
  939. A problem with this approach is that the underlying instance will
  940. continue to run even if the software using it stop or are
  941. restarted. To work around this you can use a "lazy umount". Before
  942. mounting over top the mount point with the new instance of mergerfs
  943. issue: `umount -l <mergerfs_mountpoint>`. Or you can let mergerfs do
  944. it by setting the option `lazy-umount-mountpoint=true`.
  945. # RUNTIME INTERFACES
  946. ## RUNTIME CONFIG
  947. #### .mergerfs pseudo file ####
  948. ```
  949. <mountpoint>/.mergerfs
  950. ```
  951. There is a pseudo file available at the mount point which allows for
  952. the runtime modification of certain **mergerfs** options. The file
  953. will not show up in **readdir** but can be **stat**'ed and manipulated
  954. via [{list,get,set}xattrs](http://linux.die.net/man/2/listxattr)
  955. calls.
  956. Any changes made at runtime are **not** persisted. If you wish for
  957. values to persist they must be included as options wherever you
  958. configure the mounting of mergerfs (/etc/fstab).
  959. ##### Keys #####
  960. Use `getfattr -d /mountpoint/.mergerfs` or `xattr -l
  961. /mountpoint/.mergerfs` to see all supported keys. Some are
  962. informational and therefore read-only. `setxattr` will return EINVAL
  963. (invalid argument) on read-only keys.
  964. ##### Values #####
  965. Same as the command line.
  966. ###### user.mergerfs.branches ######
  967. Used to query or modify the list of branches. When modifying there are
  968. several shortcuts to easy manipulation of the list.
  969. | Value | Description |
  970. |--------------|-------------|
  971. | [list] | set |
  972. | +<[list] | prepend |
  973. | +>[list] | append |
  974. | -[list] | remove all values provided |
  975. | -< | remove first in list |
  976. | -> | remove last in list |
  977. `xattr -w user.mergerfs.branches +</mnt/drive3 /mnt/pool/.mergerfs`
  978. The `=NC`, `=RO`, `=RW` syntax works just as on the command line.
  979. ##### Example #####
  980. ```
  981. [trapexit:/mnt/mergerfs] $ getfattr -d .mergerfs
  982. user.mergerfs.branches="/mnt/a=RW:/mnt/b=RW"
  983. user.mergerfs.minfreespace="4294967295"
  984. user.mergerfs.moveonenospc="false"
  985. ...
  986. [trapexit:/mnt/mergerfs] $ getfattr -n user.mergerfs.category.search .mergerfs
  987. user.mergerfs.category.search="ff"
  988. [trapexit:/mnt/mergerfs] $ setfattr -n user.mergerfs.category.search -v newest .mergerfs
  989. [trapexit:/mnt/mergerfs] $ getfattr -n user.mergerfs.category.search .mergerfs
  990. user.mergerfs.category.search="newest"
  991. ```
  992. #### file / directory xattrs ####
  993. While they won't show up when using `getfattr` **mergerfs** offers a
  994. number of special xattrs to query information about the files
  995. served. To access the values you will need to issue a
  996. [getxattr](http://linux.die.net/man/2/getxattr) for one of the
  997. following:
  998. * **user.mergerfs.basepath**: the base mount point for the file given the current getattr policy
  999. * **user.mergerfs.relpath**: the relative path of the file from the perspective of the mount point
  1000. * **user.mergerfs.fullpath**: the full path of the original file given the getattr policy
  1001. * **user.mergerfs.allpaths**: a NUL ('\0') separated list of full paths to all files found
  1002. ## SIGNALS
  1003. * USR1: This will cause mergerfs to send invalidation notifications to
  1004. the kernel for all files. This will cause all unused files to be
  1005. released from memory.
  1006. * USR2: Trigger a general cleanup of currently unused memory. A more
  1007. thorough version of what happens every ~15 minutes.
  1008. ## IOCTLS
  1009. Found in `fuse_ioctl.cpp`:
  1010. ```C++
  1011. typedef char IOCTL_BUF[4096];
  1012. #define IOCTL_APP_TYPE 0xDF
  1013. #define IOCTL_FILE_INFO _IOWR(IOCTL_APP_TYPE,0,IOCTL_BUF)
  1014. #define IOCTL_GC _IO(IOCTL_APP_TYPE,1)
  1015. #define IOCTL_GC1 _IO(IOCTL_APP_TYPE,2)
  1016. #define IOCTL_INVALIDATE_ALL_NODES _IO(IOCTL_APP_TYPE,3)
  1017. ```
  1018. * IOCTL\_FILE\_INFO: Same as the "file / directory xattrs" mentioned
  1019. above. Use a buffer size of 4096 bytes. Pass in a string of
  1020. "basepath", "relpath", "fullpath", or "allpaths". Receive details in
  1021. same buffer.
  1022. * IOCTL\_GC: Triggers a thorough garbage collection of excess
  1023. memory. Same as SIGUSR2.
  1024. * IOCTL\_GC1: Triggers a simple garbage collection of excess
  1025. memory. Same as what happens every 15 minutes normally.
  1026. * IOCTL\_INVALIDATE\_ALL\_NODES: Same as SIGUSR1. Send invalidation
  1027. notifications to the kernel for all files causing unused files to be
  1028. released from memory.
  1029. # TOOLING
  1030. ## preload.so
  1031. EXPERIMENTAL
  1032. For some time there has been work to enable passthrough IO in
  1033. FUSE. Passthrough IO would allow for near native performance with
  1034. regards to reads and writes (at the expense of certain mergerfs
  1035. features.) However, there have been several complications which have
  1036. kept the feature from making it into the mainline Linux kernel. Until
  1037. that feature is available there are two methods to provide similar
  1038. functionality. One method is using the LD_PRELOAD feature of the
  1039. dynamic linker. The other leveraging ptrace to intercept
  1040. syscalls. Each has their disadvantages. At the moment only a preload
  1041. based tool is available. A ptrace based tool may be developed later if
  1042. there is a need.
  1043. `/usr/lib/mergerfs/preload.so`
  1044. This [preloadable
  1045. library](https://man7.org/linux/man-pages/man8/ld.so.8.html#ENVIRONMENT)
  1046. overrides the creation and opening of files in order to simulate
  1047. passthrough file IO. It catches the open/creat/fopen calls, has
  1048. mergerfs do the call, queries mergerfs for the branch the file exists
  1049. on, reopens the file on the underlying filesystem and returns that
  1050. instead. Meaning that you will get native read/write performance
  1051. because mergerfs is no longer part of the workflow. Keep in mind that
  1052. this also means certain mergerfs features that work by interrupting
  1053. the read/write workflow, such as `moveonenospc`, will no longer work.
  1054. Also understand that this will only work on dynamically linked
  1055. software. Anything statically compiled will not work. Many GoLang and
  1056. Rust apps are statically compiled.
  1057. The library will not interfere with non-mergerfs filesystems. The
  1058. library is written to always fallback to returning the mergerfs opened
  1059. file on error.
  1060. While the library was written to account for a number of edgecases
  1061. there could be some yet accounted for so please report any oddities.
  1062. Thank you to
  1063. [nohajc](https://github.com/nohajc/mergerfs-io-passthrough) for
  1064. prototyping the idea.
  1065. ### general usage
  1066. ```sh
  1067. LD_PRELOAD=/usr/lib/mergerfs/preload.so touch /mnt/mergerfs/filename
  1068. ```
  1069. ### Docker usage
  1070. Assume `/mnt/fs0` and `/mnt/fs1` are pooled with mergerfs at `/media`.
  1071. All mergerfs branch paths *must* be bind mounted into the container at
  1072. the same path as found on the host so the preload library can see them.
  1073. ```sh
  1074. docker run \
  1075. -e LD_PRELOAD=/usr/lib/mergerfs/preload.so \
  1076. -v /usr/lib/mergerfs/preload.so:/usr/lib/mergerfs/preload.so:ro \
  1077. -v /media:/data \
  1078. -v /mnt:/mnt \
  1079. ubuntu:latest \
  1080. bash
  1081. ```
  1082. or more explicitly
  1083. ```sh
  1084. docker run \
  1085. -e LD_PRELOAD=/usr/lib/mergerfs/preload.so \
  1086. -v /usr/lib/mergerfs/preload.so:/usr/lib/mergerfs/preload.so:ro \
  1087. -v /media:/data \
  1088. -v /mnt/fs0:/mnt/fs0 \
  1089. -v /mnt/fs1:/mnt/fs1 \
  1090. ubuntu:latest \
  1091. bash
  1092. ```
  1093. ### systemd unit
  1094. Use the `Environment` option to set the LD_PRELOAD variable.
  1095. * https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#Command%20lines
  1096. * https://serverfault.com/questions/413397/how-to-set-environment-variable-in-systemd-service
  1097. ```
  1098. [Service]
  1099. Environment=LD_PRELOAD=/usr/lib/mergerfs/preload.so
  1100. ```
  1101. ## Misc
  1102. * https://github.com/trapexit/mergerfs-tools
  1103. * mergerfs.ctl: A tool to make it easier to query and configure mergerfs at runtime
  1104. * mergerfs.fsck: Provides permissions and ownership auditing and the ability to fix them
  1105. * mergerfs.dedup: Will help identify and optionally remove duplicate files
  1106. * mergerfs.dup: Ensure there are at least N copies of a file across the pool
  1107. * mergerfs.balance: Rebalance files across filesystems by moving them from the most filled to the least filled
  1108. * mergerfs.consolidate: move files within a single mergerfs directory to the filesystem with most free space
  1109. * https://github.com/trapexit/scorch
  1110. * scorch: A tool to help discover silent corruption of files and keep track of files
  1111. * https://github.com/trapexit/bbf
  1112. * bbf (bad block finder): a tool to scan for and 'fix' hard drive bad blocks and find the files using those blocks
  1113. # CACHING
  1114. #### page caching
  1115. https://en.wikipedia.org/wiki/Page_cache
  1116. * cache.files=off: Disables page caching. Underlying files cached,
  1117. mergerfs files are not.
  1118. * cache.files=partial: Enables page caching. Underlying files cached,
  1119. mergerfs files cached while open.
  1120. * cache.files=full: Enables page caching. Underlying files cached,
  1121. mergerfs files cached across opens.
  1122. * cache.files=auto-full: Enables page caching. Underlying files
  1123. cached, mergerfs files cached across opens if mtime and size are
  1124. unchanged since previous open.
  1125. * cache.files=libfuse: follow traditional libfuse `direct_io`,
  1126. `kernel_cache`, and `auto_cache` arguments.
  1127. * cache.files=per-process: Enable page caching (equivalent to
  1128. `cache.files=partial`) only for processes whose 'comm' name matches
  1129. one of the values defined in `cache.files.process-names`. If the
  1130. name does not match the file open is equivalent to
  1131. `cache.files=off`.
  1132. FUSE, which mergerfs uses, offers a number of page caching modes. mergerfs tries to simplify their use via the `cache.files`
  1133. option. It can and should replace usage of `direct_io`,
  1134. `kernel_cache`, and `auto_cache`.
  1135. Due to mergerfs using FUSE and therefore being a userland process
  1136. proxying existing filesystems the kernel will double cache the content
  1137. being read and written through mergerfs. Once from the underlying
  1138. filesystem and once from mergerfs (it sees them as two separate
  1139. entities). Using `cache.files=off` will keep the double caching from
  1140. happening by disabling caching of mergerfs but this has the side
  1141. effect that *all* read and write calls will be passed to mergerfs
  1142. which may be slower than enabling caching, you lose shared `mmap`
  1143. support which can affect apps such as rtorrent, and no read-ahead will
  1144. take place. The kernel will still cache the underlying filesystem data
  1145. but that only helps so much given mergerfs will still process all
  1146. requests.
  1147. If you do enable file page caching,
  1148. `cache.files=partial|full|auto-full`, you should also enable
  1149. `dropcacheonclose` which will cause mergerfs to instruct the kernel to
  1150. flush the underlying file's page cache when the file is closed. This
  1151. behavior is the same as the rsync fadvise / drop cache patch and Feh's
  1152. nocache project.
  1153. If most files are read once through and closed (like media) it is best
  1154. to enable `dropcacheonclose` regardless of caching mode in order to
  1155. minimize buffer bloat.
  1156. It is difficult to balance memory usage, cache bloat & duplication,
  1157. and performance. Ideally mergerfs would be able to disable caching for
  1158. the files it reads/writes but allow page caching for itself. That
  1159. would limit the FUSE overhead. However, there isn't a good way to
  1160. achieve this. It would need to open all files with O_DIRECT which
  1161. places limitations on the what underlying filesystems would be
  1162. supported and complicates the code.
  1163. kernel documentation: https://www.kernel.org/doc/Documentation/filesystems/fuse-io.txt
  1164. #### entry & attribute caching
  1165. Given the relatively high cost of FUSE due to the kernel <-> userspace
  1166. round trips there are kernel side caches for file entries and
  1167. attributes. The entry cache limits the `lookup` calls to mergerfs
  1168. which ask if a file exists. The attribute cache limits the need to
  1169. make `getattr` calls to mergerfs which provide file attributes (mode,
  1170. size, type, etc.). As with the page cache these should not be used if
  1171. the underlying filesystems are being manipulated at the same time as
  1172. it could lead to odd behavior or data corruption. The options for
  1173. setting these are `cache.entry` and `cache.negative_entry` for the
  1174. entry cache and `cache.attr` for the attributes
  1175. cache. `cache.negative_entry` refers to the timeout for negative
  1176. responses to lookups (non-existent files).
  1177. #### writeback caching
  1178. When `cache.files` is enabled the default is for it to perform
  1179. writethrough caching. This behavior won't help improve performance as
  1180. each write still goes one for one through the filesystem. By enabling
  1181. the FUSE writeback cache small writes may be aggregated by the kernel
  1182. and then sent to mergerfs as one larger request. This can greatly
  1183. improve the throughput for apps which write to files
  1184. inefficiently. The amount the kernel can aggregate is limited by the
  1185. size of a FUSE message. Read the `fuse_msg_size` section for more
  1186. details.
  1187. There is a small side effect as a result of enabling writeback
  1188. caching. Underlying files won't ever be opened with O_APPEND or
  1189. O_WRONLY. The former because the kernel then manages append mode and
  1190. the latter because the kernel may request file data from mergerfs to
  1191. populate the write cache. The O_APPEND change means that if a file is
  1192. changed outside of mergerfs it could lead to corruption as the kernel
  1193. won't know the end of the file has changed. That said any time you use
  1194. caching you should keep from using the same file outside of mergerfs
  1195. at the same time.
  1196. Note that if an application is properly sizing writes then writeback
  1197. caching will have little or no effect. It will only help with writes
  1198. of sizes below the FUSE message size (128K on older kernels, 1M on
  1199. newer).
  1200. #### statfs caching
  1201. Of the syscalls used by mergerfs in policies the `statfs` / `statvfs`
  1202. call is perhaps the most expensive. It's used to find out the
  1203. available space of a filesystem and whether it is mounted
  1204. read-only. Depending on the setup and usage pattern these queries can
  1205. be relatively costly. When `cache.statfs` is enabled all calls to
  1206. `statfs` by a policy will be cached for the number of seconds its set
  1207. to.
  1208. Example: If the create policy is `mfs` and the timeout is 60 then for
  1209. that 60 seconds the same filesystem will be returned as the target for
  1210. creates because the available space won't be updated for that time.
  1211. #### symlink caching
  1212. As of version 4.20 Linux supports symlink caching. Significant
  1213. performance increases can be had in workloads which use a lot of
  1214. symlinks. Setting `cache.symlinks=true` will result in requesting
  1215. symlink caching from the kernel only if supported. As a result its
  1216. safe to enable it on systems prior to 4.20. That said it is disabled
  1217. by default for now. You can see if caching is enabled by querying the
  1218. xattr `user.mergerfs.cache.symlinks` but given it must be requested at
  1219. startup you can not change it at runtime.
  1220. #### readdir caching
  1221. As of version 4.20 Linux supports readdir caching. This can have a
  1222. significant impact on directory traversal. Especially when combined
  1223. with entry (`cache.entry`) and attribute (`cache.attr`)
  1224. caching. Setting `cache.readdir=true` will result in requesting
  1225. readdir caching from the kernel on each `opendir`. If the kernel
  1226. doesn't support readdir caching setting the option to `true` has no
  1227. effect. This option is configurable at runtime via xattr
  1228. `user.mergerfs.cache.readdir`.
  1229. #### tiered caching
  1230. Some storage technologies support what some call "tiered" caching. The
  1231. placing of usually smaller, faster storage as a transparent cache to
  1232. larger, slower storage. NVMe, SSD, Optane in front of traditional HDDs
  1233. for instance.
  1234. mergerfs does not natively support any sort of tiered caching. Most
  1235. users have no use for such a feature and its inclusion would
  1236. complicate the code. However, there are a few situations where a cache
  1237. filesystem could help with a typical mergerfs setup.
  1238. 1. Fast network, slow filesystems, many readers: You've a 10+Gbps network
  1239. with many readers and your regular filesystems can't keep up.
  1240. 2. Fast network, slow filesystems, small'ish bursty writes: You have a
  1241. 10+Gbps network and wish to transfer amounts of data less than your
  1242. cache filesystem but wish to do so quickly.
  1243. With #1 it's arguable if you should be using mergerfs at all. RAID
  1244. would probably be the better solution. If you're going to use mergerfs
  1245. there are other tactics that may help: spreading the data across
  1246. filesystems (see the mergerfs.dup tool) and setting `func.open=rand`,
  1247. using `symlinkify`, or using dm-cache or a similar technology to add
  1248. tiered cache to the underlying device.
  1249. With #2 one could use dm-cache as well but there is another solution
  1250. which requires only mergerfs and a cronjob.
  1251. 1. Create 2 mergerfs pools. One which includes just the slow devices
  1252. and one which has both the fast devices (SSD,NVME,etc.) and slow
  1253. devices.
  1254. 2. The 'cache' pool should have the cache filesystems listed first.
  1255. 3. The best `create` policies to use for the 'cache' pool would
  1256. probably be `ff`, `epff`, `lfs`, or `eplfs`. The latter two under
  1257. the assumption that the cache filesystem(s) are far smaller than the
  1258. backing filesystems. If using path preserving policies remember that
  1259. you'll need to manually create the core directories of those paths
  1260. you wish to be cached. Be sure the permissions are in sync. Use
  1261. `mergerfs.fsck` to check / correct them. You could also set the
  1262. slow filesystems mode to `NC` though that'd mean if the cache
  1263. filesystems fill you'd get "out of space" errors.
  1264. 4. Enable `moveonenospc` and set `minfreespace` appropriately. To make
  1265. sure there is enough room on the "slow" pool you might want to set
  1266. `minfreespace` to at least as large as the size of the largest
  1267. cache filesystem if not larger. This way in the worst case the
  1268. whole of the cache filesystem(s) can be moved to the other drives.
  1269. 5. Set your programs to use the cache pool.
  1270. 6. Save one of the below scripts or create you're own.
  1271. 7. Use `cron` (as root) to schedule the command at whatever frequency
  1272. is appropriate for your workflow.
  1273. ##### time based expiring
  1274. Move files from cache to backing pool based only on the last time the
  1275. file was accessed. Replace `-atime` with `-amin` if you want minutes
  1276. rather than days. May want to use the `fadvise` / `--drop-cache`
  1277. version of rsync or run rsync with the tool "nocache".
  1278. *NOTE:* The arguments to these scripts include the cache
  1279. **filesystem** itself. Not the pool with the cache filesystem. You
  1280. could have data loss if the source is the cache pool.
  1281. [mergerfs.time-based-mover](tools/mergerfs.time-based-mover?raw=1)
  1282. ##### percentage full expiring
  1283. Move the oldest file from the cache to the backing pool. Continue till
  1284. below percentage threshold.
  1285. *NOTE:* The arguments to these scripts include the cache
  1286. **filesystem** itself. Not the pool with the cache filesystem. You
  1287. could have data loss if the source is the cache pool.
  1288. [mergerfs.percent-full-mover](tools/mergerfs.percent-full-mover?raw=1)
  1289. # PERFORMANCE
  1290. mergerfs is at its core just a proxy and therefore its theoretical max
  1291. performance is that of the underlying devices. However, given it is a
  1292. FUSE filesystem working from userspace there is an increase in
  1293. overhead relative to kernel based solutions. That said the performance
  1294. can match the theoretical max but it depends greatly on the system's
  1295. configuration. Especially when adding network filesystems into the mix
  1296. there are many variables which can impact performance. Device speeds
  1297. and latency, network speeds and latency, general concurrency,
  1298. read/write sizes, etc. Unfortunately, given the number of variables it
  1299. has been difficult to find a single set of settings which provide
  1300. optimal performance. If you're having performance issues please look
  1301. over the suggestions below (including the benchmarking section.)
  1302. NOTE: be sure to read about these features before changing them to
  1303. understand what behaviors it may impact
  1304. * test theoretical performance using `nullrw` or mounting a ram disk
  1305. * enable `passthrough`
  1306. * disable `security_capability` and/or `xattr`
  1307. * increase cache timeouts `cache.attr`, `cache.entry`, `cache.negative_entry`
  1308. * enable (or disable) page caching (`cache.files`)
  1309. * enable `parallel-direct-writes`
  1310. * enable `cache.writeback`
  1311. * enable `cache.statfs`
  1312. * enable `cache.symlinks`
  1313. * enable `cache.readdir`
  1314. * change the number of worker threads
  1315. * disable `posix_acl`
  1316. * disable `async_read`
  1317. * use `symlinkify` if your data is largely static and read-only
  1318. * use tiered cache devices
  1319. * use LVM and LVM cache to place a SSD in front of your HDDs
  1320. * increase readahead: `readahead=1024`
  1321. If you come across a setting that significantly impacts performance
  1322. please contact trapexit so he may investigate further. Please test
  1323. both against your normal setup, a singular branch, and with
  1324. `nullrw=true`
  1325. # BENCHMARKING
  1326. Filesystems are complicated. They do many things and many of those are
  1327. interconnected. Additionally, the OS, drivers, hardware, etc. all can
  1328. impact performance. Therefore, when benchmarking, it is **necessary**
  1329. that the test focus as narrowly as possible.
  1330. For most throughput is the key benchmark. To test throughput `dd` is
  1331. useful but **must** be used with the correct settings in order to
  1332. ensure the filesystem or device is actually being tested. The OS can
  1333. and will cache data. Without forcing synchronous reads and writes
  1334. and/or disabling caching the values returned will not be
  1335. representative of the device's true performance.
  1336. When benchmarking through mergerfs ensure you only use 1 branch to
  1337. remove any possibility of the policies complicating the
  1338. situation. Benchmark the underlying filesystem first and then mount
  1339. mergerfs over it and test again. If you're experience speeds below
  1340. your expectation you will need to narrow down precisely which
  1341. component is leading to the slowdown. Preferably test the following in
  1342. the order listed (but not combined).
  1343. 1. Enable `nullrw` mode with `nullrw=true`. This will effectively make
  1344. reads and writes no-ops. Removing the underlying device /
  1345. filesystem from the equation. This will give us the top theoretical
  1346. speeds.
  1347. 2. Mount mergerfs over `tmpfs`. `tmpfs` is a RAM disk. Extremely high
  1348. speed and very low latency. This is a more realistic best case
  1349. scenario. Example: `mount -t tmpfs -o size=2G tmpfs /tmp/tmpfs`
  1350. 3. Mount mergerfs over a local device. NVMe, SSD, HDD, etc. If you
  1351. have more than one I'd suggest testing each of them as drives
  1352. and/or controllers (their drivers) could impact performance.
  1353. 4. Finally, if you intend to use mergerfs with a network filesystem,
  1354. either as the source of data or to combine with another through
  1355. mergerfs, test each of those alone as above.
  1356. Once you find the component which has the performance issue you can do
  1357. further testing with different options to see if they impact
  1358. performance. For reads and writes the most relevant would be:
  1359. `cache.files`, `async_read`. Less likely but relevant when using NFS
  1360. or with certain filesystems would be `security_capability`, `xattr`,
  1361. and `posix_acl`. If you find a specific system, device, filesystem,
  1362. controller, etc. that performs poorly contact trapexit so he may
  1363. investigate further.
  1364. Sometimes the problem is really the application accessing or writing
  1365. data through mergerfs. Some software use small buffer sizes which can
  1366. lead to more requests and therefore greater overhead. You can test
  1367. this out yourself by replace `bs=1M` in the examples below with `ibs`
  1368. or `obs` and using a size of `512` instead of `1M`. In one example
  1369. test using `nullrw` the write speed dropped from 4.9GB/s to 69.7MB/s
  1370. when moving from `1M` to `512`. Similar results were had when testing
  1371. reads. Small writes overhead may be improved by leveraging a write
  1372. cache but in casual tests little gain was found. More tests will need
  1373. to be done before this feature would become available. If you have an
  1374. app that appears slow with mergerfs it could be due to this. Contact
  1375. trapexit so he may investigate further.
  1376. ### write benchmark
  1377. ```
  1378. $ dd if=/dev/zero of=/mnt/mergerfs/1GB.file bs=1M count=1024 oflag=nocache conv=fdatasync status=progress
  1379. ```
  1380. ### read benchmark
  1381. ```
  1382. $ dd if=/mnt/mergerfs/1GB.file of=/dev/null bs=1M count=1024 iflag=nocache conv=fdatasync status=progress
  1383. ```
  1384. ### other benchmarks
  1385. If you are attempting to benchmark other behaviors you must ensure you
  1386. clear kernel caches before runs. In fact it would be a good deal to
  1387. run before the read and write benchmarks as well just in case.
  1388. ```
  1389. sync
  1390. echo 3 | sudo tee /proc/sys/vm/drop_caches
  1391. ```
  1392. # TIPS / NOTES
  1393. * This document is literal and thorough. If a suspected feature isn't
  1394. mentioned it doesn't exist. If certain libfuse arguments aren't
  1395. listed they probably shouldn't be used.
  1396. * Ensure you're using the latest version.
  1397. * Run mergerfs as `root`. mergerfs is designed and intended to be run
  1398. as `root` and may exibit incorrect behavior if run otherwise..
  1399. * If you don't see some directories and files you expect, policies
  1400. seem to skip branches, you get strange permission errors, etc. be
  1401. sure the underlying filesystems' permissions are all the same. Use
  1402. `mergerfs.fsck` to audit the filesystem for out of sync permissions.
  1403. * If you still have permission issues be sure you are using POSIX ACL
  1404. compliant filesystems. mergerfs doesn't generally make exceptions
  1405. for FAT, NTFS, or other non-POSIX filesystem.
  1406. * Do **not** use `cache.files=off` if you expect applications (such as
  1407. rtorrent) to use [mmap](http://linux.die.net/man/2/mmap)
  1408. files. Shared mmap is not currently supported in FUSE w/ page
  1409. caching disabled. Enabling `dropcacheonclose` is recommended when
  1410. `cache.files=partial|full|auto-full`.
  1411. * [Kodi](http://kodi.tv), [Plex](http://plex.tv),
  1412. [Subsonic](http://subsonic.org), etc. can use directory
  1413. [mtime](http://linux.die.net/man/2/stat) to more efficiently
  1414. determine whether to scan for new content rather than simply
  1415. performing a full scan. If using the default **getattr** policy of
  1416. **ff** it's possible those programs will miss an update on account
  1417. of it returning the first directory found's **stat** info and it's a
  1418. later directory on another mount which had the **mtime** recently
  1419. updated. To fix this you will want to set
  1420. **func.getattr=newest**. Remember though that this is just
  1421. **stat**. If the file is later **open**'ed or **unlink**'ed and the
  1422. policy is different for those then a completely different file or
  1423. directory could be acted on.
  1424. * Some policies mixed with some functions may result in strange
  1425. behaviors. Not that some of these behaviors and race conditions
  1426. couldn't happen outside **mergerfs** but that they are far more
  1427. likely to occur on account of the attempt to merge together multiple
  1428. sources of data which could be out of sync due to the different
  1429. policies.
  1430. * For consistency its generally best to set **category** wide policies
  1431. rather than individual **func**'s. This will help limit the
  1432. confusion of tools such as
  1433. [rsync](http://linux.die.net/man/1/rsync). However, the flexibility
  1434. is there if needed.
  1435. # KNOWN ISSUES / BUGS
  1436. #### kernel issues & bugs
  1437. [https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs](https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs)
  1438. #### directory mtime is not being updated
  1439. Remember that the default policy for `getattr` is `ff`. The
  1440. information for the first directory found will be returned. If it
  1441. wasn't the directory which had been updated then it will appear
  1442. outdated.
  1443. The reason this is the default is because any other policy would be
  1444. more expensive and for many applications it is unnecessary. To always
  1445. return the directory with the most recent mtime or a faked value based
  1446. on all found would require a scan of all filesystems.
  1447. If you always want the directory information from the one with the
  1448. most recent mtime then use the `newest` policy for `getattr`.
  1449. #### 'mv /mnt/pool/foo /mnt/disk1/foo' removes 'foo'
  1450. This is not a bug.
  1451. Run in verbose mode to better understand what's happening:
  1452. ```
  1453. $ mv -v /mnt/pool/foo /mnt/disk1/foo
  1454. copied '/mnt/pool/foo' -> '/mnt/disk1/foo'
  1455. removed '/mnt/pool/foo'
  1456. $ ls /mnt/pool/foo
  1457. ls: cannot access '/mnt/pool/foo': No such file or directory
  1458. ```
  1459. `mv`, when working across devices, is copying the source to target and
  1460. then removing the source. Since the source **is** the target in this
  1461. case, depending on the unlink policy, it will remove the just copied
  1462. file and other files across the branches.
  1463. If you want to move files to one filesystem just copy them there and
  1464. use mergerfs.dedup to clean up the old paths or manually remove them
  1465. from the branches directly.
  1466. #### cached memory appears greater than it should be
  1467. Use `cache.files=off` and/or `dropcacheonclose=true`. See the section
  1468. on page caching.
  1469. #### NFS clients returning ESTALE / Stale file handle
  1470. NFS generally does not like out of band changes. Take a look at the
  1471. section on NFS in the [#remote-filesystems](#remote-filesystems) for
  1472. more details.
  1473. #### rtorrent fails with ENODEV (No such device)
  1474. Be sure to set
  1475. `cache.files=partial|full|auto-full|per-process`. rtorrent and some
  1476. other applications use [mmap](http://linux.die.net/man/2/mmap) to read
  1477. and write to files and offer no fallback to traditional methods. FUSE
  1478. does not currently support mmap while using `direct_io`. There may be
  1479. a performance penalty on writes with `direct_io` off as well as the
  1480. problem of double caching but it's the only way to get such
  1481. applications to work. If the performance loss is too high for other
  1482. apps you can mount mergerfs twice. Once with `direct_io` enabled and
  1483. one without it. Be sure to set `dropcacheonclose=true` if not using
  1484. `direct_io`.
  1485. #### Plex doesn't work with mergerfs
  1486. It does. If you're trying to put Plex's config / metadata / database
  1487. on mergerfs you can't set `cache.files=off` because Plex is using
  1488. sqlite3 with mmap enabled. Shared mmap is not supported by Linux's
  1489. FUSE implementation when page caching is disabled. To fix this place
  1490. the data elsewhere (preferable) or enable `cache.files` (with
  1491. `dropcacheonclose=true`). Sqlite3 does not need mmap but the developer
  1492. needs to fall back to standard IO if mmap fails.
  1493. This applies to other software: Radarr, Sonarr, Lidarr, Jellyfin, etc.
  1494. I would recommend reaching out to the developers of the software
  1495. you're having troubles with and asking them to add a fallback to
  1496. regular file IO when mmap is unavailable.
  1497. If the issue is that scanning doesn't seem to pick up media then be
  1498. sure to set `func.getattr=newest` though generally a full scan will
  1499. pick up all media anyway.
  1500. #### When a program tries to move or rename a file it fails
  1501. Please read the section above regarding [rename & link](#rename--link).
  1502. The problem is that many applications do not properly handle `EXDEV`
  1503. errors which `rename` and `link` may return even though they are
  1504. perfectly valid situations which do not indicate actual device,
  1505. filesystem, or OS errors. The error will only be returned by mergerfs
  1506. if using a path preserving policy as described in the policy section
  1507. above. If you do not care about path preservation simply change the
  1508. mergerfs policy to the non-path preserving version. For example: `-o
  1509. category.create=mfs` Ideally the offending software would be fixed and
  1510. it is recommended that if you run into this problem you contact the
  1511. software's author and request proper handling of `EXDEV` errors.
  1512. #### my 32bit software has problems
  1513. Some software have problems with 64bit inode values. The symptoms can
  1514. include EOVERFLOW errors when trying to list files. You can address
  1515. this by setting `inodecalc` to one of the 32bit based algos as
  1516. described in the relevant section.
  1517. #### Samba: Moving files / directories fails
  1518. Workaround: Copy the file/directory and then remove the original
  1519. rather than move.
  1520. This isn't an issue with Samba but some SMB clients. GVFS-fuse v1.20.3
  1521. and prior (found in Ubuntu 14.04 among others) failed to handle
  1522. certain error codes correctly. Particularly **STATUS_NOT_SAME_DEVICE**
  1523. which comes from the **EXDEV** which is returned by **rename** when
  1524. the call is crossing mount points. When a program gets an **EXDEV** it
  1525. needs to explicitly take an alternate action to accomplish its
  1526. goal. In the case of **mv** or similar it tries **rename** and on
  1527. **EXDEV** falls back to a manual copying of data between the two
  1528. locations and unlinking the source. In these older versions of
  1529. GVFS-fuse if it received **EXDEV** it would translate that into
  1530. **EIO**. This would cause **mv** or most any application attempting to
  1531. move files around on that SMB share to fail with a IO error.
  1532. [GVFS-fuse v1.22.0](https://bugzilla.gnome.org/show_bug.cgi?id=734568)
  1533. and above fixed this issue but a large number of systems use the older
  1534. release. On Ubuntu the version can be checked by issuing `apt-cache
  1535. showpkg gvfs-fuse`. Most distros released in 2015 seem to have the
  1536. updated release and will work fine but older systems may
  1537. not. Upgrading gvfs-fuse or the distro in general will address the
  1538. problem.
  1539. In Apple's MacOSX 10.9 they replaced Samba (client and server) with
  1540. their own product. It appears their new client does not handle
  1541. **EXDEV** either and responds similar to older release of gvfs on
  1542. Linux.
  1543. #### Trashing files occasionally fails
  1544. This is the same issue as with Samba. `rename` returns `EXDEV` (in our
  1545. case that will really only happen with path preserving policies like
  1546. `epmfs`) and the software doesn't handle the situation well. This is
  1547. unfortunately a common failure of software which moves files
  1548. around. The standard indicates that an implementation `MAY` choose to
  1549. support non-user home directory trashing of files (which is a
  1550. `MUST`). The implementation `MAY` also support "top directory trashes"
  1551. which many probably do.
  1552. To create a `$topdir/.Trash` directory as defined in the standard use
  1553. the [mergerfs-tools](https://github.com/trapexit/mergerfs-tools) tool
  1554. `mergerfs.mktrash`.
  1555. #### Supplemental user groups
  1556. Due to the overhead of
  1557. [getgroups/setgroups](http://linux.die.net/man/2/setgroups) mergerfs
  1558. utilizes a cache. This cache is opportunistic and per thread. Each
  1559. thread will query the supplemental groups for a user when that
  1560. particular thread needs to change credentials and will keep that data
  1561. for the lifetime of the thread. This means that if a user is added to
  1562. a group it may not be picked up without the restart of
  1563. mergerfs. However, since the high level FUSE API's (at least the
  1564. standard version) thread pool dynamically grows and shrinks it's
  1565. possible that over time a thread will be killed and later a new thread
  1566. with no cache will start and query the new data.
  1567. The gid cache uses fixed storage to simplify the design and be
  1568. compatible with older systems which may not have C++11
  1569. compilers. There is enough storage for 256 users' supplemental
  1570. groups. Each user is allowed up to 32 supplemental groups. Linux >=
  1571. 2.6.3 allows up to 65535 groups per user but most other *nixs allow
  1572. far less. NFS allowing only 16. The system does handle overflow
  1573. gracefully. If the user has more than 32 supplemental groups only the
  1574. first 32 will be used. If more than 256 users are using the system
  1575. when an uncached user is found it will evict an existing user's cache
  1576. at random. So long as there aren't more than 256 active users this
  1577. should be fine. If either value is too low for your needs you will
  1578. have to modify `gidcache.hpp` to increase the values. Note that doing
  1579. so will increase the memory needed by each thread.
  1580. While not a bug some users have found when using containers that
  1581. supplemental groups defined inside the container don't work properly
  1582. with regard to permissions. This is expected as mergerfs lives outside
  1583. the container and therefore is querying the host's group
  1584. database. There might be a hack to work around this (make mergerfs
  1585. read the /etc/group file in the container) but it is not yet
  1586. implemented and would be limited to Linux and the /etc/group
  1587. DB. Preferably users would mount in the host group file into the
  1588. containers or use a standard shared user & groups technology like NIS
  1589. or LDAP.
  1590. # Remote Filesystems
  1591. Many users ask about compatibility with remote filesystems. This
  1592. section is to describe any known issues or quirks when using mergerfs
  1593. with common remote filesystems.
  1594. Keep in mind that, like with caching, it is not a good idea to change
  1595. the contents of the remote filesystem
  1596. [out-of-band](https://en.wikipedia.org/wiki/Out-of-band). Meaning that
  1597. you really shouldn't change the contents of the underlying
  1598. filesystems or mergerfs on the server hosting the remote
  1599. filesystem. Doing so can lead to weird behavior, inconsistency,
  1600. errors, and even data corruption should multiple programs try to write
  1601. or read the same data at the same time. This isn't to say you can't do
  1602. it or that data corruption is likely but it *could* happen. It is
  1603. better to always use the remote filesystem. Even on the machine
  1604. serving it.
  1605. ## NFS
  1606. [NFS](https://en.wikipedia.org/wiki/Network_File_System) is a common
  1607. remote filesystem on Unix/POSIX systems. Due to how NFS works there
  1608. are some settings which need to be set in order for mergerfs to work
  1609. with it.
  1610. It should be noted that NFS and FUSE (the technology mergerfs uses) do
  1611. not work perfectly with one another due to certain design choices in
  1612. FUSE (and mergerfs.) Due to these issues it is generally recommended
  1613. to use SMB when possible till situations change. That said mergerfs
  1614. should generally work as an export of NFS and issues discovered should
  1615. still be reported.
  1616. To ensure compatibility between mergerfs and NFS use the following
  1617. settings.
  1618. mergerfs settings:
  1619. * noforget
  1620. * inodecalc=path-hash
  1621. NFS export settings:
  1622. * fsid=UUID
  1623. * no\_root\_squash
  1624. `noforget` is needed because NFS uses the `name_to_handle_at` and
  1625. `open_by_handle_at` functions which allow a program to keep a
  1626. reference to a file without technically having it open in the typical
  1627. sense. The problem is that FUSE has no way to know that NFS has a
  1628. handle that it will later use to open the file again. As a result it
  1629. is possible for the kernel to tell mergerfs to forget about the node
  1630. and should NFS ever ask for that node's details in the future it would
  1631. have nothing to respond with. Keeping nodes around forever is not
  1632. ideal but at the moment the only way to manage the situation.
  1633. `inodecalc=path-hash` is needed because NFS is sensitive to
  1634. out-of-band changes. FUSE doesn't care if a file's inode value changes
  1635. but NFS, being stateful, does. So if you used the default inode
  1636. calculation algorithm then it is possible that if you changed a file
  1637. or updated a directory the file mergerfs will use will be on a
  1638. different branch and therefore the inode would change. This isn't an
  1639. ideal solution and others are being considered but it works for most
  1640. situations.
  1641. `fsid=UUID` is needed because FUSE filesystems don't have different
  1642. `st_dev` values which can cause issues when exporting. The easiest
  1643. thing to do is set each mergerfs export `fsid` to some random
  1644. value. An easy way to generate a random value is to use the command
  1645. line tool `uuid` or `uuidgen` or through a website such as
  1646. [uuidgenerator.net](https://www.uuidgenerator.net/).
  1647. `no_root_squash` is not strictly necessary but can lead to confusing
  1648. permission and ownership issues if root squashing is enabled.
  1649. ## SMB / CIFS
  1650. [SMB](https://en.wikipedia.org/wiki/Server_Message_Block) is a
  1651. protocol most used by Microsoft Windows systems to share file shares,
  1652. printers, etc. However, due to the popularity for Windows, it is also
  1653. supported on many other platforms including Linux. The most popular
  1654. way of supporting SMB on Linux is via the software Samba.
  1655. [Samba](https://en.wikipedia.org/wiki/Samba_(software)), and other
  1656. ways of serving Linux filesystems, via SMB should work fine with
  1657. mergerfs. The services do not tend to use the same technologies which
  1658. NFS uses and therefore don't have the same issues. There should not be
  1659. an special settings required to use mergerfs with Samba. However,
  1660. [CIFSD](https://en.wikipedia.org/wiki/CIFSD) and other programs have
  1661. not been extensively tested. If you use mergerfs with CIFSD or other
  1662. SMB servers please submit your experiences so these docs can be
  1663. updated.
  1664. ## SSHFS
  1665. [SSHFS](https://en.wikipedia.org/wiki/SSHFS) is a FUSE filesystem
  1666. leveraging SSH as the connection and transport layer. While often
  1667. simpler to setup when compared to NFS or Samba the performance can be
  1668. lacking and the project is very much in maintenance mode.
  1669. There are no known issues using sshfs with mergerfs. You may want to
  1670. use the following arguments to improve performance but your millage
  1671. may vary.
  1672. * `-o Ciphers=arcfour`
  1673. * `-o Compression=no`
  1674. More info can be found
  1675. [here](https://ideatrash.net/2016/08/odds-and-ends-optimizing-sshfs-moving.html).
  1676. ## Other
  1677. There are other remote filesystems but none popularly used to serve
  1678. mergerfs. If you use something not listed above feel free to reach out
  1679. and I will add it to the list.
  1680. # FAQ
  1681. #### How well does mergerfs scale? Is it "production ready?"
  1682. Users have reported running mergerfs on everything from a Raspberry Pi
  1683. to dual socket Xeon systems with >20 cores. I'm aware of at least a
  1684. few companies which use mergerfs in production. [Open Media
  1685. Vault](https://www.openmediavault.org) includes mergerfs as its sole
  1686. solution for pooling filesystems. The author of mergerfs had it
  1687. running for over 300 days managing 16+ devices with reasonably heavy
  1688. 24/7 read and write usage. Stopping only after the machine's power
  1689. supply died.
  1690. Most serious issues (crashes or data corruption) have been due to
  1691. [kernel
  1692. bugs](https://github.com/trapexit/mergerfs/wiki/Kernel-Issues-&-Bugs). All
  1693. of which are fixed in stable releases.
  1694. #### Can mergerfs be used with filesystems which already have data / are in use?
  1695. Yes. mergerfs is really just a proxy and does **NOT** interfere with
  1696. the normal form or function of the filesystems / mounts / paths it
  1697. manages. It is just another userland application that is acting as a
  1698. man-in-the-middle. It can't do anything that any other random piece of
  1699. software can't do.
  1700. mergerfs is **not** a traditional filesystem that takes control over
  1701. the underlying block device. mergerfs is **not** RAID. It does **not**
  1702. manipulate the data that passes through it. It does **not** shard data
  1703. across filesystems. It merely shards some **behavior** and aggregates
  1704. others.
  1705. #### Can drives/filesystems be removed from the pool at will?
  1706. Yes. See previous question's answer.
  1707. #### Can mergerfs be removed without affecting the data?
  1708. Yes. See the previous question's answer.
  1709. #### Can drives/filesystems be moved to another pool?
  1710. Yes. See the previous question's answer.
  1711. #### How do I migrate data into or out of the pool when adding/removing drives/filesystems?
  1712. You don't need to. See the previous question's answer.
  1713. #### How do I remove a drive/filesystem but keep the data in the pool?
  1714. Nothing special needs to be done. Remove the branch from mergerfs'
  1715. config and copy (rsync) the data from the removed filesystem into the
  1716. pool. Effectively the same as if it were you transfering data from one
  1717. filesystem to another.
  1718. If you wish to continue using the pool while performing the transfer
  1719. simply create another, temporary pool without the filesystem in
  1720. question and then copy the data. It would probably be a good idea to
  1721. set the branch to `RO` prior to doing this to ensure no new content is
  1722. written to the filesystem while performing the copy.
  1723. #### What policies should I use?
  1724. Unless you're doing something more niche the average user is probably
  1725. best off using `mfs` for `category.create`. It will spread files out
  1726. across your branches based on available space. Use `mspmfs` if you
  1727. want to try to colocate the data a bit more. You may want to use `lus`
  1728. if you prefer a slightly different distribution of data if you have a
  1729. mix of smaller and larger filesystems. Generally though `mfs`, `lus`,
  1730. or even `rand` are good for the general use case. If you are starting
  1731. with an imbalanced pool you can use the tool **mergerfs.balance** to
  1732. redistribute files across the pool.
  1733. If you really wish to try to colocate files based on directory you can
  1734. set `func.create` to `epmfs` or similar and `func.mkdir` to `rand` or
  1735. `eprand` depending on if you just want to colocate generally or on
  1736. specific branches. Either way the *need* to colocate is rare. For
  1737. instance: if you wish to remove the device regularly and want the data
  1738. to predictably be on that device or if you don't use backup at all and
  1739. don't wish to replace that data piecemeal. In which case using path
  1740. preservation can help but will require some manual
  1741. attention. Colocating after the fact can be accomplished using the
  1742. **mergerfs.consolidate** tool. If you don't need strict colocation
  1743. which the `ep` policies provide then you can use the `msp` based
  1744. policies which will walk back the path till finding a branch that
  1745. works.
  1746. Ultimately there is no correct answer. It is a preference or based on
  1747. some particular need. mergerfs is very easy to test and experiment
  1748. with. I suggest creating a test setup and experimenting to get a sense
  1749. of what you want.
  1750. `epmfs` is the default `category.create` policy because `ep` policies
  1751. are not going to change the general layout of the branches. It won't
  1752. place files/dirs on branches that don't already have the relative
  1753. branch. So it keeps the system in a known state. It's much easier to
  1754. stop using `epmfs` or redistribute files around the filesystem than it
  1755. is to consolidate them back.
  1756. #### What settings should I use?
  1757. Depends on what features you want. Generally speaking there are no
  1758. "wrong" settings. All settings are performance or feature related. The
  1759. best bet is to read over the available options and choose what fits
  1760. your situation. If something isn't clear from the documentation please
  1761. reach out and the documentation will be improved.
  1762. That said, for the average person, the following should be fine:
  1763. `cache.files=off,dropcacheonclose=true,category.create=mfs`
  1764. #### Why are all my files ending up on 1 filesystem?!
  1765. Did you start with empty filesystems? Did you explicitly configure a
  1766. `category.create` policy? Are you using an `existing path` / `path
  1767. preserving` policy?
  1768. The default create policy is `epmfs`. That is a path preserving
  1769. algorithm. With such a policy for `mkdir` and `create` with a set of
  1770. empty filesystems it will select only 1 filesystem when the first
  1771. directory is created. Anything, files or directories, created in that
  1772. first directory will be placed on the same branch because it is
  1773. preserving paths.
  1774. This catches a lot of new users off guard but changing the default
  1775. would break the setup for many existing users and this policy is the
  1776. safest policy as it will not change the general layout of the existing
  1777. filesystems. If you do not care about path preservation and wish your
  1778. files to be spread across all your filesystems change to `mfs` or
  1779. similar policy as described above. If you do want path preservation
  1780. you'll need to perform the manual act of creating paths on the
  1781. filesystems you want the data to land on before transferring your
  1782. data. Setting `func.mkdir=epall` can simplify managing path
  1783. preservation for `create`. Or use `func.mkdir=rand` if you're
  1784. interested in just grouping together directory content by filesystem.
  1785. #### Do hardlinks work?
  1786. Yes. See also the option `inodecalc` for how inode values are
  1787. calculated.
  1788. What mergerfs does not do is fake hard links across branches. Read
  1789. the section "rename & link" for how it works.
  1790. Remember that hardlinks will NOT work across devices. That includes
  1791. between the original filesystem and a mergerfs pool, between two
  1792. separate pools of the same underlying filesystems, or bind mounts of
  1793. paths within the mergerfs pool. The latter is common when using Docker
  1794. or Podman. Multiple volumes (bind mounts) to the same underlying
  1795. filesystem are considered different devices. There is no way to link
  1796. between them. You should mount in the highest directory in the
  1797. mergerfs pool that includes all the paths you need if you want links
  1798. to work.
  1799. #### Does FICLONE or FICLONERANGE work?
  1800. Unfortunately not. FUSE, the technology mergerfs is based on, does not
  1801. support the `clone_file_range` feature needed for it to work. mergerfs
  1802. won't even know such a request is made. The kernel will simply return
  1803. an error back to the application making the request.
  1804. Should FUSE gain the ability mergerfs will be updated to support it.
  1805. #### Can I use mergerfs without SnapRAID? SnapRAID without mergerfs?
  1806. Yes. They are completely unrelated pieces of software.
  1807. #### Can mergerfs run via Docker, Podman, Kubernetes, etc.
  1808. Yes. With Docker you'll need to include `--cap-add=SYS_ADMIN
  1809. --device=/dev/fuse --security-opt=apparmor:unconfined` or similar with
  1810. other container runtimes. You should also be running it as root or
  1811. given sufficient caps to change user and group identity as well as
  1812. have root like filesystem permissions.
  1813. Keep in mind that you **MUST** consider identity when using
  1814. containers. For example: supplemental groups will be picked up from
  1815. the container unless you properly manage users and groups by sharing
  1816. relevant /etc files or by using some other means to share identity
  1817. across containers. Similarly if you use "rootless" containers and user
  1818. namespaces to do uid/gid translations you **MUST** consider that while
  1819. managing shared files.
  1820. Also, as mentioned by [hotio](https://hotio.dev/containers/mergerfs),
  1821. with Docker you should probably be mounting with `bind-propagation`
  1822. set to `slave`.
  1823. #### Does mergerfs support CoW / copy-on-write / writes to read-only filesystems?
  1824. Not in the sense of a filesystem like BTRFS or ZFS nor in the
  1825. overlayfs or aufs sense. It does offer a
  1826. [cow-shell](http://manpages.ubuntu.com/manpages/bionic/man1/cow-shell.1.html)
  1827. like hard link breaking (copy to temp file then rename over original)
  1828. which can be useful when wanting to save space by hardlinking
  1829. duplicate files but wish to treat each name as if it were a unique and
  1830. separate file.
  1831. If you want to write to a read-only filesystem you should look at
  1832. overlayfs. You can always include the overlayfs mount into a mergerfs
  1833. pool.
  1834. #### Why can't I see my files / directories?
  1835. It's almost always a permissions issue. Unlike mhddfs and
  1836. unionfs-fuse, which runs as root and attempts to access content as
  1837. such, mergerfs always changes its credentials to that of the
  1838. caller. This means that if the user does not have access to a file or
  1839. directory than neither will mergerfs. However, because mergerfs is
  1840. creating a union of paths it may be able to read some files and
  1841. directories on one filesystem but not another resulting in an
  1842. incomplete set.
  1843. Whenever you run into a split permission issue (seeing some but not
  1844. all files) try using
  1845. [mergerfs.fsck](https://github.com/trapexit/mergerfs-tools) tool to
  1846. check for and fix the mismatch. If you aren't seeing anything at all
  1847. be sure that the basic permissions are correct. The user and group
  1848. values are correct and that directories have their executable bit
  1849. set. A common mistake by users new to Linux is to `chmod -R 644` when
  1850. they should have `chmod -R u=rwX,go=rX`.
  1851. If using a network filesystem such as NFS or SMB (Samba) be sure to
  1852. pay close attention to anything regarding permissioning and
  1853. users. Root squashing and user translation for instance has bitten a
  1854. few mergerfs users. Some of these also affect the use of mergerfs from
  1855. container platforms such as Docker.
  1856. #### Why use FUSE? Why not a kernel based solution?
  1857. As with any solutions to a problem there are advantages and
  1858. disadvantages to each one.
  1859. A FUSE based solution has all the downsides of FUSE:
  1860. * Higher IO latency due to the trips in and out of kernel space
  1861. * Higher general overhead due to trips in and out of kernel space
  1862. * Double caching when using page caching
  1863. * Misc limitations due to FUSE's design
  1864. But FUSE also has a lot of upsides:
  1865. * Easier to offer a cross platform solution
  1866. * Easier forward and backward compatibility
  1867. * Easier updates for users
  1868. * Easier and faster release cadence
  1869. * Allows more flexibility in design and features
  1870. * Overall easier to write, secure, and maintain
  1871. * Much lower barrier to entry (getting code into the kernel takes a
  1872. lot of time and effort initially)
  1873. FUSE was chosen because of all the advantages listed above. The
  1874. negatives of FUSE do not outweigh the positives.
  1875. #### Is my OS's libfuse needed for mergerfs to work?
  1876. No. Normally `mount.fuse` is needed to get mergerfs (or any FUSE
  1877. filesystem to mount using the `mount` command but in vendoring the
  1878. libfuse library the `mount.fuse` app has been renamed to
  1879. `mount.mergerfs` meaning the filesystem type in `fstab` can simply be
  1880. `mergerfs`. That said there should be no harm in having it installed
  1881. and continuing to using `fuse.mergerfs` as the type in `/etc/fstab`.
  1882. If `mergerfs` doesn't work as a type it could be due to how the
  1883. `mount.mergerfs` tool was installed. Must be in `/sbin/` with proper
  1884. permissions.
  1885. #### Why was splice support removed?
  1886. After a lot of testing over the years splicing always appeared to be
  1887. at best provide equivalent performance and in cases worse
  1888. performance. Splice is not supported on other platforms forcing a
  1889. traditional read/write fallback to be provided. The splice code was
  1890. removed to simplify the codebase.
  1891. #### What should mergerfs NOT be used for?
  1892. * databases: Even if the database stored data in separate files
  1893. (mergerfs wouldn't offer much otherwise) the higher latency of the
  1894. indirection will kill performance. If it is a lightly used SQLITE
  1895. database then it may be fine but you'll need to test.
  1896. * VM images: For the same reasons as databases. VM images are accessed
  1897. very aggressively and mergerfs will introduce too much latency (if
  1898. it works at all).
  1899. * As replacement for RAID: mergerfs is just for pooling branches. If
  1900. you need that kind of device performance aggregation or high
  1901. availability you should stick with RAID.
  1902. #### Can filesystems be written to directly? Outside of mergerfs while pooled?
  1903. Yes, however it's not recommended to use the same file from within the
  1904. pool and from without at the same time (particularly
  1905. writing). Especially if using caching of any kind (cache.files,
  1906. cache.entry, cache.attr, cache.negative_entry, cache.symlinks,
  1907. cache.readdir, etc.) as there could be a conflict between cached data
  1908. and not.
  1909. #### Why do I get an "out of space" / "no space left on device" / ENOSPC error even though there appears to be lots of space available?
  1910. First make sure you've read the sections above about policies, path
  1911. preservation, branch filtering, and the options **minfreespace**,
  1912. **moveonenospc**, **statfs**, and **statfs_ignore**.
  1913. mergerfs is simply presenting a union of the content within multiple
  1914. branches. The reported free space is an aggregate of space available
  1915. within the pool (behavior modified by **statfs** and
  1916. **statfs_ignore**). It does not represent a contiguous space. In the
  1917. same way that read-only filesystems, those with quotas, or reserved
  1918. space report the full theoretical space available.
  1919. Due to path preservation, branch tagging, read-only status, and
  1920. **minfreespace** settings it is perfectly valid that `ENOSPC` / "out
  1921. of space" / "no space left on device" be returned. It is doing what
  1922. was asked of it: filtering possible branches due to those
  1923. settings. Only one error can be returned and if one of the reasons for
  1924. filtering a branch was **minfreespace** then it will be returned as
  1925. such. **moveonenospc** is only relevant to writing a file which is too
  1926. large for the filesystem it's currently on.
  1927. It is also possible that the filesystem selected has run out of
  1928. inodes. Use `df -i` to list the total and available inodes per
  1929. filesystem.
  1930. If you don't care about path preservation then simply change the
  1931. `create` policy to one which isn't. `mfs` is probably what most are
  1932. looking for. The reason it's not default is because it was originally
  1933. set to `epmfs` and changing it now would change people's setup. Such a
  1934. setting change will likely occur in mergerfs 3.
  1935. #### Why does the total available space in mergerfs not equal outside?
  1936. Are you using ext2/3/4? With reserve for root? mergerfs uses available
  1937. space for statfs calculations. If you've reserved space for root then
  1938. it won't show up.
  1939. You can remove the reserve by running: `tune2fs -m 0 <device>`
  1940. #### I notice massive slowdowns of writes when enabling cache.files.
  1941. When file caching is enabled in any form (`cache.files!=off`) it will
  1942. issue `getxattr` requests for `security.capability` prior to *every
  1943. single write*. This will usually result in a performance degradation,
  1944. especially when using a network filesystem (such as NFS or SMB.)
  1945. Unfortunately at this moment the kernel is not caching the response.
  1946. To work around this situation mergerfs offers a few solutions.
  1947. 1. Set `security_capability=false`. It will short circuit any call and
  1948. return `ENOATTR`. This still means though that mergerfs will
  1949. receive the request before every write but at least it doesn't get
  1950. passed through to the underlying filesystem.
  1951. 2. Set `xattr=noattr`. Same as above but applies to *all* calls to
  1952. getxattr. Not just `security.capability`. This will not be cached
  1953. by the kernel either but mergerfs' runtime config system will still
  1954. function.
  1955. 3. Set `xattr=nosys`. Results in mergerfs returning `ENOSYS` which
  1956. *will* be cached by the kernel. No future xattr calls will be
  1957. forwarded to mergerfs. The downside is that also means the xattr
  1958. based config and query functionality won't work either.
  1959. 4. Disable file caching. If you aren't using applications which use
  1960. `mmap` it's probably simpler to just disable it all together. The
  1961. kernel won't send the requests when caching is disabled.
  1962. #### It's mentioned that there are some security issues with mhddfs. What are they? How does mergerfs address them?
  1963. [mhddfs](https://github.com/trapexit/mhddfs) manages running as
  1964. **root** by calling
  1965. [getuid()](https://github.com/trapexit/mhddfs/blob/cae96e6251dd91e2bdc24800b4a18a74044f6672/src/main.c#L319)
  1966. and if it returns **0** then it will
  1967. [chown](http://linux.die.net/man/1/chown) the file. Not only is that a
  1968. race condition but it doesn't handle other situations. Rather than
  1969. attempting to simulate POSIX ACL behavior the proper way to manage
  1970. this is to use [seteuid](http://linux.die.net/man/2/seteuid) and
  1971. [setegid](http://linux.die.net/man/2/setegid), in effect becoming the
  1972. user making the original call, and perform the action as them. This is
  1973. what mergerfs does and why mergerfs should always run as root.
  1974. In Linux setreuid syscalls apply only to the thread. GLIBC hides this
  1975. away by using realtime signals to inform all threads to change
  1976. credentials. Taking after **Samba**, mergerfs uses
  1977. **syscall(SYS_setreuid,...)** to set the callers credentials for that
  1978. thread only. Jumping back to **root** as necessary should escalated
  1979. privileges be needed (for instance: to clone paths between
  1980. filesystems).
  1981. For non-Linux systems mergerfs uses a read-write lock and changes
  1982. credentials only when necessary. If multiple threads are to be user X
  1983. then only the first one will need to change the processes
  1984. credentials. So long as the other threads need to be user X they will
  1985. take a readlock allowing multiple threads to share the
  1986. credentials. Once a request comes in to run as user Y that thread will
  1987. attempt a write lock and change to Y's credentials when it can. If the
  1988. ability to give writers priority is supported then that flag will be
  1989. used so threads trying to change credentials don't starve. This isn't
  1990. the best solution but should work reasonably well assuming there are
  1991. few users.
  1992. # mergerfs versus X
  1993. #### mhddfs
  1994. mhddfs had not been maintained for some time and has some known
  1995. stability and security issues. mergerfs provides a superset of mhddfs'
  1996. features and should offer the same or better performance.
  1997. Below is an example of mhddfs and mergerfs setup to work similarly.
  1998. `mhddfs -o mlimit=4G,allow_other /mnt/drive1,/mnt/drive2 /mnt/pool`
  1999. `mergerfs -o minfreespace=4G,category.create=ff /mnt/drive1:/mnt/drive2 /mnt/pool`
  2000. #### aufs
  2001. aufs is mostly abandoned and no longer available in most Linux distros.
  2002. While aufs can offer better peak performance mergerfs provides more
  2003. configurability and is generally easier to use. mergerfs however does
  2004. not offer the overlay / copy-on-write (CoW) features which aufs has.
  2005. #### unionfs-fuse
  2006. unionfs-fuse is more like aufs than mergerfs in that it offers overlay /
  2007. copy-on-write (CoW) features. If you're just looking to create a union
  2008. of filesystems and want flexibility in file/directory placement then
  2009. mergerfs offers that whereas unionfs is more for overlaying read/write
  2010. filesystems over read-only ones.
  2011. #### overlayfs
  2012. overlayfs is similar to aufs and unionfs-fuse in that it also is
  2013. primarily used to layer a read/write filesystem over one or more
  2014. read-only filesystems. It does not have the ability to spread
  2015. files/directories across numerous filesystems.
  2016. #### RAID0, JBOD, drive concatenation, striping
  2017. With simple JBOD / drive concatenation / stripping / RAID0 a single
  2018. drive failure will result in full pool failure. mergerfs performs a
  2019. similar function without the possibility of catastrophic failure and
  2020. the difficulties in recovery. Drives may fail but all other
  2021. filesystems and their data will continue to be accessible.
  2022. The main practical difference with mergerfs is the fact you don't
  2023. actually have contiguous space as large as if you used those other
  2024. technologies. Meaning you can't create a 2TB file on a pool of 2 1TB
  2025. filesystems.
  2026. When combined with something like [SnapRaid](http://www.snapraid.it)
  2027. and/or an offsite backup solution you can have the flexibility of JBOD
  2028. without the single point of failure.
  2029. #### UnRAID
  2030. UnRAID is a full OS and its storage layer, as I understand, is
  2031. proprietary and closed source. Users who have experience with both
  2032. have often said they prefer the flexibility offered by mergerfs and
  2033. for some the fact it is open source is important.
  2034. There are a number of UnRAID users who use mergerfs as well though I'm
  2035. not entirely familiar with the use case.
  2036. For semi-static data mergerfs + [SnapRaid](http://www.snapraid.it)
  2037. provides a similar solution.
  2038. #### ZFS
  2039. mergerfs is very different from ZFS. mergerfs is intended to provide
  2040. flexible pooling of arbitrary filesystems (local or remote), of
  2041. arbitrary sizes, and arbitrary filesystems. For `write once, read
  2042. many` usecases such as bulk media storage. Where data integrity and
  2043. backup is managed in other ways. In those usecases ZFS can introduce a
  2044. number of costs and limitations as described
  2045. [here](http://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html),
  2046. [here](https://markmcb.com/2020/01/07/five-years-of-btrfs/), and
  2047. [here](https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSWhyNoRealReshaping).
  2048. #### StableBit's DrivePool
  2049. DrivePool works only on Windows so not as common an alternative as
  2050. other Linux solutions. If you want to use Windows then DrivePool is a
  2051. good option. Functionally the two projects work a bit
  2052. differently. DrivePool always writes to the filesystem with the most
  2053. free space and later rebalances. mergerfs does not offer rebalance but
  2054. chooses a branch at file/directory create time. DrivePool's
  2055. rebalancing can be done differently in any directory and has file
  2056. pattern matching to further customize the behavior. mergerfs, not
  2057. having rebalancing does not have these features, but similar features
  2058. are planned for mergerfs v3. DrivePool has builtin file duplication
  2059. which mergerfs does not natively support (but can be done via an
  2060. external script.)
  2061. There are a lot of misc differences between the two projects but most
  2062. features in DrivePool can be replicated with external tools in
  2063. combination with mergerfs.
  2064. Additionally DrivePool is a closed source commercial product vs
  2065. mergerfs a ISC licensed OSS project.
  2066. # SUPPORT
  2067. Filesystems are complex and difficult to debug. mergerfs, while being
  2068. just a proxy of sorts, can be difficult to debug given the large
  2069. number of possible settings it can have itself and the number of
  2070. environments it can run in. When reporting on a suspected issue
  2071. **please** include as much of the below information as possible
  2072. otherwise it will be difficult or impossible to diagnose. Also please
  2073. read the above documentation as it provides details on many previously
  2074. encountered questions/issues.
  2075. **Please make sure you are using the [latest
  2076. release](https://github.com/trapexit/mergerfs/releases) or have tried
  2077. it in comparison. Old versions, which are often included in distros
  2078. like Debian and Ubuntu, are not ever going to be updated and the issue
  2079. you are encountering may have been addressed already.**
  2080. **For commercial support or feature requests please [contact me
  2081. directly.](mailto:support@spawn.link)**
  2082. #### Information to include in bug reports
  2083. * [Information about the broader problem along with any attempted
  2084. solutions.](https://xyproblem.info)
  2085. * Solution already ruled out and why.
  2086. * Version of mergerfs: `mergerfs --version`
  2087. * mergerfs settings / arguments: from fstab, systemd unit, command
  2088. line, OMV plugin, etc.
  2089. * Version of the OS: `uname -a` and `lsb_release -a`
  2090. * List of branches, their filesystem types, sizes (before and after issue): `df -h`
  2091. * **All** information about the relevant paths and files: permissions, ownership, etc.
  2092. * **All** information about the client app making the requests: version, uid/gid
  2093. * Runtime environment:
  2094. * Is mergerfs running within a container?
  2095. * Are the client apps using mergerfs running in a container?
  2096. * A `strace` of the app having problems:
  2097. * `strace -fvTtt -s 256 -o /tmp/app.strace.txt <cmd>`
  2098. * A `strace` of mergerfs while the program is trying to do whatever it is failing to do:
  2099. * `strace -fvTtt -s 256 -p <mergerfsPID> -o /tmp/mergerfs.strace.txt`
  2100. * **Precise** directions on replicating the issue. Do not leave **anything** out.
  2101. * Try to recreate the problem in the simplest way using standard programs: `ln`, `mv`, `cp`, `ls`, `dd`, etc.
  2102. #### Contact / Issue submission
  2103. * github.com: https://github.com/trapexit/mergerfs/issues
  2104. * discord: https://discord.gg/MpAr69V
  2105. * reddit: https://www.reddit.com/r/mergerfs
  2106. #### Donations
  2107. https://github.com/trapexit/support
  2108. Development and support of a project like mergerfs requires a
  2109. significant amount of time and effort. The software is released under
  2110. the very liberal ISC license and is therefore free to use for personal
  2111. or commercial uses.
  2112. If you are a personal user and find mergerfs and its support valuable
  2113. and would like to support the project financially it would be very
  2114. much appreciated.
  2115. If you are using mergerfs commercially please consider sponsoring the
  2116. project to ensure it continues to be maintained and receive
  2117. updates. If custom features are needed feel free to [contact me
  2118. directly](mailto:support@spawn.link).
  2119. # LINKS
  2120. * https://spawn.link
  2121. * https://github.com/trapexit/mergerfs
  2122. * https://github.com/trapexit/mergerfs/wiki
  2123. * https://github.com/trapexit/mergerfs-tools
  2124. * https://github.com/trapexit/scorch
  2125. * https://github.com/trapexit/bbf