You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

2486 lines
96 KiB

7 years ago
7 years ago
9 years ago
9 years ago
9 years ago
9 years ago
5 years ago
9 years ago
9 years ago
5 years ago
9 years ago
9 years ago
9 years ago
9 years ago
7 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
5 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
7 years ago
6 years ago
6 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
9 years ago
9 years ago
6 years ago
6 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
8 years ago
5 years ago
5 years ago
5 years ago
5 years ago
5 years ago
9 years ago
7 years ago
7 years ago
9 years ago
9 years ago
9 years ago
  1. .\"t
  2. .\" Automatically generated by Pandoc 1.19.2.4
  3. .\"
  4. .TH "mergerfs" "1" "2020\-05\-25" "mergerfs user manual" ""
  5. .hy
  6. .SH NAME
  7. .PP
  8. mergerfs \- a featureful union filesystem
  9. .SH SYNOPSIS
  10. .PP
  11. mergerfs \-o<options> <branches> <mountpoint>
  12. .SH DESCRIPTION
  13. .PP
  14. \f[B]mergerfs\f[] is a union filesystem geared towards simplifying
  15. storage and management of files across numerous commodity storage
  16. devices.
  17. It is similar to \f[B]mhddfs\f[], \f[B]unionfs\f[], and \f[B]aufs\f[].
  18. .SH FEATURES
  19. .IP \[bu] 2
  20. Runs in userspace (FUSE)
  21. .IP \[bu] 2
  22. Configurable behaviors / file placement
  23. .IP \[bu] 2
  24. Support for extended attributes (xattrs)
  25. .IP \[bu] 2
  26. Support for file attributes (chattr)
  27. .IP \[bu] 2
  28. Runtime configurable (via xattrs)
  29. .IP \[bu] 2
  30. Safe to run as root
  31. .IP \[bu] 2
  32. Opportunistic credential caching
  33. .IP \[bu] 2
  34. Works with heterogeneous filesystem types
  35. .IP \[bu] 2
  36. Handling of writes to full drives (transparently move file to drive with
  37. capacity)
  38. .IP \[bu] 2
  39. Handles pool of read\-only and read/write drives
  40. .IP \[bu] 2
  41. Can turn read\-only files into symlinks to underlying file
  42. .IP \[bu] 2
  43. Hard link copy\-on\-write / CoW
  44. .IP \[bu] 2
  45. supports POSIX ACLs
  46. .SH HOW IT WORKS
  47. .PP
  48. mergerfs logically merges multiple paths together.
  49. Think a union of sets.
  50. The file/s or directory/s acted on or presented through mergerfs are
  51. based on the policy chosen for that particular action.
  52. Read more about policies below.
  53. .IP
  54. .nf
  55. \f[C]
  56. A\ \ \ \ \ \ \ \ \ +\ \ \ \ \ \ B\ \ \ \ \ \ \ \ =\ \ \ \ \ \ \ C
  57. /disk1\ \ \ \ \ \ \ \ \ \ \ /disk2\ \ \ \ \ \ \ \ \ \ \ /merged
  58. |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |
  59. +\-\-\ /dir1\ \ \ \ \ \ \ \ +\-\-\ /dir1\ \ \ \ \ \ \ \ +\-\-\ /dir1
  60. |\ \ \ |\ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ |\ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ |
  61. |\ \ \ +\-\-\ file1\ \ \ \ |\ \ \ +\-\-\ file2\ \ \ \ |\ \ \ +\-\-\ file1
  62. |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ +\-\-\ file3\ \ \ \ |\ \ \ +\-\-\ file2
  63. +\-\-\ /dir2\ \ \ \ \ \ \ \ |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ +\-\-\ file3
  64. |\ \ \ |\ \ \ \ \ \ \ \ \ \ \ \ +\-\-\ /dir3\ \ \ \ \ \ \ \ |
  65. |\ \ \ +\-\-\ file4\ \ \ \ \ \ \ \ |\ \ \ \ \ \ \ \ \ \ \ \ +\-\-\ /dir2
  66. |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ +\-\-\ file5\ \ \ |\ \ \ |
  67. +\-\-\ file6\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ +\-\-\ file4
  68. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |
  69. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ +\-\-\ /dir3
  70. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ |
  71. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ +\-\-\ file5
  72. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |
  73. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ +\-\-\ file6
  74. \f[]
  75. .fi
  76. .PP
  77. mergerfs does \f[B]not\f[] support the copy\-on\-write (CoW) behavior
  78. found in \f[B]aufs\f[] and \f[B]overlayfs\f[].
  79. You can \f[B]not\f[] mount a read\-only filesystem and write to it.
  80. However, mergerfs will ignore read\-only drives when creating new files
  81. so you can mix read\-write and read\-only drives.
  82. It also does \f[B]not\f[] split data across drives.
  83. It is not RAID0 / striping.
  84. It is simply a union.
  85. .SH TERMINOLOGY
  86. .IP \[bu] 2
  87. branch: A base path used in the pool.
  88. .IP \[bu] 2
  89. pool: The mergerfs mount.
  90. The union of the branches.
  91. .IP \[bu] 2
  92. relative path: The path in the pool relative to the branch and mount.
  93. .IP \[bu] 2
  94. policy: The algorithm used to select a file when performing a function.
  95. .IP \[bu] 2
  96. function: A filesystem call (open, unlink, create, getattr, etc.)
  97. .IP \[bu] 2
  98. category: A collection of functions (action, create, search).
  99. .IP \[bu] 2
  100. path preservation: Aspect of some policies which includes checking the
  101. path for which a file would be created.
  102. .SH BASIC SETUP
  103. .PP
  104. If you don\[aq]t already know that you have a special use case then just
  105. start with one of the following option sets.
  106. .SS You don\[aq]t need \f[C]mmap\f[]
  107. .PP
  108. \f[C]use_ino,cache.files=off,dropcacheonclose=true,allow_other,category.create=mfs\f[]
  109. .SS You do need \f[C]mmap\f[] (used by rtorrent and some other programs)
  110. .PP
  111. \f[C]use_ino,cache.files=partial,dropcacheonclose=true,allow_other,category.create=mfs\f[]
  112. .PP
  113. See the mergerfs wiki for real world
  114. deployments (https://github.com/trapexit/mergerfs/wiki/Real-World-Deployments)
  115. for comparisons / ideas.
  116. .SH OPTIONS
  117. .SS mount options
  118. .IP \[bu] 2
  119. \f[B]allow_other\f[]: A libfuse option which allows users besides the
  120. one which ran mergerfs to see the filesystem.
  121. This is required for most use\-cases.
  122. .IP \[bu] 2
  123. \f[B]minfreespace=SIZE\f[]: The minimum space value used for creation
  124. policies.
  125. Understands \[aq]K\[aq], \[aq]M\[aq], and \[aq]G\[aq] to represent
  126. kilobyte, megabyte, and gigabyte respectively.
  127. (default: 4G)
  128. .IP \[bu] 2
  129. \f[B]moveonenospc=BOOL\f[]: When enabled if a \f[B]write\f[] fails with
  130. \f[B]ENOSPC\f[] (no space left on device) or \f[B]EDQUOT\f[] (disk quota
  131. exceeded) a scan of all drives will be done looking for the drive with
  132. the most free space which is at least the size of the file plus the
  133. amount which failed to write.
  134. An attempt to move the file to that drive will occur (keeping all
  135. metadata possible) and if successful the original is unlinked and the
  136. write retried.
  137. (default: false)
  138. .IP \[bu] 2
  139. \f[B]use_ino\f[]: Causes mergerfs to supply file/directory inodes rather
  140. than libfuse.
  141. While not a default it is recommended it be enabled so that linked files
  142. share the same inode value.
  143. .IP \[bu] 2
  144. \f[B]dropcacheonclose=BOOL\f[]: When a file is requested to be closed
  145. call \f[C]posix_fadvise\f[] on it first to instruct the kernel that we
  146. no longer need the data and it can drop its cache.
  147. Recommended when \f[B]cache.files=partial|full|auto\-full\f[] to limit
  148. double caching.
  149. (default: false)
  150. .IP \[bu] 2
  151. \f[B]symlinkify=BOOL\f[]: When enabled and a file is not writable and
  152. its mtime or ctime is older than \f[B]symlinkify_timeout\f[] files will
  153. be reported as symlinks to the original files.
  154. Please read more below before using.
  155. (default: false)
  156. .IP \[bu] 2
  157. \f[B]symlinkify_timeout=INT\f[]: Time to wait, in seconds, to activate
  158. the \f[B]symlinkify\f[] behavior.
  159. (default: 3600)
  160. .IP \[bu] 2
  161. \f[B]nullrw=BOOL\f[]: Turns reads and writes into no\-ops.
  162. The request will succeed but do nothing.
  163. Useful for benchmarking mergerfs.
  164. (default: false)
  165. .IP \[bu] 2
  166. \f[B]ignorepponrename=BOOL\f[]: Ignore path preserving on rename.
  167. Typically rename and link act differently depending on the policy of
  168. \f[C]create\f[] (read below).
  169. Enabling this will cause rename and link to always use the non\-path
  170. preserving behavior.
  171. This means files, when renamed or linked, will stay on the same drive.
  172. (default: false)
  173. .IP \[bu] 2
  174. \f[B]security_capability=BOOL\f[]: If false return ENOATTR when xattr
  175. security.capability is queried.
  176. (default: true)
  177. .IP \[bu] 2
  178. \f[B]xattr=passthrough|noattr|nosys\f[]: Runtime control of xattrs.
  179. Default is to passthrough xattr requests.
  180. \[aq]noattr\[aq] will short circuit as if nothing exists.
  181. \[aq]nosys\[aq] will respond with ENOSYS as if xattrs are not supported
  182. or disabled.
  183. (default: passthrough)
  184. .IP \[bu] 2
  185. \f[B]link_cow=BOOL\f[]: When enabled if a regular file is opened which
  186. has a link count > 1 it will copy the file to a temporary file and
  187. rename over the original.
  188. Breaking the link and providing a basic copy\-on\-write function similar
  189. to cow\-shell.
  190. (default: false)
  191. .IP \[bu] 2
  192. \f[B]statfs=base|full\f[]: Controls how statfs works.
  193. \[aq]base\[aq] means it will always use all branches in statfs
  194. calculations.
  195. \[aq]full\[aq] is in effect path preserving and only includes drives
  196. where the path exists.
  197. (default: base)
  198. .IP \[bu] 2
  199. \f[B]statfs_ignore=none|ro|nc\f[]: \[aq]ro\[aq] will cause statfs
  200. calculations to ignore available space for branches mounted or tagged as
  201. \[aq]read\-only\[aq] or \[aq]no create\[aq].
  202. \[aq]nc\[aq] will ignore available space for branches tagged as \[aq]no
  203. create\[aq].
  204. (default: none)
  205. .IP \[bu] 2
  206. \f[B]posix_acl=BOOL\f[]: Enable POSIX ACL support (if supported by
  207. kernel and underlying filesystem).
  208. (default: false)
  209. .IP \[bu] 2
  210. \f[B]async_read=BOOL\f[]: Perform reads asynchronously.
  211. If disabled or unavailable the kernel will ensure there is at most one
  212. pending read request per file handle and will attempt to order requests
  213. by offset.
  214. (default: true)
  215. .IP \[bu] 2
  216. \f[B]fuse_msg_size=INT\f[]: Set the max number of pages per FUSE
  217. message.
  218. Only available on Linux >= 4.20 and ignored otherwise.
  219. (min: 1; max: 256; default: 256)
  220. .IP \[bu] 2
  221. \f[B]threads=INT\f[]: Number of threads to use in multithreaded mode.
  222. When set to zero it will attempt to discover and use the number of
  223. logical cores.
  224. If the lookup fails it will fall back to using 4.
  225. If the thread count is set negative it will look up the number of cores
  226. then divide by the absolute value.
  227. ie.
  228. threads=\-2 on an 8 core machine will result in 8 / 2 = 4 threads.
  229. There will always be at least 1 thread.
  230. NOTE: higher number of threads increases parallelism but usually
  231. decreases throughput.
  232. (default: 0)
  233. .IP \[bu] 2
  234. \f[B]fsname=STR\f[]: Sets the name of the filesystem as seen in
  235. \f[B]mount\f[], \f[B]df\f[], etc.
  236. Defaults to a list of the source paths concatenated together with the
  237. longest common prefix removed.
  238. .IP \[bu] 2
  239. \f[B]func.FUNC=POLICY\f[]: Sets the specific FUSE function\[aq]s policy.
  240. See below for the list of value types.
  241. Example: \f[B]func.getattr=newest\f[]
  242. .IP \[bu] 2
  243. \f[B]category.CATEGORY=POLICY\f[]: Sets policy of all FUSE functions in
  244. the provided category.
  245. See POLICIES section for defaults.
  246. Example: \f[B]category.create=mfs\f[]
  247. .IP \[bu] 2
  248. \f[B]cache.open=INT\f[]: \[aq]open\[aq] policy cache timeout in seconds.
  249. (default: 0)
  250. .IP \[bu] 2
  251. \f[B]cache.statfs=INT\f[]: \[aq]statfs\[aq] cache timeout in seconds.
  252. (default: 0)
  253. .IP \[bu] 2
  254. \f[B]cache.attr=INT\f[]: File attribute cache timeout in seconds.
  255. (default: 1)
  256. .IP \[bu] 2
  257. \f[B]cache.entry=INT\f[]: File name lookup cache timeout in seconds.
  258. (default: 1)
  259. .IP \[bu] 2
  260. \f[B]cache.negative_entry=INT\f[]: Negative file name lookup cache
  261. timeout in seconds.
  262. (default: 0)
  263. .IP \[bu] 2
  264. \f[B]cache.files=libfuse|off|partial|full|auto\-full\f[]: File page
  265. caching mode (default: libfuse)
  266. .IP \[bu] 2
  267. \f[B]cache.writeback=BOOL\f[]: Enable kernel writeback caching (default:
  268. false)
  269. .IP \[bu] 2
  270. \f[B]cache.symlinks=BOOL\f[]: Cache symlinks (if supported by kernel)
  271. (default: false)
  272. .IP \[bu] 2
  273. \f[B]cache.readdir=BOOL\f[]: Cache readdir (if supported by kernel)
  274. (default: false)
  275. .IP \[bu] 2
  276. \f[B]direct_io\f[]: deprecated \- Bypass page cache.
  277. Use \f[C]cache.files=off\f[] instead.
  278. (default: false)
  279. .IP \[bu] 2
  280. \f[B]kernel_cache\f[]: deprecated \- Do not invalidate data cache on
  281. file open.
  282. Use \f[C]cache.files=full\f[] instead.
  283. (default: false)
  284. .IP \[bu] 2
  285. \f[B]auto_cache\f[]: deprecated \- Invalidate data cache if file mtime
  286. or size change.
  287. Use \f[C]cache.files=auto\-full\f[] instead.
  288. (default: false)
  289. .IP \[bu] 2
  290. \f[B]async_read\f[]: deprecated \- Perform reads asynchronously.
  291. Use \f[C]async_read=true\f[] instead.
  292. .IP \[bu] 2
  293. \f[B]sync_read\f[]: deprecated \- Perform reads synchronously.
  294. Use \f[C]async_read=false\f[] instead.
  295. .PP
  296. \f[B]NOTE:\f[] Options are evaluated in the order listed so if the
  297. options are \f[B]func.rmdir=rand,category.action=ff\f[] the
  298. \f[B]action\f[] category setting will override the \f[B]rmdir\f[]
  299. setting.
  300. .SS Value Types
  301. .IP \[bu] 2
  302. BOOL = \[aq]true\[aq] | \[aq]false\[aq]
  303. .IP \[bu] 2
  304. INT = [0,MAX_INT]
  305. .IP \[bu] 2
  306. SIZE = \[aq]NNM\[aq]; NN = INT, M = \[aq]K\[aq] | \[aq]M\[aq] |
  307. \[aq]G\[aq] | \[aq]T\[aq]
  308. .IP \[bu] 2
  309. STR = string
  310. .IP \[bu] 2
  311. FUNC = FUSE function
  312. .IP \[bu] 2
  313. CATEGORY = FUSE function category
  314. .IP \[bu] 2
  315. POLICY = mergerfs function policy
  316. .SS branches
  317. .PP
  318. The \[aq]branches\[aq] (formerly \[aq]srcmounts\[aq]) argument is a
  319. colon (\[aq]:\[aq]) delimited list of paths to be pooled together.
  320. It does not matter if the paths are on the same or different drives nor
  321. does it matter the filesystem.
  322. Used and available space will not be duplicated for paths on the same
  323. device and any features which aren\[aq]t supported by the underlying
  324. filesystem (such as file attributes or extended attributes) will return
  325. the appropriate errors.
  326. .PP
  327. To make it easier to include multiple branches mergerfs supports
  328. globbing (http://linux.die.net/man/7/glob).
  329. \f[B]The globbing tokens MUST be escaped when using via the shell else
  330. the shell itself will apply the glob itself.\f[]
  331. .PP
  332. Each branch can have a suffix of \f[C]=RW\f[] (read / write),
  333. \f[C]=RO\f[] (read\-only), or \f[C]=NC\f[] (no create).
  334. These suffixes work with globs as well and will apply to each path
  335. found.
  336. \f[C]RW\f[] is the default behavior and those paths will be eligible for
  337. all policy categories.
  338. \f[C]RO\f[] will exclude those paths from \f[C]create\f[] and
  339. \f[C]action\f[] policies (just as a filesystem being mounted \f[C]ro\f[]
  340. would).
  341. \f[C]NC\f[] will exclude those paths from \f[C]create\f[] policies (you
  342. can\[aq]t create but you can change / delete).
  343. .IP
  344. .nf
  345. \f[C]
  346. #\ mergerfs\ \-o\ allow_other,use_ino\ /mnt/disk\\*:/mnt/cdrom\ /media/drives
  347. \f[]
  348. .fi
  349. .PP
  350. The above line will use all mount points in /mnt prefixed with
  351. \f[B]disk\f[] and the \f[B]cdrom\f[].
  352. .PP
  353. To have the pool mounted at boot or otherwise accessible from related
  354. tools use \f[B]/etc/fstab\f[].
  355. .IP
  356. .nf
  357. \f[C]
  358. #\ <file\ system>\ \ \ \ \ \ \ \ <mount\ point>\ \ <type>\ \ \ \ <options>\ \ \ \ \ \ \ \ \ \ \ \ \ <dump>\ \ <pass>
  359. /mnt/disk*:/mnt/cdrom\ \ /media/drives\ \ mergerfs\ \ allow_other,use_ino\ \ \ 0\ \ \ \ \ \ \ 0
  360. \f[]
  361. .fi
  362. .PP
  363. \f[B]NOTE:\f[] the globbing is done at mount or xattr update time (see
  364. below).
  365. If a new directory is added matching the glob after the fact it will not
  366. be automatically included.
  367. .PP
  368. \f[B]NOTE:\f[] for mounting via \f[B]fstab\f[] to work you must have
  369. \f[B]mount.fuse\f[] installed.
  370. For Ubuntu/Debian it is included in the \f[B]fuse\f[] package.
  371. .SS fuse_msg_size
  372. .PP
  373. FUSE applications communicate with the kernel over a special character
  374. device: \f[C]/dev/fuse\f[].
  375. A large portion of the overhead associated with FUSE is the cost of
  376. going back and forth from user space and kernel space over that device.
  377. Generally speaking the fewer trips needed the better the performance
  378. will be.
  379. Reducing the number of trips can be done a number of ways.
  380. Kernel level caching and increasing message sizes being two significant
  381. ones.
  382. When it comes to reads and writes if the message size is doubled the
  383. number of trips are approximately halved.
  384. .PP
  385. In Linux 4.20 a new feature was added allowing the negotiation of the
  386. max message size.
  387. Since the size is in multiples of
  388. pages (https://en.wikipedia.org/wiki/Page_(computer_memory)) the feature
  389. is called \f[C]max_pages\f[].
  390. There is a maximum \f[C]max_pages\f[] value of 256 (1MiB) and minimum of
  391. 1 (4KiB).
  392. The default used by Linux >=4.20, and hardcoded value used before 4.20,
  393. is 32 (128KiB).
  394. In mergerfs its referred to as \f[C]fuse_msg_size\f[] to make it clear
  395. what it impacts and provide some abstraction.
  396. .PP
  397. Since there should be no downsides to increasing \f[C]fuse_msg_size\f[]
  398. / \f[C]max_pages\f[], outside a minor bump in RAM usage due to larger
  399. message buffers, mergerfs defaults the value to 256.
  400. On kernels before 4.20 the value has no effect.
  401. The reason the value is configurable is to enable experimentation and
  402. benchmarking.
  403. See the BENCHMARKING section for examples.
  404. .SS symlinkify
  405. .PP
  406. Due to the levels of indirection introduced by mergerfs and the
  407. underlying technology FUSE there can be varying levels of performance
  408. degradation.
  409. This feature will turn non\-directories which are not writable into
  410. symlinks to the original file found by the \f[C]readlink\f[] policy
  411. after the mtime and ctime are older than the timeout.
  412. .PP
  413. \f[B]WARNING:\f[] The current implementation has a known issue in which
  414. if the file is open and being used when the file is converted to a
  415. symlink then the application which has that file open will receive an
  416. error when using it.
  417. This is unlikely to occur in practice but is something to keep in mind.
  418. .PP
  419. \f[B]WARNING:\f[] Some backup solutions, such as CrashPlan, do not
  420. backup the target of a symlink.
  421. If using this feature it will be necessary to point any backup software
  422. to the original drives or configure the software to follow symlinks if
  423. such an option is available.
  424. Alternatively create two mounts.
  425. One for backup and one for general consumption.
  426. .SS nullrw
  427. .PP
  428. Due to how FUSE works there is an overhead to all requests made to a
  429. FUSE filesystem that wouldn\[aq]t exist for an in kernel one.
  430. Meaning that even a simple passthrough will have some slowdown.
  431. However, generally the overhead is minimal in comparison to the cost of
  432. the underlying I/O.
  433. By disabling the underlying I/O we can test the theoretical performance
  434. boundaries.
  435. .PP
  436. By enabling \f[C]nullrw\f[] mergerfs will work as it always does
  437. \f[B]except\f[] that all reads and writes will be no\-ops.
  438. A write will succeed (the size of the write will be returned as if it
  439. were successful) but mergerfs does nothing with the data it was given.
  440. Similarly a read will return the size requested but won\[aq]t touch the
  441. buffer.
  442. .PP
  443. See the BENCHMARKING section for suggestions on how to test.
  444. .SS xattr
  445. .PP
  446. Runtime extended attribute support can be managed via the \f[C]xattr\f[]
  447. option.
  448. By default it will passthrough any xattr calls.
  449. Given xattr support is rarely used and can have significant performance
  450. implications mergerfs allows it to be disabled at runtime.
  451. The performance problems mostly comes when file caching is enabled.
  452. The kernel will send a \f[C]getxattr\f[] for
  453. \f[C]security.capability\f[] \f[I]before every single write\f[].
  454. It doesn\[aq]t cache the responses to any \f[C]getxattr\f[].
  455. This might be addressed in the future but for now mergerfs can really
  456. only offer the following workarounds.
  457. .PP
  458. \f[C]noattr\f[] will cause mergerfs to short circuit all xattr calls and
  459. return ENOATTR where appropriate.
  460. mergerfs still gets all the requests but they will not be forwarded on
  461. to the underlying filesystems.
  462. The runtime control will still function in this mode.
  463. .PP
  464. \f[C]nosys\f[] will cause mergerfs to return ENOSYS for any xattr call.
  465. The difference with \f[C]noattr\f[] is that the kernel will cache this
  466. fact and itself short circuit future calls.
  467. This is more efficient than \f[C]noattr\f[] but will cause mergerfs\[aq]
  468. runtime control via the hidden file to stop working.
  469. .SH FUNCTIONS / POLICIES / CATEGORIES
  470. .PP
  471. The POSIX filesystem API is made up of a number of functions.
  472. \f[B]creat\f[], \f[B]stat\f[], \f[B]chown\f[], etc.
  473. In mergerfs most of the core functions are grouped into 3 categories:
  474. \f[B]action\f[], \f[B]create\f[], and \f[B]search\f[].
  475. These functions and categories can be assigned a policy which dictates
  476. what file or directory is chosen when performing that behavior.
  477. Any policy can be assigned to a function or category though some may not
  478. be very useful in practice.
  479. For instance: \f[B]rand\f[] (random) may be useful for file creation
  480. (create) but could lead to very odd behavior if used for \f[C]chmod\f[]
  481. if there were more than one copy of the file.
  482. .PP
  483. Some functions, listed in the category \f[C]N/A\f[] below, can not be
  484. assigned the normal policies.
  485. All functions which work on file handles use the handle which was
  486. acquired by \f[C]open\f[] or \f[C]create\f[].
  487. \f[C]readdir\f[] has no real need for a policy given the purpose is
  488. merely to return a list of entries in a directory.
  489. \f[C]statfs\f[]\[aq]s behavior can be modified via other options.
  490. That said many times the current FUSE kernel driver will not always
  491. provide the file handle when a client calls \f[C]fgetattr\f[],
  492. \f[C]fchown\f[], \f[C]fchmod\f[], \f[C]futimens\f[], \f[C]ftruncate\f[],
  493. etc.
  494. This means it will call the regular, path based, versions.
  495. .PP
  496. When using policies which are based on a branch\[aq]s available space
  497. the base path provided is used.
  498. Not the full path to the file in question.
  499. Meaning that sub mounts won\[aq]t be considered in the space
  500. calculations.
  501. The reason is that it doesn\[aq]t really work for non\-path preserving
  502. policies and can lead to non\-obvious behaviors.
  503. .SS Function / Category classifications
  504. .PP
  505. .TS
  506. tab(@);
  507. lw(7.9n) lw(62.1n).
  508. T{
  509. Category
  510. T}@T{
  511. FUSE Functions
  512. T}
  513. _
  514. T{
  515. action
  516. T}@T{
  517. chmod, chown, link, removexattr, rename, rmdir, setxattr, truncate,
  518. unlink, utimens
  519. T}
  520. T{
  521. create
  522. T}@T{
  523. create, mkdir, mknod, symlink
  524. T}
  525. T{
  526. search
  527. T}@T{
  528. access, getattr, getxattr, ioctl (directories), listxattr, open,
  529. readlink
  530. T}
  531. T{
  532. N/A
  533. T}@T{
  534. fchmod, fchown, futimens, ftruncate, fallocate, fgetattr, fsync, ioctl
  535. (files), read, readdir, release, statfs, write, copy_file_range
  536. T}
  537. .TE
  538. .PP
  539. In cases where something may be searched (to confirm a directory exists
  540. across all source mounts) \f[B]getattr\f[] will be used.
  541. .SS Path Preservation
  542. .PP
  543. Policies, as described below, are of two basic types.
  544. \f[C]path\ preserving\f[] and \f[C]non\-path\ preserving\f[].
  545. .PP
  546. All policies which start with \f[C]ep\f[] (\f[B]epff\f[],
  547. \f[B]eplfs\f[], \f[B]eplus\f[], \f[B]epmfs\f[], \f[B]eprand\f[]) are
  548. \f[C]path\ preserving\f[].
  549. \f[C]ep\f[] stands for \f[C]existing\ path\f[].
  550. .PP
  551. A path preserving policy will only consider drives where the relative
  552. path being accessed already exists.
  553. .PP
  554. When using non\-path preserving policies paths will be cloned to target
  555. drives as necessary.
  556. .SS Filters
  557. .PP
  558. Policies basically search branches and create a list of files / paths
  559. for functions to work on.
  560. The policy is responsible for filtering and sorting.
  561. The policy type defines the sorting but filtering is mostly uniform as
  562. described below.
  563. .IP \[bu] 2
  564. No \f[B]search\f[] policies filter.
  565. .IP \[bu] 2
  566. All \f[B]action\f[] policies will filter out branches which are mounted
  567. \f[B]read\-only\f[] or tagged as \f[B]RO (read\-only)\f[].
  568. .IP \[bu] 2
  569. All \f[B]create\f[] policies will filter out branches which are mounted
  570. \f[B]read\-only\f[], tagged \f[B]RO (read\-only)\f[] or \f[B]NC (no
  571. create)\f[], or has available space less than \f[C]minfreespace\f[].
  572. .PP
  573. If all branches are filtered an error will be returned.
  574. Typically \f[B]EROFS\f[] (read\-only filesystem) or \f[B]ENOSPC\f[] (no
  575. space left on device) depending on the reasons.
  576. .SS Policy descriptions
  577. .PP
  578. .TS
  579. tab(@);
  580. lw(16.6n) lw(53.4n).
  581. T{
  582. Policy
  583. T}@T{
  584. Description
  585. T}
  586. _
  587. T{
  588. all
  589. T}@T{
  590. Search category: same as \f[B]epall\f[].
  591. Action category: same as \f[B]epall\f[].
  592. Create category: for \f[B]mkdir\f[], \f[B]mknod\f[], and
  593. \f[B]symlink\f[] it will apply to all branches.
  594. \f[B]create\f[] works like \f[B]ff\f[].
  595. T}
  596. T{
  597. epall (existing path, all)
  598. T}@T{
  599. Search category: same as \f[B]epff\f[] (but more expensive because it
  600. doesn\[aq]t stop after finding a valid branch).
  601. Action category: apply to all found.
  602. Create category: for \f[B]mkdir\f[], \f[B]mknod\f[], and
  603. \f[B]symlink\f[] it will apply to all found.
  604. \f[B]create\f[] works like \f[B]epff\f[] (but more expensive because it
  605. doesn\[aq]t stop after finding a valid branch).
  606. T}
  607. T{
  608. epff (existing path, first found)
  609. T}@T{
  610. Given the order of the branches, as defined at mount time or configured
  611. at runtime, act on the first one found where the relative path exists.
  612. T}
  613. T{
  614. eplfs (existing path, least free space)
  615. T}@T{
  616. Of all the branches on which the relative path exists choose the drive
  617. with the least free space.
  618. T}
  619. T{
  620. eplus (existing path, least used space)
  621. T}@T{
  622. Of all the branches on which the relative path exists choose the drive
  623. with the least used space.
  624. T}
  625. T{
  626. epmfs (existing path, most free space)
  627. T}@T{
  628. Of all the branches on which the relative path exists choose the drive
  629. with the most free space.
  630. T}
  631. T{
  632. eprand (existing path, random)
  633. T}@T{
  634. Calls \f[B]epall\f[] and then randomizes.
  635. Returns 1.
  636. T}
  637. T{
  638. erofs
  639. T}@T{
  640. Exclusively return \f[B]\-1\f[] with \f[B]errno\f[] set to
  641. \f[B]EROFS\f[] (read\-only filesystem).
  642. T}
  643. T{
  644. ff (first found)
  645. T}@T{
  646. Search category: same as \f[B]epff\f[].
  647. Action category: same as \f[B]epff\f[].
  648. Create category: Given the order of the drives, as defined at mount time
  649. or configured at runtime, act on the first one found.
  650. T}
  651. T{
  652. lfs (least free space)
  653. T}@T{
  654. Search category: same as \f[B]eplfs\f[].
  655. Action category: same as \f[B]eplfs\f[].
  656. Create category: Pick the drive with the least available free space.
  657. T}
  658. T{
  659. lus (least used space)
  660. T}@T{
  661. Search category: same as \f[B]eplus\f[].
  662. Action category: same as \f[B]eplus\f[].
  663. Create category: Pick the drive with the least used space.
  664. T}
  665. T{
  666. mfs (most free space)
  667. T}@T{
  668. Search category: same as \f[B]epmfs\f[].
  669. Action category: same as \f[B]epmfs\f[].
  670. Create category: Pick the drive with the most available free space.
  671. T}
  672. T{
  673. newest
  674. T}@T{
  675. Pick the file / directory with the largest mtime.
  676. T}
  677. T{
  678. rand (random)
  679. T}@T{
  680. Calls \f[B]all\f[] and then randomizes.
  681. Returns 1.
  682. T}
  683. .TE
  684. .PP
  685. \f[B]NOTE:\f[] If you are using an underlying filesystem that reserves
  686. blocks such as ext2, ext3, or ext4 be aware that mergerfs respects the
  687. reservation by using \f[C]f_bavail\f[] (number of free blocks for
  688. unprivileged users) rather than \f[C]f_bfree\f[] (number of free blocks)
  689. in policy calculations.
  690. \f[B]df\f[] does NOT use \f[C]f_bavail\f[], it uses \f[C]f_bfree\f[], so
  691. direct comparisons between \f[B]df\f[] output and mergerfs\[aq] policies
  692. is not appropriate.
  693. .SS Defaults
  694. .PP
  695. .TS
  696. tab(@);
  697. l l.
  698. T{
  699. Category
  700. T}@T{
  701. Policy
  702. T}
  703. _
  704. T{
  705. action
  706. T}@T{
  707. epall
  708. T}
  709. T{
  710. create
  711. T}@T{
  712. epmfs
  713. T}
  714. T{
  715. search
  716. T}@T{
  717. ff
  718. T}
  719. .TE
  720. .SS ioctl
  721. .PP
  722. When \f[C]ioctl\f[] is used with an open file then it will use the file
  723. handle which was created at the original \f[C]open\f[] call.
  724. However, when using \f[C]ioctl\f[] with a directory mergerfs will use
  725. the \f[C]open\f[] policy to find the directory to act on.
  726. .SS unlink
  727. .PP
  728. In FUSE there is an opaque "file handle" which is created by
  729. \f[C]open\f[], \f[C]create\f[], or \f[C]opendir\f[], passed to the
  730. kernel, and then is passed back to the FUSE userland application by the
  731. kernel.
  732. Unfortunately, the FUSE kernel driver does not always send the file
  733. handle when it theoretically could/should.
  734. This complicates certain behaviors / workflows particularly in the high
  735. level API.
  736. As a result mergerfs is currently doing a few hacky things.
  737. .PP
  738. libfuse2 and libfuse3, when using the high level API, will rename names
  739. to \f[C]\&.fuse_hiddenXXXXXX\f[] if the file is open when unlinked or
  740. renamed over.
  741. It does this so the file is still available when a request referencing
  742. the now missing file is made.
  743. This file however keeps a \f[C]rmdir\f[] from succeeding and can be
  744. picked up by software reading directories.
  745. .PP
  746. The change mergerfs has done is that if a file is open when an unlink or
  747. rename happens it will open the file and keep it open till closed by all
  748. those who opened it prior.
  749. When a request comes in referencing that file and it doesn\[aq]t include
  750. a file handle it will instead use the file handle created at
  751. unlink/rename time.
  752. .PP
  753. This won\[aq]t result in technically proper behavior but close enough
  754. for many usecases.
  755. .PP
  756. The plan is to rewrite mergerfs to use the low level API so these
  757. invasive libfuse changes are no longer necessary.
  758. .SS rename & link
  759. .PP
  760. \f[B]NOTE:\f[] If you\[aq]re receiving errors from software when files
  761. are moved / renamed / linked then you should consider changing the
  762. create policy to one which is \f[B]not\f[] path preserving, enabling
  763. \f[C]ignorepponrename\f[], or contacting the author of the offending
  764. software and requesting that \f[C]EXDEV\f[] (cross device / improper
  765. link) be properly handled.
  766. .PP
  767. \f[C]rename\f[] and \f[C]link\f[] are tricky functions in a union
  768. filesystem.
  769. \f[C]rename\f[] only works within a single filesystem or device.
  770. If a rename can\[aq]t be done atomically due to the source and
  771. destination paths existing on different mount points it will return
  772. \f[B]\-1\f[] with \f[B]errno = EXDEV\f[] (cross device / improper link).
  773. So if a \f[C]rename\f[]\[aq]s source and target are on different drives
  774. within the pool it creates an issue.
  775. .PP
  776. Originally mergerfs would return EXDEV whenever a rename was requested
  777. which was cross directory in any way.
  778. This made the code simple and was technically compliant with POSIX
  779. requirements.
  780. However, many applications fail to handle EXDEV at all and treat it as a
  781. normal error or otherwise handle it poorly.
  782. Such apps include: gvfsd\-fuse v1.20.3 and prior, Finder / CIFS/SMB
  783. client in Apple OSX 10.9+, NZBGet, Samba\[aq]s recycling bin feature.
  784. .PP
  785. As a result a compromise was made in order to get most software to work
  786. while still obeying mergerfs\[aq] policies.
  787. Below is the basic logic.
  788. .IP \[bu] 2
  789. If using a \f[B]create\f[] policy which tries to preserve directory
  790. paths (epff,eplfs,eplus,epmfs)
  791. .IP \[bu] 2
  792. Using the \f[B]rename\f[] policy get the list of files to rename
  793. .IP \[bu] 2
  794. For each file attempt rename:
  795. .RS 2
  796. .IP \[bu] 2
  797. If failure with ENOENT (no such file or directory) run \f[B]create\f[]
  798. policy
  799. .IP \[bu] 2
  800. If create policy returns the same drive as currently evaluating then
  801. clone the path
  802. .IP \[bu] 2
  803. Re\-attempt rename
  804. .RE
  805. .IP \[bu] 2
  806. If \f[B]any\f[] of the renames succeed the higher level rename is
  807. considered a success
  808. .IP \[bu] 2
  809. If \f[B]no\f[] renames succeed the first error encountered will be
  810. returned
  811. .IP \[bu] 2
  812. On success:
  813. .RS 2
  814. .IP \[bu] 2
  815. Remove the target from all drives with no source file
  816. .IP \[bu] 2
  817. Remove the source from all drives which failed to rename
  818. .RE
  819. .IP \[bu] 2
  820. If using a \f[B]create\f[] policy which does \f[B]not\f[] try to
  821. preserve directory paths
  822. .IP \[bu] 2
  823. Using the \f[B]rename\f[] policy get the list of files to rename
  824. .IP \[bu] 2
  825. Using the \f[B]getattr\f[] policy get the target path
  826. .IP \[bu] 2
  827. For each file attempt rename:
  828. .RS 2
  829. .IP \[bu] 2
  830. If the source drive != target drive:
  831. .IP \[bu] 2
  832. Clone target path from target drive to source drive
  833. .IP \[bu] 2
  834. Rename
  835. .RE
  836. .IP \[bu] 2
  837. If \f[B]any\f[] of the renames succeed the higher level rename is
  838. considered a success
  839. .IP \[bu] 2
  840. If \f[B]no\f[] renames succeed the first error encountered will be
  841. returned
  842. .IP \[bu] 2
  843. On success:
  844. .RS 2
  845. .IP \[bu] 2
  846. Remove the target from all drives with no source file
  847. .IP \[bu] 2
  848. Remove the source from all drives which failed to rename
  849. .RE
  850. .PP
  851. The the removals are subject to normal entitlement checks.
  852. .PP
  853. The above behavior will help minimize the likelihood of EXDEV being
  854. returned but it will still be possible.
  855. .PP
  856. \f[B]link\f[] uses the same strategy but without the removals.
  857. .SS readdir
  858. .PP
  859. readdir (http://linux.die.net/man/3/readdir) is different from all other
  860. filesystem functions.
  861. While it could have its own set of policies to tweak its behavior at
  862. this time it provides a simple union of files and directories found.
  863. Remember that any action or information queried about these files and
  864. directories come from the respective function.
  865. For instance: an \f[B]ls\f[] is a \f[B]readdir\f[] and for each
  866. file/directory returned \f[B]getattr\f[] is called.
  867. Meaning the policy of \f[B]getattr\f[] is responsible for choosing the
  868. file/directory which is the source of the metadata you see in an
  869. \f[B]ls\f[].
  870. .SS statfs / statvfs
  871. .PP
  872. statvfs (http://linux.die.net/man/2/statvfs) normalizes the source
  873. drives based on the fragment size and sums the number of adjusted blocks
  874. and inodes.
  875. This means you will see the combined space of all sources.
  876. Total, used, and free.
  877. The sources however are dedupped based on the drive so multiple sources
  878. on the same drive will not result in double counting its space.
  879. Filesystems mounted further down the tree of the branch will not be
  880. included when checking the mount\[aq]s stats.
  881. .PP
  882. The options \f[C]statfs\f[] and \f[C]statfs_ignore\f[] can be used to
  883. modify \f[C]statfs\f[] behavior.
  884. .SH BUILD / UPDATE
  885. .PP
  886. \f[B]NOTE:\f[] Prebuilt packages can be found at:
  887. https://github.com/trapexit/mergerfs/releases
  888. .PP
  889. First get the code from github (https://github.com/trapexit/mergerfs).
  890. .IP
  891. .nf
  892. \f[C]
  893. $\ git\ clone\ https://github.com/trapexit/mergerfs.git
  894. $\ #\ or
  895. $\ wget\ https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs\-<ver>.tar.gz
  896. \f[]
  897. .fi
  898. .SS Debian / Ubuntu
  899. .IP
  900. .nf
  901. \f[C]
  902. $\ cd\ mergerfs
  903. $\ sudo\ tools/install\-build\-pkgs
  904. $\ make\ deb
  905. $\ sudo\ dpkg\ \-i\ ../mergerfs_version_arch.deb
  906. \f[]
  907. .fi
  908. .SS RHEL / CentOS /Fedora
  909. .IP
  910. .nf
  911. \f[C]
  912. $\ su\ \-
  913. #\ cd\ mergerfs
  914. #\ tools/install\-build\-pkgs
  915. #\ make\ rpm
  916. #\ rpm\ \-i\ rpmbuild/RPMS/<arch>/mergerfs\-<verion>.<arch>.rpm
  917. \f[]
  918. .fi
  919. .SS Generically
  920. .PP
  921. Have git, g++, make, python installed.
  922. .IP
  923. .nf
  924. \f[C]
  925. $\ cd\ mergerfs
  926. $\ make
  927. $\ sudo\ make\ install
  928. \f[]
  929. .fi
  930. .SS Build options
  931. .IP
  932. .nf
  933. \f[C]
  934. $\ make\ help
  935. usage:\ make
  936. make\ USE_XATTR=0\ \ \ \ \ \ \-\ build\ program\ without\ xattrs\ functionality
  937. make\ STATIC=1\ \ \ \ \ \ \ \ \ \-\ build\ static\ binary
  938. make\ LTO=1\ \ \ \ \ \ \ \ \ \ \ \ \-\ build\ with\ link\ time\ optimization
  939. \f[]
  940. .fi
  941. .SH RUNTIME CONFIG
  942. .SS .mergerfs pseudo file
  943. .IP
  944. .nf
  945. \f[C]
  946. <mountpoint>/.mergerfs
  947. \f[]
  948. .fi
  949. .PP
  950. There is a pseudo file available at the mount point which allows for the
  951. runtime modification of certain \f[B]mergerfs\f[] options.
  952. The file will not show up in \f[B]readdir\f[] but can be
  953. \f[B]stat\f[]\[aq]ed and manipulated via
  954. {list,get,set}xattrs (http://linux.die.net/man/2/listxattr) calls.
  955. .PP
  956. Any changes made at runtime are \f[B]not\f[] persisted.
  957. If you wish for values to persist they must be included as options
  958. wherever you configure the mounting of mergerfs (/etc/fstab).
  959. .SS Keys
  960. .PP
  961. Use \f[C]xattr\ \-l\ /mountpoint/.mergerfs\f[] to see all supported
  962. keys.
  963. Some are informational and therefore read\-only.
  964. \f[C]setxattr\f[] will return EINVAL (invalid argument) on read\-only
  965. keys.
  966. .SS Values
  967. .PP
  968. Same as the command line.
  969. .SS user.mergerfs.branches
  970. .PP
  971. \f[B]NOTE:\f[] formerly \f[C]user.mergerfs.srcmounts\f[] but said key is
  972. still supported.
  973. .PP
  974. Used to query or modify the list of branches.
  975. When modifying there are several shortcuts to easy manipulation of the
  976. list.
  977. .PP
  978. .TS
  979. tab(@);
  980. l l.
  981. T{
  982. Value
  983. T}@T{
  984. Description
  985. T}
  986. _
  987. T{
  988. [list]
  989. T}@T{
  990. set
  991. T}
  992. T{
  993. +<[list]
  994. T}@T{
  995. prepend
  996. T}
  997. T{
  998. +>[list]
  999. T}@T{
  1000. append
  1001. T}
  1002. T{
  1003. \-[list]
  1004. T}@T{
  1005. remove all values provided
  1006. T}
  1007. T{
  1008. \-<
  1009. T}@T{
  1010. remove first in list
  1011. T}
  1012. T{
  1013. \->
  1014. T}@T{
  1015. remove last in list
  1016. T}
  1017. .TE
  1018. .PP
  1019. \f[C]xattr\ \-w\ user.mergerfs.branches\ +</mnt/drive3\ /mnt/pool/.mergerfs\f[]
  1020. .PP
  1021. The \f[C]=NC\f[], \f[C]=RO\f[], \f[C]=RW\f[] syntax works just as on the
  1022. command line.
  1023. .SS Example
  1024. .IP
  1025. .nf
  1026. \f[C]
  1027. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-l\ .mergerfs
  1028. user.mergerfs.branches:\ /mnt/a=RW:/mnt/b=RW
  1029. user.mergerfs.minfreespace:\ 4294967295
  1030. user.mergerfs.moveonenospc:\ false
  1031. \&...
  1032. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.category.search\ .mergerfs
  1033. ff
  1034. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-w\ user.mergerfs.category.search\ newest\ .mergerfs
  1035. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.category.search\ .mergerfs
  1036. newest
  1037. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-w\ user.mergerfs.branches\ +/mnt/c\ .mergerfs
  1038. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.branches\ .mergerfs
  1039. /mnt/a:/mnt/b:/mnt/c
  1040. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-w\ user.mergerfs.branches\ =/mnt/c\ .mergerfs
  1041. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.branches\ .mergerfs
  1042. /mnt/c
  1043. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-w\ user.mergerfs.branches\ \[aq]+</mnt/a:/mnt/b\[aq]\ .mergerfs
  1044. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.branches\ .mergerfs
  1045. /mnt/a:/mnt/b:/mnt/c
  1046. \f[]
  1047. .fi
  1048. .SS file / directory xattrs
  1049. .PP
  1050. While they won\[aq]t show up when using
  1051. listxattr (http://linux.die.net/man/2/listxattr) \f[B]mergerfs\f[]
  1052. offers a number of special xattrs to query information about the files
  1053. served.
  1054. To access the values you will need to issue a
  1055. getxattr (http://linux.die.net/man/2/getxattr) for one of the following:
  1056. .IP \[bu] 2
  1057. \f[B]user.mergerfs.basepath\f[]: the base mount point for the file given
  1058. the current getattr policy
  1059. .IP \[bu] 2
  1060. \f[B]user.mergerfs.relpath\f[]: the relative path of the file from the
  1061. perspective of the mount point
  1062. .IP \[bu] 2
  1063. \f[B]user.mergerfs.fullpath\f[]: the full path of the original file
  1064. given the getattr policy
  1065. .IP \[bu] 2
  1066. \f[B]user.mergerfs.allpaths\f[]: a NUL (\[aq]\[aq]) separated list of
  1067. full paths to all files found
  1068. .IP
  1069. .nf
  1070. \f[C]
  1071. [trapexit:/mnt/mergerfs]\ $\ ls
  1072. A\ B\ C
  1073. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.fullpath\ A
  1074. /mnt/a/full/path/to/A
  1075. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.basepath\ A
  1076. /mnt/a
  1077. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.relpath\ A
  1078. /full/path/to/A
  1079. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.allpaths\ A\ |\ tr\ \[aq]\\0\[aq]\ \[aq]\\n\[aq]
  1080. /mnt/a/full/path/to/A
  1081. /mnt/b/full/path/to/A
  1082. \f[]
  1083. .fi
  1084. .SH UPGRADE
  1085. .PP
  1086. mergerfs can be upgraded live by mounting on top of the previous
  1087. version.
  1088. Simply install the new version of mergerfs and follow the instructions
  1089. below.
  1090. .PP
  1091. Add \f[C]nonempty\f[] to your mergerfs option list and call mergerfs
  1092. again or if using \f[C]/etc/fstab\f[] call for it to mount again.
  1093. Existing open files and such will continue to work fine though they
  1094. won\[aq]t see runtime changes since any such change would be the new
  1095. mount.
  1096. If you plan on changing settings with the new mount you should / could
  1097. apply those before mounting the new version.
  1098. .IP
  1099. .nf
  1100. \f[C]
  1101. $\ sudo\ mount\ /mnt/mergerfs
  1102. $\ mount\ |\ grep\ mergerfs
  1103. media\ on\ /mnt/mergerfs\ type\ fuse.mergerfs\ (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
  1104. media\ on\ /mnt/mergerfs\ type\ fuse.mergerfs\ (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
  1105. \f[]
  1106. .fi
  1107. .SH TOOLING
  1108. .IP \[bu] 2
  1109. https://github.com/trapexit/mergerfs\-tools
  1110. .IP \[bu] 2
  1111. mergerfs.ctl: A tool to make it easier to query and configure mergerfs
  1112. at runtime
  1113. .IP \[bu] 2
  1114. mergerfs.fsck: Provides permissions and ownership auditing and the
  1115. ability to fix them
  1116. .IP \[bu] 2
  1117. mergerfs.dedup: Will help identify and optionally remove duplicate files
  1118. .IP \[bu] 2
  1119. mergerfs.dup: Ensure there are at least N copies of a file across the
  1120. pool
  1121. .IP \[bu] 2
  1122. mergerfs.balance: Rebalance files across drives by moving them from the
  1123. most filled to the least filled
  1124. .IP \[bu] 2
  1125. mergerfs.consolidate: move files within a single mergerfs directory to
  1126. the drive with most free space
  1127. .IP \[bu] 2
  1128. mergerfs.mktrash: Creates FreeDesktop.org Trash specification compatible
  1129. directories on a mergerfs mount
  1130. .IP \[bu] 2
  1131. https://github.com/trapexit/scorch
  1132. .IP \[bu] 2
  1133. scorch: A tool to help discover silent corruption of files and keep
  1134. track of files
  1135. .IP \[bu] 2
  1136. https://github.com/trapexit/bbf
  1137. .IP \[bu] 2
  1138. bbf (bad block finder): a tool to scan for and \[aq]fix\[aq] hard drive
  1139. bad blocks and find the files using those blocks
  1140. .SH CACHING
  1141. .SS page caching
  1142. .PP
  1143. https://en.wikipedia.org/wiki/Page_cache
  1144. .PP
  1145. tl;dr: * cache.files=off: Disables page caching.
  1146. Underlying files cached, mergerfs files are not.
  1147. * cache.files=partial: Enables page caching.
  1148. Underlying files cached, mergerfs files cached while open.
  1149. * cache.files=full: Enables page caching.
  1150. Underlying files cached, mergerfs files cached across opens.
  1151. * cache.files=auto\-full: Enables page caching.
  1152. Underlying files cached, mergerfs files cached across opens if mtime and
  1153. size are unchanged since previous open.
  1154. * cache.files=libfuse: follow traditional libfuse \f[C]direct_io\f[],
  1155. \f[C]kernel_cache\f[], and \f[C]auto_cache\f[] arguments.
  1156. .PP
  1157. FUSE, which mergerfs uses, offers a number of page caching modes.
  1158. mergerfs tries to simplify their use via the \f[C]cache.files\f[]
  1159. option.
  1160. It can and should replace usage of \f[C]direct_io\f[],
  1161. \f[C]kernel_cache\f[], and \f[C]auto_cache\f[].
  1162. .PP
  1163. Due to mergerfs using FUSE and therefore being a userland process
  1164. proxying existing filesystems the kernel will double cache the content
  1165. being read and written through mergerfs.
  1166. Once from the underlying filesystem and once from mergerfs (it sees them
  1167. as two separate entities).
  1168. Using \f[C]cache.files=off\f[] will keep the double caching from
  1169. happening by disabling caching of mergerfs but this has the side effect
  1170. that \f[I]all\f[] read and write calls will be passed to mergerfs which
  1171. may be slower than enabling caching, you lose shared \f[C]mmap\f[]
  1172. support which can affect apps such as rtorrent, and no read\-ahead will
  1173. take place.
  1174. The kernel will still cache the underlying filesystem data but that only
  1175. helps so much given mergerfs will still process all requests.
  1176. .PP
  1177. If you do enable file page caching,
  1178. \f[C]cache.files=partial|full|auto\-full\f[], you should also enable
  1179. \f[C]dropcacheonclose\f[] which will cause mergerfs to instruct the
  1180. kernel to flush the underlying file\[aq]s page cache when the file is
  1181. closed.
  1182. This behavior is the same as the rsync fadvise / drop cache patch and
  1183. Feh\[aq]s nocache project.
  1184. .PP
  1185. If most files are read once through and closed (like media) it is best
  1186. to enable \f[C]dropcacheonclose\f[] regardless of caching mode in order
  1187. to minimize buffer bloat.
  1188. .PP
  1189. It is difficult to balance memory usage, cache bloat & duplication, and
  1190. performance.
  1191. Ideally mergerfs would be able to disable caching for the files it
  1192. reads/writes but allow page caching for itself.
  1193. That would limit the FUSE overhead.
  1194. However, there isn\[aq]t a good way to achieve this.
  1195. It would need to open all files with O_DIRECT which places limitations
  1196. on the what underlying filesystems would be supported and complicates
  1197. the code.
  1198. .PP
  1199. kernel documentation:
  1200. https://www.kernel.org/doc/Documentation/filesystems/fuse\-io.txt
  1201. .SS entry & attribute caching
  1202. .PP
  1203. Given the relatively high cost of FUSE due to the kernel <\-> userspace
  1204. round trips there are kernel side caches for file entries and
  1205. attributes.
  1206. The entry cache limits the \f[C]lookup\f[] calls to mergerfs which ask
  1207. if a file exists.
  1208. The attribute cache limits the need to make \f[C]getattr\f[] calls to
  1209. mergerfs which provide file attributes (mode, size, type, etc.).
  1210. As with the page cache these should not be used if the underlying
  1211. filesystems are being manipulated at the same time as it could lead to
  1212. odd behavior or data corruption.
  1213. The options for setting these are \f[C]cache.entry\f[] and
  1214. \f[C]cache.negative_entry\f[] for the entry cache and
  1215. \f[C]cache.attr\f[] for the attributes cache.
  1216. \f[C]cache.negative_entry\f[] refers to the timeout for negative
  1217. responses to lookups (non\-existent files).
  1218. .SS writeback caching
  1219. .PP
  1220. When \f[C]cache.files\f[] is enabled the default is for it to perform
  1221. writethrough caching.
  1222. This behavior won\[aq]t help improve performance as each write still
  1223. goes one for one through the filesystem.
  1224. By enabling the FUSE writeback cache small writes may be aggregated by
  1225. the kernel and then sent to mergerfs as one larger request.
  1226. This can greatly improve the throughput for apps which write to files
  1227. inefficiently.
  1228. The amount the kernel can aggregate is limited by the size of a FUSE
  1229. message.
  1230. Read the \f[C]fuse_msg_size\f[] section for more details.
  1231. .PP
  1232. There is a small side effect as a result of enabling wrtieback caching.
  1233. Underlying files won\[aq]t ever be opened with O_APPEND or O_WRONLY.
  1234. The former because the kernel then manages append mode and the latter
  1235. because the kernel may request file data from mergerfs to populate the
  1236. write cache.
  1237. The O_APPEND change means that if a file is changed outside of mergerfs
  1238. it could lead to corruption as the kernel won\[aq]t know the end of the
  1239. file has changed.
  1240. That said any time you use caching you should keep from using the same
  1241. file outside of mergerfs at the same time.
  1242. .PP
  1243. Note that if an application is properly sizing writes then writeback
  1244. caching will have little or no effect.
  1245. It will only help with writes of sizes below the FUSE message size (128K
  1246. on older kernels, 1M on newer).
  1247. .SS policy caching
  1248. .PP
  1249. Policies are run every time a function (with a policy as mentioned
  1250. above) is called.
  1251. These policies can be expensive depending on mergerfs\[aq] setup and
  1252. client usage patterns.
  1253. Generally we wouldn\[aq]t want to cache policy results because it may
  1254. result in stale responses if the underlying drives are used directly.
  1255. .PP
  1256. The \f[C]open\f[] policy cache will cache the result of an \f[C]open\f[]
  1257. policy for a particular input for \f[C]cache.open\f[] seconds or until
  1258. the file is unlinked.
  1259. Each file close (release) will randomly chose to clean up the cache of
  1260. expired entries.
  1261. .PP
  1262. This cache is really only useful in cases where you have a large number
  1263. of branches and \f[C]open\f[] is called on the same files repeatedly
  1264. (like \f[B]Transmission\f[] which opens and closes a file on every
  1265. read/write presumably to keep file handle usage low).
  1266. .SS statfs caching
  1267. .PP
  1268. Of the syscalls used by mergerfs in policies the \f[C]statfs\f[] /
  1269. \f[C]statvfs\f[] call is perhaps the most expensive.
  1270. It\[aq]s used to find out the available space of a drive and whether it
  1271. is mounted read\-only.
  1272. Depending on the setup and usage pattern these queries can be relatively
  1273. costly.
  1274. When \f[C]cache.statfs\f[] is enabled all calls to \f[C]statfs\f[] by a
  1275. policy will be cached for the number of seconds its set to.
  1276. .PP
  1277. Example: If the create policy is \f[C]mfs\f[] and the timeout is 60 then
  1278. for that 60 seconds the same drive will be returned as the target for
  1279. creates because the available space won\[aq]t be updated for that time.
  1280. .SS symlink caching
  1281. .PP
  1282. As of version 4.20 Linux supports symlink caching.
  1283. Significant performance increases can be had in workloads which use a
  1284. lot of symlinks.
  1285. Setting \f[C]cache.symlinks=true\f[] will result in requesting symlink
  1286. caching from the kernel only if supported.
  1287. As a result its safe to enable it on systems prior to 4.20.
  1288. That said it is disabled by default for now.
  1289. You can see if caching is enabled by querying the xattr
  1290. \f[C]user.mergerfs.cache.symlinks\f[] but given it must be requested at
  1291. startup you can not change it at runtime.
  1292. .SS readdir caching
  1293. .PP
  1294. As of version 4.20 Linux supports readdir caching.
  1295. This can have a significant impact on directory traversal.
  1296. Especially when combined with entry (\f[C]cache.entry\f[]) and attribute
  1297. (\f[C]cache.attr\f[]) caching.
  1298. Setting \f[C]cache.readdir=true\f[] will result in requesting readdir
  1299. caching from the kernel on each \f[C]opendir\f[].
  1300. If the kernel doesn\[aq]t support readdir caching setting the option to
  1301. \f[C]true\f[] has no effect.
  1302. This option is configurable at runtime via xattr
  1303. \f[C]user.mergerfs.cache.readdir\f[].
  1304. .SS tiered caching
  1305. .PP
  1306. Some storage technologies support what some call "tiered" caching.
  1307. The placing of usually smaller, faster storage as a transparent cache to
  1308. larger, slower storage.
  1309. NVMe, SSD, Optane in front of traditional HDDs for instance.
  1310. .PP
  1311. MergerFS does not natively support any sort of tiered caching.
  1312. Most users have no use for such a feature and its inclusion would
  1313. complicate the code.
  1314. However, there are a few situations where a cache drive could help with
  1315. a typical mergerfs setup.
  1316. .IP "1." 3
  1317. Fast network, slow drives, many readers: You\[aq]ve a 10+Gbps network
  1318. with many readers and your regular drives can\[aq]t keep up.
  1319. .IP "2." 3
  1320. Fast network, slow drives, small\[aq]ish bursty writes: You have a
  1321. 10+Gbps network and wish to transfer amounts of data less than your
  1322. cache drive but wish to do so quickly.
  1323. .PP
  1324. With #1 its arguable if you should be using mergerfs at all.
  1325. RAID would probably be the better solution.
  1326. If you\[aq]re going to use mergerfs there are other tactics that may
  1327. help: spreading the data across drives (see the mergerfs.dup tool) and
  1328. setting \f[C]func.open=rand\f[], using \f[C]symlinkify\f[], or using
  1329. dm\-cache or a similar technology to add tiered cache to the underlying
  1330. device.
  1331. .PP
  1332. With #2 one could use dm\-cache as well but there is another solution
  1333. which requires only mergerfs and a cronjob.
  1334. .IP "1." 3
  1335. Create 2 mergerfs pools.
  1336. One which includes just the slow drives and one which has both the fast
  1337. drives (SSD,NVME,etc.) and slow drives.
  1338. .IP "2." 3
  1339. The \[aq]cache\[aq] pool should have the cache drives listed first.
  1340. .IP "3." 3
  1341. The best \f[C]create\f[] policies to use for the \[aq]cache\[aq] pool
  1342. would probably be \f[C]ff\f[], \f[C]epff\f[], \f[C]lfs\f[], or
  1343. \f[C]eplfs\f[].
  1344. The latter two under the assumption that the cache drive(s) are far
  1345. smaller than the backing drives.
  1346. If using path preserving policies remember that you\[aq]ll need to
  1347. manually create the core directories of those paths you wish to be
  1348. cached.
  1349. Be sure the permissions are in sync.
  1350. Use \f[C]mergerfs.fsck\f[] to check / correct them.
  1351. You could also tag the slow drives as \f[C]=NC\f[] though that\[aq]d
  1352. mean if the cache drives fill you\[aq]d get "out of space" errors.
  1353. .IP "4." 3
  1354. Enable \f[C]moveonenospc\f[] and set \f[C]minfreespace\f[]
  1355. appropriately.
  1356. To make sure there is enough room on the "slow" pool you might want to
  1357. set \f[C]minfreespace\f[] to at least as large as the size of the
  1358. largest cache drive if not larger.
  1359. This way in the worst case the whole of the cache drive(s) can be moved
  1360. to the other drives.
  1361. .IP "5." 3
  1362. Set your programs to use the cache pool.
  1363. .IP "6." 3
  1364. Save one of the below scripts or create you\[aq]re own.
  1365. .IP "7." 3
  1366. Use \f[C]cron\f[] (as root) to schedule the command at whatever
  1367. frequency is appropriate for your workflow.
  1368. .SS time based expiring
  1369. .PP
  1370. Move files from cache to backing pool based only on the last time the
  1371. file was accessed.
  1372. Replace \f[C]\-atime\f[] with \f[C]\-amin\f[] if you want minutes rather
  1373. than days.
  1374. May want to use the \f[C]fadvise\f[] / \f[C]\-\-drop\-cache\f[] version
  1375. of rsync or run rsync with the tool "nocache".
  1376. .IP
  1377. .nf
  1378. \f[C]
  1379. #!/bin/bash
  1380. if\ [\ $#\ !=\ 3\ ];\ then
  1381. \ \ echo\ "usage:\ $0\ <cache\-drive>\ <backing\-pool>\ <days\-old>"
  1382. \ \ exit\ 1
  1383. fi
  1384. CACHE="${1}"
  1385. BACKING="${2}"
  1386. N=${3}
  1387. find\ "${CACHE}"\ \-type\ f\ \-atime\ +${N}\ \-printf\ \[aq]%P\\n\[aq]\ |\ \\
  1388. \ \ rsync\ \-\-files\-from=\-\ \-axqHAXWES\ \-\-preallocate\ \-\-remove\-source\-files\ "${CACHE}/"\ "${BACKING}/"
  1389. \f[]
  1390. .fi
  1391. .SS percentage full expiring
  1392. .PP
  1393. Move the oldest file from the cache to the backing pool.
  1394. Continue till below percentage threshold.
  1395. .IP
  1396. .nf
  1397. \f[C]
  1398. #!/bin/bash
  1399. if\ [\ $#\ !=\ 3\ ];\ then
  1400. \ \ echo\ "usage:\ $0\ <cache\-drive>\ <backing\-pool>\ <percentage>"
  1401. \ \ exit\ 1
  1402. fi
  1403. CACHE="${1}"
  1404. BACKING="${2}"
  1405. PERCENTAGE=${3}
  1406. set\ \-o\ errexit
  1407. while\ [\ $(df\ \-\-output=pcent\ "${CACHE}"\ |\ grep\ \-v\ Use\ |\ cut\ \-d\[aq]%\[aq]\ \-f1)\ \-gt\ ${PERCENTAGE}\ ]
  1408. do
  1409. \ \ \ \ FILE=$(find\ "${CACHE}"\ \-type\ f\ \-printf\ \[aq]%A\@\ %P\\n\[aq]\ |\ \\
  1410. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ sort\ |\ \\
  1411. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ head\ \-n\ 1\ |\ \\
  1412. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ cut\ \-d\[aq]\ \[aq]\ \-f2\-)
  1413. \ \ \ \ test\ \-n\ "${FILE}"
  1414. \ \ \ \ rsync\ \-axqHAXWES\ \-\-preallocate\ \-\-remove\-source\-files\ "${CACHE}/./${FILE}"\ "${BACKING}/"
  1415. done
  1416. \f[]
  1417. .fi
  1418. .SH PERFORMANCE
  1419. .PP
  1420. mergerfs is at its core just a proxy and therefore its theoretical max
  1421. performance is that of the underlying devices.
  1422. However, given it is a FUSE filesystem working from userspace there is
  1423. an increase in overhead relative to kernel based solutions.
  1424. That said the performance can match the theoretical max but it depends
  1425. greatly on the system\[aq]s configuration.
  1426. Especially when adding network filesystems into the mix there are many
  1427. variables which can impact performance.
  1428. Drive speeds and latency, network speeds and latency, general
  1429. concurrency, read/write sizes, etc.
  1430. Unfortunately, given the number of variables it has been difficult to
  1431. find a single set of settings which provide optimal performance.
  1432. If you\[aq]re having performance issues please look over the suggestions
  1433. below (including the benchmarking section.)
  1434. .PP
  1435. NOTE: be sure to read about these features before changing them to
  1436. understand what behaviors it may impact
  1437. .IP \[bu] 2
  1438. enable (or disable) \f[C]splice_move\f[], \f[C]splice_read\f[], and
  1439. \f[C]splice_write\f[]
  1440. .IP \[bu] 2
  1441. disable \f[C]security_capability\f[] and/or \f[C]xattr\f[]
  1442. .IP \[bu] 2
  1443. increase cache timeouts \f[C]cache.attr\f[], \f[C]cache.entry\f[],
  1444. \f[C]cache.negative_entry\f[]
  1445. .IP \[bu] 2
  1446. enable (or disable) page caching (\f[C]cache.files\f[])
  1447. .IP \[bu] 2
  1448. enable \f[C]cache.writeback\f[]
  1449. .IP \[bu] 2
  1450. enable \f[C]cache.open\f[]
  1451. .IP \[bu] 2
  1452. enable \f[C]cache.statfs\f[]
  1453. .IP \[bu] 2
  1454. enable \f[C]cache.symlinks\f[]
  1455. .IP \[bu] 2
  1456. enable \f[C]cache.readdir\f[]
  1457. .IP \[bu] 2
  1458. change the number of worker threads
  1459. .IP \[bu] 2
  1460. disable \f[C]posix_acl\f[]
  1461. .IP \[bu] 2
  1462. disable \f[C]async_read\f[]
  1463. .IP \[bu] 2
  1464. test theoretical performance using \f[C]nullrw\f[] or mounting a ram
  1465. disk
  1466. .IP \[bu] 2
  1467. use \f[C]symlinkify\f[] if your data is largely static and read\-only
  1468. .IP \[bu] 2
  1469. use tiered cache drives
  1470. .IP \[bu] 2
  1471. use LVM and LVM cache to place a SSD in front of your HDDs
  1472. .PP
  1473. If you come across a setting that significantly impacts performance
  1474. please contact trapexit so he may investigate further.
  1475. .SH BENCHMARKING
  1476. .PP
  1477. Filesystems are complicated.
  1478. They do many things and many of those are interconnected.
  1479. Additionally, the OS, drivers, hardware, etc.
  1480. all can impact performance.
  1481. Therefore, when benchmarking, it is \f[B]necessary\f[] that the test
  1482. focus as narrowly as possible.
  1483. .PP
  1484. For most throughput is the key benchmark.
  1485. To test throughput \f[C]dd\f[] is useful but \f[B]must\f[] be used with
  1486. the correct settings in order to ensure the filesystem or device is
  1487. actually being tested.
  1488. The OS can and will cache data.
  1489. Without forcing synchronous reads and writes and/or disabling caching
  1490. the values returned will not be representative of the device\[aq]s true
  1491. performance.
  1492. .PP
  1493. When benchmarking through mergerfs ensure you only use 1 branch to
  1494. remove any possibility of the policies complicating the situation.
  1495. Benchmark the underlying filesystem first and then mount mergerfs over
  1496. it and test again.
  1497. If you\[aq]re experience speeds below your expectation you will need to
  1498. narrow down precisely which component is leading to the slowdown.
  1499. Preferably test the following in the order listed (but not combined).
  1500. .IP "1." 3
  1501. Enable \f[C]nullrw\f[] mode with \f[C]nullrw=true\f[].
  1502. This will effectively make reads and writes no\-ops.
  1503. Removing the underlying device / filesystem from the equation.
  1504. This will give us the top theoretical speeds.
  1505. .IP "2." 3
  1506. Mount mergerfs over \f[C]tmpfs\f[].
  1507. \f[C]tmpfs\f[] is a RAM disk.
  1508. Extremely high speed and very low latency.
  1509. This is a more realistic best case scenario.
  1510. Example: \f[C]mount\ \-t\ tmpfs\ \-o\ size=2G\ tmpfs\ /tmp/tmpfs\f[]
  1511. .IP "3." 3
  1512. Mount mergerfs over a local drive.
  1513. NVMe, SSD, HDD, etc.
  1514. If you have more than one I\[aq]d suggest testing each of them as drives
  1515. and/or controllers (their drivers) could impact performance.
  1516. .IP "4." 3
  1517. Finally, if you intend to use mergerfs with a network filesystem, either
  1518. as the source of data or to combine with another through mergerfs, test
  1519. each of those alone as above.
  1520. .PP
  1521. Once you find the component which has the performance issue you can do
  1522. further testing with different options to see if they impact
  1523. performance.
  1524. For reads and writes the most relevant would be: \f[C]cache.files\f[],
  1525. \f[C]async_read\f[], \f[C]splice_move\f[], \f[C]splice_read\f[],
  1526. \f[C]splice_write\f[].
  1527. Less likely but relevant when using NFS or with certain filesystems
  1528. would be \f[C]security_capability\f[], \f[C]xattr\f[], and
  1529. \f[C]posix_acl\f[].
  1530. If you find a specific system, drive, filesystem, controller, etc.
  1531. that performs poorly contact trapexit so he may investigate further.
  1532. .PP
  1533. Sometimes the problem is really the application accessing or writing
  1534. data through mergerfs.
  1535. Some software use small buffer sizes which can lead to more requests and
  1536. therefore greater overhead.
  1537. You can test this out yourself by replace \f[C]bs=1M\f[] in the examples
  1538. below with \f[C]ibs\f[] or \f[C]obs\f[] and using a size of \f[C]512\f[]
  1539. instead of \f[C]1M\f[].
  1540. In one example test using \f[C]nullrw\f[] the write speed dropped from
  1541. 4.9GB/s to 69.7MB/s when moving from \f[C]1M\f[] to \f[C]512\f[].
  1542. Similar results were had when testing reads.
  1543. Small writes overhead may be improved by leveraging a write cache but in
  1544. casual tests little gain was found.
  1545. More tests will need to be done before this feature would become
  1546. available.
  1547. If you have an app that appears slow with mergerfs it could be due to
  1548. this.
  1549. Contact trapexit so he may investigate further.
  1550. .SS write benchmark
  1551. .IP
  1552. .nf
  1553. \f[C]
  1554. $\ dd\ if=/dev/zero\ of=/mnt/mergerfs/1GB.file\ bs=1M\ count=1024\ oflag=nocache\ conv=fdatasync\ status=progress
  1555. \f[]
  1556. .fi
  1557. .SS read benchmark
  1558. .IP
  1559. .nf
  1560. \f[C]
  1561. $\ dd\ if=/mnt/mergerfs/1GB.file\ of=/dev/null\ bs=1M\ count=1024\ iflag=nocache\ conv=fdatasync\ status=progress
  1562. \f[]
  1563. .fi
  1564. .SH TIPS / NOTES
  1565. .IP \[bu] 2
  1566. \f[B]use_ino\f[] will only work when used with mergerfs 2.18.0 and
  1567. above.
  1568. .IP \[bu] 2
  1569. Run mergerfs as \f[C]root\f[] (with \f[B]allow_other\f[]) unless
  1570. you\[aq]re merging paths which are owned by the same user otherwise
  1571. strange permission issues may arise.
  1572. .IP \[bu] 2
  1573. https://github.com/trapexit/backup\-and\-recovery\-howtos : A set of
  1574. guides / howtos on creating a data storage system, backing it up,
  1575. maintaining it, and recovering from failure.
  1576. .IP \[bu] 2
  1577. If you don\[aq]t see some directories and files you expect in a merged
  1578. point or policies seem to skip drives be sure the user has permission to
  1579. all the underlying directories.
  1580. Use \f[C]mergerfs.fsck\f[] to audit the drive for out of sync
  1581. permissions.
  1582. .IP \[bu] 2
  1583. Do \f[B]not\f[] use \f[C]cache.files=off\f[] (or \f[C]direct_io\f[]) if
  1584. you expect applications (such as rtorrent) to
  1585. mmap (http://linux.die.net/man/2/mmap) files.
  1586. Shared mmap is not currently supported in FUSE w/ \f[C]direct_io\f[]
  1587. enabled.
  1588. Enabling \f[C]dropcacheonclose\f[] is recommended when
  1589. \f[C]cache.files=partial|full|auto\-full\f[] or
  1590. \f[C]direct_io=false\f[].
  1591. .IP \[bu] 2
  1592. Since POSIX functions give only a singular error or success its
  1593. difficult to determine the proper behavior when applying the function to
  1594. multiple targets.
  1595. \f[B]mergerfs\f[] will return an error only if all attempts of an action
  1596. fail.
  1597. Any success will lead to a success returned.
  1598. This means however that some odd situations may arise.
  1599. .IP \[bu] 2
  1600. Kodi (http://kodi.tv), Plex (http://plex.tv),
  1601. Subsonic (http://subsonic.org), etc.
  1602. can use directory mtime (http://linux.die.net/man/2/stat) to more
  1603. efficiently determine whether to scan for new content rather than simply
  1604. performing a full scan.
  1605. If using the default \f[B]getattr\f[] policy of \f[B]ff\f[] its possible
  1606. those programs will miss an update on account of it returning the first
  1607. directory found\[aq]s \f[B]stat\f[] info and its a later directory on
  1608. another mount which had the \f[B]mtime\f[] recently updated.
  1609. To fix this you will want to set \f[B]func.getattr=newest\f[].
  1610. Remember though that this is just \f[B]stat\f[].
  1611. If the file is later \f[B]open\f[]\[aq]ed or \f[B]unlink\f[]\[aq]ed and
  1612. the policy is different for those then a completely different file or
  1613. directory could be acted on.
  1614. .IP \[bu] 2
  1615. Some policies mixed with some functions may result in strange behaviors.
  1616. Not that some of these behaviors and race conditions couldn\[aq]t happen
  1617. outside \f[B]mergerfs\f[] but that they are far more likely to occur on
  1618. account of the attempt to merge together multiple sources of data which
  1619. could be out of sync due to the different policies.
  1620. .IP \[bu] 2
  1621. For consistency its generally best to set \f[B]category\f[] wide
  1622. policies rather than individual \f[B]func\f[]\[aq]s.
  1623. This will help limit the confusion of tools such as
  1624. rsync (http://linux.die.net/man/1/rsync).
  1625. However, the flexibility is there if needed.
  1626. .SH KNOWN ISSUES / BUGS
  1627. .SS directory mtime is not being updated
  1628. .PP
  1629. Remember that the default policy for \f[C]getattr\f[] is \f[C]ff\f[].
  1630. The information for the first directory found will be returned.
  1631. If it wasn\[aq]t the directory which had been updated then it will
  1632. appear outdated.
  1633. .PP
  1634. The reason this is the default is because any other policy would be more
  1635. expensive and for many applications it is unnecessary.
  1636. To always return the directory with the most recent mtime or a faked
  1637. value based on all found would require a scan of all drives.
  1638. .PP
  1639. If you always want the directory information from the one with the most
  1640. recent mtime then use the \f[C]newest\f[] policy for \f[C]getattr\f[].
  1641. .SS \[aq]mv /mnt/pool/foo /mnt/disk1/foo\[aq] removes \[aq]foo\[aq]
  1642. .PP
  1643. This is not a bug.
  1644. .PP
  1645. Run in verbose mode to better understand what\[aq]s happening:
  1646. .IP
  1647. .nf
  1648. \f[C]
  1649. $\ mv\ \-v\ /mnt/pool/foo\ /mnt/disk1/foo
  1650. copied\ \[aq]/mnt/pool/foo\[aq]\ \->\ \[aq]/mnt/disk1/foo\[aq]
  1651. removed\ \[aq]/mnt/pool/foo\[aq]
  1652. $\ ls\ /mnt/pool/foo
  1653. ls:\ cannot\ access\ \[aq]/mnt/pool/foo\[aq]:\ No\ such\ file\ or\ directory
  1654. \f[]
  1655. .fi
  1656. .PP
  1657. \f[C]mv\f[], when working across devices, is copying the source to
  1658. target and then removing the source.
  1659. Since the source \f[B]is\f[] the target in this case, depending on the
  1660. unlink policy, it will remove the just copied file and other files
  1661. across the branches.
  1662. .PP
  1663. If you want to move files to one drive just copy them there and use
  1664. mergerfs.dedup to clean up the old paths or manually remove them from
  1665. the branches directly.
  1666. .SS cached memory appears greater than it should be
  1667. .PP
  1668. Use \f[C]cache.files=off\f[] and/or \f[C]dropcacheonclose=true\f[].
  1669. See the section on page caching.
  1670. .SS NFS clients returning ESTALE / Stale file handle
  1671. .PP
  1672. Be sure to use \f[C]noforget\f[] and \f[C]use_ino\f[] arguments.
  1673. .SS NFS clients don\[aq]t work
  1674. .PP
  1675. Some NFS clients appear to fail when a mergerfs mount is exported.
  1676. Kodi in particular seems to have issues.
  1677. .PP
  1678. Try enabling the \f[C]use_ino\f[] option.
  1679. Some have reported that it fixes the issue.
  1680. .SS rtorrent fails with ENODEV (No such device)
  1681. .PP
  1682. Be sure to set \f[C]cache.files=partial|full|auto\-full\f[] or turn off
  1683. \f[C]direct_io\f[].
  1684. rtorrent and some other applications use
  1685. mmap (http://linux.die.net/man/2/mmap) to read and write to files and
  1686. offer no fallback to traditional methods.
  1687. FUSE does not currently support mmap while using \f[C]direct_io\f[].
  1688. There may be a performance penalty on writes with \f[C]direct_io\f[] off
  1689. as well as the problem of double caching but it\[aq]s the only way to
  1690. get such applications to work.
  1691. If the performance loss is too high for other apps you can mount
  1692. mergerfs twice.
  1693. Once with \f[C]direct_io\f[] enabled and one without it.
  1694. Be sure to set \f[C]dropcacheonclose=true\f[] if not using
  1695. \f[C]direct_io\f[].
  1696. .SS rtorrent fails with files >= 4GiB
  1697. .PP
  1698. This is a kernel bug with mmap and FUSE on 32bit platforms.
  1699. A fix should become available for all LTS releases.
  1700. .PP
  1701. https://marc.info/?l=linux\-fsdevel&m=155550785230874&w=2
  1702. .SS Crashing on OpenVZ
  1703. .PP
  1704. There appears to be a bug in the OpenVZ kernel with regard to how it
  1705. handles ioctl calls.
  1706. It is making invalid requests which leads to a crash.
  1707. As of 2019\-12\-10 there is a bug report filed with OpenVZ but it is not
  1708. yet fixed.
  1709. .SS Plex doesn\[aq]t work with mergerfs
  1710. .PP
  1711. It does.
  1712. If you\[aq]re trying to put Plex\[aq]s config / metadata on mergerfs you
  1713. have to leave \f[C]direct_io\f[] off because Plex is using sqlite3 which
  1714. apparently needs mmap.
  1715. mmap doesn\[aq]t work with \f[C]direct_io\f[].
  1716. To fix this place the data elsewhere or disable \f[C]direct_io\f[] (with
  1717. \f[C]dropcacheonclose=true\f[]).
  1718. Sqlite3 does not need mmap but the developer needs to fall back to
  1719. standard IO if mmap fails.
  1720. .PP
  1721. If the issue is that scanning doesn\[aq]t seem to pick up media then be
  1722. sure to set \f[C]func.getattr=newest\f[] as mentioned above.
  1723. .SS mmap performance is really bad
  1724. .PP
  1725. There is/was a bug (https://lkml.org/lkml/2016/3/16/260) in caching
  1726. which affects overall performance of mmap through FUSE in Linux 4.x
  1727. kernels.
  1728. It is fixed in 4.4.10 and 4.5.4 (https://lkml.org/lkml/2016/5/11/59).
  1729. .SS When a program tries to move or rename a file it fails
  1730. .PP
  1731. Please read the section above regarding rename & link (#rename--link).
  1732. .PP
  1733. The problem is that many applications do not properly handle
  1734. \f[C]EXDEV\f[] errors which \f[C]rename\f[] and \f[C]link\f[] may return
  1735. even though they are perfectly valid situations which do not indicate
  1736. actual drive or OS errors.
  1737. The error will only be returned by mergerfs if using a path preserving
  1738. policy as described in the policy section above.
  1739. If you do not care about path preservation simply change the mergerfs
  1740. policy to the non\-path preserving version.
  1741. For example: \f[C]\-o\ category.create=mfs\f[]
  1742. .PP
  1743. Ideally the offending software would be fixed and it is recommended that
  1744. if you run into this problem you contact the software\[aq]s author and
  1745. request proper handling of \f[C]EXDEV\f[] errors.
  1746. .SS Samba: Moving files / directories fails
  1747. .PP
  1748. Workaround: Copy the file/directory and then remove the original rather
  1749. than move.
  1750. .PP
  1751. This isn\[aq]t an issue with Samba but some SMB clients.
  1752. GVFS\-fuse v1.20.3 and prior (found in Ubuntu 14.04 among others) failed
  1753. to handle certain error codes correctly.
  1754. Particularly \f[B]STATUS_NOT_SAME_DEVICE\f[] which comes from the
  1755. \f[B]EXDEV\f[] which is returned by \f[B]rename\f[] when the call is
  1756. crossing mount points.
  1757. When a program gets an \f[B]EXDEV\f[] it needs to explicitly take an
  1758. alternate action to accomplish its goal.
  1759. In the case of \f[B]mv\f[] or similar it tries \f[B]rename\f[] and on
  1760. \f[B]EXDEV\f[] falls back to a manual copying of data between the two
  1761. locations and unlinking the source.
  1762. In these older versions of GVFS\-fuse if it received \f[B]EXDEV\f[] it
  1763. would translate that into \f[B]EIO\f[].
  1764. This would cause \f[B]mv\f[] or most any application attempting to move
  1765. files around on that SMB share to fail with a IO error.
  1766. .PP
  1767. GVFS\-fuse v1.22.0 (https://bugzilla.gnome.org/show_bug.cgi?id=734568)
  1768. and above fixed this issue but a large number of systems use the older
  1769. release.
  1770. On Ubuntu the version can be checked by issuing
  1771. \f[C]apt\-cache\ showpkg\ gvfs\-fuse\f[].
  1772. Most distros released in 2015 seem to have the updated release and will
  1773. work fine but older systems may not.
  1774. Upgrading gvfs\-fuse or the distro in general will address the problem.
  1775. .PP
  1776. In Apple\[aq]s MacOSX 10.9 they replaced Samba (client and server) with
  1777. their own product.
  1778. It appears their new client does not handle \f[B]EXDEV\f[] either and
  1779. responds similar to older release of gvfs on Linux.
  1780. .SS Trashing files occasionally fails
  1781. .PP
  1782. This is the same issue as with Samba.
  1783. \f[C]rename\f[] returns \f[C]EXDEV\f[] (in our case that will really
  1784. only happen with path preserving policies like \f[C]epmfs\f[]) and the
  1785. software doesn\[aq]t handle the situation well.
  1786. This is unfortunately a common failure of software which moves files
  1787. around.
  1788. The standard indicates that an implementation \f[C]MAY\f[] choose to
  1789. support non\-user home directory trashing of files (which is a
  1790. \f[C]MUST\f[]).
  1791. The implementation \f[C]MAY\f[] also support "top directory trashes"
  1792. which many probably do.
  1793. .PP
  1794. To create a \f[C]$topdir/.Trash\f[] directory as defined in the standard
  1795. use the mergerfs\-tools (https://github.com/trapexit/mergerfs-tools)
  1796. tool \f[C]mergerfs.mktrash\f[].
  1797. .SS tar: Directory renamed before its status could be extracted
  1798. .PP
  1799. Make sure to use the \f[C]use_ino\f[] option.
  1800. .SS Supplemental user groups
  1801. .PP
  1802. Due to the overhead of
  1803. getgroups/setgroups (http://linux.die.net/man/2/setgroups) mergerfs
  1804. utilizes a cache.
  1805. This cache is opportunistic and per thread.
  1806. Each thread will query the supplemental groups for a user when that
  1807. particular thread needs to change credentials and will keep that data
  1808. for the lifetime of the thread.
  1809. This means that if a user is added to a group it may not be picked up
  1810. without the restart of mergerfs.
  1811. However, since the high level FUSE API\[aq]s (at least the standard
  1812. version) thread pool dynamically grows and shrinks it\[aq]s possible
  1813. that over time a thread will be killed and later a new thread with no
  1814. cache will start and query the new data.
  1815. .PP
  1816. The gid cache uses fixed storage to simplify the design and be
  1817. compatible with older systems which may not have C++11 compilers.
  1818. There is enough storage for 256 users\[aq] supplemental groups.
  1819. Each user is allowed up to 32 supplemental groups.
  1820. Linux >= 2.6.3 allows up to 65535 groups per user but most other *nixs
  1821. allow far less.
  1822. NFS allowing only 16.
  1823. The system does handle overflow gracefully.
  1824. If the user has more than 32 supplemental groups only the first 32 will
  1825. be used.
  1826. If more than 256 users are using the system when an uncached user is
  1827. found it will evict an existing user\[aq]s cache at random.
  1828. So long as there aren\[aq]t more than 256 active users this should be
  1829. fine.
  1830. If either value is too low for your needs you will have to modify
  1831. \f[C]gidcache.hpp\f[] to increase the values.
  1832. Note that doing so will increase the memory needed by each thread.
  1833. .PP
  1834. While not a bug some users have found when using containers that
  1835. supplemental groups defined inside the container don\[aq]t work properly
  1836. with regard to permissions.
  1837. This is expected as mergerfs lives outside the container and therefore
  1838. is querying the host\[aq]s group database.
  1839. There might be a hack to work around this (make mergerfs read the
  1840. /etc/group file in the container) but it is not yet implemented and
  1841. would be limited to Linux and the /etc/group DB.
  1842. Preferably users would mount in the host group file into the containers
  1843. or use a standard shared user & groups technology like NIS or LDAP.
  1844. .SS mergerfs or libfuse crashing
  1845. .PP
  1846. First...
  1847. always upgrade to the latest version unless told otherwise.
  1848. .PP
  1849. If using mergerfs below 2.22.0:
  1850. .PP
  1851. If suddenly the mergerfs mount point disappears and
  1852. \f[C]Transport\ endpoint\ is\ not\ connected\f[] is returned when
  1853. attempting to perform actions within the mount directory \f[B]and\f[]
  1854. the version of libfuse (use \f[C]mergerfs\ \-v\f[] to find the version)
  1855. is older than \f[C]2.9.4\f[] its likely due to a bug in libfuse.
  1856. Affected versions of libfuse can be found in Debian Wheezy, Ubuntu
  1857. Precise and others.
  1858. .PP
  1859. In order to fix this please install newer versions of libfuse.
  1860. If using a Debian based distro (Debian,Ubuntu,Mint) you can likely just
  1861. install newer versions of
  1862. libfuse (https://packages.debian.org/unstable/libfuse2) and
  1863. fuse (https://packages.debian.org/unstable/fuse) from the repo of a
  1864. newer release.
  1865. .PP
  1866. If using mergerfs at or above 2.22.0:
  1867. .PP
  1868. First upgrade if possible, check the known bugs section, and contact
  1869. trapexit.
  1870. .SS mergerfs appears to be crashing or exiting
  1871. .PP
  1872. There seems to be an issue with Linux version \f[C]4.9.0\f[] and above
  1873. in which an invalid message appears to be transmitted to libfuse (used
  1874. by mergerfs) causing it to exit.
  1875. No messages will be printed in any logs as its not a proper crash.
  1876. Debugging of the issue is still ongoing and can be followed via the
  1877. fuse\-devel
  1878. thread (https://sourceforge.net/p/fuse/mailman/message/35662577).
  1879. .SS mergerfs under heavy load and memory pressure leads to kernel panic
  1880. .PP
  1881. https://lkml.org/lkml/2016/9/14/527
  1882. .IP
  1883. .nf
  1884. \f[C]
  1885. [25192.515454]\ kernel\ BUG\ at\ /build/linux\-a2WvEb/linux\-4.4.0/mm/workingset.c:346!
  1886. [25192.517521]\ invalid\ opcode:\ 0000\ [#1]\ SMP
  1887. [25192.519602]\ Modules\ linked\ in:\ netconsole\ ip6t_REJECT\ nf_reject_ipv6\ ipt_REJECT\ nf_reject_ipv4\ configfs\ binfmt_misc\ veth\ bridge\ stp\ llc\ nf_conntrack_ipv6\ nf_defrag_ipv6\ xt_conntrack\ ip6table_filter\ ip6_tables\ xt_multiport\ iptable_filter\ ipt_MASQUERADE\ nf_nat_masquerade_ipv4\ xt_comment\ xt_nat\ iptable_nat\ nf_conntrack_ipv4\ nf_defrag_ipv4\ nf_nat_ipv4\ nf_nat\ nf_conntrack\ xt_CHECKSUM\ xt_tcpudp\ iptable_mangle\ ip_tables\ x_tables\ intel_rapl\ x86_pkg_temp_thermal\ intel_powerclamp\ eeepc_wmi\ asus_wmi\ coretemp\ sparse_keymap\ kvm_intel\ ppdev\ kvm\ irqbypass\ mei_me\ 8250_fintek\ input_leds\ serio_raw\ parport_pc\ tpm_infineon\ mei\ shpchp\ mac_hid\ parport\ lpc_ich\ autofs4\ drbg\ ansi_cprng\ dm_crypt\ algif_skcipher\ af_alg\ btrfs\ raid456\ async_raid6_recov\ async_memcpy\ async_pq\ async_xor\ async_tx\ xor\ raid6_pq\ libcrc32c\ raid0\ multipath\ linear\ raid10\ raid1\ i915\ crct10dif_pclmul\ crc32_pclmul\ aesni_intel\ i2c_algo_bit\ aes_x86_64\ drm_kms_helper\ lrw\ gf128mul\ glue_helper\ ablk_helper\ syscopyarea\ cryptd\ sysfillrect\ sysimgblt\ fb_sys_fops\ drm\ ahci\ r8169\ libahci\ mii\ wmi\ fjes\ video\ [last\ unloaded:\ netconsole]
  1888. [25192.540910]\ CPU:\ 2\ PID:\ 63\ Comm:\ kswapd0\ Not\ tainted\ 4.4.0\-36\-generic\ #55\-Ubuntu
  1889. [25192.543411]\ Hardware\ name:\ System\ manufacturer\ System\ Product\ Name/P8H67\-M\ PRO,\ BIOS\ 3904\ 04/27/2013
  1890. [25192.545840]\ task:\ ffff88040cae6040\ ti:\ ffff880407488000\ task.ti:\ ffff880407488000
  1891. [25192.548277]\ RIP:\ 0010:[<ffffffff811ba501>]\ \ [<ffffffff811ba501>]\ shadow_lru_isolate+0x181/0x190
  1892. [25192.550706]\ RSP:\ 0018:ffff88040748bbe0\ \ EFLAGS:\ 00010002
  1893. [25192.553127]\ RAX:\ 0000000000001c81\ RBX:\ ffff8802f91ee928\ RCX:\ ffff8802f91eeb38
  1894. [25192.555544]\ RDX:\ ffff8802f91ee938\ RSI:\ ffff8802f91ee928\ RDI:\ ffff8804099ba2c0
  1895. [25192.557914]\ RBP:\ ffff88040748bc08\ R08:\ 000000000001a7b6\ R09:\ 000000000000003f
  1896. [25192.560237]\ R10:\ 000000000001a750\ R11:\ 0000000000000000\ R12:\ ffff8804099ba2c0
  1897. [25192.562512]\ R13:\ ffff8803157e9680\ R14:\ ffff8803157e9668\ R15:\ ffff8804099ba2c8
  1898. [25192.564724]\ FS:\ \ 0000000000000000(0000)\ GS:ffff88041f280000(0000)\ knlGS:0000000000000000
  1899. [25192.566990]\ CS:\ \ 0010\ DS:\ 0000\ ES:\ 0000\ CR0:\ 0000000080050033
  1900. [25192.569201]\ CR2:\ 00007ffabb690000\ CR3:\ 0000000001e0a000\ CR4:\ 00000000000406e0
  1901. [25192.571419]\ Stack:
  1902. [25192.573550]\ \ ffff8804099ba2c0\ ffff88039e4f86f0\ ffff8802f91ee928\ ffff8804099ba2c8
  1903. [25192.575695]\ \ ffff88040748bd08\ ffff88040748bc58\ ffffffff811b99bf\ 0000000000000052
  1904. [25192.577814]\ \ 0000000000000000\ ffffffff811ba380\ 000000000000008a\ 0000000000000080
  1905. [25192.579947]\ Call\ Trace:
  1906. [25192.582022]\ \ [<ffffffff811b99bf>]\ __list_lru_walk_one.isra.3+0x8f/0x130
  1907. [25192.584137]\ \ [<ffffffff811ba380>]\ ?\ memcg_drain_all_list_lrus+0x190/0x190
  1908. [25192.586165]\ \ [<ffffffff811b9a83>]\ list_lru_walk_one+0x23/0x30
  1909. [25192.588145]\ \ [<ffffffff811ba544>]\ scan_shadow_nodes+0x34/0x50
  1910. [25192.590074]\ \ [<ffffffff811a0e9d>]\ shrink_slab.part.40+0x1ed/0x3d0
  1911. [25192.591985]\ \ [<ffffffff811a53da>]\ shrink_zone+0x2ca/0x2e0
  1912. [25192.593863]\ \ [<ffffffff811a64ce>]\ kswapd+0x51e/0x990
  1913. [25192.595737]\ \ [<ffffffff811a5fb0>]\ ?\ mem_cgroup_shrink_node_zone+0x1c0/0x1c0
  1914. [25192.597613]\ \ [<ffffffff810a0808>]\ kthread+0xd8/0xf0
  1915. [25192.599495]\ \ [<ffffffff810a0730>]\ ?\ kthread_create_on_node+0x1e0/0x1e0
  1916. [25192.601335]\ \ [<ffffffff8182e34f>]\ ret_from_fork+0x3f/0x70
  1917. [25192.603193]\ \ [<ffffffff810a0730>]\ ?\ kthread_create_on_node+0x1e0/0x1e0
  1918. \f[]
  1919. .fi
  1920. .PP
  1921. There is a bug in the kernel.
  1922. A work around appears to be turning off \f[C]splice\f[].
  1923. Don\[aq]t add the \f[C]splice_*\f[] arguments or add
  1924. \f[C]no_splice_write,no_splice_move,no_splice_read\f[].
  1925. This, however, is not guaranteed to work.
  1926. .SS rm: fts_read failed: No such file or directory
  1927. .PP
  1928. Please update.
  1929. This is only happened to mergerfs versions at or below v2.25.x and will
  1930. not occur in more recent versions.
  1931. .SH FAQ
  1932. .SS How well does mergerfs scale? Is it "production ready?"
  1933. .PP
  1934. Users have reported running mergerfs on everything from a Raspberry Pi
  1935. to dual socket Xeon systems with >20 cores.
  1936. I\[aq]m aware of at least a few companies which use mergerfs in
  1937. production.
  1938. Open Media Vault (https://www.openmediavault.org) includes mergerfs as
  1939. its sole solution for pooling drives.
  1940. .SS Can mergerfs be used with drives which already have data / are in
  1941. use?
  1942. .PP
  1943. Yes.
  1944. MergerFS is a proxy and does \f[B]NOT\f[] interfere with the normal form
  1945. or function of the drives / mounts / paths it manages.
  1946. .PP
  1947. MergerFS is \f[B]not\f[] a traditional filesystem.
  1948. MergerFS is \f[B]not\f[] RAID.
  1949. It does \f[B]not\f[] manipulate the data that passes through it.
  1950. It does \f[B]not\f[] shard data across drives.
  1951. It merely shards some \f[B]behavior\f[] and aggregates others.
  1952. .SS Can mergerfs be removed without affecting the data?
  1953. .PP
  1954. See the previous question\[aq]s answer.
  1955. .SS What policies should I use?
  1956. .PP
  1957. Unless you\[aq]re doing something more niche the average user is
  1958. probably best off using \f[C]mfs\f[] for \f[C]category.create\f[].
  1959. It will spread files out across your branches based on available space.
  1960. You may want to use \f[C]lus\f[] if you prefer a slightly different
  1961. distribution of data if you have a mix of smaller and larger drives.
  1962. Generally though \f[C]mfs\f[], \f[C]lus\f[], or even \f[C]rand\f[] are
  1963. good for the general use case.
  1964. If you are starting with an imbalanced pool you can use the tool
  1965. \f[B]mergerfs.balance\f[] to redistribute files across the pool.
  1966. .PP
  1967. If you really wish to try to colocate files based on directory you can
  1968. set \f[C]func.create\f[] to \f[C]epmfs\f[] or similar and
  1969. \f[C]func.mkdir\f[] to \f[C]rand\f[] or \f[C]eprand\f[] depending on if
  1970. you just want to colocate generally or on specific branches.
  1971. Either way the \f[I]need\f[] to colocate is rare.
  1972. For instance: if you wish to remove the drive regularly and want the
  1973. data to predictably be on that drive or if you don\[aq]t use backup at
  1974. all and don\[aq]t wish to replace that data piecemeal.
  1975. In which case using path preservation can help but will require some
  1976. manual attention.
  1977. Colocating after the fact can be accomplished using the
  1978. \f[B]mergerfs.consolidate\f[] tool.
  1979. .PP
  1980. Ultimately there is no correct answer.
  1981. It is a preference or based on some particular need.
  1982. mergerfs is very easy to test and experiment with.
  1983. I suggest creating a test setup and experimenting to get a sense of what
  1984. you want.
  1985. .PP
  1986. The reason \f[C]mfs\f[] is not the default \f[C]category.create\f[]
  1987. policy is historical.
  1988. When/if a 3.X gets released it will be changed to minimize confusion
  1989. people often have with path preserving policies.
  1990. .SS What settings should I use?
  1991. .PP
  1992. Depends on what features you want.
  1993. Generally speaking there are no "wrong" settings.
  1994. All settings are performance or feature related.
  1995. The best bet is to read over the available options and choose what fits
  1996. your situation.
  1997. If something isn\[aq]t clear from the documentation please reach out and
  1998. the documentation will be improved.
  1999. .PP
  2000. That said, for the average person, the following should be fine:
  2001. .PP
  2002. \f[C]\-o\ use_ino,cache.files=off,dropcacheonclose=true,allow_other,category.create=mfs\f[]
  2003. .SS Why are all my files ending up on 1 drive?!
  2004. .PP
  2005. Did you start with empty drives?
  2006. Did you explicitly configure a \f[C]category.create\f[] policy?
  2007. Are you using a path preserving policy?
  2008. .PP
  2009. The default create policy is \f[C]epmfs\f[].
  2010. That is a path preserving algorithm.
  2011. With such a policy for \f[C]mkdir\f[] and \f[C]create\f[] with a set of
  2012. empty drives it will select only 1 drive when the first directory is
  2013. created.
  2014. Anything, files or directories, created in that first directory will be
  2015. placed on the same branch because it is preserving paths.
  2016. .PP
  2017. This catches a lot of new users off guard but changing the default would
  2018. break the setup for many existing users.
  2019. If you do not care about path preservation and wish your files to be
  2020. spread across all your drives change to \f[C]mfs\f[] or similar policy
  2021. as described above.
  2022. If you do want path preservation you\[aq]ll need to perform the manual
  2023. act of creating paths on the drives you want the data to land on before
  2024. transferring your data.
  2025. Setting \f[C]func.mkdir=epall\f[] can simplify managing path
  2026. preservation for \f[C]create\f[].
  2027. Or use \f[C]func.mkdir=rand\f[] if you\[aq]re intersted in just grouping
  2028. together directory content by drive.
  2029. .SS Do hard links work?
  2030. .PP
  2031. Yes.
  2032. You need to use \f[C]use_ino\f[] to support proper reporting of inodes.
  2033. .PP
  2034. What mergerfs does not do is fake hard links across branches.
  2035. Read the section "rename & link" for how it works.
  2036. .SS Does mergerfs support CoW / copy\-on\-write?
  2037. .PP
  2038. Not in the sense of a filesystem like BTRFS or ZFS nor in the overlayfs
  2039. or aufs sense.
  2040. It does offer a
  2041. cow\-shell (http://manpages.ubuntu.com/manpages/bionic/man1/cow-shell.1.html)
  2042. like hard link breaking (copy to temp file then rename over original)
  2043. which can be useful when wanting to save space by hardlinking duplicate
  2044. files but wish to treat each name as if it were a unique and separate
  2045. file.
  2046. .SS Why can\[aq]t I see my files / directories?
  2047. .PP
  2048. It\[aq]s almost always a permissions issue.
  2049. Unlike mhddfs and unionfs\-fuse, which runs as root and attempts to
  2050. access content as such, mergerfs always changes its credentials to that
  2051. of the caller.
  2052. This means that if the user does not have access to a file or directory
  2053. than neither will mergerfs.
  2054. However, because mergerfs is creating a union of paths it may be able to
  2055. read some files and directories on one drive but not another resulting
  2056. in an incomplete set.
  2057. .PP
  2058. Whenever you run into a split permission issue (seeing some but not all
  2059. files) try using
  2060. mergerfs.fsck (https://github.com/trapexit/mergerfs-tools) tool to check
  2061. for and fix the mismatch.
  2062. If you aren\[aq]t seeing anything at all be sure that the basic
  2063. permissions are correct.
  2064. The user and group values are correct and that directories have their
  2065. executable bit set.
  2066. A common mistake by users new to Linux is to \f[C]chmod\ \-R\ 644\f[]
  2067. when they should have \f[C]chmod\ \-R\ u=rwX,go=rX\f[].
  2068. .PP
  2069. If using a network filesystem such as NFS, SMB, CIFS (Samba) be sure to
  2070. pay close attention to anything regarding permissioning and users.
  2071. Root squashing and user translation for instance has bitten a few
  2072. mergerfs users.
  2073. Some of these also affect the use of mergerfs from container platforms
  2074. such as Docker.
  2075. .SS Is my OS\[aq]s libfuse needed for mergerfs to work?
  2076. .PP
  2077. No.
  2078. Normally \f[C]mount.fuse\f[] is needed to get mergerfs (or any FUSE
  2079. filesystem to mount using the \f[C]mount\f[] command but in vendoring
  2080. the libfuse library the \f[C]mount.fuse\f[] app has been renamed to
  2081. \f[C]mount.mergerfs\f[] meaning the filesystem type in \f[C]fstab\f[]
  2082. can simply be \f[C]mergerfs\f[].
  2083. .SS Why was libfuse embedded into mergerfs?
  2084. .IP "1." 3
  2085. A significant number of users use mergerfs on distros with old versions
  2086. of libfuse which have serious bugs.
  2087. Requiring updated versions of libfuse on those distros isn\[aq]t
  2088. practical (no package offered, user inexperience, etc.).
  2089. The only practical way to provide a stable runtime on those systems was
  2090. to "vendor" / embed the library into the project.
  2091. .IP "2." 3
  2092. mergerfs was written to use the high level API.
  2093. There are a number of limitations in the HLAPI that make certain
  2094. features difficult or impossible to implement.
  2095. While some of these features could be patched into newer versions of
  2096. libfuse without breaking the public API some of them would require hacky
  2097. code to provide backwards compatibility.
  2098. While it may still be worth working with upstream to address these
  2099. issues in future versions, since the library needs to be vendored for
  2100. stability and compatibility reasons it is preferable / easier to modify
  2101. the API.
  2102. Longer term the plan is to rewrite mergerfs to use the low level API.
  2103. .SS Why did support for system libfuse get removed?
  2104. .PP
  2105. See above first.
  2106. .PP
  2107. If/when mergerfs is rewritten to use the low\-level API then it\[aq]ll
  2108. be plausible to support system libfuse but till then its simply too much
  2109. work to manage the differences across the versions.
  2110. .SS Why use mergerfs over mhddfs?
  2111. .PP
  2112. mhddfs is no longer maintained and has some known stability and security
  2113. issues (see below).
  2114. MergerFS provides a superset of mhddfs\[aq] features and should offer
  2115. the same or maybe better performance.
  2116. .PP
  2117. Below is an example of mhddfs and mergerfs setup to work similarly.
  2118. .PP
  2119. \f[C]mhddfs\ \-o\ mlimit=4G,allow_other\ /mnt/drive1,/mnt/drive2\ /mnt/pool\f[]
  2120. .PP
  2121. \f[C]mergerfs\ \-o\ minfreespace=4G,allow_other,category.create=ff\ /mnt/drive1:/mnt/drive2\ /mnt/pool\f[]
  2122. .SS Why use mergerfs over aufs?
  2123. .PP
  2124. aufs is mostly abandoned and no longer available in many distros.
  2125. .PP
  2126. While aufs can offer better peak performance mergerfs provides more
  2127. configurability and is generally easier to use.
  2128. mergerfs however does not offer the overlay / copy\-on\-write (CoW)
  2129. features which aufs and overlayfs have.
  2130. .SS Why use mergerfs over unionfs?
  2131. .PP
  2132. UnionFS is more like aufs than mergerfs in that it offers overlay / CoW
  2133. features.
  2134. If you\[aq]re just looking to create a union of drives and want
  2135. flexibility in file/directory placement then mergerfs offers that
  2136. whereas unionfs is more for overlaying RW filesystems over RO ones.
  2137. .SS Why use mergerfs over overlayfs?
  2138. .PP
  2139. Same reasons as with unionfs.
  2140. .SS Why use mergerfs over LVM/ZFS/BTRFS/RAID0 drive concatenation /
  2141. striping?
  2142. .PP
  2143. With simple JBOD / drive concatenation / stripping / RAID0 a single
  2144. drive failure will result in full pool failure.
  2145. mergerfs performs a similar function without the possibility of
  2146. catastrophic failure and the difficulties in recovery.
  2147. Drives may fail, however, all other data will continue to be accessible.
  2148. .PP
  2149. When combined with something like SnapRaid (http://www.snapraid.it)
  2150. and/or an offsite backup solution you can have the flexibility of JBOD
  2151. without the single point of failure.
  2152. .SS Why use mergerfs over ZFS?
  2153. .PP
  2154. MergerFS is not intended to be a replacement for ZFS.
  2155. MergerFS is intended to provide flexible pooling of arbitrary drives
  2156. (local or remote), of arbitrary sizes, and arbitrary filesystems.
  2157. For \f[C]write\ once,\ read\ many\f[] usecases such as bulk media
  2158. storage.
  2159. Where data integrity and backup is managed in other ways.
  2160. In that situation ZFS can introduce a number of costs and limitations as
  2161. described
  2162. here (http://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html),
  2163. here (https://markmcb.com/2020/01/07/five-years-of-btrfs/), and
  2164. here (https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSWhyNoRealReshaping).
  2165. .SS Why use mergerfs over UnRAID?
  2166. .PP
  2167. UnRAID is a full OS and its storage layer, as I understand, is
  2168. proprietary and closed source.
  2169. Users who have experience with both have said they perfer the flexibilty
  2170. offered by mergerfs and for some the fact it is free and open source is
  2171. important.
  2172. .PP
  2173. There are a number of UnRAID users who use mergerfs as well though
  2174. I\[aq]m not entirely familiar with the use case.
  2175. .SS What should mergerfs NOT be used for?
  2176. .IP \[bu] 2
  2177. databases: Even if the database stored data in separate files (mergerfs
  2178. wouldn\[aq]t offer much otherwise) the higher latency of the indirection
  2179. will kill performance.
  2180. If it is a lightly used SQLITE database then it may be fine but
  2181. you\[aq]ll need to test.
  2182. .IP \[bu] 2
  2183. VM images: For the same reasons as databases.
  2184. VM images are accessed very aggressively and mergerfs will introduce too
  2185. much latency (if it works at all).
  2186. .IP \[bu] 2
  2187. As replacement for RAID: mergerfs is just for pooling branches.
  2188. If you need that kind of device performance aggregation or high
  2189. availability you should stick with RAID.
  2190. .SS Can drives be written to directly? Outside of mergerfs while pooled?
  2191. .PP
  2192. Yes, however its not recommended to use the same file from within the
  2193. pool and from without at the same time (particularly writing).
  2194. Especially if using caching of any kind (cache.files, cache.entry,
  2195. cache.attr, cache.negative_entry, cache.symlinks, cache.readdir, etc.)
  2196. as there could be a conflict between cached data and not.
  2197. .SS Why do I get an "out of space" / "no space left on device" / ENOSPC
  2198. error even though there appears to be lots of space available?
  2199. .PP
  2200. First make sure you\[aq]ve read the sections above about policies, path
  2201. preservation, branch filtering, and the options \f[B]minfreespace\f[],
  2202. \f[B]moveonenospc\f[], \f[B]statfs\f[], and \f[B]statfs_ignore\f[].
  2203. .PP
  2204. mergerfs is simply presenting a union of the content within multiple
  2205. branches.
  2206. The reported free space is an aggregate of space available within the
  2207. pool (behavior modified by \f[B]statfs\f[] and \f[B]statfs_ignore\f[]).
  2208. It does not represent a contiguous space.
  2209. In the same way that read\-only filesystems, those with quotas, or
  2210. reserved space report the full theoretical space available.
  2211. .PP
  2212. Due to path preservation, branch tagging, read\-only status, and
  2213. \f[B]minfreespace\f[] settings it is perfectly valid that
  2214. \f[C]ENOSPC\f[] / "out of space" / "no space left on device" be
  2215. returned.
  2216. It is doing what was asked of it: filtering possible branches due to
  2217. those settings.
  2218. Only one error can be returned and if one of the reasons for filtering a
  2219. branch was \f[B]minfreespace\f[] then it will be returned as such.
  2220. \f[B]moveonenospc\f[] is only relevant to writing a file which is too
  2221. large for the drive its currently on.
  2222. .PP
  2223. It is also possible that the filesystem selected has run out of inodes.
  2224. Use \f[C]df\ \-i\f[] to list the total and available inodes per
  2225. filesystem.
  2226. .PP
  2227. If you don\[aq]t care about path preservation then simply change the
  2228. \f[C]create\f[] policy to one which isn\[aq]t.
  2229. \f[C]mfs\f[] is probably what most are looking for.
  2230. The reason its not default is because it was originally set to
  2231. \f[C]epmfs\f[] and changing it now would change people\[aq]s setup.
  2232. Such a setting change will likely occur in mergerfs 3.
  2233. .SS Why does the total available space in mergerfs not equal outside?
  2234. .PP
  2235. Are you using ext4?
  2236. With reserve for root?
  2237. mergerfs uses available space for statfs calculations.
  2238. If you\[aq]ve reserved space for root then it won\[aq]t show up.
  2239. .SS Can mergerfs mounts be exported over NFS?
  2240. .PP
  2241. Yes.
  2242. Due to current usage of libfuse by mergerfs and how NFS interacts with
  2243. it it is necessary to add \f[C]noforget\f[] to mergerfs options to keep
  2244. from getting "stale file handle" errors.
  2245. .PP
  2246. Some clients (Kodi) have issues in which the contents of the NFS mount
  2247. will not be presented but users have found that enabling the
  2248. \f[C]use_ino\f[] option often fixes that problem.
  2249. .SS Can mergerfs mounts be exported over Samba / SMB?
  2250. .PP
  2251. Yes.
  2252. While some users have reported problems it appears to always be related
  2253. to how Samba is setup in relation to permissions.
  2254. .SS How are inodes calculated?
  2255. .PP
  2256. https://github.com/trapexit/mergerfs/blob/master/src/fs_inode.hpp
  2257. .PP
  2258. Originally tried to simply OR st_ino and (st_dev << 32) for 64bit
  2259. systems.
  2260. After a number of years someone finally ran into a collision that lead
  2261. to some problems.
  2262. Traditionally \f[C]dev_t\f[] was 16bit and \f[C]ino_t\f[] was 32bit so
  2263. merging into one 64bit value worked but with both types being able to be
  2264. up to 64bit that is no longer as simple.
  2265. A proper hash seems like the best compromise.
  2266. While totally unique inodes are preferred the overhead which would be
  2267. needed does not seem to be outweighed by the benefits.
  2268. .PP
  2269. While atypical, yes, inodes can be reused and not refer to the same
  2270. file.
  2271. The internal id used to reference a file in FUSE is different from the
  2272. inode value presented.
  2273. The former is the \f[C]nodeid\f[] and is actually a tuple of
  2274. (nodeid,generation).
  2275. That tuple is not user facing.
  2276. The inode is merely metadata passed through the kernel and found using
  2277. the \f[C]stat\f[] family of calls or \f[C]readdir\f[].
  2278. .PP
  2279. From FUSE docs regarding \f[C]use_ino\f[]:
  2280. .IP
  2281. .nf
  2282. \f[C]
  2283. Honor\ the\ st_ino\ field\ in\ the\ functions\ getattr()\ and
  2284. fill_dir().\ This\ value\ is\ used\ to\ fill\ in\ the\ st_ino\ field
  2285. in\ the\ stat(2),\ lstat(2),\ fstat(2)\ functions\ and\ the\ d_ino
  2286. field\ in\ the\ readdir(2)\ function.\ The\ filesystem\ does\ not
  2287. have\ to\ guarantee\ uniqueness,\ however\ some\ applications
  2288. rely\ on\ this\ value\ being\ unique\ for\ the\ whole\ filesystem.
  2289. Note\ that\ this\ does\ *not*\ affect\ the\ inode\ that\ libfuse
  2290. and\ the\ kernel\ use\ internally\ (also\ called\ the\ "nodeid").
  2291. \f[]
  2292. .fi
  2293. .PP
  2294. Generally collision, if it occurs, shouldn\[aq]t be a problem.
  2295. You can turn off the calculation by not using \f[C]use_ino\f[].
  2296. In the future it might be worth creating different strategies for users
  2297. to select from.
  2298. .SS I notice massive slowdowns of writes when enabling cache.files.
  2299. .PP
  2300. When file caching is enabled in any form (\f[C]cache.files!=off\f[] or
  2301. \f[C]direct_io=false\f[]) it will issue \f[C]getxattr\f[] requests for
  2302. \f[C]security.capability\f[] prior to \f[I]every single write\f[].
  2303. This will usually result in a performance degregation, especially when
  2304. using a network filesystem (such as NFS or CIFS/SMB/Samba.)
  2305. Unfortunately at this moment the kernel is not caching the response.
  2306. .PP
  2307. To work around this situation mergerfs offers a few solutions.
  2308. .IP "1." 3
  2309. Set \f[C]security_capability=false\f[].
  2310. It will short curcuit any call and return \f[C]ENOATTR\f[].
  2311. This still means though that mergerfs will receive the request before
  2312. every write but at least it doesn\[aq]t get passed through to the
  2313. underlying filesystem.
  2314. .IP "2." 3
  2315. Set \f[C]xattr=noattr\f[].
  2316. Same as above but applies to \f[I]all\f[] calls to getxattr.
  2317. Not just \f[C]security.capability\f[].
  2318. This will not be cached by the kernel either but mergerfs\[aq] runtime
  2319. config system will still function.
  2320. .IP "3." 3
  2321. Set \f[C]xattr=nosys\f[].
  2322. Results in mergerfs returning \f[C]ENOSYS\f[] which \f[I]will\f[] be
  2323. cached by the kernel.
  2324. No future xattr calls will be forwarded to mergerfs.
  2325. The downside is that also means the xattr based config and query
  2326. functionality won\[aq]t work either.
  2327. .IP "4." 3
  2328. Disable file caching.
  2329. If you aren\[aq]t using applications which use \f[C]mmap\f[] it\[aq]s
  2330. probably simplier to just disable it all together.
  2331. The kernel won\[aq]t send the requests when caching is disabled.
  2332. .SS What are these .fuse_hidden files?
  2333. .PP
  2334. Please upgrade.
  2335. mergerfs >= 2.26.0 will not have these temporary files.
  2336. See the notes on \f[C]unlink\f[].
  2337. .SS It\[aq]s mentioned that there are some security issues with mhddfs.
  2338. What are they? How does mergerfs address them?
  2339. .PP
  2340. mhddfs (https://github.com/trapexit/mhddfs) manages running as
  2341. \f[B]root\f[] by calling
  2342. getuid() (https://github.com/trapexit/mhddfs/blob/cae96e6251dd91e2bdc24800b4a18a74044f6672/src/main.c#L319)
  2343. and if it returns \f[B]0\f[] then it will
  2344. chown (http://linux.die.net/man/1/chown) the file.
  2345. Not only is that a race condition but it doesn\[aq]t handle other
  2346. situations.
  2347. Rather than attempting to simulate POSIX ACL behavior the proper way to
  2348. manage this is to use seteuid (http://linux.die.net/man/2/seteuid) and
  2349. setegid (http://linux.die.net/man/2/setegid), in effect becoming the
  2350. user making the original call, and perform the action as them.
  2351. This is what mergerfs does and why mergerfs should always run as root.
  2352. .PP
  2353. In Linux setreuid syscalls apply only to the thread.
  2354. GLIBC hides this away by using realtime signals to inform all threads to
  2355. change credentials.
  2356. Taking after \f[B]Samba\f[], mergerfs uses
  2357. \f[B]syscall(SYS_setreuid,...)\f[] to set the callers credentials for
  2358. that thread only.
  2359. Jumping back to \f[B]root\f[] as necessary should escalated privileges
  2360. be needed (for instance: to clone paths between drives).
  2361. .PP
  2362. For non\-Linux systems mergerfs uses a read\-write lock and changes
  2363. credentials only when necessary.
  2364. If multiple threads are to be user X then only the first one will need
  2365. to change the processes credentials.
  2366. So long as the other threads need to be user X they will take a readlock
  2367. allowing multiple threads to share the credentials.
  2368. Once a request comes in to run as user Y that thread will attempt a
  2369. write lock and change to Y\[aq]s credentials when it can.
  2370. If the ability to give writers priority is supported then that flag will
  2371. be used so threads trying to change credentials don\[aq]t starve.
  2372. This isn\[aq]t the best solution but should work reasonably well
  2373. assuming there are few users.
  2374. .SH SUPPORT
  2375. .PP
  2376. Filesystems are complex and difficult to debug.
  2377. mergerfs, while being just a proxy of sorts, is also very difficult to
  2378. debug given the large number of possible settings it can have itself and
  2379. the massive number of environments it can run in.
  2380. When reporting on a suspected issue \f[B]please, please\f[] include as
  2381. much of the below information as possible otherwise it will be difficult
  2382. or impossible to diagnose.
  2383. Also please make sure to read all of the above documentation as it
  2384. includes nearly every known system or user issue previously encountered.
  2385. .PP
  2386. \f[B]Please make sure you are using the latest
  2387. release (https://github.com/trapexit/mergerfs/releases) or have tried it
  2388. in comparison. Old versions, which are often included in distros like
  2389. Debian and Ubuntu, are not ever going to be updated and your bug may
  2390. have been addressed already.\f[]
  2391. .SS Information to include in bug reports
  2392. .IP \[bu] 2
  2393. Version of mergerfs: \f[C]mergerfs\ \-V\f[]
  2394. .IP \[bu] 2
  2395. mergerfs settings: from \f[C]/etc/fstab\f[] or command line execution
  2396. .IP \[bu] 2
  2397. Version of Linux: \f[C]uname\ \-a\f[]
  2398. .IP \[bu] 2
  2399. Versions of any additional software being used
  2400. .IP \[bu] 2
  2401. List of drives, their filesystems, and sizes (before and after issue):
  2402. \f[C]df\ \-h\f[]
  2403. .IP \[bu] 2
  2404. \f[B]All\f[] information about the relevant branches and paths:
  2405. permissions, etc.
  2406. .IP \[bu] 2
  2407. A \f[C]strace\f[] of the app having problems:
  2408. .IP \[bu] 2
  2409. \f[C]strace\ \-f\ \-o\ /tmp/app.strace.txt\ <cmd>\f[]
  2410. .IP \[bu] 2
  2411. A \f[C]strace\f[] of mergerfs while the program is trying to do whatever
  2412. it\[aq]s failing to do:
  2413. .IP \[bu] 2
  2414. \f[C]strace\ \-f\ \-p\ <mergerfsPID>\ \-o\ /tmp/mergerfs.strace.txt\f[]
  2415. .IP \[bu] 2
  2416. \f[B]Precise\f[] directions on replicating the issue.
  2417. Do not leave \f[B]anything\f[] out.
  2418. .IP \[bu] 2
  2419. Try to recreate the problem in the simplist way using standard programs.
  2420. .SS Contact / Issue submission
  2421. .IP \[bu] 2
  2422. github.com: https://github.com/trapexit/mergerfs/issues
  2423. .IP \[bu] 2
  2424. email: trapexit\@spawn.link
  2425. .IP \[bu] 2
  2426. twitter: https://twitter.com/_trapexit
  2427. .IP \[bu] 2
  2428. reddit: https://www.reddit.com/user/trapexit
  2429. .IP \[bu] 2
  2430. discord: https://discord.gg/MpAr69V
  2431. .SS Support development
  2432. .PP
  2433. This software is free to use and released under a very liberal license.
  2434. That said if you like this software and would like to support its
  2435. development donations are welcome.
  2436. .IP \[bu] 2
  2437. PayPal: https://paypal.me/trapexit
  2438. .IP \[bu] 2
  2439. GitHub Sponsors: https://github.com/sponsors/trapexit
  2440. .IP \[bu] 2
  2441. Patreon: https://www.patreon.com/trapexit
  2442. .IP \[bu] 2
  2443. SubscribeStar: https://www.subscribestar.com/trapexit
  2444. .IP \[bu] 2
  2445. Ko\-Fi: https://ko\-fi.com/trapexit
  2446. .IP \[bu] 2
  2447. Bitcoin (BTC): 1DfoUd2m5WCxJAMvcFuvDpT4DR2gWX2PWb
  2448. .IP \[bu] 2
  2449. Bitcoin Cash (BCH): qrf257j0l09yxty4kur8dk2uma8p5vntdcpks72l8z
  2450. .IP \[bu] 2
  2451. Ethereum (ETH): 0xb486C0270fF75872Fc51d85879b9c15C380E66CA
  2452. .IP \[bu] 2
  2453. Litecoin (LTC): LW1rvHRPWtm2NUEMhJpP4DjHZY1FaJ1WYs
  2454. .IP \[bu] 2
  2455. Basic Attention Token (BAT): 0xE651d4900B4C305284Da43E2e182e9abE149A87A
  2456. .IP \[bu] 2
  2457. Zcash (ZEC): t1ZwTgmbQF23DJrzqbAmw8kXWvU2xUkkhTt
  2458. .IP \[bu] 2
  2459. Zcoin (XZC): a8L5Vz35KdCQe7Y7urK2pcCGau7JsqZ5Gw
  2460. .SH LINKS
  2461. .IP \[bu] 2
  2462. https://spawn.link
  2463. .IP \[bu] 2
  2464. https://github.com/trapexit/mergerfs
  2465. .IP \[bu] 2
  2466. https://github.com/trapexit/mergerfs\-tools
  2467. .IP \[bu] 2
  2468. https://github.com/trapexit/scorch
  2469. .IP \[bu] 2
  2470. https://github.com/trapexit/bbf
  2471. .IP \[bu] 2
  2472. https://github.com/trapexit/backup\-and\-recovery\-howtos
  2473. .SH AUTHORS
  2474. Antonio SJ Musumeci <trapexit@spawn.link>.