You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

1614 lines
60 KiB

9 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
7 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
7 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
7 years ago
7 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
7 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
8 years ago
9 years ago
7 years ago
9 years ago
9 years ago
  1. .\"t
  2. .\" Automatically generated by Pandoc 1.19.2.4
  3. .\"
  4. .TH "mergerfs" "1" "2018\-07\-25" "mergerfs user manual" ""
  5. .hy
  6. .SH NAME
  7. .PP
  8. mergerfs \- a featureful union filesystem
  9. .SH SYNOPSIS
  10. .PP
  11. mergerfs \-o<options> <srcmounts> <mountpoint>
  12. .SH DESCRIPTION
  13. .PP
  14. \f[B]mergerfs\f[] is a union filesystem geared towards simplifying
  15. storage and management of files across numerous commodity storage
  16. devices.
  17. It is similar to \f[B]mhddfs\f[], \f[B]unionfs\f[], and \f[B]aufs\f[].
  18. .SH FEATURES
  19. .IP \[bu] 2
  20. Runs in userspace (FUSE)
  21. .IP \[bu] 2
  22. Configurable behaviors
  23. .IP \[bu] 2
  24. Support for extended attributes (xattrs)
  25. .IP \[bu] 2
  26. Support for file attributes (chattr)
  27. .IP \[bu] 2
  28. Runtime configurable (via xattrs)
  29. .IP \[bu] 2
  30. Safe to run as root
  31. .IP \[bu] 2
  32. Opportunistic credential caching
  33. .IP \[bu] 2
  34. Works with heterogeneous filesystem types
  35. .IP \[bu] 2
  36. Handling of writes to full drives (transparently move file to drive with
  37. capacity)
  38. .IP \[bu] 2
  39. Handles pool of readonly and read/write drives
  40. .IP \[bu] 2
  41. Turn read\-only files into symlinks to increase read performance
  42. .SH How it works
  43. .PP
  44. mergerfs logically merges multiple paths together.
  45. Think a union of sets.
  46. The file/s or directory/s acted on or presented through mergerfs are
  47. based on the policy chosen for that particular action.
  48. Read more about policies below.
  49. .IP
  50. .nf
  51. \f[C]
  52. A\ \ \ \ \ \ \ \ \ +\ \ \ \ \ \ B\ \ \ \ \ \ \ \ =\ \ \ \ \ \ \ C
  53. /disk1\ \ \ \ \ \ \ \ \ \ \ /disk2\ \ \ \ \ \ \ \ \ \ \ /merged
  54. |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |
  55. +\-\-\ /dir1\ \ \ \ \ \ \ \ +\-\-\ /dir1\ \ \ \ \ \ \ \ +\-\-\ /dir1
  56. |\ \ \ |\ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ |\ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ |
  57. |\ \ \ +\-\-\ file1\ \ \ \ |\ \ \ +\-\-\ file2\ \ \ \ |\ \ \ +\-\-\ file1
  58. |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ +\-\-\ file3\ \ \ \ |\ \ \ +\-\-\ file2
  59. +\-\-\ /dir2\ \ \ \ \ \ \ \ |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ +\-\-\ file3
  60. |\ \ \ |\ \ \ \ \ \ \ \ \ \ \ \ +\-\-\ /dir3\ \ \ \ \ \ \ \ |
  61. |\ \ \ +\-\-\ file4\ \ \ \ \ \ \ \ |\ \ \ \ \ \ \ \ \ \ \ \ +\-\-\ /dir2
  62. |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ +\-\-\ file5\ \ \ |\ \ \ |
  63. +\-\-\ file6\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ +\-\-\ file4
  64. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |
  65. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ +\-\-\ /dir3
  66. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ |
  67. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |\ \ \ +\-\-\ file5
  68. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ |
  69. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ +\-\-\ file6
  70. \f[]
  71. .fi
  72. .PP
  73. mergerfs does \f[B]not\f[] support the copy\-on\-write (CoW) behavior
  74. found in \f[B]aufs\f[] and \f[B]overlayfs\f[].
  75. You can \f[B]not\f[] mount a read\-only filesystem and write to it.
  76. However, mergerfs will ignore read\-only drives when creating new files
  77. so you can mix rw and ro drives.
  78. .SH OPTIONS
  79. .SS mount options
  80. .IP \[bu] 2
  81. \f[B]defaults\f[]: a shortcut for FUSE\[aq]s \f[B]atomic_o_trunc\f[],
  82. \f[B]auto_cache\f[], \f[B]big_writes\f[], \f[B]default_permissions\f[],
  83. \f[B]splice_move\f[], \f[B]splice_read\f[], and \f[B]splice_write\f[].
  84. These options seem to provide the best performance.
  85. .IP \[bu] 2
  86. \f[B]allow_other\f[]: a libfuse option which allows users besides the
  87. one which ran mergerfs to see the filesystem.
  88. This is required for most use\-cases.
  89. .IP \[bu] 2
  90. \f[B]direct_io\f[]: causes FUSE to bypass caching which can increase
  91. write speeds at the detriment of reads.
  92. Note that not enabling \f[C]direct_io\f[] will cause double caching of
  93. files and therefore less memory for caching generally (enable
  94. \f[B]dropcacheonclose\f[] to help with this problem).
  95. However, \f[C]mmap\f[] does not work when \f[C]direct_io\f[] is enabled.
  96. .IP \[bu] 2
  97. \f[B]minfreespace=value\f[]: the minimum space value used for creation
  98. policies.
  99. Understands \[aq]K\[aq], \[aq]M\[aq], and \[aq]G\[aq] to represent
  100. kilobyte, megabyte, and gigabyte respectively.
  101. (default: 4G)
  102. .IP \[bu] 2
  103. \f[B]moveonenospc=true|false\f[]: when enabled (set to \f[B]true\f[]) if
  104. a \f[B]write\f[] fails with \f[B]ENOSPC\f[] or \f[B]EDQUOT\f[] a scan of
  105. all drives will be done looking for the drive with the most free space
  106. which is at least the size of the file plus the amount which failed to
  107. write.
  108. An attempt to move the file to that drive will occur (keeping all
  109. metadata possible) and if successful the original is unlinked and the
  110. write retried.
  111. (default: false)
  112. .IP \[bu] 2
  113. \f[B]use_ino\f[]: causes mergerfs to supply file/directory inodes rather
  114. than libfuse.
  115. While not a default it is generally recommended it be enabled so that
  116. hard linked files share the same inode value.
  117. .IP \[bu] 2
  118. \f[B]hard_remove\f[]: force libfuse to immedately remove files when
  119. unlinked.
  120. This can have a very minor performance impact in some cases but is
  121. generally recommended since there are subtle race conditions which can
  122. occur when removing large sets of files & directories.
  123. .IP \[bu] 2
  124. \f[B]dropcacheonclose=true|false\f[]: when a file is requested to be
  125. closed call \f[C]posix_fadvise\f[] on it first to instruct the kernel
  126. that we no longer need the data and it can drop its cache.
  127. Recommended when \f[B]direct_io\f[] is not enabled to limit double
  128. caching.
  129. (default: false)
  130. .IP \[bu] 2
  131. \f[B]symlinkify=true|false\f[]: when enabled (set to \f[B]true\f[]) and
  132. a file is not writable and its mtime or ctime is older than
  133. \f[B]symlinkify_timeout\f[] files will be reported as symlinks to the
  134. original files.
  135. Please read more below before using.
  136. (default: false)
  137. .IP \[bu] 2
  138. \f[B]symlinkify_timeout=value\f[]: time to wait, in seconds, to activate
  139. the \f[B]symlinkify\f[] behavior.
  140. (default: 3600)
  141. .IP \[bu] 2
  142. \f[B]nullrw=true|false\f[]: turns reads and writes into no\-ops.
  143. The request will succeed but do nothing.
  144. Useful for benchmarking mergerfs.
  145. (default: false)
  146. .IP \[bu] 2
  147. \f[B]ignorepponrename=true|false\f[]: ignore path preserving on rename.
  148. Typically rename and link act differently depending on the policy of
  149. \f[C]create\f[] (read below).
  150. Enabling this will cause rename and link to always use the non\-path
  151. preserving behavior.
  152. This means files, when renamed or linked, will stay on the same drive.
  153. (default: false)
  154. .IP \[bu] 2
  155. \f[B]threads=num\f[]: number of threads to use in multithreaded mode.
  156. When set to zero (the default) it will attempt to discover and use the
  157. number of logical cores.
  158. If the lookup fails it will fall back to using 4.
  159. If the thread count is set negative it will look up the number of cores
  160. then divide by the absolute value.
  161. ie.
  162. threads=\-2 on an 8 core machine will result in 8 / 2 = 4 threads.
  163. There will always be at least 1 thread.
  164. NOTE: higher number of threads increases parallelism but usually
  165. decreases throughput.
  166. (default: number of cores)
  167. .IP \[bu] 2
  168. \f[B]fsname=name\f[]: sets the name of the filesystem as seen in
  169. \f[B]mount\f[], \f[B]df\f[], etc.
  170. Defaults to a list of the source paths concatenated together with the
  171. longest common prefix removed.
  172. .IP \[bu] 2
  173. \f[B]func.<func>=<policy>\f[]: sets the specific FUSE function\[aq]s
  174. policy.
  175. See below for the list of value types.
  176. Example: \f[B]func.getattr=newest\f[]
  177. .IP \[bu] 2
  178. \f[B]category.<category>=<policy>\f[]: Sets policy of all FUSE functions
  179. in the provided category.
  180. Example: \f[B]category.create=mfs\f[]
  181. .PP
  182. \f[B]NOTE:\f[] Options are evaluated in the order listed so if the
  183. options are \f[B]func.rmdir=rand,category.action=ff\f[] the
  184. \f[B]action\f[] category setting will override the \f[B]rmdir\f[]
  185. setting.
  186. .SS srcmounts
  187. .PP
  188. The srcmounts (source mounts) argument is a colon (\[aq]:\[aq])
  189. delimited list of paths to be included in the pool.
  190. It does not matter if the paths are on the same or different drives nor
  191. does it matter the filesystem.
  192. Used and available space will not be duplicated for paths on the same
  193. device and any features which aren\[aq]t supported by the underlying
  194. filesystem (such as file attributes or extended attributes) will return
  195. the appropriate errors.
  196. .PP
  197. To make it easier to include multiple source mounts mergerfs supports
  198. globbing (http://linux.die.net/man/7/glob).
  199. \f[B]The globbing tokens MUST be escaped when using via the shell else
  200. the shell itself will expand it.\f[]
  201. .IP
  202. .nf
  203. \f[C]
  204. $\ mergerfs\ \-o\ defaults,allow_other,use_ino\ /mnt/disk\\*:/mnt/cdrom\ /media/drives
  205. \f[]
  206. .fi
  207. .PP
  208. The above line will use all mount points in /mnt prefixed with
  209. \f[B]disk\f[] and the \f[B]cdrom\f[].
  210. .PP
  211. To have the pool mounted at boot or otherwise accessable from related
  212. tools use \f[B]/etc/fstab\f[].
  213. .IP
  214. .nf
  215. \f[C]
  216. #\ <file\ system>\ \ \ \ \ \ \ \ <mount\ point>\ \ <type>\ \ \ \ \ \ \ \ \ <options>\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ <dump>\ \ <pass>
  217. /mnt/disk*:/mnt/cdrom\ \ /media/drives\ \ fuse.mergerfs\ \ defaults,allow_other,use_ino,hard_remove\ \ 0\ \ \ \ \ \ \ 0
  218. \f[]
  219. .fi
  220. .PP
  221. \f[B]NOTE:\f[] the globbing is done at mount or xattr update time (see
  222. below).
  223. If a new directory is added matching the glob after the fact it will not
  224. be automatically included.
  225. .PP
  226. \f[B]NOTE:\f[] for mounting via \f[B]fstab\f[] to work you must have
  227. \f[B]mount.fuse\f[] installed.
  228. For Ubuntu/Debian it is included in the \f[B]fuse\f[] package.
  229. .SS symlinkify
  230. .PP
  231. Due to the levels of indirection introduced by mergerfs and the
  232. underlying technology FUSE there can be varying levels of performance
  233. degredation.
  234. This feature will turn non\-directories which are not writable into
  235. symlinks to the original file found by the \f[C]readlink\f[] policy
  236. after the mtime and ctime are older than the timeout.
  237. .PP
  238. \f[B]WARNING:\f[] The current implementation has a known issue in which
  239. if the file is open and being used when the file is converted to a
  240. symlink then the application which has that file open will receive an
  241. error when using it.
  242. This is unlikely to occur in practice but is something to keep in mind.
  243. .PP
  244. \f[B]WARNING:\f[] Some backup solutions, such as CrashPlan, do not
  245. backup the target of a symlink.
  246. If using this feature it will be necessary to point any backup software
  247. to the original drives or configure the software to follow symlinks if
  248. such an option is available.
  249. Alternatively create two mounts.
  250. One for backup and one for general consumption.
  251. .SS nullrw
  252. .PP
  253. Due to how FUSE works there is an overhead to all requests made to a
  254. FUSE filesystem.
  255. Meaning that even a simple passthrough will have some slowdown.
  256. However, generally the overhead is minimal in comparison to the cost of
  257. the underlying I/O.
  258. By disabling the underlying I/O we can test the theoretical performance
  259. boundries.
  260. .PP
  261. By enabling \f[C]nullrw\f[] mergerfs will work as it always does
  262. \f[B]except\f[] that all reads and writes will be no\-ops.
  263. A write will succeed (the size of the write will be returned as if it
  264. were successful) but mergerfs does nothing with the data it was given.
  265. Similarly a read will return the size requested but won\[aq]t touch the
  266. buffer.
  267. .PP
  268. Example:
  269. .IP
  270. .nf
  271. \f[C]
  272. $\ dd\ if=/dev/zero\ of=/path/to/mergerfs/mount/benchmark\ ibs=1M\ obs=512\ count=1024
  273. 1024+0\ records\ in
  274. 2097152+0\ records\ out
  275. 1073741824\ bytes\ (1.1\ GB,\ 1.0\ GiB)\ copied,\ 15.4067\ s,\ 69.7\ MB/s
  276. $\ dd\ if=/dev/zero\ of=/path/to/mergerfs/mount/benchmark\ ibs=1M\ obs=1M\ count=1024
  277. 1024+0\ records\ in
  278. 1024+0\ records\ out
  279. 1073741824\ bytes\ (1.1\ GB,\ 1.0\ GiB)\ copied,\ 0.219585\ s,\ 4.9\ GB/s
  280. $\ dd\ if=/path/to/mergerfs/mount/benchmark\ of=/dev/null\ bs=512\ count=102400
  281. 102400+0\ records\ in
  282. 102400+0\ records\ out
  283. 52428800\ bytes\ (52\ MB,\ 50\ MiB)\ copied,\ 0.757991\ s,\ 69.2\ MB/s
  284. $\ dd\ if=/path/to/mergerfs/mount/benchmark\ of=/dev/null\ bs=1M\ count=1024
  285. 1024+0\ records\ in
  286. 1024+0\ records\ out
  287. 1073741824\ bytes\ (1.1\ GB,\ 1.0\ GiB)\ copied,\ 0.18405\ s,\ 5.8\ GB/s
  288. \f[]
  289. .fi
  290. .PP
  291. It\[aq]s important to test with different \f[C]obs\f[] (output block
  292. size) values since the relative overhead is greater with smaller values.
  293. As you can see above the size of a read or write can massively impact
  294. theoretical performance.
  295. If an application performs much worse through mergerfs it could very
  296. well be that it doesn\[aq]t optimally size its read and write requests.
  297. .SH FUNCTIONS / POLICIES / CATEGORIES
  298. .PP
  299. The POSIX filesystem API has a number of functions.
  300. \f[B]creat\f[], \f[B]stat\f[], \f[B]chown\f[], etc.
  301. In mergerfs these functions are grouped into 3 categories:
  302. \f[B]action\f[], \f[B]create\f[], and \f[B]search\f[].
  303. Functions and categories can be assigned a policy which dictates how
  304. \f[B]mergerfs\f[] behaves.
  305. Any policy can be assigned to a function or category though some may not
  306. be very useful in practice.
  307. For instance: \f[B]rand\f[] (random) may be useful for file creation
  308. (create) but could lead to very odd behavior if used for \f[C]chmod\f[]
  309. (though only if there were more than one copy of the file).
  310. .PP
  311. Policies, when called to create, will ignore drives which are readonly.
  312. This allows for readonly and read/write drives to be mixed together.
  313. Note that the drive must be explicitly mounted with the \f[B]ro\f[]
  314. mount option for this to work.
  315. .SS Function / Category classifications
  316. .PP
  317. .TS
  318. tab(@);
  319. lw(7.9n) lw(62.1n).
  320. T{
  321. Category
  322. T}@T{
  323. FUSE Functions
  324. T}
  325. _
  326. T{
  327. action
  328. T}@T{
  329. chmod, chown, link, removexattr, rename, rmdir, setxattr, truncate,
  330. unlink, utimens
  331. T}
  332. T{
  333. create
  334. T}@T{
  335. create, mkdir, mknod, symlink
  336. T}
  337. T{
  338. search
  339. T}@T{
  340. access, getattr, getxattr, ioctl, listxattr, open, readlink
  341. T}
  342. T{
  343. N/A
  344. T}@T{
  345. fallocate, fgetattr, fsync, ftruncate, ioctl, read, readdir, release,
  346. statfs, write
  347. T}
  348. .TE
  349. .PP
  350. Due to FUSE limitations \f[B]ioctl\f[] behaves differently if its acting
  351. on a directory.
  352. It\[aq]ll use the \f[B]getattr\f[] policy to find and open the directory
  353. before issuing the \f[B]ioctl\f[].
  354. In other cases where something may be searched (to confirm a directory
  355. exists across all source mounts) \f[B]getattr\f[] will also be used.
  356. .SS Path Preservation
  357. .PP
  358. Policies, as described below, are of two core types.
  359. \f[C]path\ preserving\f[] and \f[C]non\-path\ preserving\f[].
  360. .PP
  361. All policies which start with \f[C]ep\f[] (\f[B]epff\f[],
  362. \f[B]eplfs\f[], \f[B]eplus\f[], \f[B]epmfs\f[], \f[B]eprand\f[]) are
  363. \f[C]path\ preserving\f[].
  364. \f[C]ep\f[] stands for \f[C]existing\ path\f[].
  365. .PP
  366. As the descriptions explain a path preserving policy will only consider
  367. drives where the relative path being accessed already exists.
  368. .PP
  369. When using non\-path preserving policies where something is created
  370. paths will be copied to target drives as necessary.
  371. .SS Policy descriptions
  372. .PP
  373. .TS
  374. tab(@);
  375. lw(16.6n) lw(53.4n).
  376. T{
  377. Policy
  378. T}@T{
  379. Description
  380. T}
  381. _
  382. T{
  383. all
  384. T}@T{
  385. Search category: acts like \f[B]ff\f[].
  386. Action category: apply to all found.
  387. Create category: for \f[B]mkdir\f[], \f[B]mknod\f[], and
  388. \f[B]symlink\f[] it will apply to all found.
  389. \f[B]create\f[] works like \f[B]ff\f[].
  390. It will exclude readonly drives and those with free space less than
  391. \f[B]minfreespace\f[].
  392. T}
  393. T{
  394. epall (existing path, all)
  395. T}@T{
  396. Search category: acts like \f[B]epff\f[].
  397. Action category: apply to all found.
  398. Create category: for \f[B]mkdir\f[], \f[B]mknod\f[], and
  399. \f[B]symlink\f[] it will apply to all existing paths found.
  400. \f[B]create\f[] works like \f[B]epff\f[].
  401. Excludes readonly drives and those with free space less than
  402. \f[B]minfreespace\f[].
  403. T}
  404. T{
  405. epff (existing path, first found)
  406. T}@T{
  407. Given the order of the drives, as defined at mount time or configured at
  408. runtime, act on the first one found where the relative path already
  409. exists.
  410. For \f[B]create\f[] category functions it will exclude readonly drives
  411. and those with free space less than \f[B]minfreespace\f[] (unless there
  412. is no other option).
  413. Falls back to \f[B]ff\f[].
  414. T}
  415. T{
  416. eplfs (existing path, least free space)
  417. T}@T{
  418. Of all the drives on which the relative path exists choose the drive
  419. with the least free space.
  420. For \f[B]create\f[] category functions it will exclude readonly drives
  421. and those with free space less than \f[B]minfreespace\f[].
  422. Falls back to \f[B]lfs\f[].
  423. T}
  424. T{
  425. eplus (existing path, least used space)
  426. T}@T{
  427. Of all the drives on which the relative path exists choose the drive
  428. with the least used space.
  429. For \f[B]create\f[] category functions it will exclude readonly drives
  430. and those with free space less than \f[B]minfreespace\f[].
  431. Falls back to \f[B]lus\f[].
  432. T}
  433. T{
  434. epmfs (existing path, most free space)
  435. T}@T{
  436. Of all the drives on which the relative path exists choose the drive
  437. with the most free space.
  438. For \f[B]create\f[] category functions it will exclude readonly drives
  439. and those with free space less than \f[B]minfreespace\f[].
  440. Falls back to \f[B]mfs\f[].
  441. T}
  442. T{
  443. eprand (existing path, random)
  444. T}@T{
  445. Calls \f[B]epall\f[] and then randomizes.
  446. Otherwise behaves the same as \f[B]epall\f[].
  447. T}
  448. T{
  449. erofs
  450. T}@T{
  451. Exclusively return \f[B]\-1\f[] with \f[B]errno\f[] set to
  452. \f[B]EROFS\f[] (Read\-only filesystem).
  453. By setting \f[B]create\f[] functions to this you can in effect turn the
  454. filesystem mostly readonly.
  455. T}
  456. T{
  457. ff (first found)
  458. T}@T{
  459. Given the order of the drives, as defined at mount time or configured at
  460. runtime, act on the first one found.
  461. For \f[B]create\f[] category functions it will exclude readonly drives
  462. and those with free space less than \f[B]minfreespace\f[] (unless there
  463. is no other option).
  464. T}
  465. T{
  466. lfs (least free space)
  467. T}@T{
  468. Pick the drive with the least available free space.
  469. For \f[B]create\f[] category functions it will exclude readonly drives
  470. and those with free space less than \f[B]minfreespace\f[].
  471. Falls back to \f[B]mfs\f[].
  472. T}
  473. T{
  474. lus (least used space)
  475. T}@T{
  476. Pick the drive with the least used space.
  477. For \f[B]create\f[] category functions it will exclude readonly drives
  478. and those with free space less than \f[B]minfreespace\f[].
  479. Falls back to \f[B]mfs\f[].
  480. T}
  481. T{
  482. mfs (most free space)
  483. T}@T{
  484. Pick the drive with the most available free space.
  485. For \f[B]create\f[] category functions it will exclude readonly drives.
  486. Falls back to \f[B]ff\f[].
  487. T}
  488. T{
  489. newest
  490. T}@T{
  491. Pick the file / directory with the largest mtime.
  492. For \f[B]create\f[] category functions it will exclude readonly drives
  493. and those with free space less than \f[B]minfreespace\f[] (unless there
  494. is no other option).
  495. T}
  496. T{
  497. rand (random)
  498. T}@T{
  499. Calls \f[B]all\f[] and then randomizes.
  500. T}
  501. .TE
  502. .SS Defaults
  503. .PP
  504. .TS
  505. tab(@);
  506. l l.
  507. T{
  508. Category
  509. T}@T{
  510. Policy
  511. T}
  512. _
  513. T{
  514. action
  515. T}@T{
  516. all
  517. T}
  518. T{
  519. create
  520. T}@T{
  521. epmfs
  522. T}
  523. T{
  524. search
  525. T}@T{
  526. ff
  527. T}
  528. .TE
  529. .SS rename & link
  530. .PP
  531. \f[B]NOTE:\f[] If you\[aq]re receiving errors from software when files
  532. are moved / renamed then you should consider changing the create policy
  533. to one which is \f[B]not\f[] path preserving, enabling
  534. \f[C]ignorepponrename\f[], or contacting the author of the offending
  535. software and requesting that \f[C]EXDEV\f[] be properly handled.
  536. .PP
  537. rename (http://man7.org/linux/man-pages/man2/rename.2.html) is a tricky
  538. function in a merged system.
  539. Under normal situations rename only works within a single filesystem or
  540. device.
  541. If a rename can\[aq]t be done atomically due to the source and
  542. destination paths existing on different mount points it will return
  543. \f[B]\-1\f[] with \f[B]errno = EXDEV\f[] (cross device).
  544. .PP
  545. Originally mergerfs would return EXDEV whenever a rename was requested
  546. which was cross directory in any way.
  547. This made the code simple and was technically complient with POSIX
  548. requirements.
  549. However, many applications fail to handle EXDEV at all and treat it as a
  550. normal error or otherwise handle it poorly.
  551. Such apps include: gvfsd\-fuse v1.20.3 and prior, Finder / CIFS/SMB
  552. client in Apple OSX 10.9+, NZBGet, Samba\[aq]s recycling bin feature.
  553. .PP
  554. As a result a compromise was made in order to get most software to work
  555. while still obeying mergerfs\[aq] policies.
  556. Below is the rather complicated logic.
  557. .IP \[bu] 2
  558. If using a \f[B]create\f[] policy which tries to preserve directory
  559. paths (epff,eplfs,eplus,epmfs)
  560. .IP \[bu] 2
  561. Using the \f[B]rename\f[] policy get the list of files to rename
  562. .IP \[bu] 2
  563. For each file attempt rename:
  564. .RS 2
  565. .IP \[bu] 2
  566. If failure with ENOENT run \f[B]create\f[] policy
  567. .IP \[bu] 2
  568. If create policy returns the same drive as currently evaluating then
  569. clone the path
  570. .IP \[bu] 2
  571. Re\-attempt rename
  572. .RE
  573. .IP \[bu] 2
  574. If \f[B]any\f[] of the renames succeed the higher level rename is
  575. considered a success
  576. .IP \[bu] 2
  577. If \f[B]no\f[] renames succeed the first error encountered will be
  578. returned
  579. .IP \[bu] 2
  580. On success:
  581. .RS 2
  582. .IP \[bu] 2
  583. Remove the target from all drives with no source file
  584. .IP \[bu] 2
  585. Remove the source from all drives which failed to rename
  586. .RE
  587. .IP \[bu] 2
  588. If using a \f[B]create\f[] policy which does \f[B]not\f[] try to
  589. preserve directory paths
  590. .IP \[bu] 2
  591. Using the \f[B]rename\f[] policy get the list of files to rename
  592. .IP \[bu] 2
  593. Using the \f[B]getattr\f[] policy get the target path
  594. .IP \[bu] 2
  595. For each file attempt rename:
  596. .RS 2
  597. .IP \[bu] 2
  598. If the source drive != target drive:
  599. .IP \[bu] 2
  600. Clone target path from target drive to source drive
  601. .IP \[bu] 2
  602. Rename
  603. .RE
  604. .IP \[bu] 2
  605. If \f[B]any\f[] of the renames succeed the higher level rename is
  606. considered a success
  607. .IP \[bu] 2
  608. If \f[B]no\f[] renames succeed the first error encountered will be
  609. returned
  610. .IP \[bu] 2
  611. On success:
  612. .RS 2
  613. .IP \[bu] 2
  614. Remove the target from all drives with no source file
  615. .IP \[bu] 2
  616. Remove the source from all drives which failed to rename
  617. .RE
  618. .PP
  619. The the removals are subject to normal entitlement checks.
  620. .PP
  621. The above behavior will help minimize the likelihood of EXDEV being
  622. returned but it will still be possible.
  623. .PP
  624. \f[B]link\f[] uses the same basic strategy.
  625. .SS readdir
  626. .PP
  627. readdir (http://linux.die.net/man/3/readdir) is different from all other
  628. filesystem functions.
  629. While it could have it\[aq]s own set of policies to tweak its behavior
  630. at this time it provides a simple union of files and directories found.
  631. Remember that any action or information queried about these files and
  632. directories come from the respective function.
  633. For instance: an \f[B]ls\f[] is a \f[B]readdir\f[] and for each
  634. file/directory returned \f[B]getattr\f[] is called.
  635. Meaning the policy of \f[B]getattr\f[] is responsible for choosing the
  636. file/directory which is the source of the metadata you see in an
  637. \f[B]ls\f[].
  638. .SS statvfs
  639. .PP
  640. statvfs (http://linux.die.net/man/2/statvfs) normalizes the source
  641. drives based on the fragment size and sums the number of adjusted blocks
  642. and inodes.
  643. This means you will see the combined space of all sources.
  644. Total, used, and free.
  645. The sources however are dedupped based on the drive so multiple sources
  646. on the same drive will not result in double counting it\[aq]s space.
  647. .SH BUILDING
  648. .PP
  649. \f[B]NOTE:\f[] Prebuilt packages can be found at:
  650. https://github.com/trapexit/mergerfs/releases
  651. .PP
  652. First get the code from github (http://github.com/trapexit/mergerfs).
  653. .IP
  654. .nf
  655. \f[C]
  656. $\ git\ clone\ https://github.com/trapexit/mergerfs.git
  657. $\ #\ or
  658. $\ wget\ https://github.com/trapexit/mergerfs/releases/download/<ver>/mergerfs\-<ver>.tar.gz
  659. \f[]
  660. .fi
  661. .SS Debian / Ubuntu
  662. .IP
  663. .nf
  664. \f[C]
  665. $\ sudo\ apt\-get\ \-y\ update
  666. $\ sudo\ apt\-get\ \-y\ install\ git\ make
  667. $\ cd\ mergerfs
  668. $\ make\ install\-build\-pkgs
  669. $\ #\ build\-essential\ git\ g++\ debhelper\ libattr1\-dev\ python\ automake\ libtool\ lsb\-release
  670. $\ make\ deb
  671. $\ sudo\ dpkg\ \-i\ ../mergerfs_version_arch.deb
  672. \f[]
  673. .fi
  674. .SS Fedora
  675. .IP
  676. .nf
  677. \f[C]
  678. $\ su\ \-
  679. #\ dnf\ \-y\ update
  680. #\ dnf\ \-y\ install\ git\ make
  681. #\ cd\ mergerfs
  682. #\ make\ install\-build\-pkgs
  683. #\ #\ rpm\-build\ libattr\-devel\ gcc\-c++\ which\ python\ automake\ libtool\ gettext\-devel
  684. #\ make\ rpm
  685. #\ rpm\ \-i\ rpmbuild/RPMS/<arch>/mergerfs\-<verion>.<arch>.rpm
  686. \f[]
  687. .fi
  688. .SS Generically
  689. .PP
  690. Have git, g++, make, python, libattr1, automake, libtool installed.
  691. .IP
  692. .nf
  693. \f[C]
  694. $\ cd\ mergerfs
  695. $\ make
  696. $\ sudo\ make\ install
  697. \f[]
  698. .fi
  699. .SH RUNTIME
  700. .SS .mergerfs pseudo file
  701. .IP
  702. .nf
  703. \f[C]
  704. <mountpoint>/.mergerfs
  705. \f[]
  706. .fi
  707. .PP
  708. There is a pseudo file available at the mount point which allows for the
  709. runtime modification of certain \f[B]mergerfs\f[] options.
  710. The file will not show up in \f[B]readdir\f[] but can be
  711. \f[B]stat\f[]\[aq]ed and manipulated via
  712. {list,get,set}xattrs (http://linux.die.net/man/2/listxattr) calls.
  713. .PP
  714. Even if xattrs are disabled for mergerfs the
  715. {list,get,set}xattrs (http://linux.die.net/man/2/listxattr) calls
  716. against this pseudo file will still work.
  717. .PP
  718. Any changes made at runtime are \f[B]not\f[] persisted.
  719. If you wish for values to persist they must be included as options
  720. wherever you configure the mounting of mergerfs (/etc/fstab).
  721. .SS Keys
  722. .PP
  723. Use \f[C]xattr\ \-l\ /mount/point/.mergerfs\f[] to see all supported
  724. keys.
  725. Some are informational and therefore readonly.
  726. .SS user.mergerfs.srcmounts
  727. .PP
  728. Used to query or modify the list of source mounts.
  729. When modifying there are several shortcuts to easy manipulation of the
  730. list.
  731. .PP
  732. .TS
  733. tab(@);
  734. l l.
  735. T{
  736. Value
  737. T}@T{
  738. Description
  739. T}
  740. _
  741. T{
  742. [list]
  743. T}@T{
  744. set
  745. T}
  746. T{
  747. +<[list]
  748. T}@T{
  749. prepend
  750. T}
  751. T{
  752. +>[list]
  753. T}@T{
  754. append
  755. T}
  756. T{
  757. \-[list]
  758. T}@T{
  759. remove all values provided
  760. T}
  761. T{
  762. \-<
  763. T}@T{
  764. remove first in list
  765. T}
  766. T{
  767. \->
  768. T}@T{
  769. remove last in list
  770. T}
  771. .TE
  772. .PP
  773. \f[C]xattr\ \-w\ user.mergerfs.srcmounts\ +</mnt/drive3\ /mnt/pool/.mergerfs\f[]
  774. .SS minfreespace
  775. .PP
  776. Input: interger with an optional multiplier suffix.
  777. \f[B]K\f[], \f[B]M\f[], or \f[B]G\f[].
  778. .PP
  779. Output: value in bytes
  780. .SS moveonenospc
  781. .PP
  782. Input: \f[B]true\f[] and \f[B]false\f[]
  783. .PP
  784. Ouput: \f[B]true\f[] or \f[B]false\f[]
  785. .SS categories / funcs
  786. .PP
  787. Input: short policy string as described elsewhere in this document
  788. .PP
  789. Output: the policy string except for categories where its funcs have
  790. multiple types.
  791. In that case it will be a comma separated list
  792. .SS Example
  793. .IP
  794. .nf
  795. \f[C]
  796. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-l\ .mergerfs
  797. user.mergerfs.srcmounts:\ /mnt/a:/mnt/b
  798. user.mergerfs.minfreespace:\ 4294967295
  799. user.mergerfs.moveonenospc:\ false
  800. \&...
  801. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.category.search\ .mergerfs
  802. ff
  803. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-w\ user.mergerfs.category.search\ newest\ .mergerfs
  804. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.category.search\ .mergerfs
  805. newest
  806. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-w\ user.mergerfs.srcmounts\ +/mnt/c\ .mergerfs
  807. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.srcmounts\ .mergerfs
  808. /mnt/a:/mnt/b:/mnt/c
  809. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-w\ user.mergerfs.srcmounts\ =/mnt/c\ .mergerfs
  810. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.srcmounts\ .mergerfs
  811. /mnt/c
  812. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-w\ user.mergerfs.srcmounts\ \[aq]+</mnt/a:/mnt/b\[aq]\ .mergerfs
  813. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.srcmounts\ .mergerfs
  814. /mnt/a:/mnt/b:/mnt/c
  815. \f[]
  816. .fi
  817. .SS file / directory xattrs
  818. .PP
  819. While they won\[aq]t show up when using
  820. listxattr (http://linux.die.net/man/2/listxattr) \f[B]mergerfs\f[]
  821. offers a number of special xattrs to query information about the files
  822. served.
  823. To access the values you will need to issue a
  824. getxattr (http://linux.die.net/man/2/getxattr) for one of the following:
  825. .IP \[bu] 2
  826. \f[B]user.mergerfs.basepath:\f[] the base mount point for the file given
  827. the current getattr policy
  828. .IP \[bu] 2
  829. \f[B]user.mergerfs.relpath:\f[] the relative path of the file from the
  830. perspective of the mount point
  831. .IP \[bu] 2
  832. \f[B]user.mergerfs.fullpath:\f[] the full path of the original file
  833. given the getattr policy
  834. .IP \[bu] 2
  835. \f[B]user.mergerfs.allpaths:\f[] a NUL (\[aq]\[aq]) separated list of
  836. full paths to all files found
  837. .IP
  838. .nf
  839. \f[C]
  840. [trapexit:/mnt/mergerfs]\ $\ ls
  841. A\ B\ C
  842. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.fullpath\ A
  843. /mnt/a/full/path/to/A
  844. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.basepath\ A
  845. /mnt/a
  846. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.relpath\ A
  847. /full/path/to/A
  848. [trapexit:/mnt/mergerfs]\ $\ xattr\ \-p\ user.mergerfs.allpaths\ A\ |\ tr\ \[aq]\\0\[aq]\ \[aq]\\n\[aq]
  849. /mnt/a/full/path/to/A
  850. /mnt/b/full/path/to/A
  851. \f[]
  852. .fi
  853. .SH TOOLING
  854. .IP \[bu] 2
  855. https://github.com/trapexit/mergerfs\-tools
  856. .IP \[bu] 2
  857. mergerfs.ctl: A tool to make it easier to query and configure mergerfs
  858. at runtime
  859. .IP \[bu] 2
  860. mergerfs.fsck: Provides permissions and ownership auditing and the
  861. ability to fix them
  862. .IP \[bu] 2
  863. mergerfs.dedup: Will help identify and optionally remove duplicate files
  864. .IP \[bu] 2
  865. mergerfs.dup: Ensure there are at least N copies of a file across the
  866. pool
  867. .IP \[bu] 2
  868. mergerfs.balance: Rebalance files across drives by moving them from the
  869. most filled to the least filled
  870. .IP \[bu] 2
  871. mergerfs.mktrash: Creates FreeDesktop.org Trash specification compatible
  872. directories on a mergerfs mount
  873. .IP \[bu] 2
  874. https://github.com/trapexit/scorch
  875. .IP \[bu] 2
  876. scorch: A tool to help discover silent corruption of files
  877. .IP \[bu] 2
  878. https://github.com/trapexit/bbf
  879. .IP \[bu] 2
  880. bbf (bad block finder): a tool to scan for and \[aq]fix\[aq] hard drive
  881. bad blocks and find the files using those blocks
  882. .SH CACHING
  883. .PP
  884. MergerFS does not natively support any sort of caching.
  885. Most users have no use for such a feature and it would greatly
  886. complicate the code.
  887. However, there are a few situations where a cache drive could help with
  888. a typical mergerfs setup.
  889. .IP "1." 3
  890. Fast network, slow drives, many readers: You\[aq]ve a 10+Gbps network
  891. with many readers and your regular drives can\[aq]t keep up.
  892. .IP "2." 3
  893. Fast network, slow drives, small\[aq]ish bursty writes: You have a
  894. 10+Gbps network and wish to transfer amounts of data less than your
  895. cache drive but wish to do so quickly.
  896. .PP
  897. The below will mostly address usecase #2.
  898. It will also work for #1 assuming the data is regularly accessed and was
  899. placed into the system via this method.
  900. Otherwise a similar script may need to be written to populate the cache
  901. from the backing pool.
  902. .IP "1." 3
  903. Create 2 mergerfs pools.
  904. One which includes just the backing drives and one which has both the
  905. cache drives (SSD,NVME,etc.) and backing drives.
  906. .IP "2." 3
  907. The \[aq]cache\[aq] pool should have the cache drives listed first.
  908. .IP "3." 3
  909. The best policies to use for the \[aq]cache\[aq] pool would probably be
  910. \f[C]ff\f[], \f[C]epff\f[], \f[C]lfs\f[], or \f[C]eplfs\f[].
  911. The latter two under the assumption that the cache drive(s) are far
  912. smaller than the backing drives.
  913. If using path preserving policies remember that you\[aq]ll need to
  914. manually create the core directories of those paths you wish to be
  915. cached.
  916. (Be sure the permissions are in sync.
  917. Use \f[C]mergerfs.fsck\f[] to check / correct them.)
  918. .IP "4." 3
  919. Enable \f[C]moveonenospc\f[] and set \f[C]minfreespace\f[]
  920. appropriately.
  921. .IP "5." 3
  922. Set your programs to use the cache pool.
  923. .IP "6." 3
  924. Save one of the below scripts.
  925. .IP "7." 3
  926. Use \f[C]crontab\f[] (as root) to schedule the command at whatever
  927. frequency is appropriate for your workflow.
  928. .SS Time based expiring
  929. .PP
  930. Move files from cache to backing pool based only on the last time the
  931. file was accessed.
  932. .IP
  933. .nf
  934. \f[C]
  935. #!/bin/bash
  936. if\ [\ $#\ !=\ 3\ ];\ then
  937. \ \ echo\ "usage:\ $0\ <cache\-drive>\ <backing\-pool>\ <days\-old>"
  938. \ \ exit\ 1
  939. fi
  940. CACHE="${1}"
  941. BACKING="${2}"
  942. N=${3}
  943. find\ "${CACHE}"\ \-type\ f\ \-atime\ +${N}\ \-printf\ \[aq]%P\\n\[aq]\ |\ \\
  944. \ \ rsync\ \-\-files\-from=\-\ \-aq\ \-\-remove\-source\-files\ "${CACHE}/"\ "${BACKING}/"
  945. \f[]
  946. .fi
  947. .SS Percentage full expiring
  948. .PP
  949. Move the oldest file from the cache to the backing pool.
  950. Continue till below percentage threshold.
  951. .IP
  952. .nf
  953. \f[C]
  954. #!/bin/bash
  955. if\ [\ $#\ !=\ 3\ ];\ then
  956. \ \ echo\ "usage:\ $0\ <cache\-drive>\ <backing\-pool>\ <percentage>"
  957. \ \ exit\ 1
  958. fi
  959. CACHE="${1}"
  960. BACKING="${2}"
  961. PERCENTAGE=${3}
  962. set\ \-o\ errexit
  963. while\ [\ $(df\ \-\-output=pcent\ "${CACHE}"\ |\ grep\ \-v\ Use\ |\ cut\ \-d\[aq]%\[aq]\ \-f1)\ \-gt\ ${PERCENTAGE}\ ]
  964. do
  965. \ \ \ \ FILE=$(find\ "${CACHE}"\ \-type\ f\ \-printf\ \[aq]%A\@\ %P\\n\[aq]\ |\ \\
  966. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ sort\ |\ \\
  967. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ head\ \-n\ 1\ |\ \\
  968. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ cut\ \-d\[aq]\ \[aq]\ \-f2\-)
  969. \ \ \ \ test\ \-n\ "${FILE}"
  970. \ \ \ \ rsync\ \-aq\ \-\-remove\-source\-files\ "${CACHE}/./${FILE}"\ "${BACKING}/"
  971. done
  972. \f[]
  973. .fi
  974. .SH TIPS / NOTES
  975. .IP \[bu] 2
  976. The recommended base options are
  977. \f[B]defaults,allow_other,direct_io,use_ino,hard_remove\f[].
  978. (\f[B]use_ino\f[] will only work when used with mergerfs 2.18.0 and
  979. above.)
  980. .IP \[bu] 2
  981. Run mergerfs as \f[C]root\f[] unless you\[aq]re merging paths which are
  982. owned by the same user otherwise strange permission issues may arise.
  983. .IP \[bu] 2
  984. https://github.com/trapexit/backup\-and\-recovery\-howtos : A set of
  985. guides / howtos on creating a data storage system, backing it up,
  986. maintaining it, and recovering from failure.
  987. .IP \[bu] 2
  988. If you don\[aq]t see some directories and files you expect in a merged
  989. point or policies seem to skip drives be sure the user has permission to
  990. all the underlying directories.
  991. Use \f[C]mergerfs.fsck\f[] to audit the drive for out of sync
  992. permissions.
  993. .IP \[bu] 2
  994. Do \f[B]not\f[] use \f[C]direct_io\f[] if you expect applications (such
  995. as rtorrent) to mmap (http://linux.die.net/man/2/mmap) files.
  996. It is not currently supported in FUSE w/ \f[C]direct_io\f[] enabled.
  997. Enabling \f[C]dropcacheonclose\f[] is recommended when
  998. \f[C]direct_io\f[] is disabled.
  999. .IP \[bu] 2
  1000. Since POSIX gives you only error or success on calls its difficult to
  1001. determine the proper behavior when applying the behavior to multiple
  1002. targets.
  1003. \f[B]mergerfs\f[] will return an error only if all attempts of an action
  1004. fail.
  1005. Any success will lead to a success returned.
  1006. This means however that some odd situations may arise.
  1007. .IP \[bu] 2
  1008. Kodi (http://kodi.tv), Plex (http://plex.tv),
  1009. Subsonic (http://subsonic.org), etc.
  1010. can use directory mtime (http://linux.die.net/man/2/stat) to more
  1011. efficiently determine whether to scan for new content rather than simply
  1012. performing a full scan.
  1013. If using the default \f[B]getattr\f[] policy of \f[B]ff\f[] its possible
  1014. those programs will miss an update on account of it returning the first
  1015. directory found\[aq]s \f[B]stat\f[] info and its a later directory on
  1016. another mount which had the \f[B]mtime\f[] recently updated.
  1017. To fix this you will want to set \f[B]func.getattr=newest\f[].
  1018. Remember though that this is just \f[B]stat\f[].
  1019. If the file is later \f[B]open\f[]\[aq]ed or \f[B]unlink\f[]\[aq]ed and
  1020. the policy is different for those then a completely different file or
  1021. directory could be acted on.
  1022. .IP \[bu] 2
  1023. Some policies mixed with some functions may result in strange behaviors.
  1024. Not that some of these behaviors and race conditions couldn\[aq]t happen
  1025. outside \f[B]mergerfs\f[] but that they are far more likely to occur on
  1026. account of the attempt to merge together multiple sources of data which
  1027. could be out of sync due to the different policies.
  1028. .IP \[bu] 2
  1029. For consistency its generally best to set \f[B]category\f[] wide
  1030. policies rather than individual \f[B]func\f[]\[aq]s.
  1031. This will help limit the confusion of tools such as
  1032. rsync (http://linux.die.net/man/1/rsync).
  1033. However, the flexibility is there if needed.
  1034. .SH KNOWN ISSUES / BUGS
  1035. .SS directory mtime is not being updated
  1036. .PP
  1037. Remember that the default policy for \f[C]getattr\f[] is \f[C]ff\f[].
  1038. The information for the first directory found will be returned.
  1039. If it wasn\[aq]t the directory which had been updated then it will
  1040. appear outdated.
  1041. .PP
  1042. The reason this is the default is because any other policy would be far
  1043. more expensive and for many applications it is unnecessary.
  1044. To always return the directory with the most recent mtime or a faked
  1045. value based on all found would require a scan of all drives.
  1046. That alone is far more expensive than \f[C]ff\f[] but would also
  1047. possibly spin up sleeping drives.
  1048. .PP
  1049. If you always want the directory information from the one with the most
  1050. recent mtime then use the \f[C]newest\f[] policy for \f[C]getattr\f[].
  1051. .SS cached memory appears greater than it should be
  1052. .PP
  1053. Use the \f[C]direct_io\f[] option as described above.
  1054. Due to what mergerfs is doing there ends up being two caches of a file
  1055. under normal usage.
  1056. One from the underlying filesystem and one from mergerfs.
  1057. Enabling \f[C]direct_io\f[] removes the mergerfs cache.
  1058. This saves on memory but means the kernel needs to communicate with
  1059. mergerfs more often and can therefore result in slower speeds.
  1060. .PP
  1061. Since enabling \f[C]direct_io\f[] disables \f[C]mmap\f[] this is not an
  1062. ideal situation however write speeds should be increased.
  1063. .PP
  1064. If \f[C]direct_io\f[] is disabled it is probably a good idea to enable
  1065. \f[C]dropcacheonclose\f[] to minimize double caching.
  1066. .SS NFS clients don\[aq]t work
  1067. .PP
  1068. Some NFS clients appear to fail when a mergerfs mount is exported.
  1069. Kodi in particular seems to have issues.
  1070. .PP
  1071. Try enabling the \f[C]use_ino\f[] option.
  1072. Some have reported that it fixes the issue.
  1073. .SS rtorrent fails with ENODEV (No such device)
  1074. .PP
  1075. Be sure to turn off \f[C]direct_io\f[].
  1076. rtorrent and some other applications use
  1077. mmap (http://linux.die.net/man/2/mmap) to read and write to files and
  1078. offer no failback to traditional methods.
  1079. FUSE does not currently support mmap while using \f[C]direct_io\f[].
  1080. There will be a performance penalty on writes with \f[C]direct_io\f[]
  1081. off as well as the problem of double caching but it\[aq]s the only way
  1082. to get such applications to work.
  1083. If the performance loss is too high for other apps you can mount
  1084. mergerfs twice.
  1085. Once with \f[C]direct_io\f[] enabled and one without it.
  1086. .SS Plex doesn\[aq]t work with mergerfs
  1087. .PP
  1088. It does.
  1089. If you\[aq]re trying to put Plex\[aq]s config / metadata on mergerfs you
  1090. have to leave \f[C]direct_io\f[] off because Plex is using sqlite which
  1091. apparently needs mmap.
  1092. mmap doesn\[aq]t work with \f[C]direct_io\f[].
  1093. .PP
  1094. If the issue is that scanning doesn\[aq]t seem to pick up media then be
  1095. sure to set \f[C]func.getattr=newest\f[] as mentioned above.
  1096. .SS mmap performance is really bad
  1097. .PP
  1098. There is a bug (https://lkml.org/lkml/2016/3/16/260) in caching which
  1099. affects overall performance of mmap through FUSE in Linux 4.x kernels.
  1100. It is fixed in 4.4.10 and 4.5.4 (https://lkml.org/lkml/2016/5/11/59).
  1101. .SS When a program tries to move or rename a file it fails
  1102. .PP
  1103. Please read the section above regarding rename & link (#rename--link).
  1104. .PP
  1105. The problem is that many applications do not properly handle
  1106. \f[C]EXDEV\f[] errors which \f[C]rename\f[] and \f[C]link\f[] may return
  1107. even though they are perfectly valid situations which do not indicate
  1108. actual drive or OS errors.
  1109. The error will only be returned by mergerfs if using a path preserving
  1110. policy as described in the policy section above.
  1111. If you do not care about path preservation simply change the mergerfs
  1112. policy to the non\-path preserving version.
  1113. For example: \f[C]\-o\ category.create=mfs\f[]
  1114. .PP
  1115. Ideally the offending software would be fixed and it is recommended that
  1116. if you run into this problem you contact the software\[aq]s author and
  1117. request proper handling of \f[C]EXDEV\f[] errors.
  1118. .SS Samba: Moving files / directories fails
  1119. .PP
  1120. Workaround: Copy the file/directory and then remove the original rather
  1121. than move.
  1122. .PP
  1123. This isn\[aq]t an issue with Samba but some SMB clients.
  1124. GVFS\-fuse v1.20.3 and prior (found in Ubuntu 14.04 among others) failed
  1125. to handle certain error codes correctly.
  1126. Particularly \f[B]STATUS_NOT_SAME_DEVICE\f[] which comes from the
  1127. \f[B]EXDEV\f[] which is returned by \f[B]rename\f[] when the call is
  1128. crossing mount points.
  1129. When a program gets an \f[B]EXDEV\f[] it needs to explicitly take an
  1130. alternate action to accomplish it\[aq]s goal.
  1131. In the case of \f[B]mv\f[] or similar it tries \f[B]rename\f[] and on
  1132. \f[B]EXDEV\f[] falls back to a manual copying of data between the two
  1133. locations and unlinking the source.
  1134. In these older versions of GVFS\-fuse if it received \f[B]EXDEV\f[] it
  1135. would translate that into \f[B]EIO\f[].
  1136. This would cause \f[B]mv\f[] or most any application attempting to move
  1137. files around on that SMB share to fail with a IO error.
  1138. .PP
  1139. GVFS\-fuse v1.22.0 (https://bugzilla.gnome.org/show_bug.cgi?id=734568)
  1140. and above fixed this issue but a large number of systems use the older
  1141. release.
  1142. On Ubuntu the version can be checked by issuing
  1143. \f[C]apt\-cache\ showpkg\ gvfs\-fuse\f[].
  1144. Most distros released in 2015 seem to have the updated release and will
  1145. work fine but older systems may not.
  1146. Upgrading gvfs\-fuse or the distro in general will address the problem.
  1147. .PP
  1148. In Apple\[aq]s MacOSX 10.9 they replaced Samba (client and server) with
  1149. their own product.
  1150. It appears their new client does not handle \f[B]EXDEV\f[] either and
  1151. responds similar to older release of gvfs on Linux.
  1152. .SS Trashing files occasionally fails
  1153. .PP
  1154. This is the same issue as with Samba.
  1155. \f[C]rename\f[] returns \f[C]EXDEV\f[] (in our case that will really
  1156. only happen with path preserving policies like \f[C]epmfs\f[]) and the
  1157. software doesn\[aq]t handle the situtation well.
  1158. This is unfortunately a common failure of software which moves files
  1159. around.
  1160. The standard indicates that an implementation \f[C]MAY\f[] choose to
  1161. support non\-user home directory trashing of files (which is a
  1162. \f[C]MUST\f[]).
  1163. The implementation \f[C]MAY\f[] also support "top directory trashes"
  1164. which many probably do.
  1165. .PP
  1166. To create a \f[C]$topdir/.Trash\f[] directory as defined in the standard
  1167. use the mergerfs\-tools (https://github.com/trapexit/mergerfs-tools)
  1168. tool \f[C]mergerfs.mktrash\f[].
  1169. .SS Supplemental user groups
  1170. .PP
  1171. Due to the overhead of
  1172. getgroups/setgroups (http://linux.die.net/man/2/setgroups) mergerfs
  1173. utilizes a cache.
  1174. This cache is opportunistic and per thread.
  1175. Each thread will query the supplemental groups for a user when that
  1176. particular thread needs to change credentials and will keep that data
  1177. for the lifetime of the thread.
  1178. This means that if a user is added to a group it may not be picked up
  1179. without the restart of mergerfs.
  1180. However, since the high level FUSE API\[aq]s (at least the standard
  1181. version) thread pool dynamically grows and shrinks it\[aq]s possible
  1182. that over time a thread will be killed and later a new thread with no
  1183. cache will start and query the new data.
  1184. .PP
  1185. The gid cache uses fixed storage to simplify the design and be
  1186. compatible with older systems which may not have C++11 compilers.
  1187. There is enough storage for 256 users\[aq] supplemental groups.
  1188. Each user is allowed upto 32 supplemental groups.
  1189. Linux >= 2.6.3 allows upto 65535 groups per user but most other *nixs
  1190. allow far less.
  1191. NFS allowing only 16.
  1192. The system does handle overflow gracefully.
  1193. If the user has more than 32 supplemental groups only the first 32 will
  1194. be used.
  1195. If more than 256 users are using the system when an uncached user is
  1196. found it will evict an existing user\[aq]s cache at random.
  1197. So long as there aren\[aq]t more than 256 active users this should be
  1198. fine.
  1199. If either value is too low for your needs you will have to modify
  1200. \f[C]gidcache.hpp\f[] to increase the values.
  1201. Note that doing so will increase the memory needed by each thread.
  1202. .SS mergerfs or libfuse crashing
  1203. .PP
  1204. \f[B]NOTE:\f[] as of mergerfs 2.22.0 it includes the most recent version
  1205. of libfuse so any crash should be reported.
  1206. For older releases continue reading...
  1207. .PP
  1208. If suddenly the mergerfs mount point disappears and
  1209. \f[C]Transport\ endpoint\ is\ not\ connected\f[] is returned when
  1210. attempting to perform actions within the mount directory \f[B]and\f[]
  1211. the version of libfuse (use \f[C]mergerfs\ \-v\f[] to find the version)
  1212. is older than \f[C]2.9.4\f[] its likely due to a bug in libfuse.
  1213. Affected versions of libfuse can be found in Debian Wheezy, Ubuntu
  1214. Precise and others.
  1215. .PP
  1216. In order to fix this please install newer versions of libfuse.
  1217. If using a Debian based distro (Debian,Ubuntu,Mint) you can likely just
  1218. install newer versions of
  1219. libfuse (https://packages.debian.org/unstable/libfuse2) and
  1220. fuse (https://packages.debian.org/unstable/fuse) from the repo of a
  1221. newer release.
  1222. .SS mergerfs appears to be crashing or exiting
  1223. .PP
  1224. There seems to be an issue with Linux version \f[C]4.9.0\f[] and above
  1225. in which an invalid message appears to be transmitted to libfuse (used
  1226. by mergerfs) causing it to exit.
  1227. No messages will be printed in any logs as its not a proper crash.
  1228. Debugging of the issue is still ongoing and can be followed via the
  1229. fuse\-devel
  1230. thread (https://sourceforge.net/p/fuse/mailman/message/35662577).
  1231. .SS mergerfs under heavy load and memory preasure leads to kernel panic
  1232. .PP
  1233. https://lkml.org/lkml/2016/9/14/527
  1234. .IP
  1235. .nf
  1236. \f[C]
  1237. [25192.515454]\ kernel\ BUG\ at\ /build/linux\-a2WvEb/linux\-4.4.0/mm/workingset.c:346!
  1238. [25192.517521]\ invalid\ opcode:\ 0000\ [#1]\ SMP
  1239. [25192.519602]\ Modules\ linked\ in:\ netconsole\ ip6t_REJECT\ nf_reject_ipv6\ ipt_REJECT\ nf_reject_ipv4\ configfs\ binfmt_misc\ veth\ bridge\ stp\ llc\ nf_conntrack_ipv6\ nf_defrag_ipv6\ xt_conntrack\ ip6table_filter\ ip6_tables\ xt_multiport\ iptable_filter\ ipt_MASQUERADE\ nf_nat_masquerade_ipv4\ xt_comment\ xt_nat\ iptable_nat\ nf_conntrack_ipv4\ nf_defrag_ipv4\ nf_nat_ipv4\ nf_nat\ nf_conntrack\ xt_CHECKSUM\ xt_tcpudp\ iptable_mangle\ ip_tables\ x_tables\ intel_rapl\ x86_pkg_temp_thermal\ intel_powerclamp\ eeepc_wmi\ asus_wmi\ coretemp\ sparse_keymap\ kvm_intel\ ppdev\ kvm\ irqbypass\ mei_me\ 8250_fintek\ input_leds\ serio_raw\ parport_pc\ tpm_infineon\ mei\ shpchp\ mac_hid\ parport\ lpc_ich\ autofs4\ drbg\ ansi_cprng\ dm_crypt\ algif_skcipher\ af_alg\ btrfs\ raid456\ async_raid6_recov\ async_memcpy\ async_pq\ async_xor\ async_tx\ xor\ raid6_pq\ libcrc32c\ raid0\ multipath\ linear\ raid10\ raid1\ i915\ crct10dif_pclmul\ crc32_pclmul\ aesni_intel\ i2c_algo_bit\ aes_x86_64\ drm_kms_helper\ lrw\ gf128mul\ glue_helper\ ablk_helper\ syscopyarea\ cryptd\ sysfillrect\ sysimgblt\ fb_sys_fops\ drm\ ahci\ r8169\ libahci\ mii\ wmi\ fjes\ video\ [last\ unloaded:\ netconsole]
  1240. [25192.540910]\ CPU:\ 2\ PID:\ 63\ Comm:\ kswapd0\ Not\ tainted\ 4.4.0\-36\-generic\ #55\-Ubuntu
  1241. [25192.543411]\ Hardware\ name:\ System\ manufacturer\ System\ Product\ Name/P8H67\-M\ PRO,\ BIOS\ 3904\ 04/27/2013
  1242. [25192.545840]\ task:\ ffff88040cae6040\ ti:\ ffff880407488000\ task.ti:\ ffff880407488000
  1243. [25192.548277]\ RIP:\ 0010:[<ffffffff811ba501>]\ \ [<ffffffff811ba501>]\ shadow_lru_isolate+0x181/0x190
  1244. [25192.550706]\ RSP:\ 0018:ffff88040748bbe0\ \ EFLAGS:\ 00010002
  1245. [25192.553127]\ RAX:\ 0000000000001c81\ RBX:\ ffff8802f91ee928\ RCX:\ ffff8802f91eeb38
  1246. [25192.555544]\ RDX:\ ffff8802f91ee938\ RSI:\ ffff8802f91ee928\ RDI:\ ffff8804099ba2c0
  1247. [25192.557914]\ RBP:\ ffff88040748bc08\ R08:\ 000000000001a7b6\ R09:\ 000000000000003f
  1248. [25192.560237]\ R10:\ 000000000001a750\ R11:\ 0000000000000000\ R12:\ ffff8804099ba2c0
  1249. [25192.562512]\ R13:\ ffff8803157e9680\ R14:\ ffff8803157e9668\ R15:\ ffff8804099ba2c8
  1250. [25192.564724]\ FS:\ \ 0000000000000000(0000)\ GS:ffff88041f280000(0000)\ knlGS:0000000000000000
  1251. [25192.566990]\ CS:\ \ 0010\ DS:\ 0000\ ES:\ 0000\ CR0:\ 0000000080050033
  1252. [25192.569201]\ CR2:\ 00007ffabb690000\ CR3:\ 0000000001e0a000\ CR4:\ 00000000000406e0
  1253. [25192.571419]\ Stack:
  1254. [25192.573550]\ \ ffff8804099ba2c0\ ffff88039e4f86f0\ ffff8802f91ee928\ ffff8804099ba2c8
  1255. [25192.575695]\ \ ffff88040748bd08\ ffff88040748bc58\ ffffffff811b99bf\ 0000000000000052
  1256. [25192.577814]\ \ 0000000000000000\ ffffffff811ba380\ 000000000000008a\ 0000000000000080
  1257. [25192.579947]\ Call\ Trace:
  1258. [25192.582022]\ \ [<ffffffff811b99bf>]\ __list_lru_walk_one.isra.3+0x8f/0x130
  1259. [25192.584137]\ \ [<ffffffff811ba380>]\ ?\ memcg_drain_all_list_lrus+0x190/0x190
  1260. [25192.586165]\ \ [<ffffffff811b9a83>]\ list_lru_walk_one+0x23/0x30
  1261. [25192.588145]\ \ [<ffffffff811ba544>]\ scan_shadow_nodes+0x34/0x50
  1262. [25192.590074]\ \ [<ffffffff811a0e9d>]\ shrink_slab.part.40+0x1ed/0x3d0
  1263. [25192.591985]\ \ [<ffffffff811a53da>]\ shrink_zone+0x2ca/0x2e0
  1264. [25192.593863]\ \ [<ffffffff811a64ce>]\ kswapd+0x51e/0x990
  1265. [25192.595737]\ \ [<ffffffff811a5fb0>]\ ?\ mem_cgroup_shrink_node_zone+0x1c0/0x1c0
  1266. [25192.597613]\ \ [<ffffffff810a0808>]\ kthread+0xd8/0xf0
  1267. [25192.599495]\ \ [<ffffffff810a0730>]\ ?\ kthread_create_on_node+0x1e0/0x1e0
  1268. [25192.601335]\ \ [<ffffffff8182e34f>]\ ret_from_fork+0x3f/0x70
  1269. [25192.603193]\ \ [<ffffffff810a0730>]\ ?\ kthread_create_on_node+0x1e0/0x1e0
  1270. \f[]
  1271. .fi
  1272. .PP
  1273. There is a bug in the kernel.
  1274. A work around appears to be turning off \f[C]splice\f[].
  1275. Add \f[C]no_splice_write,no_splice_move,no_splice_read\f[] to
  1276. mergerfs\[aq] options.
  1277. Should be placed after \f[C]defaults\f[] if it is used since it will
  1278. turn them on.
  1279. This however is not guaranteed to work.
  1280. .SS rm: fts_read failed: No such file or directory
  1281. .PP
  1282. Not \f[I]really\f[] a bug.
  1283. The FUSE library will move files when asked to delete them as a way to
  1284. deal with certain edge cases and then later delete that file when its
  1285. clear the file is no longer needed.
  1286. This however can lead to two issues.
  1287. One is that these hidden files are noticed by \f[C]rm\ \-rf\f[] or
  1288. \f[C]find\f[] when scanning directories and they may try to remove them
  1289. and they might have disappeared already.
  1290. There is nothing \f[I]wrong\f[] about this happening but it can be
  1291. annoying.
  1292. The second issue is that a directory might not be able to removed on
  1293. account of the hidden file being still there.
  1294. .PP
  1295. Using the \f[B]hard_remove\f[] option will make it so these temporary
  1296. files are not used and files are deleted immedately.
  1297. .SH FAQ
  1298. .SS How well does mergerfs scale? Is it "production ready?"
  1299. .PP
  1300. Users have reported running mergerfs on everything from a Raspberry Pi
  1301. to dual socket Xeon systems with >20 cores.
  1302. I\[aq]m aware of at least a few companies which use mergerfs in
  1303. production.
  1304. Open Media Vault (https://www.openmediavault.org) includes mergerfs is
  1305. it\[aq]s sole solution for pooling drives.
  1306. .SS Can mergerfs be used with drives which already have data / are in
  1307. use?
  1308. .PP
  1309. Yes.
  1310. MergerFS is a proxy and does \f[B]NOT\f[] interfere with the normal form
  1311. or function of the drives / mounts / paths it manages.
  1312. .PP
  1313. MergerFS is \f[B]not\f[] an actual filesystem.
  1314. MergerFS is \f[B]not\f[] RAID.
  1315. It does \f[B]not\f[] manipulate the data that passes through it.
  1316. It does \f[B]not\f[] shard data across drives.
  1317. It merely shards some \f[B]behavior\f[] and aggregates others.
  1318. .SS Can mergerfs be removed without affecting the data?
  1319. .PP
  1320. See the previous question\[aq]s answer.
  1321. .SS Why can\[aq]t I see my files / directories?
  1322. .PP
  1323. It\[aq]s almost always a permissions issue.
  1324. Unlike mhddfs, which runs as root and attempts to access content as
  1325. such, mergerfs always changes it\[aq]s credentials to that of the
  1326. caller.
  1327. This means that if the user doesn\[aq]t have access to a file or
  1328. directory than neither will mergerfs.
  1329. However, because mergerfs is creating a union of paths it may be able to
  1330. read some files and directories on one drive but not another resulting
  1331. in an incomplete set.
  1332. .PP
  1333. Whenever you run into a split permission issue (seeing some but not all
  1334. files) try using
  1335. mergerfs.fsck (https://github.com/trapexit/mergerfs-tools) tool to check
  1336. for and fix the mismatch.
  1337. If you aren\[aq]t seeing anything at all be sure that the basic
  1338. permissions are correct.
  1339. The user and group values are correct and that directories have their
  1340. executable bit set.
  1341. A common mistake by users new to Linux is to \f[C]chmod\ \-R\ 644\f[]
  1342. when they should have \f[C]chmod\ \-R\ u=rwX,go=rX\f[].
  1343. .PP
  1344. If using a network filesystem such as NFS, SMB, CIFS (Samba) be sure to
  1345. pay close attention to anything regarding permissioning and users.
  1346. Root squashing and user translation for instance has bitten a few
  1347. mergerfs users.
  1348. Some of these also affect the use of mergerfs from container platforms
  1349. such as Docker.
  1350. .SS Why is only one drive being used?
  1351. .PP
  1352. Are you using a path preserving policy?
  1353. The default policy for file creation is \f[C]epmfs\f[].
  1354. That means only the drives with the path preexisting will be considered
  1355. when creating a file.
  1356. If you don\[aq]t care about where files and directories are created you
  1357. likely shouldn\[aq]t be using a path preserving policy and instead
  1358. something like \f[C]mfs\f[].
  1359. .PP
  1360. This can be especially apparent when filling an empty pool from an
  1361. external source.
  1362. If you do want path preservation you\[aq]ll need to perform the manual
  1363. act of creating paths on the drives you want the data to land on before
  1364. transfering your data.
  1365. .SS Why use mergerfs over mhddfs?
  1366. .PP
  1367. mhddfs is no longer maintained and has some known stability and security
  1368. issues (see below).
  1369. MergerFS provides a superset of mhddfs\[aq] features and should offer
  1370. the same or maybe better performance.
  1371. .PP
  1372. Below is an example of mhddfs and mergerfs setup to work similarly.
  1373. .PP
  1374. \f[C]mhddfs\ \-o\ mlimit=4G,allow_other\ /mnt/drive1,/mnt/drive2\ /mnt/pool\f[]
  1375. .PP
  1376. \f[C]mergerfs\ \-o\ minfreespace=4G,defaults,allow_other,category.create=ff\ /mnt/drive1:/mnt/drive2\ /mnt/pool\f[]
  1377. .SS Why use mergerfs over aufs?
  1378. .PP
  1379. While aufs can offer better peak performance mergerfs provides more
  1380. configurability and is generally easier to use.
  1381. mergerfs however does not offer the overlay / copy\-on\-write (CoW)
  1382. features which aufs and overlayfs have.
  1383. .SS Why use mergerfs over unionfs?
  1384. .PP
  1385. UnionFS is more like aufs then mergerfs in that it offers overlay / CoW
  1386. features.
  1387. If you\[aq]re just looking to create a union of drives and want
  1388. flexibility in file/directory placement then mergerfs offers that
  1389. whereas unionfs is more for overlaying RW filesystems over RO ones.
  1390. .SS Why use mergerfs over LVM/ZFS/BTRFS/RAID0 drive concatenation /
  1391. striping?
  1392. .PP
  1393. With simple JBOD / drive concatenation / stripping / RAID0 a single
  1394. drive failure will result in full pool failure.
  1395. mergerfs performs a similar behavior without the possibility of
  1396. catastrophic failure and the difficulties in recovery.
  1397. Drives may fail however all other data will continue to be accessable.
  1398. .PP
  1399. When combined with something like SnapRaid (http://www.snapraid.it)
  1400. and/or an offsite backup solution you can have the flexibilty of JBOD
  1401. without the single point of failure.
  1402. .SS Why use mergerfs over ZFS?
  1403. .PP
  1404. MergerFS is not intended to be a replacement for ZFS.
  1405. MergerFS is intended to provide flexible pooling of arbitrary drives
  1406. (local or remote), of arbitrary sizes, and arbitrary filesystems.
  1407. For \f[C]write\ once,\ read\ many\f[] usecases such as bulk media
  1408. storage.
  1409. Where data integrity and backup is managed in other ways.
  1410. In that situation ZFS can introduce major maintance and cost burdens as
  1411. described
  1412. here (http://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html).
  1413. .SS Can drives be written to directly? Outside of mergerfs while pooled?
  1414. .PP
  1415. Yes.
  1416. It will be represented immediately in the pool as the policies
  1417. perscribe.
  1418. .SS Why do I get an "out of space" error even though the system says
  1419. there\[aq]s lots of space left?
  1420. .PP
  1421. First make sure you\[aq]ve read the sections above about policies, path
  1422. preserving, and the \f[B]moveonenospc\f[] option.
  1423. .PP
  1424. Remember that mergerfs is simply presenting a logical merging of the
  1425. contents of the pooled drives.
  1426. The reported free space is the aggregate space available \f[B]not\f[]
  1427. the contiguous space available.
  1428. MergerFS does not split files across drives.
  1429. If the writing of a file fills an underlying drive and
  1430. \f[B]moveonenospc\f[] is disabled it will return an ENOSPC (No space
  1431. left on device) error.
  1432. .PP
  1433. If \f[B]moveonenospc\f[] is enabled but there exists no drives with
  1434. enough space for the file and the data to be written (or the drive
  1435. happened to fill up as the file was being moved) it will error
  1436. indicating there isn\[aq]t enough space.
  1437. .PP
  1438. It is also possible that the filesystem selected has run out of inodes.
  1439. Use \f[C]df\ \-i\f[] to list the total and available inodes per
  1440. filesystem.
  1441. In the future it might be worth considering the number of inodes
  1442. available when making placement decisions in order to minimize this
  1443. situation.
  1444. .SS Can mergerfs mounts be exported over NFS?
  1445. .PP
  1446. Yes.
  1447. Some clients (Kodi) have issues in which the contents of the NFS mount
  1448. will not be presented but users have found that enabling the
  1449. \f[C]use_ino\f[] option often fixes that problem.
  1450. .SS Can mergerfs mounts be exported over Samba / SMB?
  1451. .PP
  1452. Yes.
  1453. While some users have reported problems it appears to always be related
  1454. to how Samba is setup in relation to permissions.
  1455. .SS How are inodes calculated?
  1456. .PP
  1457. mergerfs\-inode = (original\-inode | (device\-id << 32))
  1458. .PP
  1459. While \f[C]ino_t\f[] is 64 bits only a few filesystems use more than 32.
  1460. Similarly, while \f[C]dev_t\f[] is also 64 bits it was traditionally 16
  1461. bits.
  1462. Bitwise or\[aq]ing them together should work most of the time.
  1463. While totally unique inodes are preferred the overhead which would be
  1464. needed does not seem to outweighted by the benefits.
  1465. .PP
  1466. While atypical, yes, inodes can be reused and not refer to the same
  1467. file.
  1468. The internal id used to reference a file in FUSE is different from the
  1469. inode value presented.
  1470. The former is the \f[C]nodeid\f[] and is actually a tuple of
  1471. (nodeid,generation).
  1472. That tuple is not user facing.
  1473. The inode is merely metadata passed through the kernel and found using
  1474. the \f[C]stat\f[] family of calls or \f[C]readdir\f[].
  1475. .PP
  1476. From FUSE docs regarding \f[C]use_ino\f[]:
  1477. .IP
  1478. .nf
  1479. \f[C]
  1480. Honor\ the\ st_ino\ field\ in\ the\ functions\ getattr()\ and
  1481. fill_dir().\ This\ value\ is\ used\ to\ fill\ in\ the\ st_ino\ field
  1482. in\ the\ stat(2),\ lstat(2),\ fstat(2)\ functions\ and\ the\ d_ino
  1483. field\ in\ the\ readdir(2)\ function.\ The\ filesystem\ does\ not
  1484. have\ to\ guarantee\ uniqueness,\ however\ some\ applications
  1485. rely\ on\ this\ value\ being\ unique\ for\ the\ whole\ filesystem.
  1486. Note\ that\ this\ does\ *not*\ affect\ the\ inode\ that\ libfuse
  1487. and\ the\ kernel\ use\ internally\ (also\ called\ the\ "nodeid").
  1488. \f[]
  1489. .fi
  1490. .SS It\[aq]s mentioned that there are some security issues with mhddfs.
  1491. What are they? How does mergerfs address them?
  1492. .PP
  1493. mhddfs (https://github.com/trapexit/mhddfs) manages running as
  1494. \f[B]root\f[] by calling
  1495. getuid() (https://github.com/trapexit/mhddfs/blob/cae96e6251dd91e2bdc24800b4a18a74044f6672/src/main.c#L319)
  1496. and if it returns \f[B]0\f[] then it will
  1497. chown (http://linux.die.net/man/1/chown) the file.
  1498. Not only is that a race condition but it doesn\[aq]t handle many other
  1499. situations.
  1500. Rather than attempting to simulate POSIX ACL behavior the proper way to
  1501. manage this is to use seteuid (http://linux.die.net/man/2/seteuid) and
  1502. setegid (http://linux.die.net/man/2/setegid), in effect becoming the
  1503. user making the original call, and perform the action as them.
  1504. This is what mergerfs does.
  1505. .PP
  1506. In Linux setreuid syscalls apply only to the thread.
  1507. GLIBC hides this away by using realtime signals to inform all threads to
  1508. change credentials.
  1509. Taking after \f[B]Samba\f[], mergerfs uses
  1510. \f[B]syscall(SYS_setreuid,...)\f[] to set the callers credentials for
  1511. that thread only.
  1512. Jumping back to \f[B]root\f[] as necessary should escalated privileges
  1513. be needed (for instance: to clone paths between drives).
  1514. .PP
  1515. For non\-Linux systems mergerfs uses a read\-write lock and changes
  1516. credentials only when necessary.
  1517. If multiple threads are to be user X then only the first one will need
  1518. to change the processes credentials.
  1519. So long as the other threads need to be user X they will take a readlock
  1520. allowing multiple threads to share the credentials.
  1521. Once a request comes in to run as user Y that thread will attempt a
  1522. write lock and change to Y\[aq]s credentials when it can.
  1523. If the ability to give writers priority is supported then that flag will
  1524. be used so threads trying to change credentials don\[aq]t starve.
  1525. This isn\[aq]t the best solution but should work reasonably well
  1526. assuming there are few users.
  1527. .SH SUPPORT
  1528. .PP
  1529. Filesystems are very complex and difficult to debug.
  1530. mergerfs, while being just a proxy of sorts, is also very difficult to
  1531. debug given the large number of possible settings it can have itself and
  1532. the massive number of environments it can run in.
  1533. When reporting on a suspected issue \f[B]please, please\f[] include as
  1534. much of the below information as possible otherwise it will be difficult
  1535. or impossible to diagnose.
  1536. Also please make sure to read all of the above documentation as it
  1537. includes nearly every known system or user issue previously encountered.
  1538. .SS Information to include in bug reports
  1539. .IP \[bu] 2
  1540. Version of mergerfs: \f[C]mergerfs\ \-V\f[]
  1541. .IP \[bu] 2
  1542. mergerfs settings: from \f[C]/etc/fstab\f[] or command line execution
  1543. .IP \[bu] 2
  1544. Version of Linux: \f[C]uname\ \-a\f[]
  1545. .IP \[bu] 2
  1546. Versions of any additional software being used
  1547. .IP \[bu] 2
  1548. List of drives, their filesystems, and sizes (before and after issue):
  1549. \f[C]df\ \-h\f[]
  1550. .IP \[bu] 2
  1551. A \f[C]strace\f[] of the app having problems:
  1552. .IP \[bu] 2
  1553. \f[C]strace\ \-f\ \-o\ /tmp/app.strace.txt\ <cmd>\f[]
  1554. .IP \[bu] 2
  1555. A \f[C]strace\f[] of mergerfs while the program is trying to do whatever
  1556. it\[aq]s failing to do:
  1557. .IP \[bu] 2
  1558. \f[C]strace\ \-f\ \-p\ <mergerfsPID>\ \-o\ /tmp/mergerfs.strace.txt\f[]
  1559. .IP \[bu] 2
  1560. \f[B]Precise\f[] directions on replicating the issue.
  1561. Do not leave \f[B]anything\f[] out.
  1562. .IP \[bu] 2
  1563. Try to recreate the problem in the simplist way using standard programs.
  1564. .SS Contact / Issue submission
  1565. .IP \[bu] 2
  1566. github.com: https://github.com/trapexit/mergerfs/issues
  1567. .IP \[bu] 2
  1568. email: trapexit\@spawn.link
  1569. .IP \[bu] 2
  1570. twitter: https://twitter.com/_trapexit
  1571. .SS Support development
  1572. .PP
  1573. This software is free to use and released under a very liberal license.
  1574. That said if you like this software and would like to support its
  1575. development donations are welcome.
  1576. .IP \[bu] 2
  1577. Bitcoin (BTC): 12CdMhEPQVmjz3SSynkAEuD5q9JmhTDCZA
  1578. .IP \[bu] 2
  1579. Bitcoin Cash (BCH): 1AjPqZZhu7GVEs6JFPjHmtsvmDL4euzMzp
  1580. .IP \[bu] 2
  1581. Ethereum (ETH): 0x09A166B11fCC127324C7fc5f1B572255b3046E94
  1582. .IP \[bu] 2
  1583. Litecoin (LTC): LXAsq6yc6zYU3EbcqyWtHBrH1Ypx4GjUjm
  1584. .IP \[bu] 2
  1585. Ripple (XRP): rNACR2hqGjpbHuCKwmJ4pDpd2zRfuRATcE
  1586. .IP \[bu] 2
  1587. PayPal: trapexit\@spawn.link
  1588. .IP \[bu] 2
  1589. Patreon: https://www.patreon.com/trapexit
  1590. .SH LINKS
  1591. .IP \[bu] 2
  1592. http://github.com/trapexit/mergerfs
  1593. .IP \[bu] 2
  1594. http://github.com/trapexit/mergerfs\-tools
  1595. .IP \[bu] 2
  1596. http://github.com/trapexit/scorch
  1597. .IP \[bu] 2
  1598. http://github.com/trapexit/backup\-and\-recovery\-howtos
  1599. .SH AUTHORS
  1600. Antonio SJ Musumeci <trapexit@spawn.link>.