Browse Source

add 'ignore path preserving on rename' feature

pull/421/head
Antonio SJ Musumeci 8 years ago
parent
commit
a2bddec8c7
  1. 9
      README.md
  2. 23
      man/mergerfs.1
  3. 1
      src/config.cpp
  4. 1
      src/config.hpp
  5. 4
      src/getxattr.cpp
  6. 2
      src/link.cpp
  7. 2
      src/listxattr.cpp
  8. 5
      src/option_parser.cpp
  9. 2
      src/rename.cpp
  10. 4
      src/setxattr.cpp

9
README.md

@ -41,6 +41,7 @@ mergerfs -o<options> <srcmounts> <mountpoint>
* **symlinkify**: when enabled (set to **true**) and a file is not writable and its mtime or ctime is older than **symlinkify_timeout** files will be reported as symlinks to the original files. Please read more below before using. (default: false) * **symlinkify**: when enabled (set to **true**) and a file is not writable and its mtime or ctime is older than **symlinkify_timeout** files will be reported as symlinks to the original files. Please read more below before using. (default: false)
* **symlinkify_timeout**: time to wait, in seconds, to activate the **symlinkify** behavior. (default: 3600) * **symlinkify_timeout**: time to wait, in seconds, to activate the **symlinkify** behavior. (default: 3600)
* **nullrw**: turns reads and writes into no-ops. The request will succeed but do nothing. Useful for benchmarking mergerfs. (default: false) * **nullrw**: turns reads and writes into no-ops. The request will succeed but do nothing. Useful for benchmarking mergerfs. (default: false)
* **ignorepponrename**: ignore path preserving on rename. Typically rename and link act differently depending on the policy of `create` (read below). Enabling this will cause rename and link to always use the non-path preserving behavior. This means files, when renamed or linked, will stay on the same drive.
* **fsname**: sets the name of the filesystem as seen in **mount**, **df**, etc. Defaults to a list of the source paths concatenated together with the longest common prefix removed. * **fsname**: sets the name of the filesystem as seen in **mount**, **df**, etc. Defaults to a list of the source paths concatenated together with the longest common prefix removed.
* **func.<func>=<policy>**: sets the specific FUSE function's policy. See below for the list of value types. Example: **func.getattr=newest** * **func.<func>=<policy>**: sets the specific FUSE function's policy. See below for the list of value types. Example: **func.getattr=newest**
* **category.<category>=<policy>**: Sets policy of all FUSE functions in the provided category. Example: **category.create=mfs** * **category.<category>=<policy>**: Sets policy of all FUSE functions in the provided category. Example: **category.create=mfs**
@ -165,7 +166,7 @@ When using non-path preserving policies where something is created paths will be
#### rename & link #### #### rename & link ####
**NOTE:** If you're receiving errors from software when files are moved / renamed then you should consider changing the create policy to one which is **not** path preserving or contacting the author of the offending software and requesting that `EXDEV` be properly handled.
**NOTE:** If you're receiving errors from software when files are moved / renamed then you should consider changing the create policy to one which is **not** path preserving, enabling `ignorepponrename`, or contacting the author of the offending software and requesting that `EXDEV` be properly handled.
[rename](http://man7.org/linux/man-pages/man2/rename.2.html) is a tricky function in a merged system. Under normal situations rename only works within a single filesystem or device. If a rename can't be done atomically due to the source and destination paths existing on different mount points it will return **-1** with **errno = EXDEV** (cross device). [rename](http://man7.org/linux/man-pages/man2/rename.2.html) is a tricky function in a merged system. Under normal situations rename only works within a single filesystem or device. If a rename can't be done atomically due to the source and destination paths existing on different mount points it will return **-1** with **errno = EXDEV** (cross device).
@ -407,6 +408,12 @@ Try enabling the `use_ino` option. Some have reported that it fixes the issue.
Be sure to turn off `direct_io`. rtorrent and some other applications use [mmap](http://linux.die.net/man/2/mmap) to read and write to files and offer no failback to traditional methods. FUSE does not currently support mmap while using `direct_io`. There will be a performance penalty on writes with `direct_io` off as well as the problem of double caching but it's the only way to get such applications to work. If the performance loss is too high for other apps you can mount mergerfs twice. Once with `direct_io` enabled and one without it. Be sure to turn off `direct_io`. rtorrent and some other applications use [mmap](http://linux.die.net/man/2/mmap) to read and write to files and offer no failback to traditional methods. FUSE does not currently support mmap while using `direct_io`. There will be a performance penalty on writes with `direct_io` off as well as the problem of double caching but it's the only way to get such applications to work. If the performance loss is too high for other apps you can mount mergerfs twice. Once with `direct_io` enabled and one without it.
#### Plex doesn't work with mergerfs
It does. If you're trying to put Plex's config / metadata on mergerfs you have to leave `direct_io` off because Plex is using sqlite which apparently needs mmap. mmap doesn't work with `direct_io`.
If the issue is that scanning doesn't seem to pick up media then be sure to set `func.getattr=newest` as mentioned above.
#### mmap performance is really bad #### mmap performance is really bad
There [is a bug](https://lkml.org/lkml/2016/3/16/260) in caching which affects overall performance of mmap through FUSE in Linux 4.x kernels. It is fixed in [4.4.10 and 4.5.4](https://lkml.org/lkml/2016/5/11/59). There [is a bug](https://lkml.org/lkml/2016/3/16/260) in caching which affects overall performance of mmap through FUSE in Linux 4.x kernels. It is fixed in [4.4.10 and 4.5.4](https://lkml.org/lkml/2016/5/11/59).

23
man/mergerfs.1

@ -96,6 +96,13 @@ The request will succeed but do nothing.
Useful for benchmarking mergerfs. Useful for benchmarking mergerfs.
(default: false) (default: false)
.IP \[bu] 2 .IP \[bu] 2
\f[B]ignorepponrename\f[]: ignore path preserving on rename.
Typically rename and link act differently depending on the policy of
\f[C]create\f[] (read below).
Enabling this will cause rename and link to always use the non\-path
preserving behavior.
This means files, when renamed or linked, will stay on the same drive.
.IP \[bu] 2
\f[B]fsname\f[]: sets the name of the filesystem as seen in \f[B]fsname\f[]: sets the name of the filesystem as seen in
\f[B]mount\f[], \f[B]df\f[], etc. \f[B]mount\f[], \f[B]df\f[], etc.
Defaults to a list of the source paths concatenated together with the Defaults to a list of the source paths concatenated together with the
@ -463,9 +470,9 @@ T}
.PP .PP
\f[B]NOTE:\f[] If you\[aq]re receiving errors from software when files \f[B]NOTE:\f[] If you\[aq]re receiving errors from software when files
are moved / renamed then you should consider changing the create policy are moved / renamed then you should consider changing the create policy
to one which is \f[B]not\f[] path preserving or contacting the author of
the offending software and requesting that \f[C]EXDEV\f[] be properly
handled.
to one which is \f[B]not\f[] path preserving, enabling
\f[C]ignorepponrename\f[], or contacting the author of the offending
software and requesting that \f[C]EXDEV\f[] be properly handled.
.PP .PP
rename (http://man7.org/linux/man-pages/man2/rename.2.html) is a tricky rename (http://man7.org/linux/man-pages/man2/rename.2.html) is a tricky
function in a merged system. function in a merged system.
@ -920,6 +927,16 @@ to get such applications to work.
If the performance loss is too high for other apps you can mount If the performance loss is too high for other apps you can mount
mergerfs twice. mergerfs twice.
Once with \f[C]direct_io\f[] enabled and one without it. Once with \f[C]direct_io\f[] enabled and one without it.
.SS Plex doesn\[aq]t work with mergerfs
.PP
It does.
If you\[aq]re trying to put Plex\[aq]s config / metadata on mergerfs you
have to leave \f[C]direct_io\f[] off because Plex is using sqlite which
apparently needs mmap.
mmap doesn\[aq]t work with \f[C]direct_io\f[].
.PP
If the issue is that scanning doesn\[aq]t seem to pick up media then be
sure to set \f[C]func.getattr=newest\f[] as mentioned above.
.SS mmap performance is really bad .SS mmap performance is really bad
.PP .PP
There is a bug (https://lkml.org/lkml/2016/3/16/260) in caching which There is a bug (https://lkml.org/lkml/2016/3/16/260) in caching which

1
src/config.cpp

@ -44,6 +44,7 @@ namespace mergerfs
symlinkify(false), symlinkify(false),
symlinkify_timeout(3600), symlinkify_timeout(3600),
nullrw(false), nullrw(false),
ignorepponrename(false),
POLICYINIT(access), POLICYINIT(access),
POLICYINIT(chmod), POLICYINIT(chmod),
POLICYINIT(chown), POLICYINIT(chown),

1
src/config.hpp

@ -52,6 +52,7 @@ namespace mergerfs
bool symlinkify; bool symlinkify;
time_t symlinkify_timeout; time_t symlinkify_timeout;
bool nullrw; bool nullrw;
bool ignorepponrename;
public: public:
const Policy *policies[FuseFunc::Enum::END]; const Policy *policies[FuseFunc::Enum::END];

4
src/getxattr.cpp

@ -193,6 +193,10 @@ _getxattr_controlfile(const Config &config,
_getxattr_controlfile_bool(config.symlinkify,attrvalue); _getxattr_controlfile_bool(config.symlinkify,attrvalue);
else if(attr[2] == "symlinkify_timeout") else if(attr[2] == "symlinkify_timeout")
_getxattr_controlfile_time_t(config.symlinkify_timeout,attrvalue); _getxattr_controlfile_time_t(config.symlinkify_timeout,attrvalue);
else if(attr[2] == "nullrw")
_getxattr_controlfile_bool(config.nullrw,attrvalue);
else if(attr[2] == "ignorepponrename")
_getxattr_controlfile_bool(config.ignorepponrename,attrvalue);
else if(attr[2] == "policies") else if(attr[2] == "policies")
_getxattr_controlfile_policies(config,attrvalue); _getxattr_controlfile_policies(config,attrvalue);
else if(attr[2] == "version") else if(attr[2] == "version")

2
src/link.cpp

@ -240,7 +240,7 @@ namespace mergerfs
const ugid::Set ugid(fc->uid,fc->gid); const ugid::Set ugid(fc->uid,fc->gid);
const rwlock::ReadGuard readlock(&config.srcmountslock); const rwlock::ReadGuard readlock(&config.srcmountslock);
if(config.create->path_preserving())
if(config.create->path_preserving() && !config.ignorepponrename)
return _link_preserve_path(config.getattr, return _link_preserve_path(config.getattr,
config.link, config.link,
config.create, config.create,

2
src/listxattr.cpp

@ -49,6 +49,8 @@ _listxattr_controlfile(char *list,
("user.mergerfs.dropcacheonclose") ("user.mergerfs.dropcacheonclose")
("user.mergerfs.symlinkify") ("user.mergerfs.symlinkify")
("user.mergerfs.symlinkify_timeout") ("user.mergerfs.symlinkify_timeout")
("user.mergerfs.nullrw")
("user.mergerfs.ignorepponrename")
("user.mergerfs.policies") ("user.mergerfs.policies")
("user.mergerfs.version") ("user.mergerfs.version")
("user.mergerfs.pid"); ("user.mergerfs.pid");

5
src/option_parser.cpp

@ -192,6 +192,8 @@ parse_and_process_kv_arg(Config &config,
rv = parse_and_process(value,config.symlinkify_timeout); rv = parse_and_process(value,config.symlinkify_timeout);
else if(key == "nullrw") else if(key == "nullrw")
rv = parse_and_process(value,config.nullrw); rv = parse_and_process(value,config.nullrw);
else if(key == "ignorepponrename")
rv = parse_and_process(value,config.ignorepponrename);
} }
if(rv == -1) if(rv == -1)
@ -294,6 +296,9 @@ usage(void)
" timeout in seconds before will turn to symlinks.\n" " timeout in seconds before will turn to symlinks.\n"
" default=3600\n" " default=3600\n"
" -o nullrw=<bool> Disables reads and writes. For benchmarking.\n" " -o nullrw=<bool> Disables reads and writes. For benchmarking.\n"
" -o ignorepponrename=<bool>\n"
" Ignore path preserving when performing renames\n"
" and links. default = false"
<< std::endl; << std::endl;
} }

2
src/rename.cpp

@ -301,7 +301,7 @@ namespace mergerfs
const ugid::Set ugid(fc->uid,fc->gid); const ugid::Set ugid(fc->uid,fc->gid);
const rwlock::ReadGuard readlock(&config.srcmountslock); const rwlock::ReadGuard readlock(&config.srcmountslock);
if(config.create->path_preserving())
if(config.create->path_preserving() && !config.ignorepponrename)
return _rename_preserve_path(config.getattr, return _rename_preserve_path(config.getattr,
config.rename, config.rename,
config.create, config.create,

4
src/setxattr.cpp

@ -293,6 +293,10 @@ _setxattr_controlfile(Config &config,
return _setxattr_time_t(attrval, return _setxattr_time_t(attrval,
flags, flags,
config.symlinkify_timeout); config.symlinkify_timeout);
else if(attr[2] == "ignorepponrename")
return _setxattr_bool(attrval,
flags,
config.ignorepponrename);
break; break;
case 4: case 4:

Loading…
Cancel
Save