Browse Source

Merge pull request #241 from trapexit/policy-ff

add minfreespace checks to policy ff's create and remove fwfs
pull/242/head
Antonio SJ Musumeci 9 years ago
parent
commit
4a63c000ca
  1. 9
      README.md
  2. 18
      man/mergerfs.1
  3. 18
      src/fs.cpp
  4. 3
      src/fs.hpp
  5. 2
      src/policy.cpp
  6. 3
      src/policy.hpp
  7. 61
      src/policy_ff.cpp
  8. 86
      src/policy_fwfs.cpp
  9. 52
      src/policy_newest.cpp

9
README.md

@ -33,7 +33,7 @@ mergerfs -o<options> <srcmounts> <mountpoint>
* **defaults**: a shortcut for FUSE's **atomic_o_trunc**, **auto_cache**, **big_writes**, **default_permissions**, **splice_move**, **splice_read**, and **splice_write**. These options seem to provide the best performance. * **defaults**: a shortcut for FUSE's **atomic_o_trunc**, **auto_cache**, **big_writes**, **default_permissions**, **splice_move**, **splice_read**, and **splice_write**. These options seem to provide the best performance.
* **direct_io**: causes FUSE to bypass an addition caching step which can increase write speeds at the detriment of read speed. * **direct_io**: causes FUSE to bypass an addition caching step which can increase write speeds at the detriment of read speed.
* **minfreespace**: the minimum space value used for the **lfs**, **fwfs**, **eplfs**, & **epmfs** policies. Understands 'K', 'M', and 'G' to represent kilobyte, megabyte, and gigabyte respectively. (default: 4G)
* **minfreespace**: the minimum space value used for creation policies. Understands 'K', 'M', and 'G' to represent kilobyte, megabyte, and gigabyte respectively. (default: 4G)
* **moveonenospc**: when enabled (set to **true**) if a **write** fails with **ENOSPC** a scan of all drives will be done looking for the drive with most free space which is at least the size of the file plus the amount which failed to write. An attempt to move the file to that drive will occur (keeping all metadata possible) and if successful the original is unlinked and the write retried. (default: false) * **moveonenospc**: when enabled (set to **true**) if a **write** fails with **ENOSPC** a scan of all drives will be done looking for the drive with most free space which is at least the size of the file plus the amount which failed to write. An attempt to move the file to that drive will occur (keeping all metadata possible) and if successful the original is unlinked and the write retried. (default: false)
* **func.<func>=<policy>**: sets the specific FUSE function's policy. See below for the list of value types. Example: **func.getattr=newest** * **func.<func>=<policy>**: sets the specific FUSE function's policy. See below for the list of value types. Example: **func.getattr=newest**
* **category.<category>=<policy>**: Sets policy of all FUSE functions in the provided category. Example: **category.create=mfs** * **category.<category>=<policy>**: Sets policy of all FUSE functions in the provided category. Example: **category.create=mfs**
@ -78,7 +78,7 @@ Due to FUSE limitations **ioctl** behaves differently if its acting on a directo
#### Policy descriptions #### #### Policy descriptions ####
Generally speaking most policies when called to create will filter out drives which are readonly or have less than `minfreespace`.
Most policies when called to create will filter out drives which are readonly or have less than **minfreespace**.
| Policy | Description | | Policy | Description |
|--------------|-------------| |--------------|-------------|
@ -87,7 +87,6 @@ Generally speaking most policies when called to create will filter out drives wh
| epmfs (existing path, most free space) | If the path exists on multiple drives use the one with the most free space. Falls back to **mfs**. | | epmfs (existing path, most free space) | If the path exists on multiple drives use the one with the most free space. Falls back to **mfs**. |
| erofs | Exclusively return **-1** with **errno** set to **EROFS**. By setting **create** functions to this you can in effect turn the filesystem readonly. | | erofs | Exclusively return **-1** with **errno** set to **EROFS**. By setting **create** functions to this you can in effect turn the filesystem readonly. |
| ff (first found) | Given the order of the drives, as defined at mount time or when configured via xattr interface, act on the first one found. | | ff (first found) | Given the order of the drives, as defined at mount time or when configured via xattr interface, act on the first one found. |
| fwfs (first with free space) | Pick the first drive which has at least **minfreespace**. Falls back to **mfs**. |
| lfs (least free space) | Pick the drive with the least available free space but more than **minfreespace**. Falls back to **mfs**. | | lfs (least free space) | Pick the drive with the least available free space but more than **minfreespace**. Falls back to **mfs**. |
| mfs (most free space) | Use the drive with the most available free space. Falls back to **ff**. | | mfs (most free space) | Use the drive with the most available free space. Falls back to **ff**. |
| newest (newest file) | Pick the file / directory with the largest mtime. | | newest (newest file) | Pick the file / directory with the largest mtime. |
@ -245,9 +244,9 @@ user.mergerfs.moveonenospc: false
[trapexit:/tmp/mount] $ xattr -p user.mergerfs.category.search .mergerfs [trapexit:/tmp/mount] $ xattr -p user.mergerfs.category.search .mergerfs
ff ff
[trapexit:/tmp/mount] $ xattr -w user.mergerfs.category.search fwfs .mergerfs
[trapexit:/tmp/mount] $ xattr -w user.mergerfs.category.search newest .mergerfs
[trapexit:/tmp/mount] $ xattr -p user.mergerfs.category.search .mergerfs [trapexit:/tmp/mount] $ xattr -p user.mergerfs.category.search .mergerfs
fwfs
newest
[trapexit:/tmp/mount] $ xattr -w user.mergerfs.srcmounts +/tmp/c .mergerfs [trapexit:/tmp/mount] $ xattr -w user.mergerfs.srcmounts +/tmp/c .mergerfs
[trapexit:/tmp/mount] $ xattr -p user.mergerfs.srcmounts .mergerfs [trapexit:/tmp/mount] $ xattr -p user.mergerfs.srcmounts .mergerfs

18
man/mergerfs.1

@ -44,8 +44,8 @@ These options seem to provide the best performance.
\f[B]direct_io\f[]: causes FUSE to bypass an addition caching step which \f[B]direct_io\f[]: causes FUSE to bypass an addition caching step which
can increase write speeds at the detriment of read speed. can increase write speeds at the detriment of read speed.
.IP \[bu] 2 .IP \[bu] 2
\f[B]minfreespace\f[]: the minimum space value used for the
\f[B]lfs\f[], \f[B]fwfs\f[], \f[B]eplfs\f[], & \f[B]epmfs\f[] policies.
\f[B]minfreespace\f[]: the minimum space value used for creation
policies.
Understands \[aq]K\[aq], \[aq]M\[aq], and \[aq]G\[aq] to represent Understands \[aq]K\[aq], \[aq]M\[aq], and \[aq]G\[aq] to represent
kilobyte, megabyte, and gigabyte respectively. kilobyte, megabyte, and gigabyte respectively.
(default: 4G) (default: 4G)
@ -161,8 +161,8 @@ In other cases where something may be searched (to confirm a directory
exists across all source mounts) then \f[B]getattr\f[] will be used. exists across all source mounts) then \f[B]getattr\f[] will be used.
.SS Policy descriptions .SS Policy descriptions
.PP .PP
Generally speaking most policies when called to create will filter out
drives which are readonly or have less than \f[C]minfreespace\f[].
Most policies when called to create will filter out drives which are
readonly or have less than \f[B]minfreespace\f[].
.PP .PP
.TS .TS
tab(@); tab(@);
@ -212,12 +212,6 @@ Given the order of the drives, as defined at mount time or when
configured via xattr interface, act on the first one found. configured via xattr interface, act on the first one found.
T} T}
T{ T{
fwfs (first with free space)
T}@T{
Pick the first drive which has at least \f[B]minfreespace\f[].
Falls back to \f[B]mfs\f[].
T}
T{
lfs (least free space) lfs (least free space)
T}@T{ T}@T{
Pick the drive with the least available free space but more than Pick the drive with the least available free space but more than
@ -533,9 +527,9 @@ user.mergerfs.moveonenospc:\ false
[trapexit:/tmp/mount]\ $\ xattr\ \-p\ user.mergerfs.category.search\ .mergerfs [trapexit:/tmp/mount]\ $\ xattr\ \-p\ user.mergerfs.category.search\ .mergerfs
ff ff
[trapexit:/tmp/mount]\ $\ xattr\ \-w\ user.mergerfs.category.search\ fwfs\ .mergerfs
[trapexit:/tmp/mount]\ $\ xattr\ \-w\ user.mergerfs.category.search\ newest\ .mergerfs
[trapexit:/tmp/mount]\ $\ xattr\ \-p\ user.mergerfs.category.search\ .mergerfs [trapexit:/tmp/mount]\ $\ xattr\ \-p\ user.mergerfs.category.search\ .mergerfs
fwfs
newest
[trapexit:/tmp/mount]\ $\ xattr\ \-w\ user.mergerfs.srcmounts\ +/tmp/c\ .mergerfs [trapexit:/tmp/mount]\ $\ xattr\ \-w\ user.mergerfs.srcmounts\ +/tmp/c\ .mergerfs
[trapexit:/tmp/mount]\ $\ xattr\ \-p\ user.mergerfs.srcmounts\ .mergerfs [trapexit:/tmp/mount]\ $\ xattr\ \-p\ user.mergerfs.srcmounts\ .mergerfs

18
src/fs.cpp

@ -70,6 +70,24 @@ namespace fs
return exists(path,st); return exists(path,st);
} }
bool
exists(const string &path,
bool &readonly,
size_t &spaceavail)
{
bool rv;
struct statvfs st;
rv = exists(path,st);
if(rv)
{
readonly = StatVFS::readonly(st);
spaceavail = StatVFS::spaceavail(st);
}
return rv;
}
bool bool
exists_on_rw_fs(const string &path, exists_on_rw_fs(const string &path,
struct statvfs &st) struct statvfs &st)

3
src/fs.hpp

@ -31,6 +31,9 @@ namespace fs
bool exists(const string &path, bool exists(const string &path,
struct statvfs &st); struct statvfs &st);
bool exists(const string &path); bool exists(const string &path);
bool exists(const string &path,
bool &readonly,
size_t &spaceavail);
bool exists_on_rw_fs(const string &path, bool exists_on_rw_fs(const string &path,
struct statvfs &st); struct statvfs &st);

2
src/policy.cpp

@ -34,7 +34,6 @@ namespace mergerfs
(POLICY(epmfs,PRESERVES_PATH)) (POLICY(epmfs,PRESERVES_PATH))
(POLICY(erofs,DOESNT_PRESERVE_PATH)) (POLICY(erofs,DOESNT_PRESERVE_PATH))
(POLICY(ff,DOESNT_PRESERVE_PATH)) (POLICY(ff,DOESNT_PRESERVE_PATH))
(POLICY(fwfs,DOESNT_PRESERVE_PATH))
(POLICY(lfs,DOESNT_PRESERVE_PATH)) (POLICY(lfs,DOESNT_PRESERVE_PATH))
(POLICY(mfs,DOESNT_PRESERVE_PATH)) (POLICY(mfs,DOESNT_PRESERVE_PATH))
(POLICY(newest,DOESNT_PRESERVE_PATH)) (POLICY(newest,DOESNT_PRESERVE_PATH))
@ -50,7 +49,6 @@ namespace mergerfs
CONST_POLICY(epmfs); CONST_POLICY(epmfs);
CONST_POLICY(erofs); CONST_POLICY(erofs);
CONST_POLICY(ff); CONST_POLICY(ff);
CONST_POLICY(fwfs);
CONST_POLICY(lfs); CONST_POLICY(lfs);
CONST_POLICY(mfs); CONST_POLICY(mfs);
CONST_POLICY(newest); CONST_POLICY(newest);

3
src/policy.hpp

@ -47,7 +47,6 @@ namespace mergerfs
epmfs, epmfs,
erofs, erofs,
ff, ff,
fwfs,
lfs, lfs,
mfs, mfs,
newest, newest,
@ -100,7 +99,6 @@ namespace mergerfs
static int epmfs(CType,cstrvec&,const char *,csize_t,cstrptrvec&); static int epmfs(CType,cstrvec&,const char *,csize_t,cstrptrvec&);
static int erofs(CType,cstrvec&,const char *,csize_t,cstrptrvec&); static int erofs(CType,cstrvec&,const char *,csize_t,cstrptrvec&);
static int ff(CType,cstrvec&,const char *,csize_t,cstrptrvec&); static int ff(CType,cstrvec&,const char *,csize_t,cstrptrvec&);
static int fwfs(CType,cstrvec&,const char *,csize_t,cstrptrvec&);
static int lfs(CType,cstrvec&,const char *,csize_t,cstrptrvec&); static int lfs(CType,cstrvec&,const char *,csize_t,cstrptrvec&);
static int mfs(CType,cstrvec&,const char *,csize_t,cstrptrvec&); static int mfs(CType,cstrvec&,const char *,csize_t,cstrptrvec&);
static int newest(CType,cstrvec&,const char *,csize_t,cstrptrvec&); static int newest(CType,cstrvec&,const char *,csize_t,cstrptrvec&);
@ -172,7 +170,6 @@ namespace mergerfs
static const Policy &epmfs; static const Policy &epmfs;
static const Policy &erofs; static const Policy &erofs;
static const Policy &ff; static const Policy &ff;
static const Policy &fwfs;
static const Policy &lfs; static const Policy &lfs;
static const Policy &mfs; static const Policy &mfs;
static const Policy &newest; static const Policy &newest;

61
src/policy_ff.cpp

@ -15,7 +15,6 @@
*/ */
#include <errno.h> #include <errno.h>
#include <sys/statvfs.h>
#include <string> #include <string>
#include <vector> #include <vector>
@ -23,7 +22,6 @@
#include "fs.hpp" #include "fs.hpp"
#include "fs_path.hpp" #include "fs_path.hpp"
#include "policy.hpp" #include "policy.hpp"
#include "statvfs_util.hpp"
using std::string; using std::string;
using std::vector; using std::vector;
@ -31,54 +29,54 @@ using std::size_t;
static static
int int
_ff_rw(const vector<string> &basepaths,
vector<const string*> &paths)
_ff(const vector<string> &basepaths,
const char *fusepath,
vector<const string*> &paths)
{ {
struct statvfs st;
const string *fallback = NULL;
string fullpath;
for(size_t i = 0, ei = basepaths.size(); i != ei; i++) for(size_t i = 0, ei = basepaths.size(); i != ei; i++)
{ {
const string *basepath = &basepaths[i]; const string *basepath = &basepaths[i];
if(!fs::exists(basepath->c_str(),st))
continue;
fs::path::make(basepath,fusepath,fullpath);
if(StatVFS::readonly(st))
{
if(fallback == NULL)
fallback = basepath;
continue;
}
if(!fs::exists(fullpath))
continue;
paths.push_back(basepath); paths.push_back(basepath);
return POLICY_SUCCESS; return POLICY_SUCCESS;
} }
if(fallback == NULL)
return POLICY_FAIL_ENOENT;
paths.push_back(fallback);
return POLICY_SUCCESS;
return POLICY_FAIL_ENOENT;
} }
static static
int int
_ff_ro(const vector<string> &basepaths,
const char *fusepath,
vector<const string*> &paths)
_ff_create(const vector<string> &basepaths,
const size_t minfreespace,
vector<const string*> &paths)
{ {
string fullpath;
bool readonly;
size_t spaceavail;
const string *fallback;
fallback = NULL;
for(size_t i = 0, ei = basepaths.size(); i != ei; i++) for(size_t i = 0, ei = basepaths.size(); i != ei; i++)
{ {
const string *basepath = &basepaths[i]; const string *basepath = &basepaths[i];
fs::path::make(basepath,fusepath,fullpath);
if(!fs::exists(*basepath,readonly,spaceavail))
continue;
if(!fs::exists(fullpath))
if(readonly)
continue;
if(fallback == NULL)
fallback = basepath;
if(spaceavail < minfreespace)
continue; continue;
paths.push_back(basepath); paths.push_back(basepath);
@ -86,7 +84,12 @@ _ff_ro(const vector<string> &basepaths,
return POLICY_SUCCESS; return POLICY_SUCCESS;
} }
return POLICY_FAIL_ENOENT;
if(fallback == NULL)
return POLICY_FAIL_ENOENT;
paths.push_back(fallback);
return POLICY_SUCCESS;
} }
namespace mergerfs namespace mergerfs
@ -99,8 +102,8 @@ namespace mergerfs
vector<const string*> &paths) vector<const string*> &paths)
{ {
if(type == Category::Enum::create) if(type == Category::Enum::create)
return _ff_rw(basepaths,paths);
return _ff_create(basepaths,minfreespace,paths);
return _ff_ro(basepaths,fusepath,paths);
return _ff(basepaths,fusepath,paths);
} }
} }

86
src/policy_fwfs.cpp

@ -1,86 +0,0 @@
/*
Copyright (c) 2016, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include <errno.h>
#include <sys/statvfs.h>
#include <string>
#include <vector>
#include "fs.hpp"
#include "fs_path.hpp"
#include "policy.hpp"
#include "statvfs_util.hpp"
using std::string;
using std::vector;
using std::size_t;
using mergerfs::Policy;
using mergerfs::Category;
static
int
_fwfs(const vector<string> &basepaths,
const char *fusepath,
const size_t minfreespace,
const bool needswritablefs,
vector<const string*> &paths)
{
string fullpath;
struct statvfs st;
for(size_t i = 0, size = basepaths.size(); i != size; i++)
{
const string *basepath = &basepaths[i];
fs::path::make(basepath,fusepath,fullpath);
if(!fs::available(fullpath,needswritablefs,st))
continue;
if(StatVFS::spaceavail(st) < minfreespace)
continue;
paths.push_back(basepath);
return POLICY_SUCCESS;
}
return (errno=ENOENT,POLICY_FAIL);
}
namespace mergerfs
{
int
Policy::Func::fwfs(const Category::Enum::Type type,
const vector<string> &basepaths,
const char *fusepath,
const size_t minfreespace,
vector<const string*> &paths)
{
int rv;
const char *fp =
((type == Category::Enum::create) ? "" : fusepath);
const bool needswritablefs =
(type == Category::Enum::create);
rv = _fwfs(basepaths,fp,minfreespace,needswritablefs,paths);
if(POLICY_FAILED(rv))
rv = Policy::Func::mfs(type,basepaths,fusepath,minfreespace,paths);
return rv;
}
}

52
src/policy_newest.cpp

@ -16,26 +16,62 @@
#include <errno.h> #include <errno.h>
#include <sys/stat.h> #include <sys/stat.h>
#include <sys/statvfs.h>
#include <string> #include <string>
#include <vector> #include <vector>
#include <limits> #include <limits>
#include "fs.hpp"
#include "fs_path.hpp" #include "fs_path.hpp"
#include "policy.hpp" #include "policy.hpp"
#include "success_fail.hpp" #include "success_fail.hpp"
#include "statvfs_util.hpp"
using std::string; using std::string;
using std::vector; using std::vector;
using std::size_t; using std::size_t;
static
int
_newest_create(const vector<string> &basepaths,
const char *fusepath,
vector<const string*> &paths)
{
time_t newest;
string fullpath;
struct stat st;
const string *newestbasepath;
newest = std::numeric_limits<time_t>::min();
newestbasepath = NULL;
for(size_t i = 0, ei = basepaths.size(); i != ei; i++)
{
const string *basepath = &basepaths[i];
fs::path::make(basepath,fusepath,fullpath);
if(!fs::exists(fullpath,st))
continue;
if(st.st_mtime < newest)
continue;
if(!fs::exists_on_rw_fs(fullpath))
continue;
newest = st.st_mtime;
newestbasepath = basepath;
}
if(newestbasepath == NULL)
return POLICY_FAIL_ENOENT;
paths.push_back(newestbasepath);
return POLICY_SUCCESS;
}
static static
int int
_newest(const vector<string> &basepaths, _newest(const vector<string> &basepaths,
const char *fusepath, const char *fusepath,
const bool needswritablefs,
vector<const string*> &paths) vector<const string*> &paths)
{ {
time_t newest; time_t newest;
@ -55,15 +91,13 @@ _newest(const vector<string> &basepaths,
continue; continue;
if(st.st_mtime < newest) if(st.st_mtime < newest)
continue; continue;
if(needswritablefs && !fs::exists_on_rw_fs(fullpath))
continue;
newest = st.st_mtime; newest = st.st_mtime;
newestbasepath = basepath; newestbasepath = basepath;
} }
if(newestbasepath == NULL) if(newestbasepath == NULL)
return (errno=ENOENT,POLICY_FAIL);
return POLICY_FAIL_ENOENT;
paths.push_back(newestbasepath); paths.push_back(newestbasepath);
@ -79,9 +113,9 @@ namespace mergerfs
const size_t minfreespace, const size_t minfreespace,
vector<const string*> &paths) vector<const string*> &paths)
{ {
const bool needswritablefs =
(type == Category::Enum::create);
if(type == Category::Enum::create)
return _newest_create(basepaths,fusepath,paths);
return _newest(basepaths,fusepath,needswritablefs,paths);
return _newest(basepaths,fusepath,paths);
} }
} }
Loading…
Cancel
Save