|
@ -13,10 +13,11 @@ and/or disabling caching the values returned will not be |
|
|
representative of the device's true performance. |
|
|
representative of the device's true performance. |
|
|
|
|
|
|
|
|
When benchmarking through mergerfs ensure you only use 1 branch to |
|
|
When benchmarking through mergerfs ensure you only use 1 branch to |
|
|
remove any possibility of the policies complicating the |
|
|
|
|
|
situation. Benchmark the underlying filesystem first and then mount |
|
|
|
|
|
mergerfs over it and test again. If you're experiencing speeds below |
|
|
|
|
|
your expectation you will need to narrow down precisely which |
|
|
|
|
|
|
|
|
remove any possibility of the policies or differences in underlying |
|
|
|
|
|
filesystems complicating the situation. Benchmark the underlying |
|
|
|
|
|
filesystem first and then mount mergerfs with that same filesystem and |
|
|
|
|
|
that. Throughput **will be** lower but if you are experiencing speeds |
|
|
|
|
|
below your expectation you will need to narrow down precisely which |
|
|
component is leading to the slowdown. Preferably test the following in |
|
|
component is leading to the slowdown. Preferably test the following in |
|
|
the order listed (but not combined). |
|
|
the order listed (but not combined). |
|
|
|
|
|
|
|
@ -24,37 +25,38 @@ the order listed (but not combined). |
|
|
reads and writes no-ops. Removing the underlying device / |
|
|
reads and writes no-ops. Removing the underlying device / |
|
|
filesystem from the equation. This will give us the top theoretical |
|
|
filesystem from the equation. This will give us the top theoretical |
|
|
speeds. |
|
|
speeds. |
|
|
2. Mount mergerfs over `tmpfs`. `tmpfs` is a RAM disk. Extremely high |
|
|
|
|
|
speed and very low latency. This is a more realistic best case |
|
|
|
|
|
scenario. Example: `mount -t tmpfs -o size=2G tmpfs /tmp/tmpfs` |
|
|
|
|
|
3. Mount mergerfs over a local device. NVMe, SSD, HDD, etc. If you |
|
|
|
|
|
have more than one I'd suggest testing each of them as drives |
|
|
|
|
|
and/or controllers (their drivers) could impact performance. |
|
|
|
|
|
4. Finally, if you intend to use mergerfs with a network filesystem, |
|
|
|
|
|
either as the source of data or to combine with another through |
|
|
|
|
|
mergerfs, test each of those alone as above. |
|
|
|
|
|
|
|
|
2. Configure mergerfs to use a `tmpfs` branch. `tmpfs` is a RAM |
|
|
|
|
|
disk. Extremely high speed and very low latency. This is a more |
|
|
|
|
|
realistic best case scenario. Example: `mount -t tmpfs -o size=2G |
|
|
|
|
|
tmpfs /tmp/tmpfs` |
|
|
|
|
|
3. Configure mergerfs to use a local device filesystem branch. NVMe, |
|
|
|
|
|
SSD, HDD, etc. Test them individually. If you have different |
|
|
|
|
|
interconnects / controllers use the same storage device when |
|
|
|
|
|
testing to ensure consistency. |
|
|
|
|
|
4. Configure mergerfs to use any network filesystems you plan to use |
|
|
|
|
|
one at a time. It may also be worth trying a different network |
|
|
|
|
|
filesystem. `NFS` vs `CIFS/SMB/Samba` vs `sshfs`, etc. |
|
|
|
|
|
|
|
|
Once you find the component which has the performance issue you can do |
|
|
Once you find the component which has the performance issue you can do |
|
|
further testing with different options to see if they impact |
|
|
further testing with different options to see if they impact |
|
|
performance. For reads and writes the most relevant would be: |
|
|
|
|
|
`cache.files`, `async_read`. Less likely but relevant when using NFS |
|
|
|
|
|
or with certain filesystems would be `security_capability`, `xattr`, |
|
|
|
|
|
and `posix_acl`. If you find a specific system, device, filesystem, |
|
|
|
|
|
controller, etc. that performs poorly contact trapexit so he may |
|
|
|
|
|
investigate further. |
|
|
|
|
|
|
|
|
performance. If you find a specific system, device, filesystem, |
|
|
|
|
|
controller, etc. that performs poorly contact the author so it can be |
|
|
|
|
|
investigated further. |
|
|
|
|
|
|
|
|
Sometimes the problem is really the application accessing or writing |
|
|
Sometimes the problem is really the application accessing or writing |
|
|
data through mergerfs. Some software use small buffer sizes which can |
|
|
data through mergerfs. Some software use small buffer sizes which can |
|
|
lead to more requests and therefore greater overhead. You can test |
|
|
lead to more requests and therefore greater overhead. You can test |
|
|
this out yourself by replacing `bs=1M` in the examples below with `ibs` |
|
|
|
|
|
or `obs` and using a size of `512` instead of `1M`. In one example |
|
|
|
|
|
test using `nullrw` the write speed dropped from 4.9GB/s to 69.7MB/s |
|
|
|
|
|
when moving from `1M` to `512`. Similar results were had when testing |
|
|
|
|
|
reads. Small writes overhead may be improved by leveraging a write |
|
|
|
|
|
cache but in casual tests little gain was found. More tests will need |
|
|
|
|
|
to be done before this feature would become available. If you have an |
|
|
|
|
|
app that appears slow with mergerfs it could be due to this. Contact |
|
|
|
|
|
trapexit so he may investigate further. |
|
|
|
|
|
|
|
|
this out yourself by replacing `bs=1M` in the examples below with |
|
|
|
|
|
`ibs` or `obs` and using a size of `512` instead of `1M`. In one |
|
|
|
|
|
example test using `nullrw` the write speed dropped from 4.9GB/s to |
|
|
|
|
|
69.7MB/s when moving from `1M` to `512`. Similar results were had when |
|
|
|
|
|
testing reads. Small writes overhead may be improved by leveraging a |
|
|
|
|
|
write cache but in casual tests little gain was found. If you have an |
|
|
|
|
|
app that appears slow with mergerfs it could be due to this. `strace` |
|
|
|
|
|
can be used with the app in question or mergerfs to see the size of |
|
|
|
|
|
read/writes. Contact the software author or worse case the mergerfs |
|
|
|
|
|
author so it may be investigated further. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### write benchmark |
|
|
### write benchmark |
|
|
|
|
|
|
|
@ -78,3 +80,9 @@ run before the read and write benchmarks as well just in case. |
|
|
sync |
|
|
sync |
|
|
echo 3 | sudo tee /proc/sys/vm/drop_caches |
|
|
echo 3 | sudo tee /proc/sys/vm/drop_caches |
|
|
``` |
|
|
``` |
|
|
|
|
|
|
|
|
|
|
|
## Additional Reading |
|
|
|
|
|
|
|
|
|
|
|
* [Tweaking Performance](performance.md) |
|
|
|
|
|
* [Options](config/options.md) |
|
|
|
|
|
* [Tips and Notes](tips_notes.md) |