From 050ab1926409edbb8ac1186b1fb83e80c17fc33f Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Fri, 4 May 2018 23:38:42 -0700 Subject: [PATCH] add design --- weed/filer/vasto_store/design.txt | 45 +++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 weed/filer/vasto_store/design.txt diff --git a/weed/filer/vasto_store/design.txt b/weed/filer/vasto_store/design.txt new file mode 100644 index 000000000..99faa29d6 --- /dev/null +++ b/weed/filer/vasto_store/design.txt @@ -0,0 +1,45 @@ +There are two main components of a filer: directories and files. + +My previous approach was to use some sequance number to generate directoryId. +However, this is not scalable. The id generation itself is a bottleneck. +It needs careful locking and deduplication checking to get a directoryId. + +In a second design, each directory is deterministically mapped to UUID version 3, +which uses MD5 to map a tuple of to a version 3 UUID. +However, this UUID3 approach is logically the same as storing the full path. + +Storing the full path is the simplest design. + +separator is a special byte, 0x00. + +When writing a file: + => fildId, file properties +For folders: + The filer breaks the directory path into folders. + for each folder: + if it is not in cache: + check whether the folder is created in the KVS, if not: + set => directory properties + if no permission for the folder: + break + + +The filer caches the most recently used folder permissions with a TTL. + So any folder permission change needs to wait TTL interval to take effect. + + + +When listing the directory: + prefix scan of using (the folder full path + separator) as the prefix + +The downside: + 1. Rename a folder will need to recursively process all sub folders and files. + 2. Move a folder will need to recursively process all sub folders and files. +So these operations are not allowed if the folder is not empty. + +Allowing: + 1. Rename a file + 2. Move a file to a different folder + 3. Delete an empty folder + +