From ba97f3cc8eddd2d3c1263741840bd68d8cc55d49 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Mon, 12 Jan 2026 15:53:27 -0800 Subject: [PATCH] Update README.md --- README.md | 76 +++++++++++++++++++++++++------------------------------ 1 file changed, 35 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 26ddd4c86..f5ab2d698 100644 --- a/README.md +++ b/README.md @@ -122,7 +122,7 @@ SeaweedFS is a simple and highly scalable distributed file system. There are two 1. to store billions of files! 2. to serve the files fast! -SeaweedFS started as an Object Store to handle small files efficiently. +SeaweedFS started as a blob store to handle small files efficiently. Instead of managing all file metadata in a central master, the central master only manages volumes on volume servers, and these volume servers manage files and their metadata. @@ -134,16 +134,12 @@ It is so simple with O(1) disk reads that you are welcome to challenge the perfo SeaweedFS started by implementing [Facebook's Haystack design paper](http://www.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf). Also, SeaweedFS implements erasure coding with ideas from -[f4: Facebook’s Warm BLOB Storage System](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-muralidhar.pdf), and has a lot of similarities with [Facebook’s Tectonic Filesystem](https://www.usenix.org/system/files/fast21-pan.pdf) +[f4: Facebook’s Warm BLOB Storage System](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-muralidhar.pdf), and has a lot of similarities with [Facebook’s Tectonic Filesystem](https://www.usenix.org/system/files/fast21-pan.pdf) and Google's Colossle -On top of the object store, optional [Filer] can support directories and POSIX attributes. +On top of the blob store, optional [Filer] can support directories and POSIX attributes. Filer is a separate linearly-scalable stateless server with customizable metadata stores, e.g., MySql, Postgres, Redis, Cassandra, HBase, Mongodb, Elastic Search, LevelDB, RocksDB, Sqlite, MemSql, TiDB, Etcd, CockroachDB, YDB, etc. -For any distributed key value stores, the large values can be offloaded to SeaweedFS. -With the fast access speed and linearly scalable capacity, -SeaweedFS can work as a distributed [Key-Large-Value store][KeyLargeValueStore]. - SeaweedFS can transparently integrate with the cloud. With hot data on local cluster, and warm data on the cloud with O(1) access time, SeaweedFS can achieve both fast local access time and elastic cloud storage capacity. @@ -153,13 +149,13 @@ Faster and cheaper than direct cloud storage! [Back to TOC](#table-of-contents) # Features # -## Additional Features ## -* Can choose no replication or different replication levels, rack and data center aware. +## Additional Blob Store Features ## +* Support different replication levels, with rack and data center aware. * Automatic master servers failover - no single point of failure (SPOF). -* Automatic Gzip compression depending on file MIME type. +* Automatic compression depending on file MIME type. * Automatic compaction to reclaim disk space after deletion or update. * [Automatic entry TTL expiration][VolumeServerTTL]. -* Any server with some disk space can add to the total storage space. +* Flexible Capacity Expansion: Any server with some disk space can add to the total storage space. * Adding/Removing servers does **not** cause any data re-balancing unless triggered by admin commands. * Optional picture resizing. * Support ETag, Accept-Range, Last-Modified, etc. @@ -167,7 +163,7 @@ Faster and cheaper than direct cloud storage! * Support rebalancing the writable and readonly volumes. * [Customizable Multiple Storage Tiers][TieredStorage]: Customizable storage disk types to balance performance and cost. * [Transparent cloud integration][CloudTier]: unlimited capacity via tiered cloud storage for warm data. -* [Erasure Coding for warm storage][ErasureCoding] Rack-Aware 10.4 erasure coding reduces storage cost and increases availability. +* [Erasure Coding for warm storage][ErasureCoding] Rack-Aware 10.4 erasure coding reduces storage cost and increases availability. Enterprise version can customize EC ratio. [Back to TOC](#table-of-contents) @@ -213,7 +209,7 @@ Faster and cheaper than direct cloud storage! [Back to TOC](#table-of-contents) -## Example: Using Seaweed Object Store ## +## Example: Using Seaweed Blob Store ## By default, the master node runs on port 9333, and the volume nodes run on port 8080. Let's start one master node, and two volume nodes on port 8080 and 8081. Ideally, they should be started from different machines. We'll use localhost as an example. @@ -233,23 +229,25 @@ SeaweedFS uses HTTP REST operations to read, write, and delete. The responses ar > weed volume -dir="/tmp/data2" -max=10 -master="localhost:9333" -port=8081 & ``` -### Write File ### +### Write A Blob ### + +A blob, also referred as a needle, a chunk, or mistakenly as a file, is just a byte array. It can have attributes, such as name, mime type, create or update time, etc. But basically it is just a byte array of a relatively small size, such as 2 MB ~ 64 MB. The size is not fixed. -To upload a file: first, send a HTTP POST, PUT, or GET request to `/dir/assign` to get an `fid` and a volume server URL: +To upload a blob: first, send a HTTP POST, PUT, or GET request to `/dir/assign` to get an `fid` and a volume server URL: ``` > curl http://localhost:9333/dir/assign {"count":1,"fid":"3,01637037d6","url":"127.0.0.1:8080","publicUrl":"localhost:8080"} ``` -Second, to store the file content, send a HTTP multi-part POST request to `url + '/' + fid` from the response: +Second, to store the blob content, send a HTTP multi-part POST request to `url + '/' + fid` from the response: ``` > curl -F file=@/home/chris/myphoto.jpg http://127.0.0.1:8080/3,01637037d6 {"name":"myphoto.jpg","size":43234,"eTag":"1cc0118e"} ``` -To update, send another POST request with updated file content. +To update, send another POST request with updated blob content. For deletion, send an HTTP DELETE request to the same `url + '/' + fid` URL: @@ -257,7 +255,7 @@ For deletion, send an HTTP DELETE request to the same `url + '/' + fid` URL: > curl -X DELETE http://127.0.0.1:8080/3,01637037d6 ``` -### Save File Id ### +### Save Blob Id ### Now, you can save the `fid`, 3,01637037d6 in this case, to a database field. @@ -269,9 +267,9 @@ The file key and file cookie are both coded in hex. You can store the . Of course, each map entry has its own space cost for the map. But usually the disk space runs out before the memory does. +All blob metadata stored on a volume server is readable from memory without disk access. Each file takes just a 16-byte map entry of <64bit key, 32bit offset, 32bit size>. Of course, each map entry has its own space cost for the map. But usually the disk space runs out before the memory does. ### Tiered Storage to the cloud ### @@ -415,6 +403,12 @@ If the hot/warm data is split as 20/80, with 20 servers, you can achieve storage [Back to TOC](#table-of-contents) +## SeaweedFS Filer ## + +Built on top of the blob store, SeaweedFS Filer adds directory structure to create a file system. The directory sturcture is an interface that is implemented in many key-value stores or databases. + +The content of a file is mapped to one or many blobs, distributed to multiple volumes on multiple volume servers. + ## Compared to Other File Systems ## Most other distributed file systems seem more complicated than necessary.