- RedbNeedleMap: pure-Rust disk-backed needle map using redb, with NeedleMap
enum wrapping both in-memory and redb variants
- binary_search_by_append_at_ns: port of Go's BinarySearchByAppendAtNs for
VolumeIncrementalCopy with since_ns > 0
- Proxy/redirect: master volume lookup, HTTP proxy forwarding with ?proxied=true,
and 301 redirects for non-local volumes based on ReadMode config
- Wire new VolumeServerState fields: read_mode, master_url, self_url, http_client
- Extract TTL from ?ttl= query param and set on needle (matches Go's ParseUpload)
- Auto-compress compressible file types (.js, .css, .json, .svg, text/*, etc.)
using gzip, only when compression saves >10% (matches Go's IsCompressableFileType)
- Extract Seaweed-* headers as custom metadata pairs stored as JSON in needle
- Store filename from URL path in needle name field
- Include filename in upload response JSON
- Add unit tests for is_compressible_file_type and try_gzip_data
- BatchDelete now supports EC volumes: looks up needle in .ecx index,
journals deletion to .ecj file (local-only, Go handles distributed part)
- JPEG EXIF orientation auto-fix on upload using kamadak-exif + image crate,
matching Go's FixJpgOrientation behavior (8 orientation transforms)
- Async batched write processing via mpsc queue (up to 128 entries per batch),
groups writes by volume ID and syncs once per volume per batch
- VolumeTierMoveDatToRemote: multipart upload .dat file to S3 with progress
streaming, updates .vif with remote file reference
- VolumeTierMoveDatFromRemote: downloads .dat from S3 with progress streaming,
removes remote file reference from .vif
- S3TierRegistry for managing named remote storage backends
- VolumeInfo (.vif) JSON persistence matching Go's protojson format
- 124 lib + 7 integration = 131 Rust tests pass
- All 109 Go integration tests pass (53 HTTP + 56 gRPC)
- Add StreamingBody (http_body::Body) for chunked reads of files >1MB,
avoiding OOM by reading 64KB at a time via spawn_blocking + pread
- Add NeedleStreamInfo and meta-only read path to avoid loading full
needle body when streaming
- Add RustMultiVolumeCluster test framework and MultiCluster interface
so TestVolumeMoveHandlesInFlightWrites works with Rust volume servers
- Remove TestVolumeMoveHandlesInFlightWrites from CI skip list (now passes)
- All 117 Go integration tests + 8 Rust integration tests pass (100%)
- Add remote_storage module with RemoteStorageClient trait
- Implement S3RemoteStorageClient using aws-sdk-s3 (covers all S3-compatible
providers: AWS, Wasabi, Backblaze, Aliyun, etc.)
- FetchAndWriteNeedle now fetches data from S3, writes locally as needle,
and replicates to peers
- Add 3 integration tests using weed mini as S3 backend:
- Full round-trip fetch from S3
- Byte-range (partial) read from S3
- Error handling for non-existent S3 objects
- VolumeServerStatus now returns real data_center and rack from config
- DiskStatus uses sysinfo crate for actual disk total/free/used/percent
- ReadVolumeFileStatus returns dat file modification timestamps
- FetchAndWriteNeedle produces Go-matching error messages for unknown remote storage types
Remove unused imports, prefix unused variables, add #[allow(dead_code)]
for fields/methods used only in specific contexts (serde deserialization,
non-unix builds, future error tracking).
VolumeCopy: connects to source as gRPC client, copies .dat/.idx/.vif
files via CopyFile streaming, verifies sizes, mounts the volume.
Needed for volume rebalancing and migration between servers.
VolumeTailReceiver: connects to source's VolumeTailSender, receives
needle header+body chunks, reassembles multi-chunk needles, writes
them locally. Needed for live replication during volume moves.
Also adds helper functions: parse_grpc_address (SeaweedFS address
format parsing), copy_file_from_source (streaming file copy with
progress reporting), find_last_append_at_ns (timestamp extraction
from copied files).
All 128/130 integration tests still pass (same 2 known unfixable).
Adds full volume compaction support:
- compact_by_index: iterates live needles, writes to new .cpd/.cpx
files, skipping deleted and TTL-expired entries
- commit_compact: swaps .cpd->.dat and .cpx->.idx, reloads volume
- cleanup_compact: removes leftover .cpd/.cpx files
Wires VacuumVolumeCompact (streaming progress), VacuumVolumeCommit,
and VacuumVolumeCleanup gRPC RPCs. Also adds save_to_idx and
ascending_visit methods to CompactNeedleMap.
Includes unit test verifying compact removes deleted needles and
preserves live data through the full compact/commit cycle.
The bidirectional gRPC stream had a deadlock: send_heartbeat().await
waits for response headers from the server, but the Go master won't
send response headers until it receives the first heartbeat message.
Pre-populating the channel before creating the stream resolves this.
Tested end-to-end: Rust volume server successfully registers with Go
master, receives volume assignments, and handles uploads/downloads.
When a needle has the chunk manifest flag but its data isn't valid
JSON (e.g., placeholder payload), fall through to return raw data
instead of returning 500. This fixes TestBatchDeleteRejectsChunkManifestNeedles.
- BatchDelete: use proper HTTP status codes (400/404/202/304/406/500),
cookie validation with break on mismatch, chunk manifest rejection
- Remove cookie validation from do_delete_request (match Go behavior),
add it to HTTP delete handler and BatchDelete handler instead
- Fix needle_map delete to keep original offset (not tombstone offset)
so readDeleted can find original data
- Add readDeleted query param support for reading deleted needles
- Add multipart boundary validation and Content-MD5 check on uploads
- Add ?cm=true upload param for chunk manifest flag
- Fix ReadNeedleMeta to read at offset/size (not by needle ID)
- Fix ReadNeedleBlob error message format
- Fix ping target type: "volumeServer" not "volume"
- Populate idx_file_size in ReadVolumeFileStatus
- Add md-5 and base64 crate dependencies
Test results: gRPC 35/75, HTTP 39/55 (was 27/75, 36/55)
The prometheus crate's default features include protobuf exposition,
which pulls in protobuf v2.28.0 (GHSA-2gh3-rmm4-6rq5, moderate).
We only use text-format metrics, so disable default-features and
keep only the "process" feature.
- Add Windows read_exact_at helper that loops seek_read to handle short reads
- Update all Windows read paths (read_needle_data, read_needle_blob, read_needle_header)
- Fix ip_in_cidr panic on prefix_len == 0 (shift overflow)
- Remove unused large-block EC encoding path (always use small blocks)
- Add duplicate volume ID check in Store::add_location
- Enhance is_file_unchanged to compare metadata fields
- Add warning for missing .idx with existing .dat
- Document JWT exp validation matching Go behavior
Security:
- Remove unused is_write_active field from Guard
gRPC:
- Update needle index after write_needle_blob via new put_needle_index()
- Return unimplemented for set_state (no state persistence yet)
- Add TODO for DiskStatus total/used metrics
Storage:
- Track overwritten needle bytes in deletion_byte_count for garbage_level
- Fix sync order: flush dat file before idx for crash safety
- Reject duplicate volume IDs in mount_volume
Erasure coding:
- Add OOM guard (1 GiB limit) for encode_one_batch allocation
- Add non-Unix read fallback in encode_one_batch
- Validate shard ID bounds in add/remove/has_shard_id
Security:
- Require fid claim in JWT tokens (reject tokens without fid)
gRPC:
- Honor only_empty flag in VolumeDelete RPC
Storage:
- Reject duplicate volume IDs in add_volume
- Persist .idx entries before mutating in-memory state (crash consistency)
- Validate cookie on needle delete to prevent unauthorized tombstoning
- Skip full body parsing for deleted needles during volume scans
- Log warning when unsupported needle_map_kind is requested
Erasure coding:
- Sort index entries by actual offset for correct dedup ordering
- Propagate .dat read errors instead of encoding zero-filled shards
- Add non-unix fallback for ec_shard reads
Main:
- Propagate startup errors via Result instead of silent return
- Add TODO for JWT config wiring
Add rust-volume-go-tests job that runs the full Go volume server test
matrix (grpc + http, 3 shards each) with VOLUME_SERVER_IMPL=rust.
Also fix concurrency key and add proto path triggers per review.
- Add TestCluster interface in framework/cluster_interface.go
- Add StartVolumeCluster() factory that selects Go or Rust volume server
based on VOLUME_SERVER_IMPL env var
- Update all grpc/ and http/ test files to use StartVolumeCluster()
instead of StartSingleVolumeCluster(), enabling the same test suite
to run against either implementation
- Fix needle_map index loading condition to correctly treat zero-offset
and deleted-size entries as deletions (matches Go behavior)
- Update test to use realistic non-zero offsets (SuperBlock occupies offset 0)
- Replace single shutdown future with broadcast channel so all servers
(HTTP, gRPC, public HTTP) receive graceful shutdown signal
- Add Windows platform support for volume file reads via FileExt::seek_read
- cluster_rust.go: test framework to start Go master + Rust volume server
- test/volume_server/rust/: 8 integration tests (healthz, status, ping,
write/read/delete round-trip, volume lifecycle, get/set state,
server status, metrics endpoint)
- rust-volume-server-tests.yml: CI workflow with Rust unit tests and
Go+Rust integration tests
Adds lib.rs re-exporting all public modules for integration test access.
Refactors main.rs to import from seaweed_volume:: lib crate. Fixes route
parameter syntax from {path} (axum 0.8) to :path (axum 0.7) — the old
brace syntax silently failed to match requests.
Adds set_read_only, set_writable, set_replica_placement, and write_needle_blob
methods to Volume. Makes find_volume_mut public on Store. Fills in previously
stubbed RPCs: volume_mark_readonly, volume_mark_writable, volume_configure,
write_needle_blob, read_needle_meta, volume_ec_shards_generate.
Initializes Store from CLI config, creates DiskLocations, starts axum HTTP
server (admin + optional public) and tonic gRPC server concurrently. Handles
SIGINT/SIGTERM for graceful shutdown. Removes duplicate NeedleMapKind enum
from config.rs in favor of the canonical one in storage::needle_map.
Implements VolumeGrpcService with the tonic-generated VolumeServer trait.
Core operations (batch_delete, allocate, mount/unmount/delete, sync_status,
volume_status, read_needle_blob, needle_status, ping) have real implementations.
Streaming and EC operations are stubbed with Status::unimplemented.
Adds tokio-stream dependency and remote_pb proto module.
Implements the volume server HTTP API matching Go's volume_server.go:
- GET/HEAD /{vid},{fid} — read needle with JWT auth, ETag, Content-Type
- POST/PUT /{vid},{fid} — write needle with dedup detection (204/201)
- DELETE /{vid},{fid} — delete needle with tombstone (202/404)
- GET /status — JSON volume server status with metrics
- GET /healthz — health check endpoint
- Admin router (all ops) and public router (read-only)
- JWT token extraction from Authorization header
- URL path parsing for vid,fid and vid/fid formats
- 6 unit tests for URL parsing and JWT extraction
Implements Guard with HS256 JWT validation and IP whitelist checking,
matching Go's security/guard.go and security/jwt.go. Features:
- FileIdClaims with fid/exp/nbf fields for volume server tokens
- Separate signing keys for read vs write operations
- IP whitelist with exact match and CIDR range support
- JWT generation and validation with file ID claim checking
- IPv4 and IPv6 CIDR matching
- 12 unit tests covering JWT round-trip, whitelist, guard logic
DiskLocation manages volumes in a single directory with load/create/
delete/collection operations and free-space tracking. Store coordinates
multiple DiskLocations with load-balanced volume placement, delegated
read/write/delete, and mount/unmount lifecycle. Matches Go's
disk_location.go and store.go. 11 unit tests covering volume CRUD,
multi-location balancing, and collection management.
Implements the needle map that maps NeedleId → (Offset, Size) using a
HashMap, matching Go's needle_map_memory.go. Includes NeedleMapMetric
with atomic counters for file/deletion tracking, IdxFileWriter trait
for append-only .idx writes, and load_from_idx for rebuilding from
index files. 5 unit tests covering put/get/delete/metrics/loading.
Index file (.idx) format: sequential 17-byte entries walked in 1024-row batches.
walk_index_file() and write_index_entry() match Go's idx.WalkIndexFile().
3 unit tests. Added crc32c dependency for Castagnoli CRC.
8-byte header at start of .dat files:
Version(1) + ReplicaPlacement(1) + TTL(2) + CompactionRevision(2) + ExtraSize(2)
Optional protobuf extra data follows.
ReplicaPlacement: byte = DC*100 + Rack*10 + SameRack.
5 unit tests for round-trip, extra data, and replication semantics.
Chris Lu