Adds full volume compaction support:
- compact_by_index: iterates live needles, writes to new .cpd/.cpx
files, skipping deleted and TTL-expired entries
- commit_compact: swaps .cpd->.dat and .cpx->.idx, reloads volume
- cleanup_compact: removes leftover .cpd/.cpx files
Wires VacuumVolumeCompact (streaming progress), VacuumVolumeCommit,
and VacuumVolumeCleanup gRPC RPCs. Also adds save_to_idx and
ascending_visit methods to CompactNeedleMap.
Includes unit test verifying compact removes deleted needles and
preserves live data through the full compact/commit cycle.
The bidirectional gRPC stream had a deadlock: send_heartbeat().await
waits for response headers from the server, but the Go master won't
send response headers until it receives the first heartbeat message.
Pre-populating the channel before creating the stream resolves this.
Tested end-to-end: Rust volume server successfully registers with Go
master, receives volume assignments, and handles uploads/downloads.
When a needle has the chunk manifest flag but its data isn't valid
JSON (e.g., placeholder payload), fall through to return raw data
instead of returning 500. This fixes TestBatchDeleteRejectsChunkManifestNeedles.
- BatchDelete: use proper HTTP status codes (400/404/202/304/406/500),
cookie validation with break on mismatch, chunk manifest rejection
- Remove cookie validation from do_delete_request (match Go behavior),
add it to HTTP delete handler and BatchDelete handler instead
- Fix needle_map delete to keep original offset (not tombstone offset)
so readDeleted can find original data
- Add readDeleted query param support for reading deleted needles
- Add multipart boundary validation and Content-MD5 check on uploads
- Add ?cm=true upload param for chunk manifest flag
- Fix ReadNeedleMeta to read at offset/size (not by needle ID)
- Fix ReadNeedleBlob error message format
- Fix ping target type: "volumeServer" not "volume"
- Populate idx_file_size in ReadVolumeFileStatus
- Add md-5 and base64 crate dependencies
Test results: gRPC 35/75, HTTP 39/55 (was 27/75, 36/55)
The prometheus crate's default features include protobuf exposition,
which pulls in protobuf v2.28.0 (GHSA-2gh3-rmm4-6rq5, moderate).
We only use text-format metrics, so disable default-features and
keep only the "process" feature.
- Add Windows read_exact_at helper that loops seek_read to handle short reads
- Update all Windows read paths (read_needle_data, read_needle_blob, read_needle_header)
- Fix ip_in_cidr panic on prefix_len == 0 (shift overflow)
- Remove unused large-block EC encoding path (always use small blocks)
- Add duplicate volume ID check in Store::add_location
- Enhance is_file_unchanged to compare metadata fields
- Add warning for missing .idx with existing .dat
- Document JWT exp validation matching Go behavior
Security:
- Remove unused is_write_active field from Guard
gRPC:
- Update needle index after write_needle_blob via new put_needle_index()
- Return unimplemented for set_state (no state persistence yet)
- Add TODO for DiskStatus total/used metrics
Storage:
- Track overwritten needle bytes in deletion_byte_count for garbage_level
- Fix sync order: flush dat file before idx for crash safety
- Reject duplicate volume IDs in mount_volume
Erasure coding:
- Add OOM guard (1 GiB limit) for encode_one_batch allocation
- Add non-Unix read fallback in encode_one_batch
- Validate shard ID bounds in add/remove/has_shard_id
Security:
- Require fid claim in JWT tokens (reject tokens without fid)
gRPC:
- Honor only_empty flag in VolumeDelete RPC
Storage:
- Reject duplicate volume IDs in add_volume
- Persist .idx entries before mutating in-memory state (crash consistency)
- Validate cookie on needle delete to prevent unauthorized tombstoning
- Skip full body parsing for deleted needles during volume scans
- Log warning when unsupported needle_map_kind is requested
Erasure coding:
- Sort index entries by actual offset for correct dedup ordering
- Propagate .dat read errors instead of encoding zero-filled shards
- Add non-unix fallback for ec_shard reads
Main:
- Propagate startup errors via Result instead of silent return
- Add TODO for JWT config wiring
Add rust-volume-go-tests job that runs the full Go volume server test
matrix (grpc + http, 3 shards each) with VOLUME_SERVER_IMPL=rust.
Also fix concurrency key and add proto path triggers per review.
- Add TestCluster interface in framework/cluster_interface.go
- Add StartVolumeCluster() factory that selects Go or Rust volume server
based on VOLUME_SERVER_IMPL env var
- Update all grpc/ and http/ test files to use StartVolumeCluster()
instead of StartSingleVolumeCluster(), enabling the same test suite
to run against either implementation
- Fix needle_map index loading condition to correctly treat zero-offset
and deleted-size entries as deletions (matches Go behavior)
- Update test to use realistic non-zero offsets (SuperBlock occupies offset 0)
- Replace single shutdown future with broadcast channel so all servers
(HTTP, gRPC, public HTTP) receive graceful shutdown signal
- Add Windows platform support for volume file reads via FileExt::seek_read
- cluster_rust.go: test framework to start Go master + Rust volume server
- test/volume_server/rust/: 8 integration tests (healthz, status, ping,
write/read/delete round-trip, volume lifecycle, get/set state,
server status, metrics endpoint)
- rust-volume-server-tests.yml: CI workflow with Rust unit tests and
Go+Rust integration tests
Adds lib.rs re-exporting all public modules for integration test access.
Refactors main.rs to import from seaweed_volume:: lib crate. Fixes route
parameter syntax from {path} (axum 0.8) to :path (axum 0.7) — the old
brace syntax silently failed to match requests.
Adds set_read_only, set_writable, set_replica_placement, and write_needle_blob
methods to Volume. Makes find_volume_mut public on Store. Fills in previously
stubbed RPCs: volume_mark_readonly, volume_mark_writable, volume_configure,
write_needle_blob, read_needle_meta, volume_ec_shards_generate.
Initializes Store from CLI config, creates DiskLocations, starts axum HTTP
server (admin + optional public) and tonic gRPC server concurrently. Handles
SIGINT/SIGTERM for graceful shutdown. Removes duplicate NeedleMapKind enum
from config.rs in favor of the canonical one in storage::needle_map.
Implements VolumeGrpcService with the tonic-generated VolumeServer trait.
Core operations (batch_delete, allocate, mount/unmount/delete, sync_status,
volume_status, read_needle_blob, needle_status, ping) have real implementations.
Streaming and EC operations are stubbed with Status::unimplemented.
Adds tokio-stream dependency and remote_pb proto module.
Implements the volume server HTTP API matching Go's volume_server.go:
- GET/HEAD /{vid},{fid} — read needle with JWT auth, ETag, Content-Type
- POST/PUT /{vid},{fid} — write needle with dedup detection (204/201)
- DELETE /{vid},{fid} — delete needle with tombstone (202/404)
- GET /status — JSON volume server status with metrics
- GET /healthz — health check endpoint
- Admin router (all ops) and public router (read-only)
- JWT token extraction from Authorization header
- URL path parsing for vid,fid and vid/fid formats
- 6 unit tests for URL parsing and JWT extraction
Implements Guard with HS256 JWT validation and IP whitelist checking,
matching Go's security/guard.go and security/jwt.go. Features:
- FileIdClaims with fid/exp/nbf fields for volume server tokens
- Separate signing keys for read vs write operations
- IP whitelist with exact match and CIDR range support
- JWT generation and validation with file ID claim checking
- IPv4 and IPv6 CIDR matching
- 12 unit tests covering JWT round-trip, whitelist, guard logic
DiskLocation manages volumes in a single directory with load/create/
delete/collection operations and free-space tracking. Store coordinates
multiple DiskLocations with load-balanced volume placement, delegated
read/write/delete, and mount/unmount lifecycle. Matches Go's
disk_location.go and store.go. 11 unit tests covering volume CRUD,
multi-location balancing, and collection management.
Implements the needle map that maps NeedleId → (Offset, Size) using a
HashMap, matching Go's needle_map_memory.go. Includes NeedleMapMetric
with atomic counters for file/deletion tracking, IdxFileWriter trait
for append-only .idx writes, and load_from_idx for rebuilding from
index files. 5 unit tests covering put/get/delete/metrics/loading.
Index file (.idx) format: sequential 17-byte entries walked in 1024-row batches.
walk_index_file() and write_index_entry() match Go's idx.WalkIndexFile().
3 unit tests. Added crc32c dependency for Castagnoli CRC.
8-byte header at start of .dat files:
Version(1) + ReplicaPlacement(1) + TTL(2) + CompactionRevision(2) + ExtraSize(2)
Optional protobuf extra data follows.
ReplicaPlacement: byte = DC*100 + Rack*10 + SameRack.
5 unit tests for round-trip, extra data, and replication semantics.
Uses crc32c crate matching Go's crc32.MakeTable(crc32.Castagnoli).
Includes legacy Value() function for backward compat with pre-3.09 volumes.
4 unit tests.
Cargo.toml with all dependencies (tonic, axum, clap, prometheus, etc.),
build.rs for tonic-build proto compilation, and copies of
volume_server.proto, master.proto, remote.proto for gRPC codegen.
* Places the CommonResponse struct at the end of all IAM responses, rather than the start.
* iam: fix error response request id layout
* iam: add XML ordering regression test
* iam: share request id generation
---------
Co-authored-by: Aaron Segal <aaron.segal@rpsolutions.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Previously, evaluateIAMPolicies created a new PolicyEngine and re-parsed
the JSON policy document for every policy on every request. This adds a
shared iamPolicyEngine field that caches compiled policies, kept in sync
by PutPolicy, DeletePolicy, and bulk config reload paths.
- PutPolicy deletes the old cache entry before setting the new one, so a
parse failure on update does not leave a stale allow.
- Log warnings when policy compilation fails instead of silently
discarding errors.
- Add test for valid-to-invalid policy update regression.
The 'set -o pipefail' line was improperly indented outside the YAML block
scalar, causing a parse error when s3.enabled=true and s3.createBuckets
were populated. Moved the line to the beginning of the script block with
correct indentation (12 spaces).
Fixes#8520
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Chris Lu
Aaron
Chris Lu
SrikanthBhandary