seaweedfs/seaweed-volume/PARITY_PLAN.md

Rust Volume Server Parity Plan

Generated: 2026-03-16

Goal

Make seaweed-volume a drop-in replacement for the Go volume server by:

• comparing every Go volume-server code path against the Rust implementation,
• recording file-level ownership and verification status,
• closing verified behavior gaps one logic change per commit,
• extending tests so regressions are caught by Go parity suites and Rust unit/integration tests.

Ground Truth

Primary Go sources:

• weed/server/volume_server.go
• weed/server/volume_server_handlers*.go
• weed/server/volume_grpc_*.go
• weed/server/constants/volume.go
• weed/storage/store*.go
• weed/storage/disk_location*.go
• weed/storage/volume*.go
• weed/storage/needle/*.go
• weed/storage/idx/*.go
• weed/storage/needle_map*.go
• weed/storage/needle_map/*.go
• weed/storage/super_block/*.go
• weed/storage/erasure_coding/*.go

Supporting Go dependencies that affect drop-in behavior:

• weed/command/volume.go
• weed/security/*.go
• weed/images/*.go
• weed/stats/*.go

Primary Rust sources:

• seaweed-volume/src/main.rs
• seaweed-volume/src/config.rs
• seaweed-volume/src/security.rs
• seaweed-volume/src/images.rs
• seaweed-volume/src/server/*.rs
• seaweed-volume/src/storage/*.rs
• seaweed-volume/src/storage/needle/*.rs
• seaweed-volume/src/storage/idx/*.rs
• seaweed-volume/src/storage/erasure_coding/*.rs
• seaweed-volume/src/remote_storage/*.rs

Audit Method

For each Go file:

1. Map it to the Rust file or files that should own the same behavior.
2. Compare exported entry points, helper functions, state transitions, wire fields, and persistence side effects.
3. Mark each file implemented, partial, missing, or needs verification.
4. Link each behavior to an existing test or add a missing test.
5. Only treat a gap as closed after code review plus local verification.

Acceptance Criteria

The Rust server is a drop-in replacement only when all of these hold:

• HTTP routes, status codes, headers, and body semantics match Go.
• gRPC RPCs match Go request validation, response fields, streaming behavior, and maintenance/read-only semantics.
• Master heartbeat and topology metadata match Go closely enough that the Go master treats Rust and Go volume servers the same.
• On-disk volume behavior matches Go for normal volumes, EC shards, tiering metadata, and readonly persistence.
• Startup flags and operational endpoints that affect production deployment behave equivalently or are explicitly documented as unsupported.
• Existing Go integration suites pass with VOLUME_SERVER_IMPL=rust.

File Matrix

HTTP server surface

Go file	Rust counterpart	Status	Comparison focus
weed/server/volume_server.go	seaweed-volume/src/main.rs, seaweed-volume/src/server/volume_server.rs, seaweed-volume/src/server/heartbeat.rs	partial	startup wiring, routers, heartbeat, shutdown, metrics/debug listeners
weed/server/volume_server_handlers.go	seaweed-volume/src/server/volume_server.rs, seaweed-volume/src/server/handlers.rs	needs verification	method dispatch, OPTIONS behavior, public/admin split
weed/server/volume_server_handlers_admin.go	seaweed-volume/src/server/handlers.rs	implemented	/status, /healthz, stats, server headers
weed/server/volume_server_handlers_helper.go	seaweed-volume/src/server/handlers.rs	needs verification	JSON encoding, request parsing, helper parity
weed/server/volume_server_handlers_read.go	seaweed-volume/src/server/handlers.rs	needs verification	JWT, conditional reads, range reads, proxy/redirect, chunk manifests, image transforms
weed/server/volume_server_handlers_ui.go	seaweed-volume/src/server/handlers.rs, embedded assets	partial	UI payload and HTML parity
weed/server/volume_server_handlers_write.go	seaweed-volume/src/server/handlers.rs, seaweed-volume/src/images.rs	needs verification	multipart parsing, metadata, compression, ts, delete semantics
weed/server/constants/volume.go	seaweed-volume/src/server/heartbeat.rs, config defaults	needs verification	heartbeat timing, constants parity

gRPC server surface

Go file	Rust counterpart	Status	Comparison focus
weed/server/volume_grpc_admin.go	seaweed-volume/src/server/grpc_server.rs	needs verification	readonly/writable, allocate/delete/configure/mount/unmount
weed/server/volume_grpc_batch_delete.go	seaweed-volume/src/server/grpc_server.rs	implemented	batch delete, EC delete path
weed/server/volume_grpc_client_to_master.go	seaweed-volume/src/server/heartbeat.rs	partial	heartbeat fields, leader changes, metrics settings from master
weed/server/volume_grpc_copy.go	seaweed-volume/src/server/grpc_server.rs	needs verification	full copy streams
weed/server/volume_grpc_copy_incremental.go	seaweed-volume/src/server/grpc_server.rs	needs verification	incremental copy binary search, timestamps
weed/server/volume_grpc_erasure_coding.go	seaweed-volume/src/server/grpc_server.rs, seaweed-volume/src/storage/erasure_coding/*.rs	needs verification	shard read/write/delete/mount/unmount/rebuild
weed/server/volume_grpc_query.go	seaweed-volume/src/server/grpc_server.rs	needs verification	query validation and error parity
weed/server/volume_grpc_read_all.go	seaweed-volume/src/server/grpc_server.rs	needs verification	read-all ordering and tail semantics
weed/server/volume_grpc_read_write.go	seaweed-volume/src/server/grpc_server.rs, seaweed-volume/src/storage/*.rs	needs verification	blob/meta/page reads, write blob semantics
weed/server/volume_grpc_remote.go	seaweed-volume/src/server/grpc_server.rs, seaweed-volume/src/remote_storage/*.rs	needs verification	remote fetch/write and tier metadata
weed/server/volume_grpc_scrub.go	seaweed-volume/src/server/grpc_server.rs, seaweed-volume/src/storage/*.rs	needs verification	scrub result semantics
weed/server/volume_grpc_state.go	seaweed-volume/src/server/grpc_server.rs	implemented	GetState/SetState/Status
weed/server/volume_grpc_tail.go	seaweed-volume/src/server/grpc_server.rs	needs verification	tail streaming and idle timeout
weed/server/volume_grpc_tier_download.go	seaweed-volume/src/server/grpc_server.rs, seaweed-volume/src/remote_storage/*.rs	needs verification	tier download stream/error paths
weed/server/volume_grpc_tier_upload.go	seaweed-volume/src/server/grpc_server.rs, seaweed-volume/src/remote_storage/*.rs	needs verification	tier upload stream/error paths
weed/server/volume_grpc_vacuum.go	seaweed-volume/src/server/grpc_server.rs, seaweed-volume/src/storage/*.rs	needs verification	compact/commit/cleanup progress and readonly transitions

Storage and persistence surface

Go file group	Rust counterpart	Status	Comparison focus
weed/storage/store.go, store_state.go	seaweed-volume/src/storage/store.rs, seaweed-volume/src/server/heartbeat.rs	partial	topology metadata, disk tags, server id, state persistence
weed/storage/store_vacuum.go	seaweed-volume/src/storage/store.rs, seaweed-volume/src/storage/volume.rs	needs verification	vacuum sequencing
weed/storage/store_ec.go, store_ec_delete.go, store_ec_scrub.go	seaweed-volume/src/storage/store.rs, seaweed-volume/src/storage/erasure_coding/*.rs	needs verification	EC lifecycle and scrub behavior
weed/storage/disk_location.go, disk_location_ec.go	seaweed-volume/src/storage/disk_location.rs, seaweed-volume/src/storage/store.rs	partial	directory UUIDs, tags, load rules, disk space checks
weed/storage/volume.go, volume_loading.go	seaweed-volume/src/storage/volume.rs	needs verification	load/reload/readonly/remote metadata
weed/storage/volume_super_block.go	seaweed-volume/src/storage/super_block.rs, seaweed-volume/src/storage/volume.rs	implemented	super block parity
weed/storage/volume_read.go, volume_read_all.go	seaweed-volume/src/storage/volume.rs, seaweed-volume/src/server/handlers.rs	needs verification	full/meta/page reads, TTL, streaming
weed/storage/volume_write.go	seaweed-volume/src/storage/volume.rs, seaweed-volume/src/server/write_queue.rs	needs verification	dedup, sync/async writes, metadata flags
weed/storage/volume_vacuum.go	seaweed-volume/src/storage/volume.rs	needs verification	compact and commit parity
weed/storage/volume_backup.go	seaweed-volume/src/storage/volume.rs, seaweed-volume/src/server/grpc_server.rs	needs verification	backup/search logic
weed/storage/volume_checking.go	seaweed-volume/src/storage/volume.rs, seaweed-volume/src/storage/idx/mod.rs, seaweed-volume/src/server/grpc_server.rs	needs verification	scrub and integrity checks
weed/storage/volume_info.go, volume_info/volume_info.go, volume_tier.go	seaweed-volume/src/storage/volume.rs, seaweed-volume/src/remote_storage/*.rs	needs verification	.vif format and tiered file metadata
weed/storage/needle/*.go	seaweed-volume/src/storage/needle/*.rs	needs verification	needle parsing, CRC, TTL, multipart metadata
weed/storage/idx/*.go	seaweed-volume/src/storage/idx/*.rs	needs verification	index walking and binary search
weed/storage/needle_map*.go, needle_map/*.go	seaweed-volume/src/storage/needle_map.rs	needs verification	map kind parity, persistence, memory behavior
weed/storage/super_block/*.go	seaweed-volume/src/storage/super_block.rs	implemented	replica placement and TTL metadata
weed/storage/erasure_coding/*.go	seaweed-volume/src/storage/erasure_coding/*.rs	needs verification	EC shard placement, encode/decode, journal deletes

Supporting runtime surface

Go file	Rust counterpart	Status	Comparison focus
weed/command/volume.go	seaweed-volume/src/config.rs, seaweed-volume/src/main.rs	partial	flags, metrics/debug listeners, startup behavior
weed/security/*.go	seaweed-volume/src/security.rs, seaweed-volume/src/main.rs	implemented	JWT and TLS loading
weed/images/*.go	seaweed-volume/src/images.rs, seaweed-volume/src/server/handlers.rs	implemented	JPEG orientation and transforms
weed/stats/*.go	seaweed-volume/src/metrics.rs, seaweed-volume/src/server/handlers.rs	partial	metrics endpoints, push-gateway integration

Verified Gaps As Of 2026-03-08

The startup/runtime gaps that were verified in the initial audit are now closed:

1. Heartbeat metadata parity Closed by 8ade1c51d and retained in current HEAD.

2. Dedicated metrics/debug listener parity Closed by fbe0e5829.

3. Master-provided metrics push settings Closed by fbe0e5829.

4. Slow-read tuning parity Closed by 66e3900dc.

There are no remaining verified gaps from the initial startup/runtime audit. The broader line-by-line comparison batches below are still required to either confirm parity or surface new gaps.

Execution Status As Of 2026-03-16

The file-by-file comparison and verification work executed in this round was:

1. Startup and harness alignment Compared weed/command/volume.go, test/volume_server/framework/cluster*.go, seaweed-volume/src/config.rs, and seaweed-volume/src/main.rs to ensure the Rust server is invoked with Go-compatible flags and is rebuilt from the current source during parity runs.

2. HTTP admin surface Compared weed/server/volume_server_handlers_admin.go against seaweed-volume/src/server/handlers.rs with emphasis on /status payload shape, disk-status fields, and volume ordering.

3. gRPC admin surface Compared weed/server/volume_grpc_admin.go against seaweed-volume/src/server/grpc_server.rs with emphasis on Ping, VolumeConfigure, readonly/writable flows, and error wrapping.

4. Storage/index layout Compared Go index-entry defaults in weed/storage/types and weed/storage/idx/*.go against the Rust default feature set in seaweed-volume/Cargo.toml and the Rust index reader/writer paths to confirm default binaries use the same offset width.

5. End-to-end parity verification Re-ran the Go HTTP and gRPC integration suites with VOLUME_SERVER_IMPL=rust after each fix to confirm wire-level compatibility.

Verified mismatches closed in this round

• Rust parity runs could reuse a stale weed-volume binary across test invocations, hiding source and feature changes from the Go harness.
• Rust defaulted to 5-byte index offsets, while the default Go go build path uses 4-byte offsets unless built with -tags 5BytesOffset.
• Rust /status omitted Go fields in both Volumes and DiskStatuses, and did not sort volumes by Id.
• Rust Ping treated an empty target as a self-ping and only performed a raw gRPC connect for filer targets; Go returns remote_time_ns=0 for the empty request and performs a real filer Ping RPC.
• Rust VolumeNeedleStatus dropped stored TTL metadata and reported data_size instead of Go’s Size field.
• Rust multipart uploads ignored form fields such as ts, ttl, and cm, and also ignored part-level Content-Encoding and Content-MD5.
• Rust only treated dl=true and dl=1 as truthy, while Go accepts the full strconv.ParseBool set such as dl=t and dl=True.

Verification commands

• VOLUME_SERVER_IMPL=rust go test -count=1 -timeout 1200s ./test/volume_server/http/...
• VOLUME_SERVER_IMPL=rust go test -count=1 -timeout 1200s ./test/volume_server/grpc/...

Execution Plan

Batch 1: startup and heartbeat

• Compare weed/command/volume.go, weed/server/volume_server.go, weed/server/volume_grpc_client_to_master.go, weed/storage/store.go, and weed/storage/disk_location.go.
• Close metadata and startup parity gaps that affect master registration and deployment compatibility.
• Add Rust unit tests for heartbeat payloads and config wiring.

Batch 2: HTTP read path

• Compare volume_server_handlers_read.go, volume_server_handlers_helper.go, and related storage read functions line by line.
• Verify JWT, path parsing, proxy/redirect, ranges, streaming, chunk manifests, image transforms, and response-header overrides.
• Extend test/volume_server/http/... and Rust handler tests where parity is not covered.

Batch 3: HTTP write/delete path

• Compare volume_server_handlers_write.go and write-related storage functions.
• Verify multipart behavior, metadata, md5, compression, unchanged writes, delete edge cases, and timestamp handling.

Batch 4: gRPC admin and lifecycle

• Compare volume_grpc_admin.go, volume_grpc_state.go, and volume_grpc_vacuum.go.
• Verify readonly/writable flows, maintenance mode, status payloads, mount/unmount/delete/configure, and vacuum transitions.

Batch 5: gRPC data movement

• Compare volume_grpc_read_write.go, copy*.go, read_all.go, tail.go, remote.go, and query.go.
• Verify stream framing, binary search, idle timeout, and remote-storage semantics.

Batch 6: storage internals

• Compare all weed/storage volume, needle, idx, needle map, and EC files line by line.
• Focus on persistence rules, readonly semantics, TTL, recovery/scrub, backup, and memory/disk map behavior.

Commit Strategy

• One commit for the audit/plan document if the document itself changes.
• One commit per logic fix.
• Every logic commit must include the smallest test addition that proves the new parity claim.