Closes the metrics gap between Rust and Go volume servers (8 → 23
metrics). Adds handler counters, vacuuming histograms, volume/disk
gauges, inflight request tracking, and concurrent limit gauges.
Centralizes request counting in store handlers instead of per-handler.
- Check for .note file (interrupted VolumeCopy) and remove partial volumes
- Validate EC shards before skipping .dat loading: check shard count,
uniform size, and expected size from .dat file
- Remove stale .cpd/.cpx compaction temp files on startup
- Skip already-loaded volumes on repeated load_existing_volumes calls
- Fix healthz test: set is_heartbeating=true in test state
- Range requests: clamp end to file size instead of returning 416 (matches Go)
- Suffix ranges larger than file size: clamp instead of skipping (matches Go)
- Set ETag response header on 201 Created write responses (matches Go SetEtag)
- Always decompress before image resize/crop even when client accepts gzip
- VolumeTierMoveDatFromRemote: don't check maintenance mode (Go doesn't)
- Fix stale integration test: stats routes are now exposed on admin router
The `/stats/disk`, `/stats/counter`, and `/stats/memory` endpoints were
implemented in `handlers.rs` but were missing from the HTTP router
registration in `volume_server.rs`. Registered them under the UI-enabled
group to match Go implementation.
- VolumeConfigure: don't fail on unmount of non-existent volume (Go returns nil)
- 304 responses include ETag/Last-Modified headers per HTTP spec
- Conditional header checks run before chunk manifest expansion
- EC encoder uses two-phase approach: 1GB large blocks then 1MB small blocks
- Compaction uses volume-level TTL (not per-needle TTL) for filtering
- VolumeConfigure does unmount/modify-disk/remount cycle matching Go
- VolumeMarkReadonly persists flag to .vif when persist=true
- AllocateVolume accepts version parameter
- Multipart boundary uses leading CRLF per RFC 2046
- MIME type override skipped for chunk manifests
- RedbNeedleMap: pure-Rust disk-backed needle map using redb, with NeedleMap
enum wrapping both in-memory and redb variants
- binary_search_by_append_at_ns: port of Go's BinarySearchByAppendAtNs for
VolumeIncrementalCopy with since_ns > 0
- Proxy/redirect: master volume lookup, HTTP proxy forwarding with ?proxied=true,
and 301 redirects for non-local volumes based on ReadMode config
- Wire new VolumeServerState fields: read_mode, master_url, self_url, http_client
- Extract TTL from ?ttl= query param and set on needle (matches Go's ParseUpload)
- Auto-compress compressible file types (.js, .css, .json, .svg, text/*, etc.)
using gzip, only when compression saves >10% (matches Go's IsCompressableFileType)
- Extract Seaweed-* headers as custom metadata pairs stored as JSON in needle
- Store filename from URL path in needle name field
- Include filename in upload response JSON
- Add unit tests for is_compressible_file_type and try_gzip_data
- BatchDelete now supports EC volumes: looks up needle in .ecx index,
journals deletion to .ecj file (local-only, Go handles distributed part)
- JPEG EXIF orientation auto-fix on upload using kamadak-exif + image crate,
matching Go's FixJpgOrientation behavior (8 orientation transforms)
- Async batched write processing via mpsc queue (up to 128 entries per batch),
groups writes by volume ID and syncs once per volume per batch
- VolumeTierMoveDatToRemote: multipart upload .dat file to S3 with progress
streaming, updates .vif with remote file reference
- VolumeTierMoveDatFromRemote: downloads .dat from S3 with progress streaming,
removes remote file reference from .vif
- S3TierRegistry for managing named remote storage backends
- VolumeInfo (.vif) JSON persistence matching Go's protojson format
- 124 lib + 7 integration = 131 Rust tests pass
- All 109 Go integration tests pass (53 HTTP + 56 gRPC)
- Add StreamingBody (http_body::Body) for chunked reads of files >1MB,
avoiding OOM by reading 64KB at a time via spawn_blocking + pread
- Add NeedleStreamInfo and meta-only read path to avoid loading full
needle body when streaming
- Add RustMultiVolumeCluster test framework and MultiCluster interface
so TestVolumeMoveHandlesInFlightWrites works with Rust volume servers
- Remove TestVolumeMoveHandlesInFlightWrites from CI skip list (now passes)
- All 117 Go integration tests + 8 Rust integration tests pass (100%)
- Add remote_storage module with RemoteStorageClient trait
- Implement S3RemoteStorageClient using aws-sdk-s3 (covers all S3-compatible
providers: AWS, Wasabi, Backblaze, Aliyun, etc.)
- FetchAndWriteNeedle now fetches data from S3, writes locally as needle,
and replicates to peers
- Add 3 integration tests using weed mini as S3 backend:
- Full round-trip fetch from S3
- Byte-range (partial) read from S3
- Error handling for non-existent S3 objects
- VolumeServerStatus now returns real data_center and rack from config
- DiskStatus uses sysinfo crate for actual disk total/free/used/percent
- ReadVolumeFileStatus returns dat file modification timestamps
- FetchAndWriteNeedle produces Go-matching error messages for unknown remote storage types
Remove unused imports, prefix unused variables, add #[allow(dead_code)]
for fields/methods used only in specific contexts (serde deserialization,
non-unix builds, future error tracking).
VolumeCopy: connects to source as gRPC client, copies .dat/.idx/.vif
files via CopyFile streaming, verifies sizes, mounts the volume.
Needed for volume rebalancing and migration between servers.
VolumeTailReceiver: connects to source's VolumeTailSender, receives
needle header+body chunks, reassembles multi-chunk needles, writes
them locally. Needed for live replication during volume moves.
Also adds helper functions: parse_grpc_address (SeaweedFS address
format parsing), copy_file_from_source (streaming file copy with
progress reporting), find_last_append_at_ns (timestamp extraction
from copied files).
All 128/130 integration tests still pass (same 2 known unfixable).
Adds full volume compaction support:
- compact_by_index: iterates live needles, writes to new .cpd/.cpx
files, skipping deleted and TTL-expired entries
- commit_compact: swaps .cpd->.dat and .cpx->.idx, reloads volume
- cleanup_compact: removes leftover .cpd/.cpx files
Wires VacuumVolumeCompact (streaming progress), VacuumVolumeCommit,
and VacuumVolumeCleanup gRPC RPCs. Also adds save_to_idx and
ascending_visit methods to CompactNeedleMap.
Includes unit test verifying compact removes deleted needles and
preserves live data through the full compact/commit cycle.
The bidirectional gRPC stream had a deadlock: send_heartbeat().await
waits for response headers from the server, but the Go master won't
send response headers until it receives the first heartbeat message.
Pre-populating the channel before creating the stream resolves this.
Tested end-to-end: Rust volume server successfully registers with Go
master, receives volume assignments, and handles uploads/downloads.
Chris Lu