During streaming reads, checks if the volume's compaction revision changed
between chunks and re-looks up the needle offset from the needle map,
matching Go's readNeedleDataInto behavior. Also threads ReadOption through
the volume read path.
Adds ReadOption with fields for meta-only reads, volume revision tracking,
slow read detection, and out-of-range flagging. Threaded through volume
read paths for future behavioral parity.
Parse CA certificate path from [https.volume] and [grpc.volume] sections
in security.toml. When configured, enables client certificate verification
using WebPkiClientVerifier for HTTP and client_ca_root for gRPC.
The previous implementation (set_len/set_len(0)) was a no-op.
Now uses fallocate(FALLOC_FL_KEEP_SIZE) on Linux to actually reserve
disk blocks without changing the visible file size.
Matches Go's architecture where EC volumes are managed per disk location,
enabling correct per-location max volume count calculation and proper
distribution of EC shards across disks.
When -max=0, dynamically calculate max volume count based on free disk
space, existing volumes, and EC shard count — matching Go's
MaybeAdjustVolumeMax(). Recalculate on each heartbeat tick and when
volume_size_limit changes from master.
Go's flag package uses single dash (-port) while clap uses double dash
(--port). Add normalize_args_vec() to convert single-dash long options
to double-dash before clap parsing, so both formats work. Update test
framework to use single-dash flags matching Go convention.
Port of Go's CompactMap: segmented sorted arrays with compressed keys.
NeedleId is split into chunk (u64) and compact key (u16), reducing
per-entry memory from ~40-48 bytes (HashMap) to ~10 bytes.
For 1M needles: ~10 MB instead of ~40-48 MB.
Three improvements to RedbNeedleMap:
- Use Durability::None on all write transactions since .idx is the
crash recovery source and redb is always rebuildable from it
- Delete stale .rdb before rebuild in load_from_idx to prevent
leftover entries surviving a crash
- Reuse existing .rdb on clean restart by storing .idx file size in
a metadata table; incrementally replay only new .idx entries when
the .idx has grown since last build
Add .rdb (redb index) cleanup to removeVolumeFiles and vacuum commit
in Go code, for compatibility with mixed Rust/Go volume server
deployments. Route .rdb through dirIdx in FileName() like .idx/.ldb.
Closes the metrics gap between Rust and Go volume servers (8 → 23
metrics). Adds handler counters, vacuuming histograms, volume/disk
gauges, inflight request tracking, and concurrent limit gauges.
Centralizes request counting in store handlers instead of per-handler.
- Check for .note file (interrupted VolumeCopy) and remove partial volumes
- Validate EC shards before skipping .dat loading: check shard count,
uniform size, and expected size from .dat file
- Remove stale .cpd/.cpx compaction temp files on startup
- Skip already-loaded volumes on repeated load_existing_volumes calls
- Fix healthz test: set is_heartbeating=true in test state
- Range requests: clamp end to file size instead of returning 416 (matches Go)
- Suffix ranges larger than file size: clamp instead of skipping (matches Go)
- Set ETag response header on 201 Created write responses (matches Go SetEtag)
- Always decompress before image resize/crop even when client accepts gzip
- VolumeTierMoveDatFromRemote: don't check maintenance mode (Go doesn't)
- Fix stale integration test: stats routes are now exposed on admin router
The `/stats/disk`, `/stats/counter`, and `/stats/memory` endpoints were
implemented in `handlers.rs` but were missing from the HTTP router
registration in `volume_server.rs`. Registered them under the UI-enabled
group to match Go implementation.
- VolumeConfigure: don't fail on unmount of non-existent volume (Go returns nil)
- 304 responses include ETag/Last-Modified headers per HTTP spec
- Conditional header checks run before chunk manifest expansion
- EC encoder uses two-phase approach: 1GB large blocks then 1MB small blocks
- Compaction uses volume-level TTL (not per-needle TTL) for filtering
- VolumeConfigure does unmount/modify-disk/remount cycle matching Go
- VolumeMarkReadonly persists flag to .vif when persist=true
- AllocateVolume accepts version parameter
- Multipart boundary uses leading CRLF per RFC 2046
- MIME type override skipped for chunk manifests
- RedbNeedleMap: pure-Rust disk-backed needle map using redb, with NeedleMap
enum wrapping both in-memory and redb variants
- binary_search_by_append_at_ns: port of Go's BinarySearchByAppendAtNs for
VolumeIncrementalCopy with since_ns > 0
- Proxy/redirect: master volume lookup, HTTP proxy forwarding with ?proxied=true,
and 301 redirects for non-local volumes based on ReadMode config
- Wire new VolumeServerState fields: read_mode, master_url, self_url, http_client
- Extract TTL from ?ttl= query param and set on needle (matches Go's ParseUpload)
- Auto-compress compressible file types (.js, .css, .json, .svg, text/*, etc.)
using gzip, only when compression saves >10% (matches Go's IsCompressableFileType)
- Extract Seaweed-* headers as custom metadata pairs stored as JSON in needle
- Store filename from URL path in needle name field
- Include filename in upload response JSON
- Add unit tests for is_compressible_file_type and try_gzip_data
- BatchDelete now supports EC volumes: looks up needle in .ecx index,
journals deletion to .ecj file (local-only, Go handles distributed part)
- JPEG EXIF orientation auto-fix on upload using kamadak-exif + image crate,
matching Go's FixJpgOrientation behavior (8 orientation transforms)
- Async batched write processing via mpsc queue (up to 128 entries per batch),
groups writes by volume ID and syncs once per volume per batch
- VolumeTierMoveDatToRemote: multipart upload .dat file to S3 with progress
streaming, updates .vif with remote file reference
- VolumeTierMoveDatFromRemote: downloads .dat from S3 with progress streaming,
removes remote file reference from .vif
- S3TierRegistry for managing named remote storage backends
- VolumeInfo (.vif) JSON persistence matching Go's protojson format
- 124 lib + 7 integration = 131 Rust tests pass
- All 109 Go integration tests pass (53 HTTP + 56 gRPC)
Chris Lu