* fix(worker): pass compaction revision and file sizes in EC volume copy
The worker EC task was sending CopyFile requests without the current
compaction revision (defaulting to 0) and with StopOffset set to
math.MaxInt64. After a vacuum compaction this caused the volume server
to reject the copy or return stale data.
Read the volume file status first and forward the compaction revision
and actual file sizes so the copy is consistent with the compacted
volume.
* propagate erasure coding task context
* fix(worker): validate volume file status and detect short copies
Reject zero dat file size from ReadVolumeFileStatus — a zero-sized
snapshot would produce 0-byte copies and broken EC shards.
After streaming, verify totalBytes matches the expected stopOffset
and return an error on short copies instead of logging success.
* fix(worker): reject zero idx file size in volume status validation
A non-empty dat with zero idx indicates an empty or corrupt volume.
Without this guard, copyFileFromSource gets stopOffset=0, produces a
0-byte .idx, passes the short-copy check, and generateEcShardsLocally
runs against a volume with no index.
* fix fake plugin volume file status
* fix plugin volume balance test fixtures
The upstream rust:alpine manifest list no longer includes linux/386,
breaking multi-platform builds. Switch the Rust volume server builder
stage to alpine:3.23 and install Rust toolchain via apk instead.
Also adds openssl-dev which is needed for the build.
* fix(filer): apply default disk type after location-prefix resolution in gRPC AssignVolume
The gRPC AssignVolume path was applying the filer's default DiskType to
the request before calling detectStorageOption. This caused the default
to shadow any disk type configured via a filer location-prefix rule,
diverging from the HTTP write path which applies the default only when
no rule matches.
Extract resolveAssignStorageOption to apply the filer default disk type
after detectStorageOption, so location-prefix rules take precedence.
* fix(filer): apply default disk type after location-prefix resolution in TUS upload path
Same class of bug as the gRPC AssignVolume fix: the TUS tusWriteData
handler called detectStorageOption0 but never applied the filer's
default DiskType when no location-prefix rule matched. This made TUS
uploads ignore the -disk flag entirely.
* fix(s3): preserve explicit directory markers during empty folder cleanup
PR #8292 switched empty-folder cleanup from per-folder implicit checks
to bucket-level policy, inadvertently dropping the check that preserved
explicitly created directories (e.g., PUT /bucket/folder/). This caused
user-created folders to be deleted when their last file was removed.
Add IsDirectoryKeyObject check in executeCleanup to skip folders that
have a MIME type set, matching the canonical pattern used throughout the
S3 listing and delete handlers.
* fix: handle ErrNotFound in IsDirectoryKeyObject for race safety
Entry may be deleted between the emptiness check and the directory
marker lookup. Treat not-found as false rather than propagating
the error, avoiding unnecessary error logging in the cleanup path.
* refactor: consolidate directory marker tests and tidy error handling
- Combine two separate test functions into a table-driven test
- Nest ErrNotFound check inside the err != nil block
* notification.kafka: add SASL authentication and TLS support (#8827)
Wire sarama SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512) and TLS
configuration into the Kafka notification producer and consumer,
enabling connections to secured Kafka clusters.
* notification.kafka: validate mTLS config
* kafka notification: validate partial mTLS config, replace panics with errors
- Reject when only one of tls_client_cert/tls_client_key is provided
- Replace three panic() calls in KafkaInput.initialize with returned errors
* kafka notification: enforce minimum TLS 1.2 for Kafka connections
* mount: add option to show system entries
* address gemini code review's suggested changes
* rename flag from -showSystemEntries to -includeSystemEntries
* meta_cache: purge hidden system entries on filer events
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* plugin scheduler: run iceberg and lifecycle lanes concurrently
The default lane serialises job types under a single admin lock
because volume-management operations share global state. Iceberg
and lifecycle lanes have no such constraint, so run each of their
job types independently in separate goroutines.
* Fix concurrent lane scheduler status
* plugin scheduler: address review feedback
- Extract collectDueJobTypes helper to deduplicate policy loading
between locked and concurrent iteration paths.
- Use atomic.Bool instead of sync.Mutex for hadJobs in the concurrent
path.
- Set lane loop state to "busy" before launching concurrent goroutines
so the lane is not reported as idle while work runs.
- Convert TestLaneRequiresLock to table-driven style.
- Add TestRunLaneSchedulerIterationLockBehavior to verify the scheduler
acquires the admin lock only for lanes that require it.
- Fix flaky TestGetLaneSchedulerStatusShowsActiveConcurrentLaneWork by
not starting background scheduler goroutines that race with the
direct runJobTypeIteration call.
* s3api: skip TTL fast-path for versioned buckets (#8757)
PutBucketLifecycleConfiguration was translating Expiration.Days into
filer.conf TTL entries for all buckets. For versioned buckets this is
wrong:
1. TTL volumes expire as a unit, destroying all data — including
noncurrent versions that should be preserved.
2. Filer-backend TTL (RocksDB compaction filter, Redis key expiry)
removes entries without triggering chunk deletion, leaving orphaned
volume data with 0 deleted bytes.
3. On AWS S3, Expiration.Days on a versioned bucket creates a delete
marker — it does not hard-delete data. TTL has no such nuance.
Fix: skip the TTL fast-path when the bucket has versioning enabled or
suspended. All lifecycle rules are evaluated at scan time by the
lifecycle worker instead.
Also fix the lifecycle worker to evaluate Expiration rules against the
latest version in .versions/ directories, which was previously skipped
entirely — only NoncurrentVersionExpiration was handled.
* lifecycle worker: handle SeaweedList error in versions dir cleanup
Do not assume the directory is empty when the list call fails — log
the error and skip the directory to avoid incorrect deletion.
* address review feedback
- Fetch version file for tag-based rules instead of reading tags from
the .versions directory entry where they are not cached.
- Handle getBucketVersioningStatus error by failing closed (treat as
versioned) to avoid creating TTL entries on transient failures.
- Capture and assert deleteExpiredObjects return values in test.
- Improve test documentation.
* ci: add Trivy CVE scan to container release workflow
* ci: pin trivy-action version and fail on HIGH/CRITICAL CVEs
Address review feedback:
- Pin aquasecurity/trivy-action to v0.28.0 instead of @master
- Add exit-code: '1' so the scan fails the job on findings
- Add comment explaining why only amd64 is scanned
* ci: pin trivy-action to SHA for v0.35.0
Tags ≤0.34.2 were compromised (GHSA-69fq-xp46-6x23). Pin to the full
commit SHA of v0.35.0 to avoid mutable tag risks.
* s3api: accept NoncurrentVersionExpiration, AbortIncompleteMultipartUpload, Expiration.Date
Update PutBucketLifecycleConfigurationHandler to accept all newly-supported
lifecycle rule types. Only Transition and NoncurrentVersionTransition are
still rejected (require storage class tier infrastructure).
Changes:
- Remove ErrNotImplemented for Expiration.Date (handled by worker at scan time)
- Only reject rules with Transition.set or NoncurrentVersionTransition.set
- Extract prefix from Filter.And when present
- Add comment explaining that non-Expiration.Days rules are evaluated by
the lifecycle worker from stored lifecycle XML, not via filer.conf TTL
The lifecycle XML is already stored verbatim in bucket metadata, so new
rule types are preserved on Get even without explicit handler support.
Filer.conf TTL entries are only created for Expiration.Days (fast path).
* s3api: skip TTL fast path for rules with tag or size filters
Rules with tag or size constraints (Filter.Tag, Filter.And with tags
or size bounds, Filter.ObjectSizeGreaterThan/LessThan) must not be
lowered to filer.conf TTL entries, because TTL applies unconditionally
to all objects under the prefix. These rules are evaluated at scan
time by the lifecycle worker which checks each object's tags and size.
Only simple Expiration.Days rules with prefix-only filters use the
TTL fast path (RocksDB compaction filter).
---------
Co-authored-by: Copilot <copilot@github.com>
* lifecycle worker: drive MPU abort from lifecycle rules
Update the multipart upload abort phase to read
AbortIncompleteMultipartUpload.DaysAfterInitiation from the parsed
lifecycle rules. Falls back to the worker config abort_mpu_days when
no lifecycle XML rule specifies the value.
This means per-bucket MPU abort thresholds are now respected when
set via PutBucketLifecycleConfiguration, instead of using a single
global worker config value for all buckets.
* lifecycle worker: only use config AbortMPUDays when no lifecycle XML exists
When a bucket has lifecycle XML (useRuleEval=true) but no
AbortIncompleteMultipartUpload rule, mpuAbortDays should be 0
(no abort), not the worker config default. The config fallback
should only apply to buckets without lifecycle XML.
* lifecycle worker: only skip .uploads at bucket root
* lifecycle worker: use per-upload rule evaluation for MPU abort
Replace the single bucket-wide mpuAbortDays with per-upload evaluation
using s3lifecycle.EvaluateMPUAbort, which respects each rule's prefix
filter and DaysAfterInitiation threshold.
Previously the code took the first enabled abort rule's days value
and applied it to all uploads, ignoring prefix scoping and multiple
rules with different thresholds.
Config fallback (abort_mpu_days) now only applies when lifecycle XML
is truly absent (xmlPresent=false), not when XML exists but has no
abort rules.
Also fix EvaluateMPUAbort to use expectedExpiryTime for midnight-UTC
semantics matching other lifecycle cutoffs.
---------
Co-authored-by: Copilot <copilot@github.com>
* lifecycle worker: add NoncurrentVersionExpiration support
Add version-aware scanning to the rule-based execution path. When the
walker encounters a .versions directory, processVersionsDirectory():
- Lists all version entries (v_<versionId>)
- Sorts by version timestamp (newest first)
- Walks non-current versions with ShouldExpireNoncurrentVersion()
which handles both NoncurrentDays and NewerNoncurrentVersions
- Extracts successor time from version IDs (both old/new format)
- Skips delete markers in noncurrent version counting
- Falls back to entry Mtime when version ID timestamp is unavailable
Helper functions:
- sortVersionsByTimestamp: insertion sort by version ID timestamp
- getEntryVersionTimestamp: extracts timestamp with Mtime fallback
* lifecycle worker: address review feedback for noncurrent versions
- Use sentinel errLimitReached in versions directory handler
- Set NoncurrentIndex on ObjectInfo for proper NewerNoncurrentVersions
evaluation
* lifecycle worker: fail closed on XML parse error, guard zero Mtime
- Fail closed when lifecycle XML exists but fails to parse, instead
of falling back to TTL which could apply broader rules
- Guard Mtime > 0 before using time.Unix(mtime, 0) to avoid mapping
unset Mtime to 1970, which would misorder versions and cause
premature expiration
* lifecycle worker: count delete markers toward NoncurrentIndex
Noncurrent delete markers should count toward the
NewerNoncurrentVersions retention threshold so data versions
get the correct position index. Previously, skipping delete
markers without incrementing the index could retain too many
versions after delete/recreate cycles.
* lifecycle worker: fix version ordering, error propagation, and fail-closed scope
1. Use full version ID comparison (CompareVersionIds) for sorting
.versions entries, not just decoded timestamps. Two versions with
the same timestamp prefix but different random suffixes were
previously misordered, potentially treating the newest version as
noncurrent and deleting it.
2. Propagate .versions listing failures to the caller instead of
swallowing them with (nil, 0). Transient filer errors on a
.versions directory now surface in the job result.
3. Narrow the fail-closed path to only malformed lifecycle XML
(errMalformedLifecycleXML). Transient filer LookupEntry errors
now fall back to TTL with a warning, matching the original intent
of "fail closed on bad config, not on network blips."
* lifecycle worker: only skip .uploads at bucket root
* lifecycle worker: sort.Slice, mixed-format test, XML presence tracking
- Replace manual insertion sort with sort.Slice in sortVersionsByVersionId
- Add TestCompareVersionIdsMixedFormats covering old/new format ordering
- Distinguish "no lifecycle XML" (nil) from "XML present but no effective
rules" (non-nil empty slice) so buckets with all-disabled rules don't
incorrectly fall back to filer.conf TTL expiration
* lifecycle worker: guard nil Attributes, use TrimSuffix in test
- Guard entry.Attributes != nil before accessing GetFileSize() and
Mtime in both listExpiredObjectsByRules and processVersionsDirectory
- Use strings.TrimPrefix/TrimSuffix in TestVersionsDirectoryNaming
to match the production code pattern
* lifecycle worker: skip TTL scan when XML present, fix test assertions
- When lifecycle XML is present but has no effective rules, skip
object scanning entirely instead of falling back to TTL path
- Test sort output against concrete expected names instead of
re-using the same comparator as the sort itself
* lifecycle worker: fix ExpiredObjectDeleteMarker to match AWS semantics
Rewrite cleanupDeleteMarkers() to only remove delete markers that are
the sole remaining version of an object. Previously, delete markers
were removed unconditionally which could resurface older versions in
versioned buckets.
New algorithm:
1. Walk bucket tree looking for .versions directories
2. Check ExtLatestVersionIsDeleteMarker from directory metadata
3. Count versions in the .versions directory
4. Only remove if count == 1 (delete marker is sole version)
5. Require an ExpiredObjectDeleteMarker=true rule (when lifecycle
XML rules are present)
6. Remove the empty .versions directory after cleanup
This phase runs after NoncurrentVersionExpiration so version counts
are accurate.
* lifecycle worker: respect prefix filter in ExpiredObjectDeleteMarker rules
Previously hasDeleteMarkerRule was a bucket-wide boolean that ignored
rule prefixes. A prefix-scoped rule like "logs/" would incorrectly
clean up delete markers in all paths.
Add matchesDeleteMarkerRule() that checks if a matching enabled
ExpiredObjectDeleteMarker rule exists for the specific object key,
respecting the rule's prefix filter. Falls back to legacy behavior
(allow cleanup) when no lifecycle XML rules are provided.
* lifecycle worker: only skip .uploads at bucket root
Check dir == bucketPath before skipping directories named .uploads.
Previously a user-created directory like data/.uploads/ at any depth
would be incorrectly skipped during lifecycle scanning.
* lifecycle worker: fix delete marker cleanup with XML-present empty rules
1. matchesDeleteMarkerRule now uses nil check (not len==0) for legacy
fallback. A non-nil empty slice means lifecycle XML was present but
had no ExpiredObjectDeleteMarker rules, so cleanup is blocked.
Previously, an empty slice triggered the legacy true path.
2. Use per-directory removedHere flag instead of cumulative cleaned
counter when deciding to remove .versions directories. Previously,
after the first successful cleanup anywhere in the bucket, every
subsequent .versions directory would be removed even if its own
delete marker was not actually deleted.
* lifecycle worker: use full filter matching for delete marker rules
matchesDeleteMarkerRule now uses s3lifecycle.MatchesFilter (exported)
instead of prefix-only matching. This ensures tag and size filters
on ExpiredObjectDeleteMarker rules are respected, preventing broader
deletions than the configured policy intends.
Add TestMatchesDeleteMarkerRule covering: nil rules (legacy), empty
rules (XML present), prefix match/mismatch, disabled rules, rules
without the flag, and tag-filtered rules against tagless markers.
---------
Co-authored-by: Copilot <copilot@github.com>
* lifecycle worker: add NoncurrentVersionExpiration support
Add version-aware scanning to the rule-based execution path. When the
walker encounters a .versions directory, processVersionsDirectory():
- Lists all version entries (v_<versionId>)
- Sorts by version timestamp (newest first)
- Walks non-current versions with ShouldExpireNoncurrentVersion()
which handles both NoncurrentDays and NewerNoncurrentVersions
- Extracts successor time from version IDs (both old/new format)
- Skips delete markers in noncurrent version counting
- Falls back to entry Mtime when version ID timestamp is unavailable
Helper functions:
- sortVersionsByTimestamp: insertion sort by version ID timestamp
- getEntryVersionTimestamp: extracts timestamp with Mtime fallback
* lifecycle worker: address review feedback for noncurrent versions
- Use sentinel errLimitReached in versions directory handler
- Set NoncurrentIndex on ObjectInfo for proper NewerNoncurrentVersions
evaluation
* lifecycle worker: fail closed on XML parse error, guard zero Mtime
- Fail closed when lifecycle XML exists but fails to parse, instead
of falling back to TTL which could apply broader rules
- Guard Mtime > 0 before using time.Unix(mtime, 0) to avoid mapping
unset Mtime to 1970, which would misorder versions and cause
premature expiration
* lifecycle worker: count delete markers toward NoncurrentIndex
Noncurrent delete markers should count toward the
NewerNoncurrentVersions retention threshold so data versions
get the correct position index. Previously, skipping delete
markers without incrementing the index could retain too many
versions after delete/recreate cycles.
* lifecycle worker: fix version ordering, error propagation, and fail-closed scope
1. Use full version ID comparison (CompareVersionIds) for sorting
.versions entries, not just decoded timestamps. Two versions with
the same timestamp prefix but different random suffixes were
previously misordered, potentially treating the newest version as
noncurrent and deleting it.
2. Propagate .versions listing failures to the caller instead of
swallowing them with (nil, 0). Transient filer errors on a
.versions directory now surface in the job result.
3. Narrow the fail-closed path to only malformed lifecycle XML
(errMalformedLifecycleXML). Transient filer LookupEntry errors
now fall back to TTL with a warning, matching the original intent
of "fail closed on bad config, not on network blips."
* lifecycle worker: only skip .uploads at bucket root
* lifecycle worker: sort.Slice, mixed-format test, XML presence tracking
- Replace manual insertion sort with sort.Slice in sortVersionsByVersionId
- Add TestCompareVersionIdsMixedFormats covering old/new format ordering
- Distinguish "no lifecycle XML" (nil) from "XML present but no effective
rules" (non-nil empty slice) so buckets with all-disabled rules don't
incorrectly fall back to filer.conf TTL expiration
* lifecycle worker: guard nil Attributes, use TrimSuffix in test
- Guard entry.Attributes != nil before accessing GetFileSize() and
Mtime in both listExpiredObjectsByRules and processVersionsDirectory
- Use strings.TrimPrefix/TrimSuffix in TestVersionsDirectoryNaming
to match the production code pattern
* lifecycle worker: skip TTL scan when XML present, fix test assertions
- When lifecycle XML is present but has no effective rules, skip
object scanning entirely instead of falling back to TTL path
- Test sort output against concrete expected names instead of
re-using the same comparator as the sort itself
---------
Co-authored-by: Copilot <copilot@github.com>
* s3api: extend lifecycle XML types with NoncurrentVersionExpiration, AbortIncompleteMultipartUpload
Add missing S3 lifecycle rule types to the XML data model:
- NoncurrentVersionExpiration with NoncurrentDays and NewerNoncurrentVersions
- NoncurrentVersionTransition with NoncurrentDays and StorageClass
- AbortIncompleteMultipartUpload with DaysAfterInitiation
- Filter.ObjectSizeGreaterThan and ObjectSizeLessThan
- And.ObjectSizeGreaterThan and ObjectSizeLessThan
- Filter.UnmarshalXML to properly parse Tag, And, and size filter elements
Each new type follows the existing set-field pattern for conditional
XML marshaling. No behavior changes - these types are not yet wired
into handlers or the lifecycle worker.
* s3lifecycle: add lifecycle rule evaluator package
New package weed/s3api/s3lifecycle/ provides a pure-function lifecycle
rule evaluation engine. The evaluator accepts flattened Rule structs and
ObjectInfo metadata, and returns the appropriate Action.
Components:
- evaluator.go: Evaluate() for per-object actions with S3 priority
ordering (delete marker > noncurrent version > current expiration),
ShouldExpireNoncurrentVersion() with NewerNoncurrentVersions support,
EvaluateMPUAbort() for multipart upload rules
- filter.go: prefix, tag, and size-based filter matching
- tags.go: ExtractTags() extracts S3 tags from filer Extended metadata,
HasTagRules() for scan-time optimization
- version_time.go: GetVersionTimestamp() extracts timestamps from
SeaweedFS version IDs (both old and new format)
Comprehensive test coverage: 54 tests covering all action types,
filter combinations, edge cases, and version ID formats.
* s3api: add UnmarshalXML for Expiration, Transition, ExpireDeleteMarker
Add UnmarshalXML methods that set the internal 'set' flag during XML
parsing. Previously these flags were only set programmatically, causing
XML round-trip to drop elements. This ensures lifecycle configurations
stored as XML survive unmarshal/marshal cycles correctly.
Add comprehensive XML round-trip tests for all lifecycle rule types
including NoncurrentVersionExpiration, AbortIncompleteMultipartUpload,
Filter with Tag/And/size constraints, and a complete Terraform-style
lifecycle configuration.
* s3lifecycle: address review feedback
- Fix version_time.go overflow: guard timestampPart > MaxInt64 before
the inversion subtraction to prevent uint64 wrap
- Make all expiry checks inclusive (!now.Before instead of now.After)
so actions trigger at the exact scheduled instant
- Add NoncurrentIndex to ObjectInfo so Evaluate() can properly handle
NewerNoncurrentVersions via ShouldExpireNoncurrentVersion()
- Add test for high-bit overflow version ID
* s3lifecycle: guard ShouldExpireNoncurrentVersion against zero SuccessorModTime
Add early return when obj.IsLatest or obj.SuccessorModTime.IsZero()
to prevent premature expiration of versions with uninitialized
successor timestamps (zero value would compute to epoch, always expired).
* lifecycle worker: detect buckets with lifecycle XML, not just filer.conf TTLs
Update the detection phase to check for stored lifecycle XML in bucket
metadata (key: s3-bucket-lifecycle-configuration-xml) in addition to
filer.conf TTL entries. A bucket is proposed for lifecycle processing if
it has lifecycle XML OR filer.conf TTLs (backward compatible).
New proposal parameters:
- has_lifecycle_xml: whether the bucket has stored lifecycle XML
- versioning_status: the bucket's versioning state (Enabled/Suspended/"")
These parameters will be used by the execution phase (subsequent PR)
to determine which evaluation path to use.
* lifecycle worker: update detection function comment to reflect XML support
* lifecycle worker: add lifecycle XML parsing and rule conversion
Add rules.go with:
- parseLifecycleXML() converts stored lifecycle XML to evaluator-friendly
s3lifecycle.Rule structs, handling Filter.Prefix, Filter.Tag, Filter.And,
size constraints, NoncurrentVersionExpiration, AbortIncompleteMultipartUpload,
Expiration.Date, and ExpiredObjectDeleteMarker
- loadLifecycleRulesFromBucket() reads lifecycle XML from bucket metadata
- parseExpirationDate() supports RFC3339 and ISO 8601 date-only formats
Comprehensive tests for all XML variants, filter types, and date formats.
* lifecycle worker: add scan-time rule evaluation for object expiration
Update executeLifecycleForBucket to try lifecycle XML evaluation first,
falling back to TTL-only evaluation when no lifecycle XML exists.
New listExpiredObjectsByRules() function:
- Walks the bucket directory tree
- Builds s3lifecycle.ObjectInfo from each filer entry
- Calls s3lifecycle.Evaluate() to check lifecycle rules
- Skips objects already handled by TTL fast path (TtlSec set)
- Extracts tags only when rules use tag-based filters (optimization)
- Skips .uploads and .versions directories (handled by other phases)
Supports Expiration.Days, Expiration.Date, Filter.Prefix, Filter.Tag,
Filter.And, and Filter.ObjectSize* in the scan-time evaluation path.
Existing TTL-based path remains for backward compatibility.
* lifecycle worker: address review feedback
- Use sentinel error (errLimitReached) instead of string matching
for scan limit detection
- Fix loadLifecycleRulesFromBucket path: use bucketsPath directly
as directory for LookupEntry instead of path.Dir which produced
the wrong parent
* lifecycle worker: fix And filter detection for size-only constraints
The And branch condition only triggered when Prefix or Tags were present,
missing the case where And contains only ObjectSizeGreaterThan or
ObjectSizeLessThan without a prefix or tags.
* lifecycle worker: address review feedback round 3
- rules.go: pass through Filter-level size constraints when Tag is
present without And (Tag+size combination was dropping sizes)
- execution.go: add doc comment to listExpiredObjectsByRules noting
that it handles non-versioned objects only; versioned objects are
handled by processVersionsDirectory
- rules_test.go: add bounds checks before indexing rules[0]
---------
Co-authored-by: Copilot <copilot@github.com>
* s3api: extend lifecycle XML types with NoncurrentVersionExpiration, AbortIncompleteMultipartUpload
Add missing S3 lifecycle rule types to the XML data model:
- NoncurrentVersionExpiration with NoncurrentDays and NewerNoncurrentVersions
- NoncurrentVersionTransition with NoncurrentDays and StorageClass
- AbortIncompleteMultipartUpload with DaysAfterInitiation
- Filter.ObjectSizeGreaterThan and ObjectSizeLessThan
- And.ObjectSizeGreaterThan and ObjectSizeLessThan
- Filter.UnmarshalXML to properly parse Tag, And, and size filter elements
Each new type follows the existing set-field pattern for conditional
XML marshaling. No behavior changes - these types are not yet wired
into handlers or the lifecycle worker.
* s3lifecycle: add lifecycle rule evaluator package
New package weed/s3api/s3lifecycle/ provides a pure-function lifecycle
rule evaluation engine. The evaluator accepts flattened Rule structs and
ObjectInfo metadata, and returns the appropriate Action.
Components:
- evaluator.go: Evaluate() for per-object actions with S3 priority
ordering (delete marker > noncurrent version > current expiration),
ShouldExpireNoncurrentVersion() with NewerNoncurrentVersions support,
EvaluateMPUAbort() for multipart upload rules
- filter.go: prefix, tag, and size-based filter matching
- tags.go: ExtractTags() extracts S3 tags from filer Extended metadata,
HasTagRules() for scan-time optimization
- version_time.go: GetVersionTimestamp() extracts timestamps from
SeaweedFS version IDs (both old and new format)
Comprehensive test coverage: 54 tests covering all action types,
filter combinations, edge cases, and version ID formats.
* s3api: add UnmarshalXML for Expiration, Transition, ExpireDeleteMarker
Add UnmarshalXML methods that set the internal 'set' flag during XML
parsing. Previously these flags were only set programmatically, causing
XML round-trip to drop elements. This ensures lifecycle configurations
stored as XML survive unmarshal/marshal cycles correctly.
Add comprehensive XML round-trip tests for all lifecycle rule types
including NoncurrentVersionExpiration, AbortIncompleteMultipartUpload,
Filter with Tag/And/size constraints, and a complete Terraform-style
lifecycle configuration.
* s3lifecycle: address review feedback
- Fix version_time.go overflow: guard timestampPart > MaxInt64 before
the inversion subtraction to prevent uint64 wrap
- Make all expiry checks inclusive (!now.Before instead of now.After)
so actions trigger at the exact scheduled instant
- Add NoncurrentIndex to ObjectInfo so Evaluate() can properly handle
NewerNoncurrentVersions via ShouldExpireNoncurrentVersion()
- Add test for high-bit overflow version ID
* s3lifecycle: guard ShouldExpireNoncurrentVersion against zero SuccessorModTime
Add early return when obj.IsLatest or obj.SuccessorModTime.IsZero()
to prevent premature expiration of versions with uninitialized
successor timestamps (zero value would compute to epoch, always expired).
* lifecycle worker: detect buckets with lifecycle XML, not just filer.conf TTLs
Update the detection phase to check for stored lifecycle XML in bucket
metadata (key: s3-bucket-lifecycle-configuration-xml) in addition to
filer.conf TTL entries. A bucket is proposed for lifecycle processing if
it has lifecycle XML OR filer.conf TTLs (backward compatible).
New proposal parameters:
- has_lifecycle_xml: whether the bucket has stored lifecycle XML
- versioning_status: the bucket's versioning state (Enabled/Suspended/"")
These parameters will be used by the execution phase (subsequent PR)
to determine which evaluation path to use.
* lifecycle worker: update detection function comment to reflect XML support
---------
Co-authored-by: Copilot <copilot@github.com>
* s3api: extend lifecycle XML types with NoncurrentVersionExpiration, AbortIncompleteMultipartUpload
Add missing S3 lifecycle rule types to the XML data model:
- NoncurrentVersionExpiration with NoncurrentDays and NewerNoncurrentVersions
- NoncurrentVersionTransition with NoncurrentDays and StorageClass
- AbortIncompleteMultipartUpload with DaysAfterInitiation
- Filter.ObjectSizeGreaterThan and ObjectSizeLessThan
- And.ObjectSizeGreaterThan and ObjectSizeLessThan
- Filter.UnmarshalXML to properly parse Tag, And, and size filter elements
Each new type follows the existing set-field pattern for conditional
XML marshaling. No behavior changes - these types are not yet wired
into handlers or the lifecycle worker.
* s3lifecycle: add lifecycle rule evaluator package
New package weed/s3api/s3lifecycle/ provides a pure-function lifecycle
rule evaluation engine. The evaluator accepts flattened Rule structs and
ObjectInfo metadata, and returns the appropriate Action.
Components:
- evaluator.go: Evaluate() for per-object actions with S3 priority
ordering (delete marker > noncurrent version > current expiration),
ShouldExpireNoncurrentVersion() with NewerNoncurrentVersions support,
EvaluateMPUAbort() for multipart upload rules
- filter.go: prefix, tag, and size-based filter matching
- tags.go: ExtractTags() extracts S3 tags from filer Extended metadata,
HasTagRules() for scan-time optimization
- version_time.go: GetVersionTimestamp() extracts timestamps from
SeaweedFS version IDs (both old and new format)
Comprehensive test coverage: 54 tests covering all action types,
filter combinations, edge cases, and version ID formats.
* s3api: add UnmarshalXML for Expiration, Transition, ExpireDeleteMarker
Add UnmarshalXML methods that set the internal 'set' flag during XML
parsing. Previously these flags were only set programmatically, causing
XML round-trip to drop elements. This ensures lifecycle configurations
stored as XML survive unmarshal/marshal cycles correctly.
Add comprehensive XML round-trip tests for all lifecycle rule types
including NoncurrentVersionExpiration, AbortIncompleteMultipartUpload,
Filter with Tag/And/size constraints, and a complete Terraform-style
lifecycle configuration.
* s3lifecycle: address review feedback
- Fix version_time.go overflow: guard timestampPart > MaxInt64 before
the inversion subtraction to prevent uint64 wrap
- Make all expiry checks inclusive (!now.Before instead of now.After)
so actions trigger at the exact scheduled instant
- Add NoncurrentIndex to ObjectInfo so Evaluate() can properly handle
NewerNoncurrentVersions via ShouldExpireNoncurrentVersion()
- Add test for high-bit overflow version ID
* s3lifecycle: guard ShouldExpireNoncurrentVersion against zero SuccessorModTime
Add early return when obj.IsLatest or obj.SuccessorModTime.IsZero()
to prevent premature expiration of versions with uninitialized
successor timestamps (zero value would compute to epoch, always expired).
---------
Co-authored-by: Copilot <copilot@github.com>
* s3: support s3:x-amz-server-side-encryption policy condition (#7680)
- Normalize x-amz-server-side-encryption header values to canonical form
(aes256 → AES256, aws:kms mixed-case → aws:kms) so StringEquals
conditions work regardless of client capitalisation
- Exempt UploadPart and UploadPartCopy from SSE Null conditions: these
actions inherit SSE from the initial CreateMultipartUpload request and
do not re-send the header, so Deny/Null("true") should not block them
- Add sse_condition_test.go covering StringEquals, Null, case-insensitive
normalisation, and multipart continuation action exemption
* s3: address review comments on SSE condition support
- Replace "inherited" sentinel in injectSSEForMultipart with "AES256" so
that StringEquals/Null conditions evaluate against a meaningful value;
add TODO noting that KMS multipart uploads need the actual algorithm
looked up from the upload state
- Rewrite TestSSECaseInsensitiveNormalization to drive normalisation
through EvaluatePolicyForRequest with a real *http.Request so regressions
in the production code path are caught; split into AES256 and aws:kms
variants to cover both normalisation branches
* s3: plumb real inherited SSE from multipart upload state into policy eval
Instead of injecting a static "AES256" sentinel for UploadPart/UploadPartCopy,
look up the actual SSE algorithm from the stored CreateMultipartUpload entry
and pass it through the evaluation chain.
Changes:
- PolicyEvaluationArgs gains InheritedSSEAlgorithm string; set by the
BucketPolicyEngine wrapper for multipart continuation actions
- injectSSEForMultipart(conditions, inheritedSSE) now accepts the real
algorithm; empty string means no SSE → Null("true") fires correctly
- IsMultipartContinuationAction exported so the s3api wrapper can use it
- BucketPolicyEngine gets a MultipartSSELookup callback (set by S3ApiServer)
that fetches the upload entry and reads SeaweedFSSSEKMSKeyID /
SeaweedFSSSES3Encryption to determine the algorithm
- S3ApiServer.getMultipartSSEAlgorithm implements the lookup via getEntry
- Tests updated: three multipart cases (AES256, aws:kms, no-SSE-must-deny)
plus UploadPartCopy coverage
* test: preserve branch when recovering bare git repo
* Replaced the standalone ensureMountClone + gitRun in Phase 5 with a new resetToCommitWithRecovery function that mirrors the existing pullFromCommitWithRecovery pattern
* fix ec.balance failing to rebalance when all nodes share all volumes (#8793)
Two bugs in doBalanceEcRack prevented rebalancing:
1. Sorting by freeEcSlot instead of actual shard count caused incorrect
empty/full node selection when nodes have different total capacities.
2. The volume-level check skipped any volume already present on the
target node. When every node has a shard of every volume (common
with many EC volumes across N nodes with N shards each), no moves
were possible.
Fix: sort by actual shard count, and use a two-pass approach - first
prefer moving shards of volumes not on the target (best diversity),
then fall back to moving specific shard IDs not yet on the target.
* add test simulating real cluster topology from issue #8793
Uses the actual node addresses and mixed max capacities (80 vs 33)
from the reporter's 14-node cluster to verify ec.balance correctly
rebalances with heterogeneous node sizes.
* fix pass comments to match 0-indexed loop variable
* filer.sync: send log file chunk fids to clients for direct volume server reads
Instead of the server reading persisted log files from volume servers, parsing
entries, and streaming them over gRPC (serial bottleneck), clients that opt in
via client_supports_metadata_chunks receive log file chunk references (fids)
and read directly from volume servers in parallel.
New proto messages:
- LogFileChunkRef: chunk fids + timestamp + filer ID for one log file
- SubscribeMetadataRequest.client_supports_metadata_chunks: client opt-in
- SubscribeMetadataResponse.log_file_refs: server sends refs during backlog
Server changes:
- CollectLogFileRefs: lists log files and returns chunk refs without any
volume server I/O (metadata-only operation)
- SubscribeMetadata/SubscribeLocalMetadata: when client opts in, sends refs
during persisted log phase, then falls back to normal streaming for
in-memory events
Client changes:
- ReadLogFileRefs: reads log files from volume servers, parses entries,
filters by path prefix, invokes processEventFn
- MetadataFollowOption.LogFileReaderFn: factory for chunk readers,
enables metadata chunks when non-nil
- Both filer_pb_tail.go and meta_aggregator.go recv loops accumulate
refs then process them at the disk→memory transition
Backward compatible: old clients don't set the flag, get existing behavior.
Ref: #8771
* filer.sync: merge entries across filers in timestamp order on client side
ReadLogFileRefs now groups refs by filer ID and merges entries from
multiple filers using a min-heap priority queue — the same algorithm
the server uses in OrderedLogVisitor + LogEntryItemPriorityQueue.
This ensures events are processed in correct timestamp order even when
log files from different filers have interleaved timestamps. Single-filer
case takes the fast path (no heap allocation).
* filer.sync: integration tests for direct-read metadata chunks
Three test categories:
1. Merge correctness (TestReadLogFileRefsMergeOrder):
Verifies entries from 3 filers are delivered in strict timestamp order,
matching the server-side OrderedLogVisitor guarantee.
2. Path filtering (TestReadLogFileRefsPathFilter):
Verifies client-side path prefix filtering works correctly.
3. Throughput comparison (TestDirectReadVsServerSideThroughput):
3 filers × 7 files × 300 events = 6300 events, 2ms per file read:
server-side: 6300 events 218ms 28,873 events/sec
direct-read: 6300 events 51ms 123,566 events/sec (4.3x)
parallel: 6300 events 17ms 378,628 events/sec (13.1x)
Direct-read eliminates gRPC send overhead per event (4.3x).
Parallel per-filer reading eliminates serial file I/O (13.1x).
* filer.sync: parallel per-filer reads with prefetching in ReadLogFileRefs
ReadLogFileRefs now has two levels of I/O overlap:
1. Cross-filer parallelism: one goroutine per filer reads its files
concurrently. Entries feed into per-filer channels, merged by the
main goroutine via min-heap (same ordering guarantee as the server's
OrderedLogVisitor).
2. Within-filer prefetching: while the current file's entries are being
consumed by the merge heap, the next file is already being read from
the volume server in a background goroutine.
Single-filer fast path avoids the heap and channels.
Test results (3 filers × 7 files × 300 events, 2ms per file read):
server-side sequential: 6300 events 212ms 29,760 events/sec
parallel + prefetch: 6300 events 36ms 177,443 events/sec
Speedup: 6.0x
* filer.sync: address all review comments on metadata chunks PR
Critical fixes:
- sendLogFileRefs: bypass pipelinedSender, send directly on gRPC stream.
Ref messages have TsNs=0 and were being incorrectly batched into the
Events field by the adaptive batching logic, corrupting ref delivery.
- readLogFileEntries: use io.ReadFull instead of reader.Read to prevent
partial reads from corrupting size values or protobuf data.
- Error handling: only skip chunk-not-found errors (matching server-side
isChunkNotFoundError). Other I/O or decode failures are propagated so
the follower can retry.
High-priority fixes:
- CollectLogFileRefs: remove incorrect +24h padding from stopTime. The
extra day caused unnecessary log file refs to be collected.
- Path filtering: ReadLogFileRefs now accepts PathFilter struct with
PathPrefix, AdditionalPathPrefixes, and DirectoriesToWatch. Uses
util.Join for path construction (avoids "//foo" on root). Excludes
/.system/log/ internal entries. Matches server-side
eachEventNotificationFn filtering logic.
Medium-priority fixes:
- CollectLogFileRefs: accept context.Context, propagate to
ListDirectoryEntries calls for cancellation support.
- NewChunkStreamReaderFromLookup: accept context.Context, propagate to
doNewChunkStreamReader.
Test fixes:
- Check error returns from ReadLogFileRefs in all test call sites.
---------
Co-authored-by: Copilot <copilot@github.com>
* sftpd: use global TLS-aware HTTP client for filer uploads (#8794)
putFile() hardcoded http:// and used http.DefaultClient, which broke
file uploads when the filer has HTTPS/TLS enabled. Switch to the global
HTTP client which reads [https.client] from security.toml and
automatically normalizes the URL scheme.
* sftpd: propagate NormalizeUrl error instead of swallowing it
* filer.sync: pipelined subscription with adaptive batching for faster catch-up
The SubscribeMetadata pipeline was fully serial: reading a log entry from a
volume server, unmarshaling, filtering, and calling stream.Send() all happened
one-at-a-time. stream.Send() blocked the entire pipeline until the client
acknowledged each event, limiting throughput to ~80 events/sec regardless of
the -concurrency setting.
Three server-side optimizations that stack:
1. Pipelined sender: decouple stream.Send() from the read loop via a buffered
channel (1024 messages). A dedicated goroutine handles gRPC delivery while
the reader continues processing the next events.
2. Adaptive batching: when event timestamps are >2min behind wall clock
(backlog catch-up), drain multiple events from the channel and pack them
into a single stream.Send() using a new `repeated events` field on
SubscribeMetadataResponse. When events are recent (real-time), send
one-by-one for low latency. Old clients ignore the new field (backward
compatible).
3. Persisted log readahead: run the OrderedLogVisitor in a background
goroutine so volume server I/O for the next log file overlaps with event
processing and gRPC delivery.
4. Event-driven aggregated subscription: replace time.Sleep(1127ms) polling
in SubscribeMetadata with notification-driven wake-up using the
MetaLogBuffer subscriber mechanism, reducing real-time latency from
~1127ms to sub-millisecond.
Combined, these create a 3-stage pipeline:
[Volume I/O → readahead buffer] → [Filter → send buffer] → [gRPC Send]
Test results (simulated backlog with 50µs gRPC latency per Send):
direct (old): 2100 events 2100 sends 168ms 12,512 events/sec
pipelined+batched: 2100 events 14 sends 40ms 52,856 events/sec
Speedup: 4.2x single-stream throughput
Ref: #8771
* filer.sync: require client opt-in for batch event delivery
Add ClientSupportsBatching field to SubscribeMetadataRequest. The server
only packs events into the Events batch field when the client explicitly
sets this flag to true. Old clients (Java SDK, third-party) that don't
set the flag get one-event-per-Send, preserving backward compatibility.
All Go callers (FollowMetadata, MetaAggregator) set the flag to true
since their recv loops already unpack batched events.
* filer.sync: clear batch Events field after Send to release references
Prevents the envelope message from holding references to the rest of the
batch after gRPC serialization, allowing the GC to collect them sooner.
* filer.sync: fix Send deadlock, add error propagation test, event-driven local subscribe
- pipelinedSender.Send: add case <-s.done to unblock when sender goroutine
exits (fixes deadlock when errCh was already consumed by a prior Send).
- pipelinedSender.reportErr: remove for-range drain on sendCh that could
block indefinitely. Send() now detects exit via s.done instead.
- SubscribeLocalMetadata: replace remaining time.Sleep(1127ms) in the
gap-detected-no-memory-data path with event-driven listenersCond.Wait(),
consistent with the rest of the subscription paths.
- Add TestPipelinedSenderErrorPropagation: verifies error surfaces via
Send and Close when the underlying stream fails.
- Replace goto with labeled break in test simulatePipeline.
* filer.sync: check error returns in test code
- direct_send: check slowStream.Send error return
- pipelined_batched_send: check sender.Close error return
- simulatePipeline: return error from sender.Close, propagate to callers
---------
Co-authored-by: Copilot <copilot@github.com>
remove min_interval_seconds from plugin workers and default vacuum interval to 17m
The worker-level min_interval_seconds was redundant with the admin-side
DetectionIntervalSeconds, complicating scheduling logic. Remove it from
vacuum, volume_balance, erasure_coding, and ec_balance handlers.
Also change the vacuum default DetectionIntervalSeconds from 2 hours to
17 minutes to match the previous default behavior.
* fix(fuse-test): harden ensureMountClone to verify .git/HEAD
On FUSE, the kernel dcache can retain a stale directory entry after
heavy git operations. Checking only os.Stat(mountClone) may find
the top-level directory but the clone internals (.git/HEAD) are gone.
Now ensureMountClone verifies .git/HEAD exists, cleans up stale
remnants with os.RemoveAll before re-cloning, and adds a brief
settle delay for the FUSE metadata cache.
* fix(fuse-test): add pull recovery loop for Phase 6 git operations
After git reset --hard on a FUSE mount, the kernel dcache can
permanently lose the clone directory entry. The existing retry logic
polls for 60+ seconds but the directory never recovers.
Add pullFromCommitWithRecovery which wraps the Phase 6 pull in a
recovery loop: if the clone directory vanishes, it removes the stale
clone, re-creates it from the bare repo, resets to the target commit,
and retries the pull (up to 3 attempts).
Also adds tryGitCommand, a non-fatal git runner that returns
(output, error) instead of calling require.NoError, enabling the
recovery loop to handle failures gracefully without aborting the test.
Fixes flaky TestGitOperations/CloneAndPull on CI.
* fix(fuse-test): make recovery loop fully non-fatal
Address review feedback:
- Extract tryEnsureMountClone (returns error) so a transient FUSE
failure during re-clone doesn't abort the test — the recovery loop
can retry instead.
- Check waitForDirEventually return after git reset --hard and return
a specific error if the directory doesn't recover.
* style(fuse-test): simplify ensureMountClone error check
* fix(fuse-test): recover bare repo when FUSE mount loses it
CI showed that not just the working clone but also the bare repo on
the FUSE mount can vanish after heavy git operations. The recovery
loop now re-creates the bare repo from the local clone (which lives
on local disk and is always available) before attempting to re-clone.
Adds tryEnsureBareRepo: checks HEAD exists in the bare repo, and if
not, re-inits and force-pushes from the local clone.
dependabot[bot]
Chris Lu
msementsov
Lars Lehtonen
Chris Lu
Simon Bråten