REMOVED DEBUG LOGS:
- PutObjectHandler: CI-DEBUG logs for object creation tracing
- putVersionedObject: CI-DEBUG logs for version metadata updates
- updateLatestVersionInDirectory: CI-DEBUG logs for retry logic
- getLatestObjectVersion: CI-DEBUG logs for version lookups and race conditions
- HeadObjectHandler: CI-DEBUG logs for latest version requests
- ListObjectVersionsHandler: CI-DEBUG logs for request params and responses
REMOVED S3-TESTS MODIFICATIONS:
- Dynamic sed patches to s3-tests Python code in GitHub Actions
- CI-S3TEST-DEBUG logging for test_bucket_list_return_data_versioning
- CI-S3TEST-DEBUG logging for test_versioning_concurrent_multi_object_delete
- CI-S3TEST-DEBUG logging for _create_objects function
RETAINED FUNCTIONALITY:
✓ Path normalization fix (removeDuplicateSlashes consistency)
✓ Retry logic with exponential backoff for filer consistency
✓ Race condition handling for .versions directory metadata
✓ All existing glog.V(1), glog.V(2), glog.V(3) logging levels
The debug logs served their purpose in identifying and fixing the root cause
of S3 versioning CI failures (path normalization inconsistency). Now that
the issues are resolved, the verbose debug output is no longer needed.
Added CI-DEBUG logs to trace:
- ListObjectVersions request parameters (bucket, prefix, maxKeys, markers)
- Response details (number of versions, delete markers, truncation status)
- First few versions in response (key, versionId, size) to avoid log spam
- Clear indication when NO VERSIONS are found
This will help diagnose:
1. Whether ListObjectVersions is being called for the failing tests
2. What response is being returned (empty vs missing versions)
3. Whether the issue is in object creation or version listing
4. The exact response structure that boto3 receives
Combined with PutObject debug logs, this provides end-to-end tracing
of versioned object lifecycle in CI environment.
ROOT CAUSE IDENTIFIED: The issue with objects like '//bar', '//testobjfoo',
'//testobjbar', and '/key' was due to inconsistent path normalization between
object upload and versioned metadata operations.
PROBLEM:
- toFilerUrl() calls removeDuplicateSlashes() normalizing '//bar' → '/bar'
- But versioned operations used raw object paths: '//bar.versions'
- This created a mismatch where version files were stored under '/bar.versions/'
but .versions directory metadata was stored under '//bar.versions'
- Filer lookups failed because paths didn't match
SOLUTION:
- Apply removeDuplicateSlashes() consistently in all versioned operations:
- putVersionedObject: normalize before creating .versions directory
- getLatestObjectVersion: normalize before looking up .versions directory
- getSpecificObjectVersion: normalize for all version operations
- deleteSpecificObjectVersion: normalize for version deletion
- Ensures all version-related paths use the same normalization as toFilerUrl()
This should resolve the persistent CI failures for objects with double slashes
in their paths, eliminating the 'filer: no entry is found in filer store' errors
that even 8 retries with exponential backoff couldn't resolve.
- Increase retry attempts from 5 to 8 for both updateLatestVersionInDirectory
and getLatestObjectVersion functions
- Increase base delay from 50ms to 100ms with exponential backoff up to 6.4s
- Add specific retry logic for 'no Extended metadata' race condition where
.versions directory exists but metadata is not yet written
- Add detailed timing logs to track retry delays and total wait times
- Addresses persistent CI failures where even 5 retries with 400ms max delay
were insufficient for filer store consistency in GitHub Actions environment
- Increase retry attempts from 3 to 5 for both updateLatestVersionInDirectory
and getLatestObjectVersion functions
- Implement exponential backoff: 50ms, 100ms, 200ms, 400ms delays
- Addresses persistent CI failures where .versions directories are not
immediately visible after creation in filer store
- Based on CI log analysis showing 50ms fixed delays were insufficient
- Maintains CI debug logging to track improved retry behavior
- Add retry logic to updateLatestVersionInDirectory to handle cases where
.versions directory creation succeeds but is not immediately visible
- Add retry logic to getLatestObjectVersion for the same consistency issue
- Use 3 retries with 50ms delays to handle filer store consistency timing
- Addresses CI failures where 'filer: no entry is found in filer store'
occurs after successful directory creation
- Maintains CI debug logging to track retry attempts and outcomes
- Add detailed logging for .versions metadata updates in putVersionedObject
- Add logging for latest version resolution in getLatestObjectVersion
- Add logging for HeadObject latest version requests
- All logs use glog.V(0) with CI-DEBUG prefix for easy filtering
- Will help diagnose timing issues between object creation and retrieval in CI
Debug logs will show:
- When .versions metadata updates start and complete
- When HeadObject tries to read latest version metadata
- Race conditions if HeadObject runs before metadata update completes
- Missing metadata if .versions directory exists but metadata keys are missing
- File access issues if version files exist but can't be accessed
* add back tests
* get put object acl
* check permission to put object acl
* rename file
* object list versions now contains owners
* set object owner
* refactoring
* Revert "add back tests"
This reverts commit 9adc507c45.
* test versioning also
* fix some versioning tests
* fall back
* fixes
Never-versioned buckets: No VersionId headers, no Status field
Pre-versioning objects: Regular files, VersionId="null", included in all operations
Post-versioning objects: Stored in .versions directories with real version IDs
Suspended versioning: Proper status handling and null version IDs
* fixes
Bucket Versioning Status Compliance
Fixed: New buckets now return no Status field (AWS S3 compliant)
Before: Always returned "Suspended" ❌
After: Returns empty VersioningConfiguration for unconfigured buckets ✅
2. Multi-Object Delete Versioning Support
Fixed: DeleteMultipleObjectsHandler now fully versioning-aware
Before: Always deleted physical files, breaking versioning ❌
After: Creates delete markers or deletes specific versions properly ✅
Added: DeleteMarker field in response structure for AWS compatibility
3. Copy Operations Versioning Support
Fixed: CopyObjectHandler and CopyObjectPartHandler now versioning-aware
Before: Only copied regular files, couldn't handle versioned sources ❌
After: Parses version IDs from copy source, creates versions in destination ✅
Added: pathToBucketObjectAndVersion() function for version ID parsing
4. Pre-versioning Object Handling
Fixed: getLatestObjectVersion() now has proper fallback logic
Before: Failed when .versions directory didn't exist ❌
After: Falls back to regular objects for pre-versioning scenarios ✅
5. Enhanced Object Version Listings
Fixed: listObjectVersions() includes both versioned AND pre-versioning objects
Before: Only showed .versions directories, ignored pre-versioning objects ❌
After: Shows complete version history with VersionId="null" for pre-versioning ✅
6. Null Version ID Handling
Fixed: getSpecificObjectVersion() properly handles versionId="null"
Before: Couldn't retrieve pre-versioning objects by version ID ❌
After: Returns regular object files for "null" version requests ✅
7. Version ID Response Headers
Fixed: PUT operations only return x-amz-version-id when appropriate
Before: Returned version IDs for non-versioned buckets ❌
After: Only returns version IDs for explicitly configured versioning ✅
* more fixes
* fix copying with versioning, multipart upload
* more fixes
* reduce volume size for easier dev test
* fix
* fix version id
* fix versioning
* Update filer_multipart.go
* fix multipart versioned upload
* more fixes
* more fixes
* fix versioning on suspended
* fixes
* fixing test_versioning_obj_suspended_copy
* Update s3api_object_versioning.go
* fix versions
* skipping test_versioning_obj_suspend_versions
* > If the versioning state has never been set on a bucket, it has no versioning state; a GetBucketVersioning request does not return a versioning state value.
* fix tests, avoid duplicated bucket creation, skip tests
* only run s3tests_boto3/functional/test_s3.py
* fix checking filer_pb.ErrNotFound
* Update weed/s3api/s3api_object_versioning.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/s3api/s3api_object_handlers_copy.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/s3api/s3api_bucket_config.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update test/s3/versioning/s3_versioning_test.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* add object versioning
* add missing file
* Update weed/s3api/s3api_object_versioning.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/s3api/s3api_object_versioning.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/s3api/s3api_object_versioning.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* ListObjectVersionsResult is better to show multiple version entries
* fix test
* Update weed/s3api/s3api_object_handlers_put.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/s3api/s3api_object_versioning.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* multiple improvements
* move PutBucketVersioningHandler into weed/s3api/s3api_bucket_handlers.go file
* duplicated code for reading bucket config, versioningEnabled, etc. try to use functions
* opportunity to cache bucket config
* error handling if bucket is not found
* in case bucket is not found
* fix build
* add object versioning tests
* remove non-existent tests
* add tests
* add versioning tests
* skip a new test
* ensure .versions directory exists before saving info into it
* fix creating version entry
* logging on creating version directory
* Update s3api_object_versioning_test.go
* retry and wait for directory creation
* revert add more logging
* Update s3api_object_versioning.go
* more debug messages
* clean up logs, and touch directory correctly
* log the .versions creation and then parent directory listing
* use mkFile instead of touch
touch is for update
* clean up data
* add versioning test in go
* change location
* if modified, latest version is moved to .versions directory, and create a new latest version
Core versioning functionality: WORKING
TestVersioningBasicWorkflow - PASS
TestVersioningDeleteMarkers - PASS
TestVersioningMultipleVersionsSameObject - PASS
TestVersioningDeleteAndRecreate - PASS
TestVersioningListWithPagination - PASS
❌ Some advanced features still failing:
ETag calculation issues (using mtime instead of proper MD5)
Specific version retrieval (EOF error)
Version deletion (internal errors)
Concurrent operations (race conditions)
* calculate multi chunk md5
Test Results - All Passing:
✅ TestBucketListReturnDataVersioning - PASS
✅ TestVersioningCreateObjectsInOrder - PASS
✅ TestVersioningBasicWorkflow - PASS
✅ TestVersioningMultipleVersionsSameObject - PASS
✅ TestVersioningDeleteMarkers - PASS
* dedupe
* fix TestVersioningErrorCases
* fix eof error of reading old versions
* get specific version also check current version
* enable integration tests for versioning
* trigger action to work for now
* Fix GitHub Actions S3 versioning tests workflow
- Fix syntax error (incorrect indentation)
- Update directory paths from weed/s3api/versioning_tests/ to test/s3/versioning/
- Add push trigger for add-object-versioning branch to enable CI during development
- Update artifact paths to match correct directory structure
* Improve CI robustness for S3 versioning tests
Makefile improvements:
- Increase server startup timeout from 30s to 90s for CI environments
- Add progressive timeout reporting (logs at 30s, full logs at 90s)
- Better error handling with server logs on failure
- Add server PID tracking for debugging
- Improved test failure reporting
GitHub Actions workflow improvements:
- Increase job timeouts to account for CI environment delays
- Add system information logging (memory, disk space)
- Add detailed failure reporting with server logs
- Add process and network diagnostics on failure
- Better error messaging and log collection
These changes should resolve the 'Server failed to start within 30 seconds' issue
that was causing the CI tests to fail.
* adjust testing volume size
* Update Makefile
* Update Makefile
* Update Makefile
* Update Makefile
* Update s3-versioning-tests.yml
* Update s3api_object_versioning.go
* Update Makefile
* do not clean up
* log received version id
* more logs
* printout response
* print out list version response
* use tmp files when put versioned object
* change to versions folder layout
* Delete weed-test.log
* test with mixed versioned and unversioned objects
* remove versionDirCache
* remove unused functions
* remove unused function
* remove fallback checking
* minor
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
chrislu
Chris Lu