* mount: implement create for rsync temp files
* mount: move access implementation out of unsupported
* mount: tighten access checks
* mount: log access group lookup failures
* mount: reset dirty pages on truncate
* mount: tighten create and root access handling
* mount: handle existing creates before quota checks
* mount: restrict access fallback when group lookup fails
When lookupSupplementaryGroupIDs returns an error, the previous code
fell through to checking only the "other" permission bits, which could
overgrant access. Require both group and other permission classes to
satisfy the mask so access is never broader than intended.
* mount: guard against nil entry in Create existing-file path
maybeLoadEntry can return OK with a nil entry or nil Attributes in
edge cases. Check before dereferencing to prevent a panic.
* mount: reopen existing file on create race without O_EXCL
When createRegularFile returns EEXIST because another process won the
race, and O_EXCL is not set, reload the winner's entry and open it
instead of propagating the error to the caller.
* mount: check parent directory permission in createRegularFile
Verify the caller has write+search (W_OK|X_OK) permission on the
parent directory before creating a file. This applies to both
Create and Mknod. Update test fixture mount mode to 0o777 so the
existing tests pass with the new check.
* mount: enforce file permission bits in AcquireHandle
Map the open flags (O_RDONLY/O_WRONLY/O_RDWR) to an access mask and
call hasAccess before handing out a file handle. This makes
AcquireHandle the single source of truth for mode-based access
control across Open, Create-existing, and Create-new paths.
---------
Co-authored-by: Copilot <copilot@github.com>
Key improvements:
- Fix concurrent map write panic in partition type cache
- Fix data races in yieldDataFiles and key map getter
- Fix response body leaks in REST catalog
- Fix index out of range in buildManifestEvaluator
- Table Metadata V3 support
- Schema evolution API
- Partitioned write throughput optimizations
- Gzipped metadata read/write support
* Add Spark Iceberg catalog integration tests and CI support
Implement comprehensive integration tests for Spark with SeaweedFS Iceberg REST catalog:
- Basic CRUD operations (Create, Read, Update, Delete) on Iceberg tables
- Namespace (database) management
- Data insertion, querying, and deletion
- Time travel capabilities via snapshot versioning
- Compatible with SeaweedFS S3 and Iceberg REST endpoints
Tests mirror the structure of existing Trino integration tests but use Spark's
Python SQL API and PySpark for testing.
Add GitHub Actions CI job for spark-iceberg-catalog-tests in s3-tables-tests.yml
to automatically run Spark integration tests on pull requests.
* fmt
* Fix Spark integration tests - code review feedback
* go mod tidy
* Add go mod tidy step to integration test jobs
Add 'go mod tidy' step before test runs for all integration test jobs:
- s3-tables-tests
- iceberg-catalog-tests
- trino-iceberg-catalog-tests
- spark-iceberg-catalog-tests
This ensures dependencies are clean before running tests.
* Fix remaining Spark operations test issues
Address final code review comments:
Setup & Initialization:
- Add waitForSparkReady() helper function that polls Spark readiness
with backoff instead of hardcoded 10-second sleep
- Extract setupSparkTestEnv() helper to reduce boilerplate duplication
between TestSparkCatalogBasicOperations and TestSparkTimeTravel
- Both tests now use helpers for consistent, reliable setup
Assertions & Validation:
- Make setup-critical operations (namespace, table creation, initial
insert) use t.Fatalf instead of t.Errorf to fail fast
- Validate setupSQL output in TestSparkTimeTravel and fail if not
'Setup complete'
- Add validation after second INSERT in TestSparkTimeTravel:
verify row count increased to 2 before time travel test
- Add context to error messages with namespace and tableName params
Code Quality:
- Remove code duplication between test functions
- All critical paths now properly validated
- Consistent error handling throughout
* Fix go vet errors in S3 Tables tests
Fixes:
1. setup_test.go (Spark):
- Add missing import: github.com/testcontainers/testcontainers-go/wait
- Use wait.ForLog instead of undefined testcontainers.NewLogStrategy
- Remove unused strings import
2. trino_catalog_test.go:
- Use net.JoinHostPort instead of fmt.Sprintf for address formatting
- Properly handles IPv6 addresses by wrapping them in brackets
* Use weed mini for simpler SeaweedFS startup
Replace complex multi-process startup (master, volume, filer, s3)
with single 'weed mini' command that starts all services together.
Benefits:
- Simpler, more reliable startup
- Single weed mini process vs 4 separate processes
- Automatic coordination between components
- Better port management with no manual coordination
Changes:
- Remove separate master, volume, filer process startup
- Use weed mini with -master.port, -filer.port, -s3.port flags
- Keep Iceberg REST as separate service (still needed)
- Increase timeout to 15s for port readiness (weed mini startup)
- Remove volumePort and filerProcess fields from TestEnvironment
- Simplify cleanup to only handle two processes (mini, iceberg rest)
* Clean up dead code and temp directory leaks
Fixes:
1. Remove dead s3Process field and cleanup:
- weed mini bundles S3 gateway, no separate process needed
- Removed s3Process field from TestEnvironment
- Removed unnecessary s3Process cleanup code
2. Fix temp config directory leak:
- Add sparkConfigDir field to TestEnvironment
- Store returned configDir in writeSparkConfig
- Clean up sparkConfigDir in Cleanup() with os.RemoveAll
- Prevents accumulation of temp directories in test runs
3. Simplify Cleanup:
- Now handles only necessary processes (weed mini, iceberg rest)
- Removes both seaweedfsDataDir and sparkConfigDir
- Cleaner shutdown sequence
* Use weed mini's built-in Iceberg REST and fix python binary
Changes:
- Add -s3.port.iceberg flag to weed mini for built-in Iceberg REST Catalog
- Remove separate 'weed server' process for Iceberg REST
- Remove icebergRestProcess field from TestEnvironment
- Simplify Cleanup() to only manage weed mini + Spark
- Add port readiness check for iceberg REST from weed mini
- Set Spark container Cmd to '/bin/sh -c sleep 3600' to keep it running
- Change python to python3 in container.Exec calls
This simplifies to truly one all-in-one weed mini process (master, filer, s3,
iceberg-rest) plus just the Spark container.
* go fmt
* clean up
* bind on a non-loopback IP for container access, aligned Iceberg metadata saves/locations with table locations, and reworked Spark time travel to use TIMESTAMP AS OF with safe timestamp extraction.
* shared mini start
* Fixed internal directory creation under /buckets so .objects paths can auto-create without failing bucket-name validation, which restores table bucket object writes
* fix path
Updated table bucket objects to write under `/buckets/<bucket>` and saved Iceberg metadata there, adjusting Spark time-travel timestamp to committed_at +1s. Rebuilt the weed binary (`go
install ./weed`) and confirmed passing tests for Spark and Trino with focused test commands.
* Updated table bucket creation to stop creating /buckets/.objects and switched Trino REST warehouse to s3://<bucket> to match Iceberg layout.
* Stabilize S3Tables integration tests
* Fix timestamp extraction and remove dead code in bucketDir
* Use table bucket as warehouse in s3tables tests
* Update trino_blog_operations_test.go
* adds the CASCADE option to handle any remaining table metadata/files in the schema directory
* skip namespace not empty
* full integration with iceberg-go
* Table Commit Operations (handleUpdateTable)
* s3tables: fix Iceberg v2 compliance and namespace properties
This commit ensures SeaweedFS Iceberg REST Catalog is compliant with
Iceberg Format Version 2 by:
- Using iceberg-go's table.NewMetadataWithUUID for strict v2 compliance.
- Explicitly initializing namespace properties to empty maps.
- Removing omitempty from required Iceberg response fields.
- Fixing CommitTableRequest unmarshaling using table.Requirements and table.Updates.
* s3tables: automate Iceberg integration tests
- Added Makefile for local test execution and cluster management.
- Added docker-compose for PyIceberg compatibility kit.
- Added Go integration test harness for PyIceberg.
- Updated GitHub CI to run Iceberg catalog tests automatically.
* s3tables: update PyIceberg test suite for compatibility
- Updated test_rest_catalog.py to use latest PyIceberg transaction APIs.
- Updated Dockerfile to include pyarrow and pandas dependencies.
- Improved namespace and table handling in integration tests.
* s3tables: address review feedback on Iceberg Catalog
- Implemented robust metadata version parsing and incrementing.
- Ensured table metadata changes are persisted during commit (handleUpdateTable).
- Standardized namespace property initialization for consistency.
- Fixed unused variable and incorrect struct field build errors.
* s3tables: finalize Iceberg REST Catalog and optimize tests
- Implemented robust metadata versioning and persistence.
- Standardized namespace property initialization.
- Optimized integration tests using pre-built Docker image.
- Added strict property persistence validation to test suite.
- Fixed build errors from previous partial updates.
* Address PR review: fix Table UUID stability, implement S3Tables UpdateTable, and support full metadata persistence individually
* fix: Iceberg catalog stable UUIDs, metadata persistence, and file writing
- Ensure table UUIDs are stable (do not regenerate on load).
- Persist full table metadata (Iceberg JSON) in s3tables extended attributes.
- Add `MetadataVersion` to explicitly track version numbers, replacing regex parsing.
- Implement `saveMetadataFile` to persist metadata JSON files to the Filer on commit.
- Update `CreateTable` and `UpdateTable` handlers to use the new logic.
* test: bind weed mini to 0.0.0.0 in integration tests to fix Docker connectivity
* Iceberg: fix metadata handling in REST catalog
- Add nil guard in createTable
- Fix updateTable to correctly load existing metadata from storage
- Ensure full metadata persistence on updates
- Populate loadTable result with parsed metadata
* S3Tables: add auth checks and fix response fields in UpdateTable
- Add CheckPermissionWithContext to UpdateTable handler
- Include TableARN and MetadataLocation in UpdateTable response
- Use ErrCodeConflict (409) for version token mismatches
* Tests: improve Iceberg catalog test infrastructure and cleanup
- Makefile: use PID file for precise process killing
- test_rest_catalog.py: remove unused variables and fix f-strings
* Iceberg: fix variable shadowing in UpdateTable
- Rename inner loop variable `req` to `requirement` to avoid shadowing outer request variable
* S3Tables: simplify MetadataVersion initialization
- Use `max(req.MetadataVersion, 1)` instead of anonymous function
* Tests: remove unicode characters from S3 tables integration test logs
- Remove unicode checkmarks from test output for cleaner logs
* Iceberg: improve metadata persistence robustness
- Fix MetadataLocation in LoadTableResult to fallback to generated location
- Improve saveMetadataFile to ensure directory hierarchy existence and robust error handling
Chris Lu
dependabot[bot]