Tree: 839028b2e0

add-ec-vacuum

add-filer-iam-grpc

add-iam-grpc-management

add_fasthttp_client

add_remote_storage

adding-message-queue-integration-tests

adjust-fsck-cutoff-default

admin/csrf-s3tables

also-delete-parent-directory-if-empty

avoid_releasing_temp_file_on_write

changing-to-zap

collect-public-metrics

copilot/fix-helm-chart-installation

copilot/fix-s3-object-tagging-issue

copilot/make-renew-interval-configurable

copilot/make-renew-interval-configurable-again

copilot/sub-pr-7677

create-table-snapshot-api-design

data_query_pushdown

dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5

dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0

detect-and-plan-ec-tasks

do-not-retry-if-error-is-NotFound

ec-disk-type-support

enhance-erasure-coding

fasthttp

feature-8113-storage-class-disk-routing

feature/mini-port-detection

feature/modernize-s3-tests

feature/s3-multi-cert-support

feature/s3tables-improvements-and-spark-tests

filer1_maintenance_branch

fix-8303-s3-lifecycle-ttl-assign

fix-GetObjectLockConfigurationHandler

fix-bucket-name-case-7910

fix-helm-fromtoml-compatibility

fix-mount-http-parallelism

fix-mount-read-throughput-7504

fix-pr-7909

fix-s3-configure-consistency

fix-s3-object-tagging-issue-7589

fix-sts-session-token-7941

fix-versioning-listing-only

fix/iceberg-stage-create-semantics

fix/windows-test-file-cleanup

ftp

gh-pages

iam-multi-file-migration

iam-permissions-and-api

improve-fuse-mount

improve-fuse-mount2

logrus

master

message_send

mount2

mq-subscribe

mq2

nfs-cookie-prefix-list-fixes

optimize-delete-lookups

original_weed_mount

pr-7412

pr/7984

pr/8140

raft-dual-write

random_access_file

refactor-needle-read-operations

refactor-volume-write

remote_overlay

remove-implicit-directory-handling

revert-5134-patch-1

revert-5819-patch-1

revert-6434-bugfix-missing-s3-audit

s3-remote-cache-singleflight

s3-select

s3tables-by-claude

sub

tcp_read

test-reverting-lock-table

test_udp

testing

testing-sdx-generation

tikv

track-mount-e2e

upgrade-versions-to-4.00

volume_buffered_writes

worker-execute-ec-tasks

0.72

0.72.release

0.73

0.74

0.75

0.76

0.77

0.90

0.91

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1.00

1.01

1.02

1.03

1.04

1.05

1.06

1.07

1.08

1.09

1.10

1.11

1.12

1.14

1.15

1.16

1.17

1.18

1.19

1.20

1.21

1.22

1.23

1.24

1.25

1.26

1.27

1.28

1.29

1.30

1.31

1.32

1.33

1.34

1.35

1.36

1.37

1.38

1.40

1.41

1.42

1.43

1.44

1.45

1.46

1.47

1.48

1.49

1.50

1.51

1.52

1.53

1.54

1.55

1.56

1.57

1.58

1.59

1.60

1.61

1.61RC

1.62

1.63

1.64

1.65

1.66

1.67

1.68

1.69

1.70

1.71

1.72

1.73

1.74

1.75

1.76

1.77

1.78

1.79

1.80

1.81

1.82

1.83

1.84

1.85

1.86

1.87

1.88

1.90

1.91

1.92

1.93

1.94

1.95

1.96

1.97

1.98

1.99

1;70

2.00

2.01

2.02

2.03

2.04

2.05

2.06

2.07

2.08

2.09

2.10

2.11

2.12

2.13

2.14

2.15

2.16

2.17

2.18

2.19

2.20

2.21

2.22

2.23

2.24

2.25

2.26

2.27

2.28

2.29

2.30

2.31

2.32

2.33

2.34

2.35

2.36

2.37

2.38

2.39

2.40

2.41

2.42

2.43

2.47

2.48

2.49

2.50

2.51

2.52

2.53

2.54

2.55

2.56

2.57

2.58

2.59

2.60

2.61

2.62

2.63

2.64

2.65

2.66

2.67

2.68

2.69

2.70

2.71

2.72

2.73

2.74

2.75

2.76

2.77

2.78

2.79

2.80

2.81

2.82

2.83

2.84

2.85

2.86

2.87

2.88

2.89

2.90

2.91

2.92

2.93

2.94

2.95

2.96

2.97

2.98

2.99

3.00

3.01

3.02

3.03

3.04

3.05

3.06

3.07

3.08

3.09

3.10

3.11

3.12

3.13

3.14

3.15

3.16

3.18

3.19

3.20

3.21

3.22

3.23

3.24

3.25

3.26

3.27

3.28

3.29

3.30

3.31

3.32

3.33

3.34

3.35

3.36

3.37

3.38

3.39

3.40

3.41

3.42

3.43

3.44

3.45

3.46

3.47

3.48

3.50

3.51

3.52

3.53

3.54

3.55

3.56

3.57

3.58

3.59

3.60

3.61

3.62

3.63

3.64

3.65

3.66

3.67

3.68

3.69

3.71

3.72

3.73

3.74

3.75

3.76

3.77

3.78

3.79

3.80

3.81

3.82

3.83

3.84

3.85

3.86

3.87

3.88

3.89

3.90

3.91

3.92

3.93

3.94

3.95

3.96

3.97

3.98

3.99

4.00

4.01

4.02

4.03

4.04

4.05

4.06

4.07

4.08

4.09

4.12

dev

helm-3.65.1

v0.69

v0.70beta

v3.33

Branches Tags

${ item.name }

Create tag ${ searchTerm }

Create branch ${ searchTerm }

from '839028b2e0'

${ noResults }

Commit Graph

180 Commits (839028b2e0c5676b309e65b1846aef55cc69bae9)

Author	SHA1	Message	Date
Chris Lu	458c12fb99	test: add Spark S3 integration regression for issue #8234 (#8249) * test: add Spark S3 regression integration test for issue 8234 * Update test/s3/spark/setup_test.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	4 days ago
Chris Lu	5a0204310c	Add Iceberg admin UI (#8246) * Add Iceberg table details view * Enhance Iceberg catalog browsing UI * Fix Iceberg UI security and logic issues - Fix selectSchema() and partitionFieldsFromFullMetadata() to always search for matching IDs instead of checking != 0 - Fix snapshotsFromFullMetadata() to defensive-copy before sorting to prevent mutating caller's slice - Fix XSS vulnerabilities in s3tables.js: replace innerHTML with textContent/createElement for user-controlled data - Fix deleteIcebergTable() to redirect to namespace tables list on details page instead of reloading - Fix data-bs-target in iceberg_namespaces.templ: remove templ.SafeURL for CSS selector - Add catalogName to delete modal data attributes for proper redirect - Remove unused hidden inputs from create table form (icebergTableBucketArn, icebergTableNamespace) * Regenerate templ files for Iceberg UI updates * Support complex Iceberg type objects in schema Change Type field from string to json.RawMessage in both IcebergSchemaFieldInfo and internal icebergSchemaField to properly handle Iceberg spec's complex type objects (e.g. {"type": "struct", "fields": [...]}). Currently test data only shows primitive string types, but this change makes the implementation defensively robust for future complex types by preserving the exact JSON representation. Add typeToString() helper and update schema extraction functions to marshal string types as JSON. Update template to convert json.RawMessage to string for display. * Regenerate templ files for Type field changes * templ * Fix additional Iceberg UI issues from code review - Fix lazy-load flag that was set before async operation completed, preventing retries on error; now sets loaded flag only after successful load and throws error to caller for proper error handling and UI updates - Add zero-time guards for CreatedAt and ModifiedAt fields in table details to avoid displaying Go zero-time values; render dash when time is zero - Add URL path escaping for all catalog/namespace/table names in URLs to prevent malformed URLs when names contain special characters like /, ?, or # - Remove redundant innerHTML clear in loadIcebergNamespaceTables that cleared twice before appending the table list - Fix selectSnapshotForMetrics to remove != 0 guard for consistency with selectSchema fix; now always searches for CurrentSnapshotID without zero-value gate - Enhance typeToString() helper to display '(complex)' for non-primitive JSON types * Regenerate templ files for Phase 3 updates * Fix template generation to use correct file paths Run templ generate from repo root instead of weed/admin directory to ensure generated _templ.go files have correct absolute paths in error messages (e.g., 'weed/admin/view/app/iceberg_table_details.templ' instead of 'app/iceberg_table_details.templ'). This ensures both 'make admin-generate' at repo root and 'make generate' in weed/admin directory produce identical output with consistent file path references. * Regenerate template files with correct path references * Validate S3 Tables names in UI - Add client-side validation for table bucket and namespace names to surface errors for invalid characters (dots/underscores) before submission - Use HTML validity messages with reportValidity for immediate feedback - Update namespace helper text to reflect actual constraints (single-level, lowercase letters, numbers, and underscores) * Regenerate templ files for namespace helper text * Fix Iceberg catalog REST link and actions * Disallow S3 object access on table buckets * Validate Iceberg layout for table bucket objects * Fix REST API link to /v1/config * merge iceberg page with table bucket page * Allowed Trino/Iceberg stats files in metadata validation * fixes - Backend/data handling: - Normalized Iceberg type display and fallback handling in weed/admin/dash/s3tables_management.go. - Fixed snapshot fallback pointer semantics in weed/admin/dash/s3tables_management.go. - Added CSRF token generation/propagation/validation for namespace create/delete in: - weed/admin/dash/csrf.go - weed/admin/dash/auth_middleware.go - weed/admin/dash/middleware.go - weed/admin/dash/s3tables_management.go - weed/admin/view/layout/layout.templ - weed/admin/static/js/s3tables.js - UI/template fixes: - Zero-time guards for CreatedAt fields in: - weed/admin/view/app/iceberg_namespaces.templ - weed/admin/view/app/iceberg_tables.templ - Fixed invalid templ-in-script interpolation and host/port rendering in: - weed/admin/view/app/iceberg_catalog.templ - weed/admin/view/app/s3tables_buckets.templ - Added data-catalog-name consistency on Iceberg delete action in weed/admin/view/app/iceberg_tables.templ. - Updated retry wording in weed/admin/static/js/s3tables.js. - Regenerated all affected _templ.go files. - S3 API/comment follow-ups: - Reused cached table-bucket validator in weed/s3api/bucket_paths.go. - Added validation-failure debug logging in weed/s3api/s3api_object_handlers_tagging.go. - Added multipart path-validation design comment in weed/s3api/s3api_object_handlers_multipart.go. - Build tooling: - Fixed templ generate working directory issues in weed/admin/Makefile (watch + pattern rule). * populate data * test/s3tables: harden populate service checks * admin: skip table buckets in object-store bucket list * admin sidebar: move object store to top-level links * admin iceberg catalog: guard zero times and escape links * admin forms: add csrf/error handling and client-side name validation * admin s3tables: fix namespace delete modal redeclaration * admin: replace native confirm dialogs with modal helpers * admin modal-alerts: remove noisy confirm usage console log * reduce logs * test/s3tables: use partitioned tables in trino and spark populate * admin file browser: normalize filer ServerAddress for HTTP parsing	4 days ago
Chris Lu	403592bb9f	Add Spark Iceberg catalog integration tests and CI support (#8242) * Add Spark Iceberg catalog integration tests and CI support Implement comprehensive integration tests for Spark with SeaweedFS Iceberg REST catalog: - Basic CRUD operations (Create, Read, Update, Delete) on Iceberg tables - Namespace (database) management - Data insertion, querying, and deletion - Time travel capabilities via snapshot versioning - Compatible with SeaweedFS S3 and Iceberg REST endpoints Tests mirror the structure of existing Trino integration tests but use Spark's Python SQL API and PySpark for testing. Add GitHub Actions CI job for spark-iceberg-catalog-tests in s3-tables-tests.yml to automatically run Spark integration tests on pull requests. * fmt * Fix Spark integration tests - code review feedback * go mod tidy * Add go mod tidy step to integration test jobs Add 'go mod tidy' step before test runs for all integration test jobs: - s3-tables-tests - iceberg-catalog-tests - trino-iceberg-catalog-tests - spark-iceberg-catalog-tests This ensures dependencies are clean before running tests. * Fix remaining Spark operations test issues Address final code review comments: Setup & Initialization: - Add waitForSparkReady() helper function that polls Spark readiness with backoff instead of hardcoded 10-second sleep - Extract setupSparkTestEnv() helper to reduce boilerplate duplication between TestSparkCatalogBasicOperations and TestSparkTimeTravel - Both tests now use helpers for consistent, reliable setup Assertions & Validation: - Make setup-critical operations (namespace, table creation, initial insert) use t.Fatalf instead of t.Errorf to fail fast - Validate setupSQL output in TestSparkTimeTravel and fail if not 'Setup complete' - Add validation after second INSERT in TestSparkTimeTravel: verify row count increased to 2 before time travel test - Add context to error messages with namespace and tableName params Code Quality: - Remove code duplication between test functions - All critical paths now properly validated - Consistent error handling throughout * Fix go vet errors in S3 Tables tests Fixes: 1. setup_test.go (Spark): - Add missing import: github.com/testcontainers/testcontainers-go/wait - Use wait.ForLog instead of undefined testcontainers.NewLogStrategy - Remove unused strings import 2. trino_catalog_test.go: - Use net.JoinHostPort instead of fmt.Sprintf for address formatting - Properly handles IPv6 addresses by wrapping them in brackets * Use weed mini for simpler SeaweedFS startup Replace complex multi-process startup (master, volume, filer, s3) with single 'weed mini' command that starts all services together. Benefits: - Simpler, more reliable startup - Single weed mini process vs 4 separate processes - Automatic coordination between components - Better port management with no manual coordination Changes: - Remove separate master, volume, filer process startup - Use weed mini with -master.port, -filer.port, -s3.port flags - Keep Iceberg REST as separate service (still needed) - Increase timeout to 15s for port readiness (weed mini startup) - Remove volumePort and filerProcess fields from TestEnvironment - Simplify cleanup to only handle two processes (mini, iceberg rest) * Clean up dead code and temp directory leaks Fixes: 1. Remove dead s3Process field and cleanup: - weed mini bundles S3 gateway, no separate process needed - Removed s3Process field from TestEnvironment - Removed unnecessary s3Process cleanup code 2. Fix temp config directory leak: - Add sparkConfigDir field to TestEnvironment - Store returned configDir in writeSparkConfig - Clean up sparkConfigDir in Cleanup() with os.RemoveAll - Prevents accumulation of temp directories in test runs 3. Simplify Cleanup: - Now handles only necessary processes (weed mini, iceberg rest) - Removes both seaweedfsDataDir and sparkConfigDir - Cleaner shutdown sequence * Use weed mini's built-in Iceberg REST and fix python binary Changes: - Add -s3.port.iceberg flag to weed mini for built-in Iceberg REST Catalog - Remove separate 'weed server' process for Iceberg REST - Remove icebergRestProcess field from TestEnvironment - Simplify Cleanup() to only manage weed mini + Spark - Add port readiness check for iceberg REST from weed mini - Set Spark container Cmd to '/bin/sh -c sleep 3600' to keep it running - Change python to python3 in container.Exec calls This simplifies to truly one all-in-one weed mini process (master, filer, s3, iceberg-rest) plus just the Spark container. * go fmt * clean up * bind on a non-loopback IP for container access, aligned Iceberg metadata saves/locations with table locations, and reworked Spark time travel to use TIMESTAMP AS OF with safe timestamp extraction. * shared mini start * Fixed internal directory creation under /buckets so .objects paths can auto-create without failing bucket-name validation, which restores table bucket object writes * fix path Updated table bucket objects to write under `/buckets/<bucket>` and saved Iceberg metadata there, adjusting Spark time-travel timestamp to committed_at +1s. Rebuilt the weed binary (`go install ./weed`) and confirmed passing tests for Spark and Trino with focused test commands. * Updated table bucket creation to stop creating /buckets/.objects and switched Trino REST warehouse to s3://<bucket> to match Iceberg layout. * Stabilize S3Tables integration tests * Fix timestamp extraction and remove dead code in bucketDir * Use table bucket as warehouse in s3tables tests * Update trino_blog_operations_test.go * adds the CASCADE option to handle any remaining table metadata/files in the schema directory * skip namespace not empty	4 days ago
Chris Lu	e6ee293c17	Add table operations test (#8241) * Add Trino blog operations test * Update test/s3tables/catalog_trino/trino_blog_operations_test.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * feat: add table bucket path helpers and filer operations - Add table object root and table location mapping directories - Implement ensureDirectory, upsertFile, deleteEntryIfExists helpers - Support table location bucket mapping for S3 access * feat: manage table bucket object roots on creation/deletion - Create .objects directory for table buckets on creation - Clean up table object bucket paths on deletion - Enable S3 operations on table bucket object roots * feat: add table location mapping for Iceberg REST - Track table location bucket mappings when tables are created/updated/deleted - Enable location-based routing for S3 operations on table data * feat: route S3 operations to table bucket object roots - Route table-s3 bucket names to mapped table paths - Route table buckets to object root directories - Support table location bucket mapping lookup * feat: emit table-s3 locations from Iceberg REST - Generate unique table-s3 bucket names with UUID suffix - Store table metadata under table bucket paths - Return table-s3 locations for Trino compatibility * fix: handle missing directories in S3 list operations - Propagate ErrNotFound from ListEntries for non-existent directories - Treat missing directories as empty results for list operations - Fixes Trino non-empty location checks on table creation * test: improve Trino CSV parsing for single-value results - Sanitize Trino output to skip jline warnings - Handle single-value CSV results without header rows - Strip quotes from numeric values in tests * refactor: use bucket path helpers throughout S3 API - Replace direct bucket path operations with helper functions - Leverage centralized table bucket routing logic - Improve maintainability with consistent path resolution * fix: add table bucket cache and improve filer error handling - Cache table bucket lookups to reduce filer overhead on repeated checks - Use filer_pb.CreateEntry and filer_pb.UpdateEntry helpers to check resp.Error - Fix delete order in handler_bucket_get_list_delete: delete table object before directory - Make location mapping errors best-effort: log and continue, don't fail API - Update table location mappings to delete stale prior bucket mappings on update - Add 1-second sleep before timestamp time travel query to ensure timestamps are in past - Fix CSV parsing: examine all lines, not skip first; handle single-value rows * fix: properly handle stale metadata location mapping cleanup - Capture oldMetadataLocation before mutation in handleUpdateTable - Update updateTableLocationMapping to accept both old and new locations - Use passed-in oldMetadataLocation to detect location changes - Delete stale mapping only when location actually changes - Pass empty string for oldLocation in handleCreateTable (new tables have no prior mapping) - Improve logging to show old -> new location transitions * refactor: cleanup imports and cache design - Remove unused 'sync' import from bucket_paths.go - Use filer_pb.UpdateEntry helper in setExtendedAttribute and deleteExtendedAttribute for consistent error handling - Add dedicated tableBucketCache map[string]bool to BucketRegistry instead of mixing concerns with metadataCache - Improve cache separation: table buckets cache is now separate from bucket metadata cache * fix: improve cache invalidation and add transient error handling Cache invalidation (critical fix): - Add tableLocationCache to BucketRegistry for location mapping lookups - Clear tableBucketCache and tableLocationCache in RemoveBucketMetadata - Prevents stale cache entries when buckets are deleted/recreated Transient error handling: - Only cache table bucket lookups when conclusive (found or ErrNotFound) - Skip caching on transient errors (network, permission, etc) - Prevents marking real table buckets as non-table due to transient failures Performance optimization: - Cache tableLocationDir results to avoid repeated filer RPCs on hot paths - tableLocationDir now checks cache before making expensive filer lookups - Cache stores empty string for 'not found' to avoid redundant lookups Code clarity: - Add comment to deleteDirectory explaining DeleteEntry response lacks Error field * go fmt * fix: mirror transient error handling in tableLocationDir and optimize bucketDir Transient error handling: - tableLocationDir now only caches definitive results - Mirrors isTableBucket behavior to prevent treating transient errors as permanent misses - Improves reliability on flaky systems or during recovery Performance optimization: - bucketDir avoids redundant isTableBucket call via bucketRoot - Directly use s3a.option.BucketsPath for regular buckets - Saves one cache lookup for every non-table bucket operation * fix: revert bucketDir optimization to preserve bucketRoot logic The optimization to directly use BucketsPath bypassed bucketRoot's logic and caused issues with S3 list operations on delimiter+prefix cases. Revert to using path.Join(s3a.bucketRoot(bucket), bucket) which properly handles all bucket types and ensures consistent path resolution across the codebase. The slight performance cost of an extra cache lookup is worth the correctness and consistency benefits. * feat: move table buckets under /buckets Add a table-bucket marker attribute, reuse bucket metadata cache for table bucket detection, and update list/validation/UI/test paths to treat table buckets as /buckets entries. * Fix S3 Tables code review issues - handler_bucket_create.go: Fix bucket existence check to properly validate entryResp.Entry before setting s3BucketExists flag (nil Entry should not indicate existing bucket) - bucket_paths.go: Add clarifying comment to bucketRoot() explaining unified buckets root path for all bucket types - file_browser_data.go: Optimize by extracting table bucket check early to avoid redundant WithFilerClient call * Fix list prefix delimiter handling * Handle list errors conservatively * Fix Trino FOR TIMESTAMP query - use past timestamp Iceberg requires the timestamp to be strictly in the past. Use current_timestamp - interval '1' second instead of current_timestamp. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	5 days ago
Chris Lu	c284e51d20	fix: multipart upload ETag calculation (#8238) * fix multipart etag * address comments * clean up * clean up * optimization * address comments * unquoted etag * dedup * upgrade * clean * etag * return quoted tag * quoted etag * debug * s3api: unify ETag retrieval and quoting across handlers Refactor newListEntry to take *S3ApiServer and use getObjectETag, and update setResponseHeaders to use the same logic. This ensures consistent ETags are returned for both listing and direct access. * s3api: implement ListObjects deduplication for versioned buckets Handle duplicate entries between the main path and the .versions directory by prioritizing the latest version when bucket versioning is enabled. * s3api: cleanup stale main file entries during versioned uploads Add explicit deletion of pre-existing "main" files when creating new versions in versioned buckets. This prevents stale entries from appearing in bucket listings and ensures consistency. * s3api: fix cleanup code placement in versioned uploads Correct the placement of rm calls in completeMultipartUpload and putVersionedObject to ensure stale main files are properly deleted during versioned uploads. * s3api: improve getObjectETag fallback for empty ExtETagKey Ensure that when ExtETagKey exists but contains an empty value, the function falls through to MD5/chunk-based calculation instead of returning an empty string. * s3api: fix test files for new newListEntry signature Update test files to use the new newListEntry signature where the first parameter is *S3ApiServer. Created mockS3ApiServer to properly test owner display name lookup functionality. * s3api: use filer.ETag for consistent Md5 handling in getEtagFromEntry Change getEtagFromEntry fallback to use filer.ETag(entry) instead of filer.ETagChunks to ensure legacy entries with Attributes.Md5 are handled consistently with the rest of the codebase. * s3api: optimize list logic and fix conditional header logging - Hoist bucket versioning check out of per-entry callback to avoid repeated getVersioningState calls - Extract appendOrDedup helper function to eliminate duplicate dedup/append logic across multiple code paths - Change If-Match mismatch logging from glog.Errorf to glog.V(3).Infof and remove DEBUG prefix for consistency * s3api: fix test mock to properly initialize IAM accounts Fixed nil pointer dereference in TestNewListEntryOwnerDisplayName by directly initializing the IdentityAccessManagement.accounts map in the test setup. This ensures newListEntry can properly look up account display names without panicking. * cleanup * s3api: remove premature main file cleanup in versioned uploads Removed incorrect cleanup logic that was deleting main files during versioned uploads. This was causing test failures because it deleted objects that should have been preserved as null versions when versioning was first enabled. The deduplication logic in listing is sufficient to handle duplicate entries without deleting files during upload. * s3api: add empty-value guard to getEtagFromEntry Added the same empty-value guard used in getObjectETag to prevent returning quoted empty strings. When ExtETagKey exists but is empty, the function now falls through to filer.ETag calculation instead of returning "". * s3api: fix listing of directory key objects with matching prefix Revert prefix handling logic to use strings.TrimPrefix instead of checking HasPrefix with empty string result. This ensures that when a directory key object exactly matches the prefix (e.g. prefix="dir/", object="dir/"), it is correctly handled as a regular entry instead of being skipped or incorrectly processed as a common prefix. Also fixed missing variable definition. * s3api: refactor list inline dedup to use appendOrDedup helper Refactored the inline deduplication logic in listFilerEntries to use the shared appendOrDedup helper function. This ensures consistent behavior and reduces code duplication. * test: fix port allocation race in s3tables integration test Updated startMiniCluster to find all required ports simultaneously using findAvailablePorts instead of sequentially. This prevents race conditions where the OS reallocates a port that was just released, causing multiple services (e.g. Filer and Volume) to be assigned the same port and fail to start.	6 days ago
Chris Lu	2163570d16	test: add CRUD tests for S3 Tables Catalog Trino integration (#8236) * test: add comprehensive CRUD tests for S3 Tables Catalog Trino integration - Add TestNamespaceCRUD: Tests complete Create-Read-Update-Delete lifecycle for namespaces - Add TestNamespaceListingPagination: Tests listing multiple namespaces with verification - Add TestNamespaceErrorHandling: Tests error handling for edge cases (IF EXISTS, IF NOT EXISTS) - Add TestSchemaIntegrationWithCatalog: Tests integration between Trino SQL and Iceberg REST Catalog All tests pass successfully and use Trino SQL interface for practical integration testing. Tests properly skip when Docker is unavailable. Use randomized namespace names to avoid conflicts in parallel execution. The tests provide comprehensive coverage of namespace/schema CRUD operations which form the foundation of the Iceberg catalog integration with Trino. * test: Address code review feedback for S3 Tables Catalog Trino CRUD tests - Extract common test setup into setupTrinoTest() helper function - Replace all fmt.Printf calls with idiomatic t.Logf - Change namespace deletion verification from t.Logf to t.Errorf for proper test failures - Enhance TestNamespaceErrorHandling with persistence verification test - Remove unnecessary fmt import - Improve test documentation with clarifying comments * test: Fix schema naming and remove verbose output logging - Fix TestNamespaceListingPagination schema name generation: use fmt.Sprintf instead of string(rune()) - Remove verbose logging of SHOW SCHEMAS output to reduce noise in test logs - Keep high-level operation logging while removing detailed result output	6 days ago
Chris Lu	a3b83f8808	test: add Trino Iceberg catalog integration test (#8228) * test: add Trino Iceberg catalog integration test - Create test/s3/catalog_trino/trino_catalog_test.go with TestTrinoIcebergCatalog - Tests integration between Trino SQL engine and SeaweedFS Iceberg REST catalog - Starts weed mini with all services and Trino in Docker container - Validates Iceberg catalog schema creation and listing operations - Uses native S3 filesystem support in Trino with path-style access - Add workflow job to s3-tables-tests.yml for CI execution * fix: preserve AWS environment credentials when replacing S3 configuration When S3 configuration is loaded from filer/db, it replaces the identities list and inadvertently removes AWS_ACCESS_KEY_ID credentials that were added from environment variables. This caused auth to remain disabled even though valid credentials were present. Fix by preserving environment-based identities when replacing the configuration and re-adding them after the replacement. This ensures environment credentials persist across configuration reloads and properly enable authentication. * fix: use correct ServerAddress format with gRPC port encoding The admin server couldn't connect to master because the master address was missing the gRPC port information. Use pb.NewServerAddress() which properly encodes both HTTP and gRPC ports in the address string. Changes: - weed/command/mini.go: Use pb.NewServerAddress for master address in admin - test/s3/policy/policy_test.go: Store and use gRPC ports for master/filer addresses This fix applies to: 1. Admin server connection to master (mini.go) 2. Test shell commands that need master/filer addresses (policy_test.go) * move * move * fix: always include gRPC port in server address encoding The NewServerAddress() function was omitting the gRPC port from the address string when it matched the port+10000 convention. However, gRPC port allocation doesn't always follow this convention - when the calculated port is busy, an alternative port is allocated. This caused a bug where: 1. Master's gRPC port was allocated as 50661 (sequential, not port+10000) 2. Address was encoded as '192.168.1.66:50660' (gRPC port omitted) 3. Admin client called ToGrpcAddress() which assumed port+10000 offset 4. Admin tried to connect to 60660 but master was on 50661 → connection failed Fix: Always include explicit gRPC port in address format (host:httpPort.grpcPort) unless gRPC port is 0. This makes addresses unambiguous and works regardless of the port allocation strategy used. Impacts: All server-to-server gRPC connections now use properly formatted addresses. * test: fix Iceberg REST API readiness check The Iceberg REST API endpoints require authentication. When checked without credentials, the API returns 403 Forbidden (not 401 Unauthorized). The readiness check now accepts both auth error codes (401/403) as indicators that the service is up and ready, it just needs credentials. This fixes the 'Iceberg REST API did not become ready' test failure. * Fix AWS SigV4 signature verification for base64-encoded payload hashes AWS SigV4 canonical requests must use hex-encoded SHA256 hashes, but the X-Amz-Content-Sha256 header may be transmitted as base64. Changes: - Added normalizePayloadHash() function to convert base64 to hex - Call normalizePayloadHash() in extractV4AuthInfoFromHeader() - Added encoding/base64 import Fixes 403 Forbidden errors on POST requests to Iceberg REST API when clients send base64-encoded content hashes in the header. Impacted services: Iceberg REST API, S3Tables * Fix AWS SigV4 signature verification for base64-encoded payload hashes AWS SigV4 canonical requests must use hex-encoded SHA256 hashes, but the X-Amz-Content-Sha256 header may be transmitted as base64. Changes: - Added normalizePayloadHash() function to convert base64 to hex - Call normalizePayloadHash() in extractV4AuthInfoFromHeader() - Added encoding/base64 import - Removed unused fmt import Fixes 403 Forbidden errors on POST requests to Iceberg REST API when clients send base64-encoded content hashes in the header. Impacted services: Iceberg REST API, S3Tables * pass sigv4 * s3api: fix identity preservation and logging levels - Ensure environment-based identities are preserved during config replacement - Update accessKeyIdent and nameToIdentity maps correctly - Downgrade informational logs to V(2) to reduce noise * test: fix trino integration test and s3 policy test - Pin Trino image version to 479 - Fix port binding to 0.0.0.0 for Docker connectivity - Fix S3 policy test hang by correctly assigning MiniClusterCtx - Improve port finding robustness in policy tests * ci: pre-pull trino image to avoid timeouts - Pull trinodb/trino:479 after Docker setup - Ensure image is ready before integration tests start * iceberg: remove unused checkAuth and improve logging - Remove unused checkAuth method - Downgrade informational logs to V(2) - Ensure loggingMiddleware uses a status writer for accurate reported codes - Narrow catch-all route to avoid interfering with other subsystems * iceberg: fix build failure by removing unused s3api import * Update iceberg.go * use warehouse * Update trino_catalog_test.go	6 days ago
Chris Lu	833bcde9f3	test: add Trino Iceberg catalog integration test - Create test/s3/catalog_trino/trino_catalog_test.go with TestTrinoIcebergCatalog - Tests integration between Trino SQL engine and SeaweedFS Iceberg REST catalog - Starts weed mini with all services and Trino in Docker container - Validates Iceberg catalog schema creation and listing operations - Uses native S3 filesystem support in Trino with path-style access - Add workflow job to s3-tables-tests.yml for CI execution	7 days ago
Chris Lu	e39a4c2041	fix flaky test	1 week ago
Chris Lu	b244bb58aa	s3tables: redesign Iceberg REST Catalog using iceberg-go and automate integration tests (#8197) * full integration with iceberg-go * Table Commit Operations (handleUpdateTable) * s3tables: fix Iceberg v2 compliance and namespace properties This commit ensures SeaweedFS Iceberg REST Catalog is compliant with Iceberg Format Version 2 by: - Using iceberg-go's table.NewMetadataWithUUID for strict v2 compliance. - Explicitly initializing namespace properties to empty maps. - Removing omitempty from required Iceberg response fields. - Fixing CommitTableRequest unmarshaling using table.Requirements and table.Updates. * s3tables: automate Iceberg integration tests - Added Makefile for local test execution and cluster management. - Added docker-compose for PyIceberg compatibility kit. - Added Go integration test harness for PyIceberg. - Updated GitHub CI to run Iceberg catalog tests automatically. * s3tables: update PyIceberg test suite for compatibility - Updated test_rest_catalog.py to use latest PyIceberg transaction APIs. - Updated Dockerfile to include pyarrow and pandas dependencies. - Improved namespace and table handling in integration tests. * s3tables: address review feedback on Iceberg Catalog - Implemented robust metadata version parsing and incrementing. - Ensured table metadata changes are persisted during commit (handleUpdateTable). - Standardized namespace property initialization for consistency. - Fixed unused variable and incorrect struct field build errors. * s3tables: finalize Iceberg REST Catalog and optimize tests - Implemented robust metadata versioning and persistence. - Standardized namespace property initialization. - Optimized integration tests using pre-built Docker image. - Added strict property persistence validation to test suite. - Fixed build errors from previous partial updates. * Address PR review: fix Table UUID stability, implement S3Tables UpdateTable, and support full metadata persistence individually * fix: Iceberg catalog stable UUIDs, metadata persistence, and file writing - Ensure table UUIDs are stable (do not regenerate on load). - Persist full table metadata (Iceberg JSON) in s3tables extended attributes. - Add `MetadataVersion` to explicitly track version numbers, replacing regex parsing. - Implement `saveMetadataFile` to persist metadata JSON files to the Filer on commit. - Update `CreateTable` and `UpdateTable` handlers to use the new logic. * test: bind weed mini to 0.0.0.0 in integration tests to fix Docker connectivity * Iceberg: fix metadata handling in REST catalog - Add nil guard in createTable - Fix updateTable to correctly load existing metadata from storage - Ensure full metadata persistence on updates - Populate loadTable result with parsed metadata * S3Tables: add auth checks and fix response fields in UpdateTable - Add CheckPermissionWithContext to UpdateTable handler - Include TableARN and MetadataLocation in UpdateTable response - Use ErrCodeConflict (409) for version token mismatches * Tests: improve Iceberg catalog test infrastructure and cleanup - Makefile: use PID file for precise process killing - test_rest_catalog.py: remove unused variables and fix f-strings * Iceberg: fix variable shadowing in UpdateTable - Rename inner loop variable `req` to `requirement` to avoid shadowing outer request variable * S3Tables: simplify MetadataVersion initialization - Use `max(req.MetadataVersion, 1)` instead of anonymous function * Tests: remove unicode characters from S3 tables integration test logs - Remove unicode checkmarks from test output for cleaner logs * Iceberg: improve metadata persistence robustness - Fix MetadataLocation in LoadTableResult to fallback to generated location - Improve saveMetadataFile to ensure directory hierarchy existence and robust error handling	1 week ago
Chris Lu	1274cf038c	s3: enforce authentication and JSON error format for Iceberg REST Catalog (#8192) * s3: enforce authentication and JSON error format for Iceberg REST Catalog * s3/iceberg: align error exception types with OpenAPI spec examples * s3api: refactor AuthenticateRequest to return identity object * s3/iceberg: propagate full identity object to request context * s3/iceberg: differentiate NotAuthorizedException and ForbiddenException * s3/iceberg: reject requests if authenticator is nil to prevent auth bypass * s3/iceberg: refactor Auth middleware to build context incrementally and use switch for error mapping * s3api: update misleading comment for authRequestWithAuthType * s3api: return ErrAccessDenied if IAM is not configured to prevent auth bypass * s3/iceberg: optimize context update in Auth middleware * s3api: export CanDo for external authorization use * s3/iceberg: enforce identity-based authorization in all API handlers * s3api: fix compilation errors by updating internal CanDo references * s3/iceberg: robust identity validation and consistent action usage in handlers * s3api: complete CanDo rename across tests and policy engine integration * s3api: fix integration tests by allowing admin access when auth is disabled and explicit gRPC ports * duckdb * create test bucket	1 week ago
Chris Lu	2bb21ea276	feat: Add Iceberg REST Catalog server and admin UI (#8175) * feat: Add Iceberg REST Catalog server Implement Iceberg REST Catalog API on a separate port (default 8181) that exposes S3 Tables metadata through the Apache Iceberg REST protocol. - Add new weed/s3api/iceberg package with REST handlers - Implement /v1/config endpoint returning catalog configuration - Implement namespace endpoints (list/create/get/head/delete) - Implement table endpoints (list/create/load/head/delete/update) - Add -port.iceberg flag to S3 standalone server (s3.go) - Add -s3.port.iceberg flag to combined server mode (server.go) - Add -s3.port.iceberg flag to mini cluster mode (mini.go) - Support prefix-based routing for multiple catalogs The Iceberg REST server reuses S3 Tables metadata storage under /table-buckets and enables DuckDB, Spark, and other Iceberg clients to connect to SeaweedFS as a catalog. * feat: Add Iceberg Catalog pages to admin UI Add admin UI pages to browse Iceberg catalogs, namespaces, and tables. - Add Iceberg Catalog menu item under Object Store navigation - Create iceberg_catalog.templ showing catalog overview with REST info - Create iceberg_namespaces.templ listing namespaces in a catalog - Create iceberg_tables.templ listing tables in a namespace - Add handlers and routes in admin_handlers.go - Add Iceberg data provider methods in s3tables_management.go - Add Iceberg data types in types.go The Iceberg Catalog pages provide visibility into the same S3 Tables data through an Iceberg-centric lens, including REST endpoint examples for DuckDB and PyIceberg. * test: Add Iceberg catalog integration tests and reorg s3tables tests - Reorganize existing s3tables tests to test/s3tables/table-buckets/ - Add new test/s3tables/catalog/ for Iceberg REST catalog tests - Add TestIcebergConfig to verify /v1/config endpoint - Add TestIcebergNamespaces to verify namespace listing - Add TestDuckDBIntegration for DuckDB connectivity (requires Docker) - Update CI workflow to use new test paths * fix: Generate proper random UUIDs for Iceberg tables Address code review feedback: - Replace placeholder UUID with crypto/rand-based UUID v4 generation - Add detailed TODO comments for handleUpdateTable stub explaining the required atomic metadata swap implementation * fix: Serve Iceberg on localhost listener when binding to different interface Address code review feedback: properly serve the localhost listener when the Iceberg server is bound to a non-localhost interface. * ci: Add Iceberg catalog integration tests to CI Add new job to run Iceberg catalog tests in CI, along with: - Iceberg package build verification - Iceberg unit tests - Iceberg go vet checks - Iceberg format checks * fix: Address code review feedback for Iceberg implementation - fix: Replace hardcoded account ID with s3_constants.AccountAdminId in buildTableBucketARN() - fix: Improve UUID generation error handling with deterministic fallback (timestamp + PID + counter) - fix: Update handleUpdateTable to return HTTP 501 Not Implemented instead of fake success - fix: Better error handling in handleNamespaceExists to distinguish 404 from 500 errors - fix: Use relative URL in template instead of hardcoded localhost:8181 - fix: Add HTTP timeout to test's waitForService function to avoid hangs - fix: Use dynamic ephemeral ports in integration tests to avoid flaky parallel failures - fix: Add Iceberg port to final port configuration logging in mini.go * fix: Address critical issues in Iceberg implementation - fix: Cache table UUIDs to ensure persistence across LoadTable calls The UUID now remains stable for the lifetime of the server session. TODO: For production, UUIDs should be persisted in S3 Tables metadata. - fix: Remove redundant URL-encoded namespace parsing mux router already decodes %1F to \x1F before passing to handlers. Redundant ReplaceAll call could cause bugs with literal %1F in namespace. * fix: Improve test robustness and reduce code duplication - fix: Make DuckDB test more robust by failing on unexpected errors Instead of silently logging errors, now explicitly check for expected conditions (extension not available) and skip the test appropriately. - fix: Extract username helper method to reduce duplication Created getUsername() helper in AdminHandlers to avoid duplicating the username retrieval logic across Iceberg page handlers. * fix: Add mutex protection to table UUID cache Protects concurrent access to the tableUUIDs map with sync.RWMutex. Uses read-lock for fast path when UUID already cached, and write-lock for generating new UUIDs. Includes double-check pattern to handle race condition between read-unlock and write-lock. * style: fix go fmt errors * feat(iceberg): persist table UUID in S3 Tables metadata * feat(admin): configure Iceberg port in Admin UI and commands * refactor: address review comments (flags, tests, handlers) - command/mini: fix tracking of explicit s3.port.iceberg flag - command/admin: add explicit -iceberg.port flag - admin/handlers: reuse getUsername helper - tests: use 127.0.0.1 for ephemeral ports and os.Stat for file size check * test: check error from FileStat in verify_gc_empty_test	1 week ago
Chris Lu	f1e27b8f30	s3: change s3 tables to use RESTful API (#8169) * s3: refactor s3 tables to use RESTful API * test/s3tables: guard empty namespaces * s3api: document tag parsing and validate get-table * s3api: limit S3Tables REST body size * Update weed/s3api/s3api_tables.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3tables/handler.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * s3api: accept encoded table bucket ARNs * s3api: validate namespaces and close body * s3api: match encoded table bucket ARNs * s3api: scope table bucket ARN routes * s3api: dedupe table bucket request builders * test/s3tables: allow list tables without namespace * s3api: validate table params and tag ARN * s3api: tighten tag handling and get-table params * s3api: loosen tag ARN route matching * Fix S3 Tables REST routing and tests * Adjust S3 Tables request parsing * Gate S3 Tables target routing * Avoid double decoding namespaces --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2 weeks ago
Chris Lu	2f155ee5ee	feat: Add S3 Tables support for Iceberg tabular data (#8147) * s3tables: extract utility and filer operations to separate modules - Move ARN parsing, path helpers, and metadata structures to utils.go - Extract all extended attribute and filer operations to filer_ops.go - Reduces code duplication and improves modularity - Improves code organization and maintainability * s3tables: split table bucket operations into focused modules - Create bucket_create.go for CreateTableBucket operation - Create bucket_get_list_delete.go for Get, List, Delete operations - Related operations grouped for better maintainability - Each file has a single, clear responsibility - Improves code clarity and makes it easier to test * s3tables: simplify handler by removing duplicate utilities - Reduce handler.go from 370 to 195 lines (47% reduction) - Remove duplicate ARN parsing and path helper functions - Remove filer operation methods moved to filer_ops.go - Remove metadata structure definitions moved to utils.go - Keep handler focused on request routing and response formatting - Maintains all functionality with improved code organization * s3tables: complete s3tables package implementation - namespace.go: namespace CRUD operations (310 lines) - table.go: table CRUD operations with Iceberg schema support (409 lines) - policy.go: resource policies and tagging operations (419 lines) - types.go: request/response types and error definitions (290 lines) - All handlers updated to use standalone utilities from utils.go - All files follow single responsibility principle * s3api: add S3 Tables integration layer - Create s3api_tables.go to integrate S3 Tables with S3 API server - Implement S3 Tables route matcher for X-Amz-Target header - Register S3 Tables routes with API router - Provide gRPC filer client interface for S3 Tables handlers - All S3 Tables operations accessible via S3 API endpoint * s3api: register S3 Tables routes in API server - Add S3 Tables route registration in s3api_server.go registerRouter method - Enable S3 Tables API operations to be routed through S3 API server - Routes handled by s3api_tables.go integration layer - Minimal changes to existing S3 API structure * test: add S3 Tables test infrastructure - Create setup.go with TestCluster and S3TablesClient definitions - Create client.go with HTTP client methods for all operations - Test utilities and client methods organized for reusability - Foundation for S3 Tables integration tests * test: add S3 Tables integration tests - Comprehensive integration tests for all 23 S3 Tables operations - Test cluster setup based on existing S3 integration tests - Tests cover: * Table bucket lifecycle (create, get, list, delete) * Namespace operations * Table CRUD with Iceberg schema * Table bucket and table policies * Resource tagging operations - Ready for CI/CD pipeline integration * ci: add S3 Tables integration tests to GitHub Actions - Create new workflow for S3 Tables integration testing - Add build verification job for s3tables package and s3api integration - Add format checking for S3 Tables code - Add go vet checks for code quality - Workflow runs on all pull requests - Includes test output logging and artifact upload on failure * s3tables: add handler_ prefix to operation handler files - Rename bucket_create.go → handler_bucket_create.go - Rename bucket_get_list_delete.go → handler_bucket_get_list_delete.go - Rename namespace.go → handler_namespace.go - Rename table.go → handler_table.go - Rename policy.go → handler_policy.go Improves file organization by clearly identifying handler implementations. No code changes, refactoring only. * s3tables test: refactor to eliminate duplicate definitions - Move all client methods to client.go - Remove duplicate types/constants from s3tables_integration_test.go - Keep setup.go for test infrastructure - Keep integration test logic in s3tables_integration_test.go - Clean up unused imports - Test compiles successfully * Delete client_methods.go * s3tables: add bucket name validation and fix error handling - Add isValidBucketName validation function for [a-z0-9_-] characters - Validate bucket name characters match ARN parsing regex - Fix error handling in WithFilerClient closure - properly check for lookup errors - Add error handling for json.Marshal calls (metadata and tags) - Improve error messages and logging * s3tables: add error handling for json.Marshal calls - Add error handling in handler_namespace.go (metadata marshaling) - Add error handling in handler_table.go (metadata and tags marshaling) - Add error handling in handler_policy.go (tag marshaling in TagResource and UntagResource) - Return proper errors with context instead of silently ignoring failures * s3tables: replace custom splitPath with stdlib functions - Remove custom splitPath implementation (23 lines) - Use filepath.Dir and filepath.Base from stdlib - More robust and handles edge cases correctly - Reduces code duplication * s3tables: improve error handling specificity in ListTableBuckets - Specifically check for 'not found' errors instead of catching all errors - Return empty list only when directory doesn't exist - Propagate other errors (network, permission) with context - Prevents masking real errors * s3api_tables: optimize action validation with map lookup - Replace O(n) slice iteration with O(1) map lookup - Move s3TablesActionsMap to package level - Avoid recreating the map on every function call - Improves performance for request validation * s3tables: implement permission checking and authorization - Add permissions.go with permission definitions and checks - Define permissions for all 21 S3 Tables operations - Add permission checking helper functions - Add getPrincipalFromRequest to extract caller identity - Implement access control in CreateTableBucket, GetTableBucket, DeleteTableBucket - Return 403 Forbidden for unauthorized operations - Only bucket owner can perform operations (extensible for future policies) - Add AuthError type for authorization failures * workflow: fix s3 tables tests path and working directory The workflow was failing because it was running inside 'weed' directory, but the tests are at the repository root. Removed working-directory default and updated relative paths to weed source. * workflow: remove emojis from echo statements * test: format s3tables client.go * workflow: fix go install path to ./weed * ci: fail s3 tables tests if any command in pipeline fails * s3tables: use path.Join for path construction and align namespace paths * s3tables: improve integration test stability and error reporting * s3tables: propagate request context to filer operations * s3tables: clean up unused code and improve error response formatting * Refine S3 Tables implementation to address code review feedback - Standardize namespace representation to []string - Improve listing logic with pagination and StartFromFileName - Enhance error handling with sentinel errors and robust checks - Add JSON encoding error logging - Fix CI workflow to use gofmt -l - Standardize timestamps in directory creation - Validate single-level namespaces * s3tables: further refinements to filer operations and utilities - Add multi-segment namespace support to ARN parsing - Refactor permission checking to use map lookup - Wrap lookup errors with ErrNotFound in filer operations - Standardize splitPath to use path package * test: improve S3 Tables client error handling and cleanup - Add detailed error reporting when decoding failure responses - Remove orphaned comments and unused sections * command: implement graceful shutdown for mini cluster - Introduce MiniClusterCtx to coordinate shutdown across mini services - Update Master, Volume, Filer, S3, and WebDAV servers to respect context cancellation - Ensure all resources are cleaned up properly during test teardown - Integrate MiniClusterCtx in s3tables integration tests * s3tables: fix pagination and enhance error handling in list/delete operations - Fix InclusiveStartFrom logic to ensure exclusive start on continued pages - Prevent duplicates in bucket, namespace, and table listings - Fail fast on listing errors during bucket and namespace deletion - Stop swallowing errors in handleListTables and return proper HTTP error responses * s3tables: align ARN formatting and optimize resource handling - Update generateTableARN to match AWS S3 Tables specification - Move defer r.Body.Close() to follow standard Go patterns - Remove unused generateNamespaceARN helper * command: fix stale error variable logging in filer serving goroutines - Use local 'err' variable instead of stale 'e' from outer scope - Applied to both TLS and non-TLS paths for local listener * s3tables: implement granular authorization and refine error responses - Remove mandatory ACTION_ADMIN at the router level - Enforce granular permissions in bucket and namespace handlers - Prioritize AccountID in ExtractPrincipalFromContext for ARN matching - Distinguish between 404 (NoSuchBucket) and 500 (InternalError) in metadata lookups - Clean up unused imports in s3api_tables.go * test: refactor S3 Tables client for DRYness and multi-segment namespaces - Implement doRequestAndDecode to eliminate HTTP boilerplate - Update client API to accept []string for namespaces to support hierarchy - Standardize error response decoding across all client methods * test: update integration tests to match refactored S3 Tables client - Pass namespaces as []string to support hierarchical structures - Adapt test calls to new client API signatures * s3tables: normalize filer errors and use standard helpers - Migrate from custom ErrNotFound to filer_pb.ErrNotFound - Use filer_pb.LookupEntry for automatic error normalization - Normalize entryExists and attribute lookups * s3tables: harden namespace validation and correct ARN parsing - Prohibit path traversal (".", "..") and "/" in namespaces - Restrict namespace characters to [a-z0-9_] for consistency - Switch to url.PathUnescape for correct decoding of ARN path components - Align ARN parsing regex with single-segment namespace validation * s3tables: improve robustness, security, and error propagation in handlers - Implement strict table name validation (prevention of path traversal and character enforcement) - Add nil checks for entry.Entry in all listing loops to prevent panics - Propagate backend errors instead of swallowing them or assuming 404 - Correctly map filer_pb.ErrNotFound to appropriate S3 error codes - Standardize existence checks across bucket, namespace, and table handlers * test: add miniClusterMutex to prevent race conditions - Introduce sync.Mutex to protect global state (os.Args, os.Chdir) - Ensure serialized initialization of the mini cluster runner - Fix intermittent race conditions during parallel test execution * s3tables: improve error handling and permission logic - Update handleGetNamespace to distinguish between 404 and 500 errors - Refactor CanManagePolicy to use CheckPermission for consistent enforcement - Ensure empty identities are correctly handled in policy management checks * s3tables: optimize regex usage and improve version token uniqueness - Pre-compile regex patterns as package-level variables to avoid re-compilation overhead on every call - Add a random component to version token generation to reduce collision probability under high concurrency * s3tables: harden auth and error handling - Add authorization checks to all S3 Tables handlers (policy, table ops) to enforce security - Improve error handling to distinguish between NotFound (404) and InternalError (500) - Fix directory FileMode usage in filer_ops - Improve test randomness for version tokens - Update permissions comments to acknowledge IAM gaps * S3 Tables: fix gRPC stream loop handling for list operations - Correctly handle io.EOF to terminate loops gracefully. - Propagate other errors to prevent silent failures. - Ensure all list results are processed effectively. * S3 Tables: validate ARN namespace to prevent path traversal - Enforce validation on decoded namespace in parseTableFromARN. - Ensures path components are safe after URL unescaping. * S3 Tables: secure API router with IAM authentication - Wrap S3 Tables handler with authenticateS3Tables. - Use AuthSignatureOnly to enforce valid credentials while delegating granular authorization to handlers. - Prevent anonymous access to all S3 Tables endpoints. * S3 Tables: fix gRPC stream loop handling in namespace handlers - Correctly handle io.EOF in handleListNamespaces and handleDeleteNamespace. - Propagate other errors to prevent silent failures or accidental data loss. - Added necessary io import. * S3 Tables: use os.ModeDir constant in filer_ops.go - Replace magic number 1<<31 with os.ModeDir for better readability. - Added necessary os import. * s3tables: improve principal extraction using identity context * s3tables: remove duplicate comment in permissions.go * s3tables test: improve error reporting on decoding failure * s3tables: implement validateTableName helper * s3tables: add table name validation and 404 propagation to policy handlers * s3tables: add table name validation and cleanup duplicated logic in table handlers * s3tables: ensure root tables directory exists before bucket creation * s3tables: implement token-based pagination for table buckets listing * s3tables: implement token-based pagination for namespace listing * s3tables: refine permission helpers to align with operation names * s3tables: return 404 in handleDeleteNamespace if namespace not found * s3tables: fix cross-namespace pagination in listTablesInAllNamespaces * s3tables test: expose pagination parameters in client list methods * s3tables test: update integration tests for new client API * s3tables: use crypto/rand for secure version token generation Replaced math/rand with crypto/rand to ensure version tokens are cryptographically secure and unpredictable for optimistic concurrency control. * s3tables: improve account ID handling and define missing error codes Updated getPrincipalFromRequest to prioritize X-Amz-Account-ID header and added getAccountID helper. Defined ErrVersionTokenMismatch and ErrCodeConflict for better optimistic concurrency support. * s3tables: update bucket handlers for multi-account support Ensured bucket ownership is correctly attributed to the authenticated account ID and updated ARNs to use the request-derived account ID. Added standard S3 existence checks for bucket deletion. * s3tables: update namespace handlers for multi-account support Updated namespace creation to use authenticated account ID for ownership and unified permission checks across all namespace operations to use the correct account principal. * s3tables: implement optimistic concurrency for table deletion Added VersionToken validation to handleDeleteTable. Refactored table listing to use request context for accurate ARN generation and fixed cross-namespace pagination issues. * s3tables: improve resource resolution and error mapping for policies and tagging Refactored resolveResourcePath to return resource type, enabling accurate NoSuchBucket vs NoSuchTable error codes. Added existence checks before deleting policies. * s3tables: enhance test robustness and resilience Updated random string generation to use crypto/rand in s3tables tests. Increased resilience of IAM distributed tests by adding "connection refused" to retryable errors. * s3tables: remove legacy principal fallback header Removed the fallback to X-Amz-Principal in getPrincipalFromRequest as S3 Tables is a new feature and does not require legacy header support. * s3tables: remove unused ExtractPrincipalFromContext function Removed the unused ExtractPrincipalFromContext utility and its accompanying iam/utils import to keep the new s3tables codebase clean. * s3tables: allow hyphens in namespace and table names Relaxed regex validation in utils.go to support hyphens in S3 Tables namespaces and table names, improving consistency with S3 bucket naming and allowing derived names from services like S3 Storage Lens. * s3tables: add isAuthError helper to handler.go * s3tables: refactor permission checks to use resource owner in bucket handlers * s3tables: refactor permission checks to use resource owner in namespace handlers * s3tables: refactor permission checks to use resource owner in table handlers * s3tables: refactor permission checks to use resource owner in policy and tagging handlers * ownerAccountID * s3tables: implement strict AWS-aligned name validation for buckets, namespaces, and tables * s3tables: enforce strict resource ownership and implement result filtering for buckets * s3tables: enforce strict resource ownership and implement result filtering for namespaces * s3tables: enforce strict resource ownership and implement result filtering for tables * s3tables: align getPrincipalFromRequest with account ID for IAM compatibility * s3tables: fix inconsistent permission check in handleCreateTableBucket * s3tables: improve pagination robustness and error handling in table listing handlers * s3tables: refactor handleDeleteTableBucket to use strongly typed AuthError * s3tables: align ARN regex patterns with S3 standards and refactor to constants * s3tables: standardize access denied errors using ErrAccessDenied constant * go fmt * s3tables: fix double-write issue in handleListTables Remove premature HTTP error writes from within WithFilerClient closure to prevent duplicate status code responses. Error handling is now consistently performed at the top level using isAuthError. * s3tables: update bucket name validation message Remove "underscores" from error message to accurately reflect that bucket names only allow lowercase letters, numbers, and hyphens. * s3tables: add table policy test coverage Add comprehensive test coverage for table policy operations: - Added PutTablePolicy, GetTablePolicy, DeleteTablePolicy methods to test client - Implemented testTablePolicy lifecycle test validating Put/Get/Delete operations - Verified error handling for missing policies * follow aws spec * s3tables: add request body size limiting Add request body size limiting (10MB) to readRequestBody method: - Define maxRequestBodySize constant to prevent unbounded reads - Use io.LimitReader to enforce size limit - Add explicit error handling for oversized requests - Prevents potential DoS attacks via large request bodies * S3 Tables API now properly enforces resource policies addressing the critical security gap where policies were created but never evaluated. * s3tables: Add upper bound validation for MaxTables parameter MaxTables is user-controlled and influences gRPC ListEntries limits via uint32(maxTables*2). Without an upper bound, very large values can overflow uint32 or cause excessively large directory scans. Cap MaxTables to 1000 and return InvalidRequest for out-of-range values, consistent with S3 MaxKeys handling. * s3tables: Add upper bound validation for MaxBuckets parameter MaxBuckets is user-controlled and used in uint32(maxBuckets*2) for ListEntries. Very large values can overflow uint32 or trigger overly expensive scans. Cap MaxBuckets to 1000 and reject out-of-range values, consistent with MaxTables handling and S3 MaxKeys validation elsewhere in the codebase. * s3tables: Validate bucket name in parseBucketNameFromARN() Enforce the same bucket name validation rules (length, characters, reserved prefixes/suffixes) when extracting from ARN. This prevents accepting ARNs that the system would never create and ensures consistency with CreateTableBucket validation. * s3tables: Fix parseTableFromARN() namespace and table name validation - Remove dead URL unescape for namespace (regex [a-z0-9_]+ cannot contain percent-escapes) - Add URL decoding and validation of extracted table name via validateTableName() to prevent callers from bypassing request validation done in other paths * s3tables: Rename tableMetadataInternal.Schema to Metadata The field name 'Schema' was confusing given it holds a *TableMetadata struct and serializes as 'metadata' in JSON. Rename to 'Metadata' for clarity and consistency with the JSON tag and intended meaning. * s3tables: Improve bucket name validation error message Replace misleading character-only error message with generic 'invalid bucket name'. The isValidBucketName() function checks multiple constraints beyond character set (length, reserved prefixes/suffixes, start/end rules), so a specific character message is inaccurate. * s3tables: Separate permission checks for tagging and untagging - Add CanTagResource() to check TagResource permission - Add CanUntagResource() to check UntagResource permission - Update CanManageTags() to check both operations (OR logic) This prevents UntagResource from incorrectly checking 'ManageTags' permission and ensures each operation validates the correct permission when per-operation permissions are enforced. * s3tables: Consolidate getPrincipalFromRequest and getAccountID into single method Both methods had identical implementations - they return the account ID from request header or fall back to handler's default. Remove the duplicate getPrincipalFromRequest and use getAccountID throughout, with updated comment explaining its dual role as both caller identity and principal for permission checks. * s3tables: Fetch bucket policy in handleListTagsForResource for permission evaluation Update handleListTagsForResource to fetch and pass bucket policy to CheckPermission, matching the behavior of handleTagResource/handleUntagResource. This enables bucket-policy-based permission grants to be evaluated for ListTagsForResource, not just ownership-based checks. * s3tables: Extract resource owner and bucket extraction into helper method Create extractResourceOwnerAndBucket() helper to consolidate the repeated pattern of unmarshaling metadata and extracting bucket name from resource path. This pattern was duplicated in handleTagResource, handleListTagsForResource, and handleUntagResource. Update all three handlers to use the helper. Also update remaining uses of getPrincipalFromRequest() (in handler_bucket_create, handler_bucket_get_list_delete, handler_namespace) to use getAccountID() after consolidating the two identical methods. * s3tables: Add log message when cluster shutdown times out The timeout path (2 second wait for graceful shutdown) was silent. Add a warning log message when it occurs to help diagnose flaky test issues and indicate when the mini cluster didn't shut down cleanly. * s3tables: Use policy_engine wildcard matcher for complete IAM compatibility Replace the custom suffix-only wildcard implementation in matchesActionPattern and matchesPrincipal with the policy_engine.MatchesWildcard function from PR #8052. This enables full wildcard support including: - Middle wildcards: s3tables:Get*Table matches GetTable - Question mark wildcards: Get? matches any single character - Combined patterns: s3tables:*Table* matches any action containing 'Table' Benefits: - Code reuse: eliminates duplicate wildcard logic - Complete IAM compatibility: supports all AWS wildcard patterns - Performance: uses efficient O(n) backtracking algorithm - Consistency: same wildcard behavior across S3 API and S3 Tables Add comprehensive unit tests covering exact matches, suffix wildcards, middle wildcards, question marks, and combined patterns for both action and principal matching. * go fmt * s3tables: Fix vet error - remove undefined c.t reference in Stop() The TestCluster.Stop() method doesn't have access to testing.T object. Remove the log statement and keep the timeout handling comment for clarity. The original intent (warning about shutdown timeout) is still captured in the code comment explaining potential issues. * clean up * s3tables: Add t field to TestCluster for logging Add *testing.T field to TestCluster struct and initialize it in startMiniCluster. This allows Stop() to properly log warnings when cluster shutdown times out. Includes the t field in the test cluster initialization and restores the logging statement in Stop(). * s3tables: Fix bucket policy error handling in permission checks Replace error-swallowing pattern where all errors from getExtendedAttribute were ignored for bucket policy reads. Now properly distinguish between: - ErrAttributeNotFound: Policy not found is expected; continue with empty policy - Other errors: Return internal server error and stop processing Applied fix to all bucket policy reads in: - handleDeleteTableBucketPolicy (line 220) - handleTagResource (line 313) - handleUntagResource (line 405) - handleListTagsForResource (line 488) - And additional occurrences in closures This prevents silent failures and ensures policy-related errors are surfaced to callers rather than being silently ignored. * s3tables: Pre-validate namespace to return 400 instead of 500 Move validateNamespace call outside of filerClient.WithFilerClient closure so that validation errors return HTTP 400 (InvalidRequest) instead of 500 (InternalError). Before: Validation error inside closure → treated as internal error → 500 After: Validation error before closure → handled as bad request → 400 This provides correct error semantics: namespace validation is an input validation issue, not a server error. * Update weed/s3api/s3tables/handler.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * s3tables: Normalize action names to include service prefix Add automatic normalization of operations to full IAM-style action names (e.g., 's3tables:CreateTableBucket') in CheckPermission(). This ensures policy statements using prefixed actions (s3tables:*) correctly match operations evaluated by permission helpers. Also fixes incorrect r.Context() passed to GetIdentityNameFromContext which expects *http.Request. Now passes r directly. * s3tables: Use policy framework for table creation authorization Replace strict ownership check in CreateTable with policy-based authorization. Now checks both namespace and bucket policies for CreateTable permission, allowing delegation via resource policies while still respecting owner bypass. Authorization logic: - Namespace policy grants CreateTable → allowed - Bucket policy grants CreateTable → allowed - Otherwise → denied (even if same owner) This enables cross-principal table creation via policies while maintaining security through explicit allow/deny semantics. * s3tables: Use policy framework for GetTable authorization Replace strict ownership check with policy-based authorization in GetTable. Now checks both table and bucket policies for GetTable permission, allowing authorized non-owners to read table metadata. Authorization logic: - Table policy grants GetTable → allowed - Bucket policy grants GetTable → allowed - Otherwise → 404 NotFound (no access disclosed) Maintains security through policy evaluation while enabling read delegation. * s3tables: Generate ARNs using resource owner account ID Change ARN generation to use resource OwnerAccountID instead of caller identity (h.getAccountID(r)). This ensures ARNs are stable and consistent regardless of which principal accesses the resource. Updated generateTableBucketARN and generateTableARN function signatures to accept ownerAccountID parameter. All call sites updated to pass the resource owner's account ID from metadata. This prevents ARN inconsistency issues when multiple principals have access to the same resource via policies. * s3tables: Fix remaining policy error handling in namespace and bucket handlers Replace silent error swallowing (err == nil) with proper error distinction for bucket policy reads. Now properly checks ErrAttributeNotFound and propagates other errors as internal server errors. Fixed 5 locations: - handleCreateNamespace (policy fetch) - handleDeleteNamespace (policy fetch) - handleListNamespaces (policy fetch) - handleGetNamespace (policy fetch) - handleGetTableBucket (policy fetch) This prevents masking of filer issues when policies cannot be read due to I/O errors or other transient failures. * ci: Pin GitHub Actions to commit SHAs for s3-tables-tests Update all action refs to use pinned commit SHAs instead of floating tags: - actions/checkout: @v6 → @8e8c483 (v4) - actions/setup-go: @v6 → @0c52d54 (v5) - actions/upload-artifact: @v6 → @65d8626 (v4) Pinned SHAs improve reproducibility and reduce supply chain risk by preventing accidental or malicious changes in action releases. Aligns with repository conventions used in other workflows (e.g., go.yml). * s3tables: Add resource ARN validation to policy evaluation Implement resource-specific policy validation to prevent over-broad permission grants. Add matchesResource and matchesResourcePattern functions to validate statement Resource fields against specific resource ARNs. Add new CheckPermissionWithResource function that includes resource ARN validation, while keeping CheckPermission unchanged for backward compatibility. This enables policies to grant access to specific resources only: - statements with Resource: "arn:aws:s3tables:...:bucket/specific-bucket/*" will only match when accessing that specific bucket - statements without Resource field match all resources (implicit *) - resource patterns support wildcards (* for any sequence, ? for single char) For future use: Handlers can call CheckPermissionWithResource with the target resource ARN to enforce resource-level access control. * Revert "ci: Pin GitHub Actions to commit SHAs for s3-tables-tests" This reverts commit 01da26fbcb. * s3tables: Remove duplicate bucket extraction logic in helper Move bucket name extraction outside the if/else block in extractResourceOwnerAndBucket since the logic is identical for both ResourceTypeTable and ResourceTypeBucket cases. This reduces code duplication and improves maintainability. The extraction pattern (parts[1] from /tables/{bucket}/...) works for both resource types, so it's now performed once before the type-specific metadata unmarshaling. * go fmt * s3tables: Fix ownership consistency across handlers Address three related ownership consistency issues: 1. CreateNamespace now sets OwnerAccountID to bucketMetadata.OwnerAccountID instead of request principal. This prevents namespaces created by delegated callers (via bucket policy) from becoming unmanageable, since ListNamespaces filters by bucket owner. 2. CreateTable now: - Fetches bucket metadata to use correct owner for bucket policy evaluation - Uses namespaceMetadata.OwnerAccountID for namespace policy checks - Uses bucketMetadata.OwnerAccountID for bucket policy checks - Sets table OwnerAccountID to namespaceMetadata.OwnerAccountID (inherited) 3. GetTable now: - Fetches bucket metadata to use correct owner for bucket policy evaluation - Uses metadata.OwnerAccountID for table policy checks - Uses bucketMetadata.OwnerAccountID for bucket policy checks This ensures: - Bucket owner retains implicit "owner always allowed" behavior even when evaluating bucket policies - Ownership hierarchy is consistent (namespace owned by bucket, table owned by namespace) - Cross-principal delegation via policies doesn't break ownership chains * s3tables: Fix ListTables authorization and policy parsing Make ListTables authorization consistent with GetTable/CreateTable: 1. ListTables authorization now evaluates policies instead of owner-only checks: - For namespace listing: checks namespace policy AND bucket policy - For bucket-wide listing: checks bucket policy - Uses CanListTables permission framework 2. Remove owner-only filter in listTablesWithClient that prevented policy-based sharing of tables. Authorization is now enforced at the handler level, so all tables in the namespace/bucket are returned to authorized callers (who have access either via ownership or policy). 3. Add flexible PolicyDocument.UnmarshalJSON to support both single-object and array forms of Statement field: - Handles: {"Statement": {...}} - Handles: {"Statement": [{...}, {...}]} - Improves AWS IAM compatibility This ensures cross-account table listing works when delegated via bucket/namespace policies, consistent with the authorization model for other operations. * go fmt * s3tables: Separate table name pattern constant for clarity Define a separate tableNamePatternStr constant for the table name component in the ARN regex, even though it currently has the same value as tableNamespacePatternStr. This improves code clarity and maintainability, making it easier to modify if the naming rules for tables and namespaces diverge in the future. * refactor --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	2 weeks ago
Chris Lu	fe856928c4	s3tables: Add t field to TestCluster for logging Add *testing.T field to TestCluster struct and initialize it in startMiniCluster. This allows Stop() to properly log warnings when cluster shutdown times out. Includes the t field in the test cluster initialization and restores the logging statement in Stop().	2 weeks ago
Chris Lu	c5eadadf5a	s3tables: Fix vet error - remove undefined c.t reference in Stop() The TestCluster.Stop() method doesn't have access to testing.T object. Remove the log statement and keep the timeout handling comment for clarity. The original intent (warning about shutdown timeout) is still captured in the code comment explaining potential issues.	2 weeks ago
Chris Lu	dbf6465b0e	s3tables: Add log message when cluster shutdown times out The timeout path (2 second wait for graceful shutdown) was silent. Add a warning log message when it occurs to help diagnose flaky test issues and indicate when the mini cluster didn't shut down cleanly.	2 weeks ago
Chris Lu	473e699368	s3tables: add table policy test coverage Add comprehensive test coverage for table policy operations: - Added PutTablePolicy, GetTablePolicy, DeleteTablePolicy methods to test client - Implemented testTablePolicy lifecycle test validating Put/Get/Delete operations - Verified error handling for missing policies	2 weeks ago
Chris Lu	dfdace9a13	s3tables: enhance test robustness and resilience Updated random string generation to use crypto/rand in s3tables tests. Increased resilience of IAM distributed tests by adding "connection refused" to retryable errors.	2 weeks ago
Chris Lu	89b85bfd5e	s3tables test: update integration tests for new client API	2 weeks ago
Chris Lu	783fe25eac	s3tables test: expose pagination parameters in client list methods	2 weeks ago
Chris Lu	9f1dd57939	s3tables test: improve error reporting on decoding failure	2 weeks ago
Chris Lu	dc4c62e742	s3tables: harden auth and error handling - Add authorization checks to all S3 Tables handlers (policy, table ops) to enforce security - Improve error handling to distinguish between NotFound (404) and InternalError (500) - Fix directory FileMode usage in filer_ops - Improve test randomness for version tokens - Update permissions comments to acknowledge IAM gaps	2 weeks ago
Chris Lu	6fc170c645	test: add miniClusterMutex to prevent race conditions - Introduce sync.Mutex to protect global state (os.Args, os.Chdir) - Ensure serialized initialization of the mini cluster runner - Fix intermittent race conditions during parallel test execution	2 weeks ago
Chris Lu	1d1634c2a2	test: update integration tests to match refactored S3 Tables client - Pass namespaces as []string to support hierarchical structures - Adapt test calls to new client API signatures	2 weeks ago
Chris Lu	44f580c24e	test: refactor S3 Tables client for DRYness and multi-segment namespaces - Implement doRequestAndDecode to eliminate HTTP boilerplate - Update client API to accept []string for namespaces to support hierarchy - Standardize error response decoding across all client methods	2 weeks ago
Chris Lu	01c17478ae	command: implement graceful shutdown for mini cluster - Introduce MiniClusterCtx to coordinate shutdown across mini services - Update Master, Volume, Filer, S3, and WebDAV servers to respect context cancellation - Ensure all resources are cleaned up properly during test teardown - Integrate MiniClusterCtx in s3tables integration tests	2 weeks ago
Chris Lu	07002cf54c	test: improve S3 Tables client error handling and cleanup - Add detailed error reporting when decoding failure responses - Remove orphaned comments and unused sections	2 weeks ago
Chris Lu	33da87452b	Refine S3 Tables implementation to address code review feedback - Standardize namespace representation to []string - Improve listing logic with pagination and StartFromFileName - Enhance error handling with sentinel errors and robust checks - Add JSON encoding error logging - Fix CI workflow to use gofmt -l - Standardize timestamps in directory creation - Validate single-level namespaces	2 weeks ago
Chris Lu	6cdd34da77	s3tables: improve integration test stability and error reporting	2 weeks ago
Chris Lu	33c1a8251a	test: format s3tables client.go	2 weeks ago
Chris Lu	22a9fbf062	Delete client_methods.go	2 weeks ago
Chris Lu	06f90028f7	s3tables test: refactor to eliminate duplicate definitions - Move all client methods to client.go - Remove duplicate types/constants from s3tables_integration_test.go - Keep setup.go for test infrastructure - Keep integration test logic in s3tables_integration_test.go - Clean up unused imports - Test compiles successfully	2 weeks ago
Chris Lu	3b1920cf43	s3tables: add handler_ prefix to operation handler files - Rename bucket_create.go → handler_bucket_create.go - Rename bucket_get_list_delete.go → handler_bucket_get_list_delete.go - Rename namespace.go → handler_namespace.go - Rename table.go → handler_table.go - Rename policy.go → handler_policy.go Improves file organization by clearly identifying handler implementations. No code changes, refactoring only.	2 weeks ago
Chris Lu	ddcdefb6b4	test: add S3 Tables integration tests - Comprehensive integration tests for all 23 S3 Tables operations - Test cluster setup based on existing S3 integration tests - Tests cover: * Table bucket lifecycle (create, get, list, delete) * Namespace operations * Table CRUD with Iceberg schema * Table bucket and table policies * Resource tagging operations - Ready for CI/CD pipeline integration	2 weeks ago
Chris Lu	d66e194284	test: add S3 Tables test infrastructure - Create setup.go with TestCluster and S3TablesClient definitions - Create client.go with HTTP client methods for all operations - Test utilities and client methods organized for reusability - Foundation for S3 Tables integration tests	2 weeks ago
Chris Lu	551a31e156	Implement IAM propagation to S3 servers (#8130) * Implement IAM propagation to S3 servers - Add PropagatingCredentialStore to propagate IAM changes to S3 servers via gRPC - Add Policy management RPCs to S3 proto and S3ApiServer - Update CredentialManager to use PropagatingCredentialStore when MasterClient is available - Wire FilerServer to enable propagation * Implement parallel IAM propagation and fix S3 cluster registration - Parallelized IAM change propagation with 10s timeout. - Refined context usage in PropagatingCredentialStore. - Added S3Type support to cluster node management. - Enabled S3 servers to register with gRPC address to the master. - Ensured IAM configuration reload after policy updates via gRPC. * Optimize IAM propagation with direct in-memory cache updates * Secure IAM propagation: Use metadata to skip persistence only on propagation * pb: refactor IAM and S3 services for unidirectional IAM propagation - Move SeaweedS3IamCache service from iam.proto to s3.proto. - Remove legacy IAM management RPCs and empty SeaweedS3 service from s3.proto. - Enforce that S3 servers only use the synchronization interface. * pb: regenerate Go code for IAM and S3 services Updated generated code following the proto refactoring of IAM synchronization services. * s3api: implement read-only mode for Embedded IAM API - Add readOnly flag to EmbeddedIamApi to reject write operations via HTTP. - Enable read-only mode by default in S3ApiServer. - Handle AccessDenied error in writeIamErrorResponse. - Embed SeaweedS3IamCacheServer in S3ApiServer. * credential: refactor PropagatingCredentialStore for unidirectional IAM flow - Update to use s3_pb.SeaweedS3IamCacheClient for propagation to S3 servers. - Propagate full Identity object via PutIdentity for consistency. - Remove redundant propagation of specific user/account/policy management RPCs. - Add timeout context for propagation calls. * s3api: implement SeaweedS3IamCacheServer for unidirectional sync - Update S3ApiServer to implement the cache synchronization gRPC interface. - Methods (PutIdentity, RemoveIdentity, etc.) now perform direct in-memory cache updates. - Register SeaweedS3IamCacheServer in command/s3.go. - Remove registration for the legacy and now empty SeaweedS3 service. * s3api: update tests for read-only IAM and propagation - Added TestEmbeddedIamReadOnly to verify rejection of write operations in read-only mode. - Update test setup to pass readOnly=false to NewEmbeddedIamApi in routing tests. - Updated EmbeddedIamApiForTest helper with read-only checks matching production behavior. * s3api: add back temporary debug logs for IAM updates Log IAM updates received via: - gRPC propagation (PutIdentity, PutPolicy, etc.) - Metadata configuration reloads (LoadS3ApiConfigurationFromCredentialManager) - Core identity management (UpsertIdentity, RemoveIdentity) * IAM: finalize propagation fix with reduced logging and clarified architecture * Allow configuring IAM read-only mode for S3 server integration tests * s3api: add defensive validation to UpsertIdentity * s3api: fix log message to reference correct IAM read-only flag * test/s3/iam: ensure WaitForS3Service checks for IAM write permissions * test: enable writable IAM in Makefile for integration tests * IAM: add GetPolicy/ListPolicies RPCs to s3.proto * S3: add GetBucketPolicy and ListBucketPolicies helpers * S3: support storing generic IAM policies in IdentityAccessManagement * S3: implement IAM policy RPCs using IdentityAccessManagement * IAM: fix stale user identity on rename propagation	2 weeks ago
Chris Lu	43229b05ce	Explicit IAM gRPC APIs for S3 Server (#8126) * Update IAM and S3 protobuf definitions for explicit IAM gRPC APIs * Refactor s3api: Extract generic ExecuteAction method for IAM operations * Implement explicit IAM gRPC APIs in S3 server * iam: remove deprecated GetConfiguration and PutConfiguration RPCs * iamapi: refactor handlers to use CredentialManager directly * s3api: refactor embedded IAM to use CredentialManager directly * server: remove deprecated configuration gRPC handlers * credential/grpc: refactor configuration calls to return error * shell: update s3.configure to list users instead of full config * s3api: fix CreateServiceAccount gRPC handler to map required fields * s3api: fix UpdateServiceAccount gRPC handler to map fields and safe status * s3api: enforce UserName in embedded IAM ListAccessKeys * test: fix test_config.json structure to match proto definition * Revert "credential/grpc: refactor configuration calls to return error" This reverts commit cde707dd8b. * Revert "server: remove deprecated configuration gRPC handlers" This reverts commit 7307e205a0. * Revert "s3api: enforce UserName in embedded IAM ListAccessKeys" This reverts commit adf727ba52. * Revert "s3api: fix UpdateServiceAccount gRPC handler to map fields and safe status" This reverts commit 6a4be3314d. * Revert "s3api: fix CreateServiceAccount gRPC handler to map required fields" This reverts commit 9bb4425f07. * Revert "shell: update s3.configure to list users instead of full config" This reverts commit f3304ead53. * Revert "s3api: refactor embedded IAM to use CredentialManager directly" This reverts commit 9012f27af8. * Revert "iamapi: refactor handlers to use CredentialManager directly" This reverts commit 3a14821223. * Revert "iam: remove deprecated GetConfiguration and PutConfiguration RPCs" This reverts commit e16e08aa00. * s3api: address IAM code review comments (error handling, logging, gRPC response mapping) * s3api: add robustness to startup by retrying KEK and IAM config loading from Filer * s3api: address IAM gRPC code review comments (safety, validation, status logic) * fix return	2 weeks ago
Chris Lu	6bf088cec9	IAM Policy Management via gRPC (#8109) * Add IAM gRPC service definition - Add GetConfiguration/PutConfiguration for config management - Add CreateUser/GetUser/UpdateUser/DeleteUser/ListUsers for user management - Add CreateAccessKey/DeleteAccessKey/GetUserByAccessKey for access key management - Methods mirror existing IAM HTTP API functionality * Add IAM gRPC handlers on filer server - Implement IamGrpcServer with CredentialManager integration - Handle configuration get/put operations - Handle user CRUD operations - Handle access key create/delete operations - All methods delegate to CredentialManager for actual storage * Wire IAM gRPC service to filer server - Add CredentialManager field to FilerOption and FilerServer - Import credential store implementations in filer command - Initialize CredentialManager from credential.toml if available - Register IAM gRPC service on filer gRPC server - Enable credential management via gRPC alongside existing filer services * Regenerate IAM protobuf with gRPC service methods * iam_pb: add Policy Management to protobuf definitions * credential: implement PolicyManager in credential stores * filer: implement IAM Policy Management RPCs * shell: add s3.policy command * test: add integration test for s3.policy * test: fix compilation errors in policy_test * pb * fmt * test * weed shell: add -policies flag to s3.configure This allows linking/unlinking IAM policies to/from identities directly from the s3.configure command. * test: verify s3.configure policy linking and fix port allocation - Added test case for linking policies to users via s3.configure - Implemented findAvailablePortPair to ensure HTTP and gRPC ports are both available, avoiding conflicts with randomized port assignments. - Updated assertion to match jsonpb output (policyNames) * credential: add StoreTypeGrpc constant * credential: add IAM gRPC store boilerplate * credential: implement identity methods in gRPC store * credential: implement policy methods in gRPC store * admin: use gRPC credential store for AdminServer This ensures that all IAM and policy changes made through the Admin UI are persisted via the Filer's IAM gRPC service instead of direct file manipulation. * shell: s3.configure use granular IAM gRPC APIs instead of full config patching * shell: s3.configure use granular IAM gRPC APIs * shell: replace deprecated ioutil with os in s3.policy * filer: use gRPC FailedPrecondition for unconfigured credential manager * test: improve s3.policy integration tests and fix error checks * ci: add s3 policy shell integration tests to github workflow * filer: fix LoadCredentialConfiguration error handling * credential/grpc: propagate unmarshal errors in GetPolicies * filer/grpc: improve error handling and validation * shell: use gRPC status codes in s3.configure * credential: document PutPolicy as create-or-replace * credential/postgres: reuse CreatePolicy in PutPolicy to deduplicate logic * shell: add timeout context and strictly enforce flags in s3.policy * iam: standardize policy content field naming in gRPC and proto * shell: extract slice helper functions in s3.configure * filer: map credential store errors to gRPC status codes * filer: add input validation for UpdateUser and CreateAccessKey * iam: improve validation in policy and config handlers * filer: ensure IAM service registration by defaulting credential manager * credential: add GetStoreName method to manager * test: verify policy deletion in integration test	3 weeks ago
Chris Lu	535be3096b	Add AWS IAM integration tests and refactor admin authorization (#8098) * Add AWS IAM integration tests and refactor admin authorization - Added AWS IAM management integration tests (User, AccessKey, Policy) - Updated test framework to support IAM client creation with JWT/OIDC - Refactored s3api authorization to be policy-driven for IAM actions - Removed hardcoded role name checks for admin privileges - Added new tests to GitHub Actions basic test matrix * test(s3/iam): add UpdateUser and UpdateAccessKey tests and fix nil pointer dereference * feat(s3api): add DeletePolicy and update tests with cleanup logic * test(s3/iam): use t.Cleanup for managed policy deletion in CreatePolicy test	3 weeks ago
Chris Lu	d664ca5ed3	fix: IAM authentication with AWS Signature V4 and environment credentials (#8099) * fix: IAM authentication with AWS Signature V4 and environment credentials Three key fixes for authenticated IAM requests to work: 1. Fix request body consumption before signature verification - iamMatcher was calling r.ParseForm() which consumed POST body - This broke AWS Signature V4 verification on subsequent reads - Now only check query string in matcher, preserving body for verification - File: weed/s3api/s3api_server.go 2. Preserve environment variable credentials across config reloads - After IAM mutations, config reload overwrote env var credentials - Extract env var loading into loadEnvironmentVariableCredentials() - Call after every config reload to persist credentials - File: weed/s3api/auth_credentials.go 3. Add authenticated IAM tests and test infrastructure - New TestIAMAuthenticated suite with AWS SDK + Signature V4 - Dynamic port allocation for independent test execution - Flag reset to prevent state leakage between tests - CI workflow to run S3 and IAM tests separately - Files: test/s3/example/*, .github/workflows/s3-example-integration-tests.yml All tests pass: - TestIAMCreateUser (unauthenticated) - TestIAMAuthenticated (with AWS Signature V4) - S3 integration tests * fmt * chore: rename test/s3/example to test/s3/normal * simplify: CI runs all integration tests in single job * Update s3-example-integration-tests.yml * ci: run each test group separately to avoid raft registry conflicts	3 weeks ago
Chris Lu	14f44379cb	test: fix flaky S3 volume encryption test (#8083) Specifically: - Use bytes.NewReader for binary data instead of strings.NewReader - Increase binary test data from 8 bytes to 1KB to avoid edge cases - Add 50ms delay between subtests to prevent overwhelming the server	3 weeks ago
Chris Lu	51735e667c	Fix S3 conditional writes with versioning (Issue #8073) (#8080) * Fix S3 conditional writes with versioning (Issue #8073) Refactors conditional header checks to properly resolve the latest object version when versioning is enabled. This prevents incorrect validation against non-versioned root objects. * Add integration test for S3 conditional writes with versioning (Issue #8073) * Refactor: Propagate internal errors in conditional header checks - Make resolveObjectEntry return errors from isVersioningConfigured - Update checkConditionalHeaders checks to return 500 on internal resolve errors * Refactor: Stricter error handling and test assertions - Propagate internal errors in checkConditionalHeaders*WithGetter functions - Enforce strict 412 PreconditionFailed check in integration test * Perf: Add early return for conditional headers + safety improvements - Add fast path to skip resolveObjectEntry when no conditional headers present - Avoids expensive getLatestObjectVersion retries in common case - Add nil checks before dereferencing pointers in integration test - Fix grammar in test comments - Remove duplicate comment in resolveObjectEntry * Refactor: Use errors.Is for robust ErrNotFound checking - Update checkConditionalHeaders* to use errors.Is(err, filer_pb.ErrNotFound) - Update resolveObjectEntry to use errors.Is for wrapped error compatibility - Remove duplicate comment lines in s3api handlers * Perf: Optimize resolveObjectEntry for conditional checks - Refactor getLatestObjectVersion to doGetLatestObjectVersion supporting variable retries - Use 1-retry path in resolveObjectEntry to avoid exponential backoff latency * Test: Enhance integration test with content verification - Verify actual object content equals expected content after successful conditional write - Add missing io and errors imports to test file * Refactor: Final refinements based on feedback - Optimize header validation by passing parsed headers to avoid redundant parsing - Simplify integration test assertions using require.Error and assert.True - Fix build errors in s3api handler and test imports * Test: Use smithy.APIError for robust error code checking - Replace string-based error checking with structured API error - Add smithy-go import for AWS SDK v2 error handling * Test: Use types.PreconditionFailed and handle io.ReadAll error - Replace smithy.APIError with more specific types.PreconditionFailed - Add proper error handling for io.ReadAll in content verification * Refactor: Use combined error checking and add nil guards - Use smithy.APIError with ErrorCode() for robust error checking - Add nil guards for entry.Attributes before accessing Mtime - Prevents potential panics when Attributes is uninitialized	3 weeks ago
Chris Lu	13dcf445a4	Fix maintenance worker panic and add EC integration tests (#8068) * Fix nil pointer panic in maintenance worker when receiving empty task assignment When a worker requests a task and none are available, the admin server sends an empty TaskAssignment message. The worker was attempting to log the task details without checking if the TaskId was empty, causing a nil pointer dereference when accessing taskAssign.Params.VolumeId. This fix adds a check for empty TaskId before processing the assignment, preventing worker crashes and improving stability in production environments. * Add EC integration test for admin-worker maintenance system Adds comprehensive integration test that verifies the end-to-end flow of erasure coding maintenance tasks: - Admin server detects volumes needing EC encoding - Workers register and receive task assignments - EC encoding is executed and verified in master topology - File read-back validation confirms data integrity The test uses unique absolute working directories for each worker to prevent ID conflicts and ensure stable worker registration. Includes proper cleanup and process management for reliable test execution. * Improve maintenance system stability and task deduplication - Add cross-type task deduplication to prevent concurrent maintenance operations on the same volume (EC, balance, vacuum) - Implement HasAnyTask check in ActiveTopology for better coordination - Increase RequestTask timeout from 5s to 30s to prevent unnecessary worker reconnections - Add TaskTypeNone sentinel for generic task checks - Update all task detectors to use HasAnyTask for conflict prevention - Improve config persistence and schema handling * Add GitHub Actions workflow for EC integration tests Adds CI workflow that runs EC integration tests on push and pull requests to master branch. The workflow: - Triggers on changes to admin, worker, or test files - Builds the weed binary - Runs the EC integration test suite - Uploads test logs as artifacts on failure for debugging This ensures the maintenance system remains stable and worker-admin integration is validated in CI. * go version 1.24 * address comments * Update maintenance_integration.go * support seconds * ec prioritize over balancing in tests	3 weeks ago
madavic	86c61e86c9	fix: S3 copying test Makefile syntax and add S3_ENDPOINT env support (#8042) * fix: S3 copying test Makefile syntax and add S3_ENDPOINT env support * fix: add weed mini to stop-seaweedfs target Ensure weed mini process is properly killed when stopping SeaweedFS, matching the process started in start-seaweedfs target. * Clean up PID file in stop-seaweedfs and clean targets Address review feedback to ensure /tmp/weed-mini.pid is removed for a clean state after tests.	4 weeks ago
Chris Lu	ee3813787e	feat(s3api): Implement S3 Policy Variables (#8039) * feat: Add AWS IAM Policy Variables support to S3 API Implements policy variables for dynamic access control in bucket policies. Supported variables: - aws:username - Extracted from principal ARN - aws:userid - User identifier (same as username in SeaweedFS) - aws:principaltype - IAMUser, IAMRole, or AssumedRole - jwt:* - Any JWT claim (e.g., jwt:preferred_username, jwt:sub) Key changes: - Added PolicyVariableRegex to detect ${...} patterns - Extended CompiledStatement with DynamicResourcePatterns, DynamicPrincipalPatterns, DynamicActionPatterns - Added Claims field to PolicyEvaluationArgs for JWT claim access - Implemented SubstituteVariables() for variable replacement from context and JWT claims - Implemented extractPrincipalVariables() for ARN parsing - Updated EvaluateConditions() to support variable substitution - Comprehensive unit and integration tests Resolves #8037 * feat: Add LDAP and PrincipalAccount variable support Completes future enhancements for policy variables: - Added ldap:* variable support for LDAP claims - ldap:username - LDAP username from claims - ldap:dn - LDAP distinguished name from claims - ldap:* - Any LDAP claim - Added aws:PrincipalAccount extraction from ARN - Extracts account ID from principal ARN - Available as ${aws:PrincipalAccount} in policies Updated SubstituteVariables() to check LDAP claims Updated extractPrincipalVariables() to extract account ID Added comprehensive tests for new variables * feat(s3api): implement IAM policy variables core logic and optimization * feat(s3api): integrate policy variables with S3 authentication and handlers * test(s3api): add integration tests for policy variables * cleanup: remove unused policy conversion files * Add S3 policy variables integration tests and path support - Add comprehensive integration tests for policy variables - Test username isolation, JWT claims, LDAP claims - Add support for IAM paths in principal ARN parsing - Add tests for principals with paths * Fix IAM Role principal variable extraction IAM Roles should not have aws:userid or aws:PrincipalAccount according to AWS behavior. Only IAM Users and Assumed Roles should have these variables. Fixes TestExtractPrincipalVariables test failures. * Security fixes and bug fixes for S3 policy variables SECURITY FIXES: - Prevent X-SeaweedFS-Principal header spoofing by clearing internal headers at start of authentication (auth_credentials.go) - Restrict policy variable substitution to safe allowlist to prevent client header injection (iam/policy/policy_engine.go) - Add core policy validation before storing bucket policies BUG FIXES: - Remove unused sid variable in evaluateStatement - Fix LDAP claim lookup to check both prefixed and unprefixed keys - Add ValidatePolicy call in PutBucketPolicyHandler These fixes prevent privilege escalation via header injection and ensure only validated identity claims are used in policy evaluation. * Additional security fixes and code cleanup SECURITY FIXES: - Fixed X-Forwarded-For spoofing by only trusting proxy headers from private/localhost IPs (s3_iam_middleware.go) - Changed context key from "sourceIP" to "aws:SourceIp" for proper policy variable substitution CODE IMPROVEMENTS: - Kept aws:PrincipalAccount for IAM Roles to support condition evaluations - Removed redundant STS principaltype override - Removed unused service variable - Cleaned up commented-out debug logging statements - Updated tests to reflect new IAM Role behavior These changes prevent IP spoofing attacks and ensure policy variables work correctly with the safe allowlist. * Add security documentation for ParseJWTToken Added comprehensive security comments explaining that ParseJWTToken is safe despite parsing without verification because: - It's only used for routing to the correct verification method - All code paths perform cryptographic verification before trusting claims - OIDC tokens: validated via validateExternalOIDCToken - STS tokens: validated via ValidateSessionToken Enhanced function documentation with clear security warnings about proper usage to prevent future misuse. * Fix IP condition evaluation to use aws:SourceIp key Fixed evaluateIPCondition in IAM policy engine to use "aws:SourceIp" instead of "sourceIP" to match the updated extractRequestContext. This fixes the failing IP-restricted role test where IP-based policy conditions were not being evaluated correctly. Updated all test cases to use the correct "aws:SourceIp" key. * Address code review feedback: optimize and clarify PERFORMANCE IMPROVEMENT: - Optimized expandPolicyVariables to use regexp.ReplaceAllStringFunc for single-pass variable substitution instead of iterating through all safe variables. This improves performance from O(n*m) to O(m) where n is the number of safe variables and m is the pattern length. CODE CLARITY: - Added detailed comment explaining LDAP claim fallback mechanism (checks both prefixed and unprefixed keys for compatibility) - Enhanced TODO comment for trusted proxy configuration with rationale and recommendations for supporting cloud load balancers, CDNs, and complex network topologies All tests passing. * Address Copilot code review feedback BUG FIXES: - Fixed type switch for int/int32/int64 - separated into individual cases since interface type switches only match the first type in multi-type cases - Fixed grammatically incorrect error message in types.go CODE QUALITY: - Removed duplicate Resource/NotResource validation (already in ValidateStatement) - Added comprehensive comment explaining isEnabled() logic and security implications - Improved trusted proxy NOTE comment to be more concise while noting limitations All tests passing. * Fix test failures after extractSourceIP security changes Updated tests to work with the security fix that only trusts X-Forwarded-For/X-Real-IP headers from private IP addresses: - Set RemoteAddr to 127.0.0.1 in tests to simulate trusted proxy - Changed context key from "sourceIP" to "aws:SourceIp" - Added test case for untrusted proxy (public RemoteAddr) - Removed invalid ValidateStatement call (validation happens in ValidatePolicy) All tests now passing. * Address remaining Gemini code review feedback CODE SAFETY: - Deep clone Action field in CompileStatement to prevent potential data races if the original policy document is modified after compilation TEST CLEANUP: - Remove debug logging (fmt.Fprintf) from engine_notresource_test.go - Remove unused imports in engine_notresource_test.go All tests passing. * Fix insecure JWT parsing in IAM auth flow SECURITY FIX: - Renamed ParseJWTToken to ParseUnverifiedJWTToken with explicit security warnings. - Refactored AuthenticateJWT to use the trusted SessionInfo returned by ValidateSessionToken instead of relying on unverified claims from the initial parse. - Refactored ValidatePresignedURLWithIAM to reuse the robust AuthenticateJWT logic, removing duplicated and insecure manual token parsing. This ensures all identity information (Role, Principal, Subject) used for authorization decisions is derived solely from cryptographically verified tokens. * Security: Fix insecure JWT claim extraction in policy engine - Refactored EvaluatePolicy to accept trusted claims from verified Identity instead of parsing unverified tokens - Updated AuthenticateJWT to populate Claims in IAMIdentity from verified sources (SessionInfo/ExternalIdentity) - Updated s3api_server and handlers to pass claims correctly - Improved isPrivateIP to support IPv6 loopback, link-local, and ULA - Fixed flaky distributed_session_consistency test with retry logic * fix(iam): populate Subject in STSSessionInfo to ensure correct identity propagation This fixes the TestS3IAMAuthentication/valid_jwt_token_authentication failure by ensuring the session subject (sub) is correctly mapped to the internal SessionInfo struct, allowing bucket ownership validation to succeed. * Optimized isPrivateIP * Create s3-policy-tests.yml * fix tests * fix tests * tests(s3/iam): simplify policy to resource-based \ (step 1) * tests(s3/iam): add explicit Deny NotResource for isolation (step 2) * fixes * policy: skip resource matching for STS trust policies to allow AssumeRole evaluation * refactor: remove debug logging and hoist policy variables for performance * test: fix TestS3IAMBucketPolicyIntegration cleanup to handle per-subtest object lifecycle * test: fix bucket name generation to comply with S3 63-char limit * test: skip TestS3IAMPolicyEnforcement until role setup is implemented * test: use weed mini for simpler test server deployment Replace 'weed server' with 'weed mini' for IAM tests to avoid port binding issues and simplify the all-in-one server deployment. This improves test reliability and execution time. * security: prevent allocation overflow in policy evaluation Add maxPoliciesForEvaluation constant to cap the number of policies evaluated in a single request. This prevents potential integer overflow when allocating slices for policy lists that may be influenced by untrusted input. Changes: - Add const maxPoliciesForEvaluation = 1024 to set an upper bound - Validate len(policies) < maxPoliciesForEvaluation before appending bucket policy - Use append() instead of make([]string, len+1) to avoid arithmetic overflow - Apply fix to both IsActionAllowed policy evaluation paths	4 weeks ago
Chris Lu	905e7e72d9	Add remote.copy.local command to copy local files to remote storage (#8033) * Add remote.copy.local command to copy local files to remote storage This new command solves the issue described in GitHub Discussion #8031 where files exist locally but are not synced to remote storage due to missing filer logs. Features: - Copies local-only files to remote storage - Supports file filtering (include/exclude patterns) - Dry run mode to preview actions - Configurable concurrency for performance - Force update option for existing remote files - Comprehensive error handling with retry logic Usage: remote.copy.local -dir=/path/to/mount/dir [options] This addresses the need to manually sync files when filer logs were deleted or when local files were never synced to remote storage. * shell: rename commandRemoteLocalSync to commandRemoteCopyLocal * test: add comprehensive remote cache integration tests * shell: fix forceUpdate logic in remote.copy.local The previous logic only allowed force updates when localEntry.RemoteEntry was not nil, which defeated the purpose of using -forceUpdate to fix inconsistencies where local metadata might be missing. Now -forceUpdate will overwrite remote files whenever they exist, regardless of local metadata state. * shell: fix code review issues in remote.copy.local - Return actual error from flag parsing instead of swallowing it - Use sync.Once to safely capture first error in concurrent operations - Add atomic counter to track actual successful copies - Protect concurrent writes to output with mutex to prevent interleaving - Fix path matching to prevent false positives with sibling directories (e.g., /mnt/remote2 no longer matches /mnt/remote) * test: address code review nitpicks in integration tests - Improve create_bucket error handling to fail on real errors - Fix test assertions to properly verify expected failures - Use case-insensitive string matching for error detection - Replace weak logging-only tests with proper assertions - Remove extra blank line in Makefile * test: remove redundant edge case tests Removed 5 tests that were either duplicates or didn't assert meaningful behavior: - TestEdgeCaseEmptyDirectory (duplicate of TestRemoteCopyLocalEmptyDirectory) - TestEdgeCaseRapidCacheUncache (no meaningful assertions) - TestEdgeCaseConcurrentCommands (only logs errors, no assertions) - TestEdgeCaseInvalidPaths (no security assertions) - TestEdgeCaseFileNamePatterns (duplicate of pattern tests in cache tests) Kept valuable stress tests: nested directories, special characters, very large files (100MB), many small files (100), and zero-byte files. * test: fix CI failures by forcing localhost IP advertising Added -ip=127.0.0.1 flag to both primary and remote weed mini commands to prevent IP auto-detection issues in CI environments. Without this flag, the master would advertise itself using the actual IP (e.g., 10.1.0.17) while binding to 127.0.0.1, causing connection refused errors when other services tried to connect to the gRPC port. * test: address final code review issues - Add proper error assertions for concurrent commands test - Require errors for invalid path tests instead of just logging - Remove unused 'match' field from pattern test struct - Add dry-run output assertion to verify expected behavior - Simplify redundant condition in remote.copy.local (remove entry.RemoteEntry check) * test: fix remote.configure tests to match actual validation rules - Use only letters in remote names (no numbers) to match validation - Relax missing parameter test expectations since validation may not be strict - Generate unique names using letter suffix instead of numbers * shell: rename pathToCopyCopy to localPath for clarity Improved variable naming in concurrent copy loop to make the code more readable and less repetitive. * test: fix remaining test failures - Remove strict error requirement for invalid paths (commands handle gracefully) - Fix TestRemoteUncacheBasic to actually test uncache instead of cache - Use simple numeric names for remote.configure tests (testcfg1234 format) to avoid validation issues with letter-only or complex name generation * test: use only letters in remote.configure test names The validation regex ^[A-Za-z][A-Za-z0-9]*$ requires names to start with a letter, but using static letter-only names avoids any potential issues with the validation. * test: remove quotes from -name parameter in remote.configure tests Single quotes were being included as part of the name value, causing validation failures. Changed from -name='testremote' to -name=testremote. * test: fix remote.configure assertion to be flexible about JSON formatting Changed from checking exact JSON format with specific spacing to just checking if the name appears in the output, since JSON formatting may vary (e.g., "name": "value" vs "name": "value").	4 weeks ago
Chris Lu	06391701ed	Add AssumeRole and AssumeRoleWithLDAPIdentity STS actions (#8003) * test: add integration tests for AssumeRole and AssumeRoleWithLDAPIdentity STS actions - Add s3_sts_assume_role_test.go with comprehensive tests for AssumeRole: * Parameter validation (missing RoleArn, RoleSessionName, invalid duration) * AWS SigV4 authentication with valid/invalid credentials * Temporary credential generation and usage - Add s3_sts_ldap_test.go with tests for AssumeRoleWithLDAPIdentity: * Parameter validation (missing LDAP credentials, RoleArn) * LDAP authentication scenarios (valid/invalid credentials) * Integration with LDAP server (when configured) - Update Makefile with new test targets: * test-sts: run all STS tests * test-sts-assume-role: run AssumeRole tests only * test-sts-ldap: run LDAP STS tests only * test-sts-suite: run tests with full service lifecycle - Enhance setup_all_tests.sh: * Add OpenLDAP container setup for LDAP testing * Create test LDAP users (testuser, ldapadmin) * Set LDAP environment variables for tests * Update cleanup to remove LDAP container - Fix setup_keycloak.sh: * Enable verbose error logging for realm creation * Improve error diagnostics Tests use fail-fast approach (t.Fatal) when server not configured, ensuring clear feedback when infrastructure is missing. * feat: implement AssumeRole and AssumeRoleWithLDAPIdentity STS actions Implement two new STS actions to match MinIO's STS feature set: **AssumeRole Implementation:** - Add handleAssumeRole with full AWS SigV4 authentication - Integrate with existing IAM infrastructure via verifyV4Signature - Validate required parameters (RoleArn, RoleSessionName) - Validate DurationSeconds (900-43200 seconds range) - Generate temporary credentials with expiration - Return AWS-compatible XML response **AssumeRoleWithLDAPIdentity Implementation:** - Add handleAssumeRoleWithLDAPIdentity handler (stub) - Validate LDAP-specific parameters (LDAPUsername, LDAPPassword) - Validate common STS parameters (RoleArn, RoleSessionName, DurationSeconds) - Return proper error messages for missing LDAP provider - Ready for LDAP provider integration **Routing Fixes:** - Add explicit routes for AssumeRole and AssumeRoleWithLDAPIdentity - Prevent IAM handler from intercepting authenticated STS requests - Ensure proper request routing priority **Handler Infrastructure:** - Add IAM field to STSHandlers for SigV4 verification - Update NewSTSHandlers to accept IAM reference - Add STS-specific error codes and response types - Implement writeSTSErrorResponse for AWS-compatible errors The AssumeRole action is fully functional and tested. AssumeRoleWithLDAPIdentity requires LDAP provider implementation. * fix: update IAM matcher to exclude STS actions from interception Update the IAM handler matcher to check for STS actions (AssumeRole, AssumeRoleWithWebIdentity, AssumeRoleWithLDAPIdentity) and exclude them from IAM handler processing. This allows STS requests to be handled by the STS fallback handler even when they include AWS SigV4 authentication. The matcher now parses the form data to check the Action parameter and returns false for STS actions, ensuring they are routed to the correct handler. Note: This is a work-in-progress fix. Tests are still showing some routing issues that need further investigation. * fix: address PR review security issues for STS handlers This commit addresses all critical security issues from PR review: Security Fixes: - Use crypto/rand for cryptographically secure credential generation instead of time.Now().UnixNano() (fixes predictable credentials) - Add sts:AssumeRole permission check via VerifyActionPermission to prevent unauthorized role assumption - Generate proper session tokens using crypto/rand instead of placeholder strings Code Quality Improvements: - Refactor DurationSeconds parsing into reusable parseDurationSeconds() helper function used by all three STS handlers - Create generateSecureCredentials() helper for consistent and secure temporary credential generation - Fix iamMatcher to check query string as fallback when Action not found in form data LDAP Provider Implementation: - Add go-ldap/ldap/v3 dependency - Create LDAPProvider implementing IdentityProvider interface with full LDAP authentication support (connect, bind, search, groups) - Update ProviderFactory to create real LDAP providers - Wire LDAP provider into AssumeRoleWithLDAPIdentity handler Test Infrastructure: - Add LDAP user creation verification step in setup_all_tests.sh * fix: address PR feedback (Round 2) - config validation & provider improvements - Implement `validateLDAPConfig` in `ProviderFactory` - Improve `LDAPProvider.Initialize`: - Support `connectionTimeout` parsing (string/int/float) from config map - Warn if `BindDN` is present but `BindPassword` is empty - Improve `LDAPProvider.GetUserInfo`: - Add fallback to `searchUserGroups` if `memberOf` returns no groups (consistent with Authenticate) * fix: address PR feedback (Round 3) - LDAP connection improvements & build fix - Improve `LDAPProvider` connection handling: - Use `net.Dialer` with configured timeout for connection establishment - Enforce TLS 1.2+ (`MinVersion: tls.VersionTLS12`) for both LDAPS and StartTLS - Fix build error in `s3api_sts.go` (format verb for ErrorCode) * fix: address PR feedback (Round 4) - LDAP hardening, Authz check & Routing fix - LDAP Provider Hardening: - Prevent re-initialization - Enforce single user match in `GetUserInfo` (was explicit only in Authenticate) - Ensure connection closure if StartTLS fails - STS Handlers: - Add robust provider detection using type assertion - **Security**: Implement authorization check (`VerifyActionPermission`) after LDAP authentication - Routing: - Update tests to reflect that STS actions are handled by STS handler, not generic IAM * fix: address PR feedback (Round 5) - JWT tokens, ARN formatting, PrincipalArn CRITICAL FIXES: - Replace standalone credential generation with STS service JWT tokens - handleAssumeRole now generates proper JWT session tokens - handleAssumeRoleWithLDAPIdentity now generates proper JWT session tokens - Session tokens can be validated across distributed instances - Fix ARN formatting in responses - Extract role name from ARN using utils.ExtractRoleNameFromArn() - Prevents malformed ARNs like "arn:aws:sts::assumed-role/arn:aws:iam::..." - Add configurable AccountId for federated users - Add AccountId field to STSConfig (defaults to "111122223333") - PrincipalArn now uses configured account ID instead of hardcoded "aws" - Enables proper trust policy validation IMPROVEMENTS: - Sanitize LDAP authentication error messages (don't leak internal details) - Remove duplicate comment in provider detection - Add utils import for ARN parsing utilities * feat: implement LDAP connection pooling to prevent resource exhaustion PERFORMANCE IMPROVEMENT: - Add connection pool to LDAPProvider (default size: 10 connections) - Reuse LDAP connections across authentication requests - Prevent file descriptor exhaustion under high load IMPLEMENTATION: - connectionPool struct with channel-based connection management - getConnection(): retrieves from pool or creates new connection - returnConnection(): returns healthy connections to pool - createConnection(): establishes new LDAP connection with TLS support - Close(): cleanup method to close all pooled connections - Connection health checking (IsClosing()) before reuse BENEFITS: - Reduced connection overhead (no TCP handshake per request) - Better resource utilization under load - Prevents "too many open files" errors - Non-blocking pool operations (creates new conn if pool empty) * fix: correct TokenGenerator access in STS handlers CRITICAL FIX: - Make TokenGenerator public in STSService (was private tokenGenerator) - Update all references from Config.TokenGenerator to TokenGenerator - Remove TokenGenerator from STSConfig (it belongs in STSService) This fixes the "NotImplemented" errors in distributed and Keycloak tests. The issue was that Round 5 changes tried to access Config.TokenGenerator which didn't exist - TokenGenerator is a field in STSService, not STSConfig. The TokenGenerator is properly initialized in STSService.Initialize() and is now accessible for JWT token generation in AssumeRole handlers. * fix: update tests to use public TokenGenerator field Following the change to make TokenGenerator public in STSService, this commit updates the test files to reference the correct public field name. This resolves compilation errors in the IAM STS test suite. * fix: update distributed tests to use valid Keycloak users Updated s3_iam_distributed_test.go to use 'admin-user' and 'read-user' which exist in the standard Keycloak setup provided by setup_keycloak.sh. This resolves 'unknown test user' errors in distributed integration tests. * fix: ensure iam_config.json exists in setup target for CI The GitHub Actions workflow calls 'make setup' which was not creating iam_config.json, causing the server to start without IAM integration enabled (iamIntegration = nil), resulting in NotImplemented errors. Now 'make setup' copies iam_config.local.json to iam_config.json if it doesn't exist, ensuring IAM is properly configured in CI. * fix(iam/ldap): fix connection pool race and rebind corruption - Add atomic 'closed' flag to connection pool to prevent racing on Close() - Rebind authenticated user connections back to service account before returning to pool - Close connections on error instead of returning potentially corrupted state to pool * fix(iam/ldap): populate standard TokenClaims fields in ValidateToken - Set Subject, Issuer, Audience, IssuedAt, and ExpiresAt to satisfy the interface - Use time.Time for timestamps as required by TokenClaims struct - Default to 1 hour TTL for LDAP tokens * fix(s3api): include account ID in STS AssumedRoleUser ARN - Consistent with AWS, include the account ID in the assumed-role ARN - Use the configured account ID from STS service if available, otherwise default to '111122223333' - Apply to both AssumeRole and AssumeRoleWithLDAPIdentity handlers - Also update .gitignore to ignore IAM test environment files * refactor(s3api): extract shared STS credential generation logic - Move common logic for session claims and credential generation to prepareSTSCredentials - Update handleAssumeRole and handleAssumeRoleWithLDAPIdentity to use the helper - Remove stale comments referencing outdated line numbers * feat(iam/ldap): make pool size configurable and add audience support - Add PoolSize to LDAPConfig (default 10) - Add Audience to LDAPConfig to align with OIDC validation - Update initialization and ValidateToken to use new fields * update tests * debug * chore(iam): cleanup debug prints and fix test config port * refactor(iam): use mapstructure for LDAP config parsing * feat(sts): implement strict trust policy validation for AssumeRole * test(iam): refactor STS tests to use AWS SDK signer * test(s3api): implement ValidateTrustPolicyForPrincipal in MockIAMIntegration * fix(s3api): ensure IAM matcher checks query string on ParseForm error * fix(sts): use crypto/rand for secure credentials and extract constants * fix(iam): fix ldap connection leaks and add insecure warning * chore(iam): improved error wrapping and test parameterization * feat(sts): add support for LDAPProviderName parameter * Update weed/iam/ldap/ldap_provider.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_sts.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix(sts): use STSErrSTSNotReady when LDAP provider is missing * fix(sts): encapsulate TokenGenerator in STSService and add getter --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	1 month ago
Chris Lu	217d8b9e0e	Fix: ListObjectVersions delimiter support (#7987) * Fix: Add delimiter support to ListObjectVersions with proper truncation - Implemented delimiter support to group keys into CommonPrefixes - Fixed critical truncation bug: now merges versions and common prefixes into single sorted list before truncation - Ensures total items never exceed MaxKeys (prevents infinite pagination loops) - Properly sets NextKeyMarker and NextVersionIdMarker for pagination - Added integration tests in test/s3/versioning/s3_versioning_delimiter_test.go - Verified behavior matches S3 API specification * Fix: Add delimiter support to ListObjectVersions with proper truncation - Implemented delimiter support to group keys into CommonPrefixes - Fixed critical truncation bug: now merges versions and common prefixes before truncation - Added safety guard for maxKeys=0 to prevent panics - Condensed verbose comments for better readability - Added robust Go integration tests with nil checks for AWS SDK pointers - Verified behavior matches S3 API specification - Resolved compilation error in integration tests - Refined pagination comments and ensured exclusive KeyMarker behavior - Refactored listObjectVersions into helper methods for better maintainability	1 month ago
promalert	9012069bd7	chore: execute goimports to format the code (#7983) * chore: execute goimports to format the code Signed-off-by: promalert <promalert@outlook.com> * goimports -w . --------- Signed-off-by: promalert <promalert@outlook.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	1 month ago