Tree: 7169fe585d

add-ec-vacuum

add-filer-iam-grpc

add-iam-grpc-management

add_fasthttp_client

add_remote_storage

adding-message-queue-integration-tests

adjust-fsck-cutoff-default

also-delete-parent-directory-if-empty

avoid_releasing_temp_file_on_write

changing-to-zap

collect-public-metrics

copilot/fix-helm-chart-installation

copilot/fix-s3-object-tagging-issue

copilot/make-renew-interval-configurable

copilot/make-renew-interval-configurable-again

copilot/sub-pr-7677

create-table-snapshot-api-design

data_query_pushdown

dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5

dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0

detect-and-plan-ec-tasks

do-not-retry-if-error-is-NotFound

ec-disk-type-support

enhance-erasure-coding

fasthttp

feature/mini-port-detection

feature/modernize-s3-tests

feature/s3-multi-cert-support

filer1_maintenance_branch

fix-GetObjectLockConfigurationHandler

fix-bucket-name-case-7910

fix-helm-fromtoml-compatibility

fix-maintenance-sorting-order

fix-mount-http-parallelism

fix-mount-read-throughput-7504

fix-pr-7909

fix-s3-configure-consistency

fix-s3-object-tagging-issue-7589

fix-sts-session-token-7941

fix-versioning-listing-only

fix/windows-test-file-cleanup

ftp

gh-pages

iam-multi-file-migration

iam-permissions-and-api

improve-fuse-mount

improve-fuse-mount2

logrus

master

message_send

mount2

mq-subscribe

mq2

nfs-cookie-prefix-list-fixes

optimize-delete-lookups

original_weed_mount

pr-7412

pr/7984

pr/8140

raft-dual-write

random_access_file

refactor-needle-read-operations

refactor-volume-write

remote_overlay

remove-implicit-directory-handling

revert-5134-patch-1

revert-5819-patch-1

revert-6434-bugfix-missing-s3-audit

s3-remote-cache-singleflight

s3-select

s3tables-by-claude

sub

tcp_read

test-reverting-lock-table

test_udp

testing

testing-sdx-generation

tikv

track-mount-e2e

upgrade-versions-to-4.00

volume_buffered_writes

worker-execute-ec-tasks

0.72

0.72.release

0.73

0.74

0.75

0.76

0.77

0.90

0.91

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1.00

1.01

1.02

1.03

1.04

1.05

1.06

1.07

1.08

1.09

1.10

1.11

1.12

1.14

1.15

1.16

1.17

1.18

1.19

1.20

1.21

1.22

1.23

1.24

1.25

1.26

1.27

1.28

1.29

1.30

1.31

1.32

1.33

1.34

1.35

1.36

1.37

1.38

1.40

1.41

1.42

1.43

1.44

1.45

1.46

1.47

1.48

1.49

1.50

1.51

1.52

1.53

1.54

1.55

1.56

1.57

1.58

1.59

1.60

1.61

1.61RC

1.62

1.63

1.64

1.65

1.66

1.67

1.68

1.69

1.70

1.71

1.72

1.73

1.74

1.75

1.76

1.77

1.78

1.79

1.80

1.81

1.82

1.83

1.84

1.85

1.86

1.87

1.88

1.90

1.91

1.92

1.93

1.94

1.95

1.96

1.97

1.98

1.99

1;70

2.00

2.01

2.02

2.03

2.04

2.05

2.06

2.07

2.08

2.09

2.10

2.11

2.12

2.13

2.14

2.15

2.16

2.17

2.18

2.19

2.20

2.21

2.22

2.23

2.24

2.25

2.26

2.27

2.28

2.29

2.30

2.31

2.32

2.33

2.34

2.35

2.36

2.37

2.38

2.39

2.40

2.41

2.42

2.43

2.47

2.48

2.49

2.50

2.51

2.52

2.53

2.54

2.55

2.56

2.57

2.58

2.59

2.60

2.61

2.62

2.63

2.64

2.65

2.66

2.67

2.68

2.69

2.70

2.71

2.72

2.73

2.74

2.75

2.76

2.77

2.78

2.79

2.80

2.81

2.82

2.83

2.84

2.85

2.86

2.87

2.88

2.89

2.90

2.91

2.92

2.93

2.94

2.95

2.96

2.97

2.98

2.99

3.00

3.01

3.02

3.03

3.04

3.05

3.06

3.07

3.08

3.09

3.10

3.11

3.12

3.13

3.14

3.15

3.16

3.18

3.19

3.20

3.21

3.22

3.23

3.24

3.25

3.26

3.27

3.28

3.29

3.30

3.31

3.32

3.33

3.34

3.35

3.36

3.37

3.38

3.39

3.40

3.41

3.42

3.43

3.44

3.45

3.46

3.47

3.48

3.50

3.51

3.52

3.53

3.54

3.55

3.56

3.57

3.58

3.59

3.60

3.61

3.62

3.63

3.64

3.65

3.66

3.67

3.68

3.69

3.71

3.72

3.73

3.74

3.75

3.76

3.77

3.78

3.79

3.80

3.81

3.82

3.83

3.84

3.85

3.86

3.87

3.88

3.89

3.90

3.91

3.92

3.93

3.94

3.95

3.96

3.97

3.98

3.99

4.00

4.01

4.02

4.03

4.04

4.05

4.06

4.07

4.08

4.09

dev

helm-3.65.1

v0.69

v0.70beta

v3.33

Branches Tags

${ item.name }

Create tag ${ searchTerm }

Create branch ${ searchTerm }

from '7169fe585d'

${ noResults }

Commit Graph

8201 Commits (7169fe585d40a50ccbd92cdab7a1f5641128721c)

Author	SHA1	Message	Date
Chris Lu	7169fe585d	Delete verify_gc_empty_test.go	1 day ago
Chris Lu	1c1f80358f	templ	1 day ago
Chris Lu	2bb21ea276	feat: Add Iceberg REST Catalog server and admin UI (#8175) * feat: Add Iceberg REST Catalog server Implement Iceberg REST Catalog API on a separate port (default 8181) that exposes S3 Tables metadata through the Apache Iceberg REST protocol. - Add new weed/s3api/iceberg package with REST handlers - Implement /v1/config endpoint returning catalog configuration - Implement namespace endpoints (list/create/get/head/delete) - Implement table endpoints (list/create/load/head/delete/update) - Add -port.iceberg flag to S3 standalone server (s3.go) - Add -s3.port.iceberg flag to combined server mode (server.go) - Add -s3.port.iceberg flag to mini cluster mode (mini.go) - Support prefix-based routing for multiple catalogs The Iceberg REST server reuses S3 Tables metadata storage under /table-buckets and enables DuckDB, Spark, and other Iceberg clients to connect to SeaweedFS as a catalog. * feat: Add Iceberg Catalog pages to admin UI Add admin UI pages to browse Iceberg catalogs, namespaces, and tables. - Add Iceberg Catalog menu item under Object Store navigation - Create iceberg_catalog.templ showing catalog overview with REST info - Create iceberg_namespaces.templ listing namespaces in a catalog - Create iceberg_tables.templ listing tables in a namespace - Add handlers and routes in admin_handlers.go - Add Iceberg data provider methods in s3tables_management.go - Add Iceberg data types in types.go The Iceberg Catalog pages provide visibility into the same S3 Tables data through an Iceberg-centric lens, including REST endpoint examples for DuckDB and PyIceberg. * test: Add Iceberg catalog integration tests and reorg s3tables tests - Reorganize existing s3tables tests to test/s3tables/table-buckets/ - Add new test/s3tables/catalog/ for Iceberg REST catalog tests - Add TestIcebergConfig to verify /v1/config endpoint - Add TestIcebergNamespaces to verify namespace listing - Add TestDuckDBIntegration for DuckDB connectivity (requires Docker) - Update CI workflow to use new test paths * fix: Generate proper random UUIDs for Iceberg tables Address code review feedback: - Replace placeholder UUID with crypto/rand-based UUID v4 generation - Add detailed TODO comments for handleUpdateTable stub explaining the required atomic metadata swap implementation * fix: Serve Iceberg on localhost listener when binding to different interface Address code review feedback: properly serve the localhost listener when the Iceberg server is bound to a non-localhost interface. * ci: Add Iceberg catalog integration tests to CI Add new job to run Iceberg catalog tests in CI, along with: - Iceberg package build verification - Iceberg unit tests - Iceberg go vet checks - Iceberg format checks * fix: Address code review feedback for Iceberg implementation - fix: Replace hardcoded account ID with s3_constants.AccountAdminId in buildTableBucketARN() - fix: Improve UUID generation error handling with deterministic fallback (timestamp + PID + counter) - fix: Update handleUpdateTable to return HTTP 501 Not Implemented instead of fake success - fix: Better error handling in handleNamespaceExists to distinguish 404 from 500 errors - fix: Use relative URL in template instead of hardcoded localhost:8181 - fix: Add HTTP timeout to test's waitForService function to avoid hangs - fix: Use dynamic ephemeral ports in integration tests to avoid flaky parallel failures - fix: Add Iceberg port to final port configuration logging in mini.go * fix: Address critical issues in Iceberg implementation - fix: Cache table UUIDs to ensure persistence across LoadTable calls The UUID now remains stable for the lifetime of the server session. TODO: For production, UUIDs should be persisted in S3 Tables metadata. - fix: Remove redundant URL-encoded namespace parsing mux router already decodes %1F to \x1F before passing to handlers. Redundant ReplaceAll call could cause bugs with literal %1F in namespace. * fix: Improve test robustness and reduce code duplication - fix: Make DuckDB test more robust by failing on unexpected errors Instead of silently logging errors, now explicitly check for expected conditions (extension not available) and skip the test appropriately. - fix: Extract username helper method to reduce duplication Created getUsername() helper in AdminHandlers to avoid duplicating the username retrieval logic across Iceberg page handlers. * fix: Add mutex protection to table UUID cache Protects concurrent access to the tableUUIDs map with sync.RWMutex. Uses read-lock for fast path when UUID already cached, and write-lock for generating new UUIDs. Includes double-check pattern to handle race condition between read-unlock and write-lock. * style: fix go fmt errors * feat(iceberg): persist table UUID in S3 Tables metadata * feat(admin): configure Iceberg port in Admin UI and commands * refactor: address review comments (flags, tests, handlers) - command/mini: fix tracking of explicit s3.port.iceberg flag - command/admin: add explicit -iceberg.port flag - admin/handlers: reuse getUsername helper - tests: use 127.0.0.1 for ephemeral ports and os.Stat for file size check * test: check error from FileStat in verify_gc_empty_test	1 day ago
Chris Lu	330bd92ddc	4.08	1 day ago
Chris Lu	ba8816e2e1	4.08	1 day ago
Chris Lu	3b9e367c1a	templ	1 day ago
Lisandro Pin	ff5a8f0579	Implement RPC skeleton for regular/EC volumes scrubbing. (#8187) * Implement RPC skeleton for regular/EC volumes scrubbing. See https://github.com/seaweedfs/seaweedfs/issues/8018 for details. * Minor proto improvements for `ScrubVolume()`, `ScrubEcVolume()`: - Add fields for scrubbing details in `ScrubVolumeResponse` and `ScrubEcVolumeResponse`, instead of reporting these through RPC errors. - Return a list of broken shards when scrubbing EC volumes, via `EcShardInfo'.	2 days ago
Lisandro Pin	345ac950b6	Add volume server RPCs to read and update state flags. (#8186) * Boostrap persistent state for volume servers. This PR implements logic load/save persistent state information for storages associated with volume servers, and reporting state changes back to masters via heartbeat messages. More work ensues! See https://github.com/seaweedfs/seaweedfs/issues/7977 for details. * Add volume server RPCs to read and update state flags.	2 days ago
Lisandro Pin	9638d37fe2	Block RPC write operations on volume servers when maintenance mode is enabled (#8115) * Boostrap persistent state for volume servers. This PR implements logic load/save persistent state information for storages associated with volume servers, and reporting state changes back to masters via heartbeat messages. More work ensues! See https://github.com/seaweedfs/seaweedfs/issues/7977 for details. * Block RPC operations writing to volume servers when maintenance mode is on.	2 days ago
Chris Lu	f23e09f58b	fix: skip exhausted blocks before creating an interval (#8180) * fix: skip exhausted blocks before creating an interval * refactor: optimize interval creation and fix logic duplication * docs: add docstring for LocateData * refactor: extract moveToNextBlock helper to deduplicate logic * fix: use int64 for block index comparison to prevent overflow * test: add unit test for LocateData boundary crossing (issue #8179) * fix: skip exhausted blocks to prevent negative interval size and panics (issue #8179) * refactor: apply review suggestions for test maintainability and code style	2 days ago
Chris Lu	621834d96a	s3tables: add Iceberg file layout validation for table buckets (#8176) * s3tables: add Iceberg file layout validation for table buckets This PR adds file layout validation for table buckets to enforce Apache Iceberg table structure. Files uploaded to table buckets must conform to the expected Iceberg layout: - metadata/ directory: contains metadata files (*.json, *.avro) - v*.metadata.json (table metadata) - snap-*.avro (snapshot manifests) - *-m*.avro (manifest files) - version-hint.text - data/ directory: contains data files (*.parquet, *.orc, *.avro) - Supports partition paths (e.g., year=2024/month=01/) - Supports bucket subdirectories The validator exports functions for use by the S3 API: - IsTableBucketPath: checks if a path is under /table-buckets/ - GetTableInfoFromPath: extracts bucket/namespace/table from path - ValidateTableBucketUpload: validates file layout for table bucket uploads - ValidateTableBucketUploadWithClient: validates with filer client access Invalid uploads receive InvalidIcebergLayout error response. * Address review comments: regex performance, error handling, stricter patterns * Fix validateMetadataFile and validateDataFile to handle subdirectories and directory creation * Fix error handling, metadata validation, reduce code duplication * Fix empty remainingPath handling for directory paths * Refactor: unify validateMetadataFile and validateDataFile * Refactor: extract UUID pattern constant * fix: allow Iceberg partition and directory paths without trailing slashes Modified validateFile to correctly handle directory paths that do not end with a trailing slash. This ensures that paths like 'data/year=2024' are validated as directories if they match partition or subdirectory patterns, rather than being incorrectly rejected as invalid files. Added comprehensive test cases for various directory and partition path combinations. * refactor: use standard path package and idiomatic returns Simplified directory and filename extraction in validateFile by using the standard path package (aliased as pathpkg). This improves readability and avoids manual string manipulation. Also updated GetTableInfoFromPath to use naked returns for named return values, aligning with Go conventions for short functions. * feat: enforce strict Iceberg top-level directories and metadata restrictions Implemented strict validation for Iceberg layout: - Bare top-level keys like 'metadata' and 'data' are now rejected; they must have a trailing slash or a subpath. - Subdirectories under 'metadata/' are now prohibited to enforce the flat structure required by Iceberg. - Updated the test suite with negative test cases and ensured proper formatting. * feat: allow table root directory markers in ValidateTableBucketUpload Modified ValidateTableBucketUpload to short-circuit and return nil when the relative path within a table is empty. This occurs when a trailing slash is used on the table directory (e.g., /table-buckets/mybucket/myns/mytable/). Added a test case 'table dir with slash' to verify this behavior. * test: add regression cases for metadata subdirs and table markers Enforced a strictly flat structure for the metadata directory by removing the "directory without trailing slash" fallback in validateFile for metadata. Added regression test cases: - metadata/nested (must fail) - /table-buckets/.../mytable/ (must pass) Verified all tests pass. * feat: reject double slashes in Iceberg table paths Modified validateDirectoryPath to return an error when encountering empty path segments, effectively rejecting double slashes like 'data//file.parquet'. Updated validateFile to use manual path splitting instead of the 'path' package for intermediate directories to ensure redundant slashes are not auto-cleaned before validation. Added regression tests for various double slash scenarios. * refactor: separate isMetadata logic in validateDirectoryPath Following reviewer feedback, refactored validateDirectoryPath to explicitly separate the handling of metadata and data paths. This improves readability and clarifies the function's intent while maintaining the strict validation rules and double-slash rejection previously implemented. * feat: validate bucket, namespace, and table path segments Updated ValidateTableBucketUpload to ensure that bucket, namespace, and table segments in the path are non-empty. This prevents invalid paths like '/table-buckets//myns/mytable/...' from being accepted during upload. Added regression tests for various empty segment scenarios. * Update weed/s3api/s3tables/iceberg_layout.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * feat: block double-slash bypass in table relative paths Added a guard in ValidateTableBucketUpload to reject tableRelativePath if it starts with a '/' or contains '//'. This ensures that paths like '/table-buckets/b/ns/t//data/file.parquet' are properly rejected and cannot bypass the layout validation. Added regression tests to verify. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	2 days ago
Chris Lu	2ee6e4f391	mount: refresh and evict hot dir cache (#8174) * mount: refresh and evict hot dir cache * mount: guard dir update window and extend TTL * mount: reuse timestamp for cache mark * Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * mount: make dir cache tuning configurable * mount: dedupe dir update notices * mount: restore invalidate-all cache helper * mount: keep hot dir tuning constants * mount: centralize cache state reset * mount: mark refresh completion time * mount: allow disabling idle eviction --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	4 days ago
Chris Lu	fe6f8d737d	mount: invalidate meta cache on follow retry (#8173) * mount: invalidate meta cache on follow retry * mount: clear cache under mount root on retry	4 days ago
Chris Lu	79722bcf30	Add s3tables shell and admin UI (#8172) * Add shared s3tables manager * Add s3tables shell commands * Add s3tables admin API * Add s3tables admin UI * Fix admin s3tables namespace create * Rename table buckets menu * Centralize s3tables tag validation * Reuse s3tables manager in admin * Extract s3tables list limit * Add s3tables bucket ARN helper * Remove write middleware from s3tables APIs * Fix bucket link and policy hint * Fix table tag parsing and nav link * Disable namespace table link on invalid ARN * Improve s3tables error decode * Return flag parse errors for s3tables tag * Accept query params for namespace create * Bind namespace create form data * Read s3tables JS data from DOM * s3tables: allow empty region ARN * shell: pass s3tables account id * shell: require account for table buckets * shell: use bucket name for namespaces * shell: use bucket name for tables * shell: use bucket name for tags * admin: add table buckets links in file browser * s3api: reuse s3tables tag validation * admin: harden s3tables UI handlers * fix admin list table buckets * allow admin s3tables access * validate s3tables bucket tags * log s3tables bucket metadata errors * rollback table bucket on owner failure * show s3tables bucket owner * add s3tables iam conditions * Add s3tables user permissions UI * Authorize s3tables using identity actions * Add s3tables permissions to user modal * Disambiguate bucket scope in user permissions * Block table bucket names that match S3 buckets * Pretty-print IAM identity JSON * Include tags in s3tables permission context * admin: refactor S3 Tables inline JavaScript into a separate file * s3tables: extend IAM policy condition operators support * shell: use LookupEntry wrapper for s3tables bucket conflict check * admin: handle buildBucketPermissions validation in create/update flows	4 days ago
Chris Lu	b2b0a38e71	s3api: allow empty region and account id in s3tables ARN (#8171) * s3api: allow empty region and account id in s3tables ARN * s3api: refactor S3 Tables ARN regex into a constant	5 days ago
Chris Lu	6a9e7360df	s3api: fix S3 Tables auth to allow auto-hashing of body (#8170) * s3api: allow auto-hashing of request body for s3tables * s3api: add unit test for s3tables body hashing	5 days ago
Chris Lu	f1e27b8f30	s3: change s3 tables to use RESTful API (#8169) * s3: refactor s3 tables to use RESTful API * test/s3tables: guard empty namespaces * s3api: document tag parsing and validate get-table * s3api: limit S3Tables REST body size * Update weed/s3api/s3api_tables.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3tables/handler.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * s3api: accept encoded table bucket ARNs * s3api: validate namespaces and close body * s3api: match encoded table bucket ARNs * s3api: scope table bucket ARN routes * s3api: dedupe table bucket request builders * test/s3tables: allow list tables without namespace * s3api: validate table params and tag ARN * s3api: tighten tag handling and get-table params * s3api: loosen tag ARN route matching * Fix S3 Tables REST routing and tests * Adjust S3 Tables request parsing * Gate S3 Tables target routing * Avoid double decoding namespaces --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	5 days ago
Lisandro Pin	9e15823855	Have masters update DataNode details based on state heartbeats from volume servers. (#8017)	5 days ago
Chris Lu	6940b7d06e	fix Filer startup failure due to JWT on / path #8149 (#8167) * fix Filer startup failure due to JWT on / path #8149 - Comment out JWT keys in security.toml.example - Revert Dockerfile.local change that enabled security by default - Exempt GET/HEAD on / from JWT check for health checks * refactor: simplify JWT bypass condition as per PR feedback	5 days ago
Chris Lu	23c25379ca	iam: add ECDSA support for OIDC token validation (#8166) * iam: add ECDSA support for OIDC token validation Fixes seaweedfs/seaweedfs#8148 * iam: refactor OIDC ECDSA tests and add failure cases - Refactored TestOIDCProviderJWTValidationECDSA to use t.Run - Added sub-tests for expired token, wrong key, invalid issuer, and invalid audience * Update weed/iam/oidc/oidc_provider_test.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * iam: improve error type assertions for OIDC invalid signature tests - Updated both RSA and ECDSA tests to specifically check for ErrProviderInvalidToken * iam: pad EC coordinates in OIDC tests to comply with RFC 7518 - Coordinates are now zero-padded to the full field size (e.g., 32 bytes for P-256) - Ensures interoperability with strict OIDC providers --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	5 days ago
Chris Lu	88c27615c4	/table-buckets	5 days ago
Chris Lu	49c66bbb2e	shell: allow spaces in arguments via quoting (#8157) (#8165) * shell: allow spaces in arguments via quoting (#8157) - updated argument splitting regex to handle quoted segments - added robust quote stripping to remove matching quotes from flags - added unit tests for regex splitting and flag parsing * shell: use robust state machine parser for command line arguments - replaced regex-based splitter with splitCommandLine state machine - added escape character support in splitCommandLine and stripQuotes - updated unit tests to include escaped quotes and single-quote literals - addressed feedback regarding escaped quotes handling (#8157) * shell: detect unbalanced quotes in stripQuotes - modified stripQuotes to return the original string if quotes are unbalanced - added test cases for unbalanced quotes in shell_liner_test.go * shell: refactor shared parsing logic into parseShellInput helper - unified splitting and unquoting logic into a single state machine - splitCommandLine now returns unquoted tokens directly - simplified processEachCmd by removing redundant unquoting loop - improved maintainability by eliminating code duplication * shell: detect trailing backslash in stripQuotes - updated parseShellInput to include escaped state in unbalanced flag - stripQuotes now returns original string if it ends with an unescaped backslash - added test case for trailing backslash in shell_liner_test.go	5 days ago
Chris Lu	94e0b902f9	shell: update fs.verify and volume.fsck for new BFS signature Updated dependent commands to match the refactored doTraverseBfsAndSaving signature and use context for channel sends.	6 days ago
Chris Lu	3b05efbdbc	shell: fix potential deadlock in fs.meta.save BFS traversal Refactored doTraverseBfsAndSaving to use context cancellation. If the saving process fails, the traversal is stopped immediately to prevent workers from blocking on the output channel.	6 days ago
Chris Lu	b91427c30f	filer: preserve existing TTL during CreateEntry/UpdateEntry gRPC calls This ensures that metadata load correctly restores entries with their original TTL instead of overwriting with the default from filer.conf. Fixes #8159.	6 days ago
Chris Lu	550a4ff761	Fix inconsistent TTL reporting in volume.list #8158 (#8164) * fix inconsistent TTL reporting in volume.list #8158 * simplify volume.list output using vi.String()	6 days ago
Chris Lu	8b61fd77b5	s3api: ensure MD5 is calculated or reused during CopyObject (#8163) * s3api: ensure MD5 is calculated or reused during CopyObject Fixes #8155 - Capture and reuse source MD5 for direct copies - Calculate MD5 for small inline objects during copy * s3api: refactor encryption logic and safe MD5 copying - Extract duplicated bucket default encryption logic into helper - Use safe append copy for MD5 slice to avoid shared modifications * refactor * avoids unnecessary MD5 recalculations for small files	6 days ago
Peter Dodd	4d513a2b3d	feat(gcs): add application default credentials fallback support (#8161) * feat(gcs): add application default credentials fallback support * refactor * Update weed/remote_storage/gcs/gcs_storage_client.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	6 days ago
Chris Lu	c52d3d1229	fix: correct chunk size in encrypted uploads (fixes #8151) (#8154) * fix: correct chunk size in encrypted uploads When uploading with encryption enabled (-encryptVolumeData flag), the chunk metadata was incorrectly storing the encrypted data size instead of the original uncompressed size. This caused data corruption when reading files back because: 1. Encrypted data is larger than original data 2. After decryption and decompression, the actual data is smaller 3. Size validation in readEncryptedUrl would fail 4. Files appeared truncated with null bytes The fix ensures uploadResult.Size uses clearDataLen (the original uncompressed size) in both cipher and non-cipher code paths, matching the expected behavior. Fixes: #8151 * test: add comprehensive tests for encrypted upload size fix Add unit tests to verify that encrypted uploads correctly store the original uncompressed data size, not the encrypted size. Tests cover: - Cipher key generation and encryption/decryption roundtrip - Compression + encryption roundtrip with size preservation - Different data sizes - Compression detection and effectiveness - Size behavior demonstration showing the bug and fix All tests verify that metadata stores the original size (19 bytes in the key example) not the encrypted size (75 bytes), which is critical for preventing data corruption on read operations. Issue: #8151 * remove emojis * Update upload_content_test.go * Delete upload_content_test.go	6 days ago
Chris Lu	d399113e0c	test: fix duplicate subtest names in permissions_test.go Rename duplicate 'combined * and ?' test cases to include singular/plural suffix for clarity and to support targeted test runs.	6 days ago
Chris Lu	c106532b79	fix: prevent MiniClusterCtx race conditions in command shutdown Capture global MiniClusterCtx into local variables before goroutine/select evaluation to prevent nil dereference/data race when context is reset to nil after nil check. Applied to filer, master, volume, and s3 commands.	6 days ago
Chris Lu	a4217dff5f	s3tables: enhance DeleteTable authorization with policy checking Fetch and evaluate table policies in DeleteTable handler to support policy-based delegation. Aligns authorization behavior with GetTable and ListTables handlers instead of only checking ownership.	6 days ago
Chris Lu	745a7e40a6	s3tables: improve bucket policy error handling in DeleteTableBucket Explicitly handle ErrAttributeNotFound vs other errors when fetching bucket policy. Return errors for non-expected failures to prevent masking filer issues and ensure correct authorization decisions.	6 days ago
Chris Lu	d5ce6a4cda	s3tables: refactor bucket name validation into single function Combine length, character, and reserved pattern validation into validateBucketName() which returns descriptive error messages. Keep isValidBucketName() for backward compatibility. This simplifies handler validation and provides better error reporting.	6 days ago
Chris Lu	2f155ee5ee	feat: Add S3 Tables support for Iceberg tabular data (#8147) * s3tables: extract utility and filer operations to separate modules - Move ARN parsing, path helpers, and metadata structures to utils.go - Extract all extended attribute and filer operations to filer_ops.go - Reduces code duplication and improves modularity - Improves code organization and maintainability * s3tables: split table bucket operations into focused modules - Create bucket_create.go for CreateTableBucket operation - Create bucket_get_list_delete.go for Get, List, Delete operations - Related operations grouped for better maintainability - Each file has a single, clear responsibility - Improves code clarity and makes it easier to test * s3tables: simplify handler by removing duplicate utilities - Reduce handler.go from 370 to 195 lines (47% reduction) - Remove duplicate ARN parsing and path helper functions - Remove filer operation methods moved to filer_ops.go - Remove metadata structure definitions moved to utils.go - Keep handler focused on request routing and response formatting - Maintains all functionality with improved code organization * s3tables: complete s3tables package implementation - namespace.go: namespace CRUD operations (310 lines) - table.go: table CRUD operations with Iceberg schema support (409 lines) - policy.go: resource policies and tagging operations (419 lines) - types.go: request/response types and error definitions (290 lines) - All handlers updated to use standalone utilities from utils.go - All files follow single responsibility principle * s3api: add S3 Tables integration layer - Create s3api_tables.go to integrate S3 Tables with S3 API server - Implement S3 Tables route matcher for X-Amz-Target header - Register S3 Tables routes with API router - Provide gRPC filer client interface for S3 Tables handlers - All S3 Tables operations accessible via S3 API endpoint * s3api: register S3 Tables routes in API server - Add S3 Tables route registration in s3api_server.go registerRouter method - Enable S3 Tables API operations to be routed through S3 API server - Routes handled by s3api_tables.go integration layer - Minimal changes to existing S3 API structure * test: add S3 Tables test infrastructure - Create setup.go with TestCluster and S3TablesClient definitions - Create client.go with HTTP client methods for all operations - Test utilities and client methods organized for reusability - Foundation for S3 Tables integration tests * test: add S3 Tables integration tests - Comprehensive integration tests for all 23 S3 Tables operations - Test cluster setup based on existing S3 integration tests - Tests cover: * Table bucket lifecycle (create, get, list, delete) * Namespace operations * Table CRUD with Iceberg schema * Table bucket and table policies * Resource tagging operations - Ready for CI/CD pipeline integration * ci: add S3 Tables integration tests to GitHub Actions - Create new workflow for S3 Tables integration testing - Add build verification job for s3tables package and s3api integration - Add format checking for S3 Tables code - Add go vet checks for code quality - Workflow runs on all pull requests - Includes test output logging and artifact upload on failure * s3tables: add handler_ prefix to operation handler files - Rename bucket_create.go → handler_bucket_create.go - Rename bucket_get_list_delete.go → handler_bucket_get_list_delete.go - Rename namespace.go → handler_namespace.go - Rename table.go → handler_table.go - Rename policy.go → handler_policy.go Improves file organization by clearly identifying handler implementations. No code changes, refactoring only. * s3tables test: refactor to eliminate duplicate definitions - Move all client methods to client.go - Remove duplicate types/constants from s3tables_integration_test.go - Keep setup.go for test infrastructure - Keep integration test logic in s3tables_integration_test.go - Clean up unused imports - Test compiles successfully * Delete client_methods.go * s3tables: add bucket name validation and fix error handling - Add isValidBucketName validation function for [a-z0-9_-] characters - Validate bucket name characters match ARN parsing regex - Fix error handling in WithFilerClient closure - properly check for lookup errors - Add error handling for json.Marshal calls (metadata and tags) - Improve error messages and logging * s3tables: add error handling for json.Marshal calls - Add error handling in handler_namespace.go (metadata marshaling) - Add error handling in handler_table.go (metadata and tags marshaling) - Add error handling in handler_policy.go (tag marshaling in TagResource and UntagResource) - Return proper errors with context instead of silently ignoring failures * s3tables: replace custom splitPath with stdlib functions - Remove custom splitPath implementation (23 lines) - Use filepath.Dir and filepath.Base from stdlib - More robust and handles edge cases correctly - Reduces code duplication * s3tables: improve error handling specificity in ListTableBuckets - Specifically check for 'not found' errors instead of catching all errors - Return empty list only when directory doesn't exist - Propagate other errors (network, permission) with context - Prevents masking real errors * s3api_tables: optimize action validation with map lookup - Replace O(n) slice iteration with O(1) map lookup - Move s3TablesActionsMap to package level - Avoid recreating the map on every function call - Improves performance for request validation * s3tables: implement permission checking and authorization - Add permissions.go with permission definitions and checks - Define permissions for all 21 S3 Tables operations - Add permission checking helper functions - Add getPrincipalFromRequest to extract caller identity - Implement access control in CreateTableBucket, GetTableBucket, DeleteTableBucket - Return 403 Forbidden for unauthorized operations - Only bucket owner can perform operations (extensible for future policies) - Add AuthError type for authorization failures * workflow: fix s3 tables tests path and working directory The workflow was failing because it was running inside 'weed' directory, but the tests are at the repository root. Removed working-directory default and updated relative paths to weed source. * workflow: remove emojis from echo statements * test: format s3tables client.go * workflow: fix go install path to ./weed * ci: fail s3 tables tests if any command in pipeline fails * s3tables: use path.Join for path construction and align namespace paths * s3tables: improve integration test stability and error reporting * s3tables: propagate request context to filer operations * s3tables: clean up unused code and improve error response formatting * Refine S3 Tables implementation to address code review feedback - Standardize namespace representation to []string - Improve listing logic with pagination and StartFromFileName - Enhance error handling with sentinel errors and robust checks - Add JSON encoding error logging - Fix CI workflow to use gofmt -l - Standardize timestamps in directory creation - Validate single-level namespaces * s3tables: further refinements to filer operations and utilities - Add multi-segment namespace support to ARN parsing - Refactor permission checking to use map lookup - Wrap lookup errors with ErrNotFound in filer operations - Standardize splitPath to use path package * test: improve S3 Tables client error handling and cleanup - Add detailed error reporting when decoding failure responses - Remove orphaned comments and unused sections * command: implement graceful shutdown for mini cluster - Introduce MiniClusterCtx to coordinate shutdown across mini services - Update Master, Volume, Filer, S3, and WebDAV servers to respect context cancellation - Ensure all resources are cleaned up properly during test teardown - Integrate MiniClusterCtx in s3tables integration tests * s3tables: fix pagination and enhance error handling in list/delete operations - Fix InclusiveStartFrom logic to ensure exclusive start on continued pages - Prevent duplicates in bucket, namespace, and table listings - Fail fast on listing errors during bucket and namespace deletion - Stop swallowing errors in handleListTables and return proper HTTP error responses * s3tables: align ARN formatting and optimize resource handling - Update generateTableARN to match AWS S3 Tables specification - Move defer r.Body.Close() to follow standard Go patterns - Remove unused generateNamespaceARN helper * command: fix stale error variable logging in filer serving goroutines - Use local 'err' variable instead of stale 'e' from outer scope - Applied to both TLS and non-TLS paths for local listener * s3tables: implement granular authorization and refine error responses - Remove mandatory ACTION_ADMIN at the router level - Enforce granular permissions in bucket and namespace handlers - Prioritize AccountID in ExtractPrincipalFromContext for ARN matching - Distinguish between 404 (NoSuchBucket) and 500 (InternalError) in metadata lookups - Clean up unused imports in s3api_tables.go * test: refactor S3 Tables client for DRYness and multi-segment namespaces - Implement doRequestAndDecode to eliminate HTTP boilerplate - Update client API to accept []string for namespaces to support hierarchy - Standardize error response decoding across all client methods * test: update integration tests to match refactored S3 Tables client - Pass namespaces as []string to support hierarchical structures - Adapt test calls to new client API signatures * s3tables: normalize filer errors and use standard helpers - Migrate from custom ErrNotFound to filer_pb.ErrNotFound - Use filer_pb.LookupEntry for automatic error normalization - Normalize entryExists and attribute lookups * s3tables: harden namespace validation and correct ARN parsing - Prohibit path traversal (".", "..") and "/" in namespaces - Restrict namespace characters to [a-z0-9_] for consistency - Switch to url.PathUnescape for correct decoding of ARN path components - Align ARN parsing regex with single-segment namespace validation * s3tables: improve robustness, security, and error propagation in handlers - Implement strict table name validation (prevention of path traversal and character enforcement) - Add nil checks for entry.Entry in all listing loops to prevent panics - Propagate backend errors instead of swallowing them or assuming 404 - Correctly map filer_pb.ErrNotFound to appropriate S3 error codes - Standardize existence checks across bucket, namespace, and table handlers * test: add miniClusterMutex to prevent race conditions - Introduce sync.Mutex to protect global state (os.Args, os.Chdir) - Ensure serialized initialization of the mini cluster runner - Fix intermittent race conditions during parallel test execution * s3tables: improve error handling and permission logic - Update handleGetNamespace to distinguish between 404 and 500 errors - Refactor CanManagePolicy to use CheckPermission for consistent enforcement - Ensure empty identities are correctly handled in policy management checks * s3tables: optimize regex usage and improve version token uniqueness - Pre-compile regex patterns as package-level variables to avoid re-compilation overhead on every call - Add a random component to version token generation to reduce collision probability under high concurrency * s3tables: harden auth and error handling - Add authorization checks to all S3 Tables handlers (policy, table ops) to enforce security - Improve error handling to distinguish between NotFound (404) and InternalError (500) - Fix directory FileMode usage in filer_ops - Improve test randomness for version tokens - Update permissions comments to acknowledge IAM gaps * S3 Tables: fix gRPC stream loop handling for list operations - Correctly handle io.EOF to terminate loops gracefully. - Propagate other errors to prevent silent failures. - Ensure all list results are processed effectively. * S3 Tables: validate ARN namespace to prevent path traversal - Enforce validation on decoded namespace in parseTableFromARN. - Ensures path components are safe after URL unescaping. * S3 Tables: secure API router with IAM authentication - Wrap S3 Tables handler with authenticateS3Tables. - Use AuthSignatureOnly to enforce valid credentials while delegating granular authorization to handlers. - Prevent anonymous access to all S3 Tables endpoints. * S3 Tables: fix gRPC stream loop handling in namespace handlers - Correctly handle io.EOF in handleListNamespaces and handleDeleteNamespace. - Propagate other errors to prevent silent failures or accidental data loss. - Added necessary io import. * S3 Tables: use os.ModeDir constant in filer_ops.go - Replace magic number 1<<31 with os.ModeDir for better readability. - Added necessary os import. * s3tables: improve principal extraction using identity context * s3tables: remove duplicate comment in permissions.go * s3tables test: improve error reporting on decoding failure * s3tables: implement validateTableName helper * s3tables: add table name validation and 404 propagation to policy handlers * s3tables: add table name validation and cleanup duplicated logic in table handlers * s3tables: ensure root tables directory exists before bucket creation * s3tables: implement token-based pagination for table buckets listing * s3tables: implement token-based pagination for namespace listing * s3tables: refine permission helpers to align with operation names * s3tables: return 404 in handleDeleteNamespace if namespace not found * s3tables: fix cross-namespace pagination in listTablesInAllNamespaces * s3tables test: expose pagination parameters in client list methods * s3tables test: update integration tests for new client API * s3tables: use crypto/rand for secure version token generation Replaced math/rand with crypto/rand to ensure version tokens are cryptographically secure and unpredictable for optimistic concurrency control. * s3tables: improve account ID handling and define missing error codes Updated getPrincipalFromRequest to prioritize X-Amz-Account-ID header and added getAccountID helper. Defined ErrVersionTokenMismatch and ErrCodeConflict for better optimistic concurrency support. * s3tables: update bucket handlers for multi-account support Ensured bucket ownership is correctly attributed to the authenticated account ID and updated ARNs to use the request-derived account ID. Added standard S3 existence checks for bucket deletion. * s3tables: update namespace handlers for multi-account support Updated namespace creation to use authenticated account ID for ownership and unified permission checks across all namespace operations to use the correct account principal. * s3tables: implement optimistic concurrency for table deletion Added VersionToken validation to handleDeleteTable. Refactored table listing to use request context for accurate ARN generation and fixed cross-namespace pagination issues. * s3tables: improve resource resolution and error mapping for policies and tagging Refactored resolveResourcePath to return resource type, enabling accurate NoSuchBucket vs NoSuchTable error codes. Added existence checks before deleting policies. * s3tables: enhance test robustness and resilience Updated random string generation to use crypto/rand in s3tables tests. Increased resilience of IAM distributed tests by adding "connection refused" to retryable errors. * s3tables: remove legacy principal fallback header Removed the fallback to X-Amz-Principal in getPrincipalFromRequest as S3 Tables is a new feature and does not require legacy header support. * s3tables: remove unused ExtractPrincipalFromContext function Removed the unused ExtractPrincipalFromContext utility and its accompanying iam/utils import to keep the new s3tables codebase clean. * s3tables: allow hyphens in namespace and table names Relaxed regex validation in utils.go to support hyphens in S3 Tables namespaces and table names, improving consistency with S3 bucket naming and allowing derived names from services like S3 Storage Lens. * s3tables: add isAuthError helper to handler.go * s3tables: refactor permission checks to use resource owner in bucket handlers * s3tables: refactor permission checks to use resource owner in namespace handlers * s3tables: refactor permission checks to use resource owner in table handlers * s3tables: refactor permission checks to use resource owner in policy and tagging handlers * ownerAccountID * s3tables: implement strict AWS-aligned name validation for buckets, namespaces, and tables * s3tables: enforce strict resource ownership and implement result filtering for buckets * s3tables: enforce strict resource ownership and implement result filtering for namespaces * s3tables: enforce strict resource ownership and implement result filtering for tables * s3tables: align getPrincipalFromRequest with account ID for IAM compatibility * s3tables: fix inconsistent permission check in handleCreateTableBucket * s3tables: improve pagination robustness and error handling in table listing handlers * s3tables: refactor handleDeleteTableBucket to use strongly typed AuthError * s3tables: align ARN regex patterns with S3 standards and refactor to constants * s3tables: standardize access denied errors using ErrAccessDenied constant * go fmt * s3tables: fix double-write issue in handleListTables Remove premature HTTP error writes from within WithFilerClient closure to prevent duplicate status code responses. Error handling is now consistently performed at the top level using isAuthError. * s3tables: update bucket name validation message Remove "underscores" from error message to accurately reflect that bucket names only allow lowercase letters, numbers, and hyphens. * s3tables: add table policy test coverage Add comprehensive test coverage for table policy operations: - Added PutTablePolicy, GetTablePolicy, DeleteTablePolicy methods to test client - Implemented testTablePolicy lifecycle test validating Put/Get/Delete operations - Verified error handling for missing policies * follow aws spec * s3tables: add request body size limiting Add request body size limiting (10MB) to readRequestBody method: - Define maxRequestBodySize constant to prevent unbounded reads - Use io.LimitReader to enforce size limit - Add explicit error handling for oversized requests - Prevents potential DoS attacks via large request bodies * S3 Tables API now properly enforces resource policies addressing the critical security gap where policies were created but never evaluated. * s3tables: Add upper bound validation for MaxTables parameter MaxTables is user-controlled and influences gRPC ListEntries limits via uint32(maxTables*2). Without an upper bound, very large values can overflow uint32 or cause excessively large directory scans. Cap MaxTables to 1000 and return InvalidRequest for out-of-range values, consistent with S3 MaxKeys handling. * s3tables: Add upper bound validation for MaxBuckets parameter MaxBuckets is user-controlled and used in uint32(maxBuckets*2) for ListEntries. Very large values can overflow uint32 or trigger overly expensive scans. Cap MaxBuckets to 1000 and reject out-of-range values, consistent with MaxTables handling and S3 MaxKeys validation elsewhere in the codebase. * s3tables: Validate bucket name in parseBucketNameFromARN() Enforce the same bucket name validation rules (length, characters, reserved prefixes/suffixes) when extracting from ARN. This prevents accepting ARNs that the system would never create and ensures consistency with CreateTableBucket validation. * s3tables: Fix parseTableFromARN() namespace and table name validation - Remove dead URL unescape for namespace (regex [a-z0-9_]+ cannot contain percent-escapes) - Add URL decoding and validation of extracted table name via validateTableName() to prevent callers from bypassing request validation done in other paths * s3tables: Rename tableMetadataInternal.Schema to Metadata The field name 'Schema' was confusing given it holds a *TableMetadata struct and serializes as 'metadata' in JSON. Rename to 'Metadata' for clarity and consistency with the JSON tag and intended meaning. * s3tables: Improve bucket name validation error message Replace misleading character-only error message with generic 'invalid bucket name'. The isValidBucketName() function checks multiple constraints beyond character set (length, reserved prefixes/suffixes, start/end rules), so a specific character message is inaccurate. * s3tables: Separate permission checks for tagging and untagging - Add CanTagResource() to check TagResource permission - Add CanUntagResource() to check UntagResource permission - Update CanManageTags() to check both operations (OR logic) This prevents UntagResource from incorrectly checking 'ManageTags' permission and ensures each operation validates the correct permission when per-operation permissions are enforced. * s3tables: Consolidate getPrincipalFromRequest and getAccountID into single method Both methods had identical implementations - they return the account ID from request header or fall back to handler's default. Remove the duplicate getPrincipalFromRequest and use getAccountID throughout, with updated comment explaining its dual role as both caller identity and principal for permission checks. * s3tables: Fetch bucket policy in handleListTagsForResource for permission evaluation Update handleListTagsForResource to fetch and pass bucket policy to CheckPermission, matching the behavior of handleTagResource/handleUntagResource. This enables bucket-policy-based permission grants to be evaluated for ListTagsForResource, not just ownership-based checks. * s3tables: Extract resource owner and bucket extraction into helper method Create extractResourceOwnerAndBucket() helper to consolidate the repeated pattern of unmarshaling metadata and extracting bucket name from resource path. This pattern was duplicated in handleTagResource, handleListTagsForResource, and handleUntagResource. Update all three handlers to use the helper. Also update remaining uses of getPrincipalFromRequest() (in handler_bucket_create, handler_bucket_get_list_delete, handler_namespace) to use getAccountID() after consolidating the two identical methods. * s3tables: Add log message when cluster shutdown times out The timeout path (2 second wait for graceful shutdown) was silent. Add a warning log message when it occurs to help diagnose flaky test issues and indicate when the mini cluster didn't shut down cleanly. * s3tables: Use policy_engine wildcard matcher for complete IAM compatibility Replace the custom suffix-only wildcard implementation in matchesActionPattern and matchesPrincipal with the policy_engine.MatchesWildcard function from PR #8052. This enables full wildcard support including: - Middle wildcards: s3tables:Get*Table matches GetTable - Question mark wildcards: Get? matches any single character - Combined patterns: s3tables:*Table* matches any action containing 'Table' Benefits: - Code reuse: eliminates duplicate wildcard logic - Complete IAM compatibility: supports all AWS wildcard patterns - Performance: uses efficient O(n) backtracking algorithm - Consistency: same wildcard behavior across S3 API and S3 Tables Add comprehensive unit tests covering exact matches, suffix wildcards, middle wildcards, question marks, and combined patterns for both action and principal matching. * go fmt * s3tables: Fix vet error - remove undefined c.t reference in Stop() The TestCluster.Stop() method doesn't have access to testing.T object. Remove the log statement and keep the timeout handling comment for clarity. The original intent (warning about shutdown timeout) is still captured in the code comment explaining potential issues. * clean up * s3tables: Add t field to TestCluster for logging Add *testing.T field to TestCluster struct and initialize it in startMiniCluster. This allows Stop() to properly log warnings when cluster shutdown times out. Includes the t field in the test cluster initialization and restores the logging statement in Stop(). * s3tables: Fix bucket policy error handling in permission checks Replace error-swallowing pattern where all errors from getExtendedAttribute were ignored for bucket policy reads. Now properly distinguish between: - ErrAttributeNotFound: Policy not found is expected; continue with empty policy - Other errors: Return internal server error and stop processing Applied fix to all bucket policy reads in: - handleDeleteTableBucketPolicy (line 220) - handleTagResource (line 313) - handleUntagResource (line 405) - handleListTagsForResource (line 488) - And additional occurrences in closures This prevents silent failures and ensures policy-related errors are surfaced to callers rather than being silently ignored. * s3tables: Pre-validate namespace to return 400 instead of 500 Move validateNamespace call outside of filerClient.WithFilerClient closure so that validation errors return HTTP 400 (InvalidRequest) instead of 500 (InternalError). Before: Validation error inside closure → treated as internal error → 500 After: Validation error before closure → handled as bad request → 400 This provides correct error semantics: namespace validation is an input validation issue, not a server error. * Update weed/s3api/s3tables/handler.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * s3tables: Normalize action names to include service prefix Add automatic normalization of operations to full IAM-style action names (e.g., 's3tables:CreateTableBucket') in CheckPermission(). This ensures policy statements using prefixed actions (s3tables:*) correctly match operations evaluated by permission helpers. Also fixes incorrect r.Context() passed to GetIdentityNameFromContext which expects *http.Request. Now passes r directly. * s3tables: Use policy framework for table creation authorization Replace strict ownership check in CreateTable with policy-based authorization. Now checks both namespace and bucket policies for CreateTable permission, allowing delegation via resource policies while still respecting owner bypass. Authorization logic: - Namespace policy grants CreateTable → allowed - Bucket policy grants CreateTable → allowed - Otherwise → denied (even if same owner) This enables cross-principal table creation via policies while maintaining security through explicit allow/deny semantics. * s3tables: Use policy framework for GetTable authorization Replace strict ownership check with policy-based authorization in GetTable. Now checks both table and bucket policies for GetTable permission, allowing authorized non-owners to read table metadata. Authorization logic: - Table policy grants GetTable → allowed - Bucket policy grants GetTable → allowed - Otherwise → 404 NotFound (no access disclosed) Maintains security through policy evaluation while enabling read delegation. * s3tables: Generate ARNs using resource owner account ID Change ARN generation to use resource OwnerAccountID instead of caller identity (h.getAccountID(r)). This ensures ARNs are stable and consistent regardless of which principal accesses the resource. Updated generateTableBucketARN and generateTableARN function signatures to accept ownerAccountID parameter. All call sites updated to pass the resource owner's account ID from metadata. This prevents ARN inconsistency issues when multiple principals have access to the same resource via policies. * s3tables: Fix remaining policy error handling in namespace and bucket handlers Replace silent error swallowing (err == nil) with proper error distinction for bucket policy reads. Now properly checks ErrAttributeNotFound and propagates other errors as internal server errors. Fixed 5 locations: - handleCreateNamespace (policy fetch) - handleDeleteNamespace (policy fetch) - handleListNamespaces (policy fetch) - handleGetNamespace (policy fetch) - handleGetTableBucket (policy fetch) This prevents masking of filer issues when policies cannot be read due to I/O errors or other transient failures. * ci: Pin GitHub Actions to commit SHAs for s3-tables-tests Update all action refs to use pinned commit SHAs instead of floating tags: - actions/checkout: @v6 → @8e8c483 (v4) - actions/setup-go: @v6 → @0c52d54 (v5) - actions/upload-artifact: @v6 → @65d8626 (v4) Pinned SHAs improve reproducibility and reduce supply chain risk by preventing accidental or malicious changes in action releases. Aligns with repository conventions used in other workflows (e.g., go.yml). * s3tables: Add resource ARN validation to policy evaluation Implement resource-specific policy validation to prevent over-broad permission grants. Add matchesResource and matchesResourcePattern functions to validate statement Resource fields against specific resource ARNs. Add new CheckPermissionWithResource function that includes resource ARN validation, while keeping CheckPermission unchanged for backward compatibility. This enables policies to grant access to specific resources only: - statements with Resource: "arn:aws:s3tables:...:bucket/specific-bucket/*" will only match when accessing that specific bucket - statements without Resource field match all resources (implicit *) - resource patterns support wildcards (* for any sequence, ? for single char) For future use: Handlers can call CheckPermissionWithResource with the target resource ARN to enforce resource-level access control. * Revert "ci: Pin GitHub Actions to commit SHAs for s3-tables-tests" This reverts commit 01da26fbcb. * s3tables: Remove duplicate bucket extraction logic in helper Move bucket name extraction outside the if/else block in extractResourceOwnerAndBucket since the logic is identical for both ResourceTypeTable and ResourceTypeBucket cases. This reduces code duplication and improves maintainability. The extraction pattern (parts[1] from /tables/{bucket}/...) works for both resource types, so it's now performed once before the type-specific metadata unmarshaling. * go fmt * s3tables: Fix ownership consistency across handlers Address three related ownership consistency issues: 1. CreateNamespace now sets OwnerAccountID to bucketMetadata.OwnerAccountID instead of request principal. This prevents namespaces created by delegated callers (via bucket policy) from becoming unmanageable, since ListNamespaces filters by bucket owner. 2. CreateTable now: - Fetches bucket metadata to use correct owner for bucket policy evaluation - Uses namespaceMetadata.OwnerAccountID for namespace policy checks - Uses bucketMetadata.OwnerAccountID for bucket policy checks - Sets table OwnerAccountID to namespaceMetadata.OwnerAccountID (inherited) 3. GetTable now: - Fetches bucket metadata to use correct owner for bucket policy evaluation - Uses metadata.OwnerAccountID for table policy checks - Uses bucketMetadata.OwnerAccountID for bucket policy checks This ensures: - Bucket owner retains implicit "owner always allowed" behavior even when evaluating bucket policies - Ownership hierarchy is consistent (namespace owned by bucket, table owned by namespace) - Cross-principal delegation via policies doesn't break ownership chains * s3tables: Fix ListTables authorization and policy parsing Make ListTables authorization consistent with GetTable/CreateTable: 1. ListTables authorization now evaluates policies instead of owner-only checks: - For namespace listing: checks namespace policy AND bucket policy - For bucket-wide listing: checks bucket policy - Uses CanListTables permission framework 2. Remove owner-only filter in listTablesWithClient that prevented policy-based sharing of tables. Authorization is now enforced at the handler level, so all tables in the namespace/bucket are returned to authorized callers (who have access either via ownership or policy). 3. Add flexible PolicyDocument.UnmarshalJSON to support both single-object and array forms of Statement field: - Handles: {"Statement": {...}} - Handles: {"Statement": [{...}, {...}]} - Improves AWS IAM compatibility This ensures cross-account table listing works when delegated via bucket/namespace policies, consistent with the authorization model for other operations. * go fmt * s3tables: Separate table name pattern constant for clarity Define a separate tableNamePatternStr constant for the table name component in the ARN regex, even though it currently has the same value as tableNamespacePatternStr. This improves code clarity and maintainability, making it easier to modify if the naming rules for tables and namespaces diverge in the future. * refactor --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	6 days ago
Chris Lu	549b65785d	refactor	6 days ago
Chris Lu	590e7efbef	s3tables: Separate table name pattern constant for clarity Define a separate tableNamePatternStr constant for the table name component in the ARN regex, even though it currently has the same value as tableNamespacePatternStr. This improves code clarity and maintainability, making it easier to modify if the naming rules for tables and namespaces diverge in the future.	6 days ago
Chris Lu	78c00e313a	go fmt	6 days ago
Chris Lu	f5d26b803b	s3tables: Fix ListTables authorization and policy parsing Make ListTables authorization consistent with GetTable/CreateTable: 1. ListTables authorization now evaluates policies instead of owner-only checks: - For namespace listing: checks namespace policy AND bucket policy - For bucket-wide listing: checks bucket policy - Uses CanListTables permission framework 2. Remove owner-only filter in listTablesWithClient that prevented policy-based sharing of tables. Authorization is now enforced at the handler level, so all tables in the namespace/bucket are returned to authorized callers (who have access either via ownership or policy). 3. Add flexible PolicyDocument.UnmarshalJSON to support both single-object and array forms of Statement field: - Handles: {"Statement": {...}} - Handles: {"Statement": [{...}, {...}]} - Improves AWS IAM compatibility This ensures cross-account table listing works when delegated via bucket/namespace policies, consistent with the authorization model for other operations.	6 days ago
Chris Lu	25b0f86bda	s3tables: Fix ownership consistency across handlers Address three related ownership consistency issues: 1. CreateNamespace now sets OwnerAccountID to bucketMetadata.OwnerAccountID instead of request principal. This prevents namespaces created by delegated callers (via bucket policy) from becoming unmanageable, since ListNamespaces filters by bucket owner. 2. CreateTable now: - Fetches bucket metadata to use correct owner for bucket policy evaluation - Uses namespaceMetadata.OwnerAccountID for namespace policy checks - Uses bucketMetadata.OwnerAccountID for bucket policy checks - Sets table OwnerAccountID to namespaceMetadata.OwnerAccountID (inherited) 3. GetTable now: - Fetches bucket metadata to use correct owner for bucket policy evaluation - Uses metadata.OwnerAccountID for table policy checks - Uses bucketMetadata.OwnerAccountID for bucket policy checks This ensures: - Bucket owner retains implicit "owner always allowed" behavior even when evaluating bucket policies - Ownership hierarchy is consistent (namespace owned by bucket, table owned by namespace) - Cross-principal delegation via policies doesn't break ownership chains	7 days ago
Chris Lu	b049e883e1	go fmt	7 days ago
Chris Lu	c99e8d4152	s3tables: Remove duplicate bucket extraction logic in helper Move bucket name extraction outside the if/else block in extractResourceOwnerAndBucket since the logic is identical for both ResourceTypeTable and ResourceTypeBucket cases. This reduces code duplication and improves maintainability. The extraction pattern (parts[1] from /tables/{bucket}/...) works for both resource types, so it's now performed once before the type-specific metadata unmarshaling.	7 days ago
Chris Lu	21584e4ac8	s3tables: Add resource ARN validation to policy evaluation Implement resource-specific policy validation to prevent over-broad permission grants. Add matchesResource and matchesResourcePattern functions to validate statement Resource fields against specific resource ARNs. Add new CheckPermissionWithResource function that includes resource ARN validation, while keeping CheckPermission unchanged for backward compatibility. This enables policies to grant access to specific resources only: - statements with Resource: "arn:aws:s3tables:...:bucket/specific-bucket/*" will only match when accessing that specific bucket - statements without Resource field match all resources (implicit *) - resource patterns support wildcards (* for any sequence, ? for single char) For future use: Handlers can call CheckPermissionWithResource with the target resource ARN to enforce resource-level access control.	7 days ago
Chris Lu	2c45b69775	s3tables: Fix remaining policy error handling in namespace and bucket handlers Replace silent error swallowing (err == nil) with proper error distinction for bucket policy reads. Now properly checks ErrAttributeNotFound and propagates other errors as internal server errors. Fixed 5 locations: - handleCreateNamespace (policy fetch) - handleDeleteNamespace (policy fetch) - handleListNamespaces (policy fetch) - handleGetNamespace (policy fetch) - handleGetTableBucket (policy fetch) This prevents masking of filer issues when policies cannot be read due to I/O errors or other transient failures.	7 days ago
Chris Lu	b7bba7e7dc	s3tables: Generate ARNs using resource owner account ID Change ARN generation to use resource OwnerAccountID instead of caller identity (h.getAccountID(r)). This ensures ARNs are stable and consistent regardless of which principal accesses the resource. Updated generateTableBucketARN and generateTableARN function signatures to accept ownerAccountID parameter. All call sites updated to pass the resource owner's account ID from metadata. This prevents ARN inconsistency issues when multiple principals have access to the same resource via policies.	7 days ago
Chris Lu	e7b2869aa9	s3tables: Use policy framework for GetTable authorization Replace strict ownership check with policy-based authorization in GetTable. Now checks both table and bucket policies for GetTable permission, allowing authorized non-owners to read table metadata. Authorization logic: - Table policy grants GetTable → allowed - Bucket policy grants GetTable → allowed - Otherwise → 404 NotFound (no access disclosed) Maintains security through policy evaluation while enabling read delegation.	7 days ago
Chris Lu	bea0f8eda0	s3tables: Use policy framework for table creation authorization Replace strict ownership check in CreateTable with policy-based authorization. Now checks both namespace and bucket policies for CreateTable permission, allowing delegation via resource policies while still respecting owner bypass. Authorization logic: - Namespace policy grants CreateTable → allowed - Bucket policy grants CreateTable → allowed - Otherwise → denied (even if same owner) This enables cross-principal table creation via policies while maintaining security through explicit allow/deny semantics.	7 days ago
Chris Lu	cf5043a9f9	s3tables: Normalize action names to include service prefix Add automatic normalization of operations to full IAM-style action names (e.g., 's3tables:CreateTableBucket') in CheckPermission(). This ensures policy statements using prefixed actions (s3tables:*) correctly match operations evaluated by permission helpers. Also fixes incorrect r.Context() passed to GetIdentityNameFromContext which expects *http.Request. Now passes r directly.	7 days ago
Ping Qiu	5c8de5e282	fix: close volumes and EC shards in tests for Windows compatibility (#8152) * fix: close volumes and EC shards in tests to prevent Windows cleanup failures On Windows, t.TempDir() cleanup fails when test files are still open because Windows enforces mandatory file locking. Add defer v.Close(), defer store.Close(), and EC volume cleanup to ensure all file handles are released before temp directory removal. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: extract closeEcVolumes helper to reduce duplication Address code review feedback by extracting the repeated EC volume cleanup loop into a closeEcVolumes() helper function. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	7 days ago
Chris Lu	ee468749bd	Update weed/s3api/s3tables/handler.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	7 days ago