seaweedfs

Commit Graph

Author	SHA1	Message	Date
Chris Lu	2f155ee5ee	feat: Add S3 Tables support for Iceberg tabular data (#8147 ) * s3tables: extract utility and filer operations to separate modules - Move ARN parsing, path helpers, and metadata structures to utils.go - Extract all extended attribute and filer operations to filer_ops.go - Reduces code duplication and improves modularity - Improves code organization and maintainability * s3tables: split table bucket operations into focused modules - Create bucket_create.go for CreateTableBucket operation - Create bucket_get_list_delete.go for Get, List, Delete operations - Related operations grouped for better maintainability - Each file has a single, clear responsibility - Improves code clarity and makes it easier to test * s3tables: simplify handler by removing duplicate utilities - Reduce handler.go from 370 to 195 lines (47% reduction) - Remove duplicate ARN parsing and path helper functions - Remove filer operation methods moved to filer_ops.go - Remove metadata structure definitions moved to utils.go - Keep handler focused on request routing and response formatting - Maintains all functionality with improved code organization * s3tables: complete s3tables package implementation - namespace.go: namespace CRUD operations (310 lines) - table.go: table CRUD operations with Iceberg schema support (409 lines) - policy.go: resource policies and tagging operations (419 lines) - types.go: request/response types and error definitions (290 lines) - All handlers updated to use standalone utilities from utils.go - All files follow single responsibility principle * s3api: add S3 Tables integration layer - Create s3api_tables.go to integrate S3 Tables with S3 API server - Implement S3 Tables route matcher for X-Amz-Target header - Register S3 Tables routes with API router - Provide gRPC filer client interface for S3 Tables handlers - All S3 Tables operations accessible via S3 API endpoint * s3api: register S3 Tables routes in API server - Add S3 Tables route registration in s3api_server.go registerRouter method - Enable S3 Tables API operations to be routed through S3 API server - Routes handled by s3api_tables.go integration layer - Minimal changes to existing S3 API structure * test: add S3 Tables test infrastructure - Create setup.go with TestCluster and S3TablesClient definitions - Create client.go with HTTP client methods for all operations - Test utilities and client methods organized for reusability - Foundation for S3 Tables integration tests * test: add S3 Tables integration tests - Comprehensive integration tests for all 23 S3 Tables operations - Test cluster setup based on existing S3 integration tests - Tests cover: * Table bucket lifecycle (create, get, list, delete) * Namespace operations * Table CRUD with Iceberg schema * Table bucket and table policies * Resource tagging operations - Ready for CI/CD pipeline integration * ci: add S3 Tables integration tests to GitHub Actions - Create new workflow for S3 Tables integration testing - Add build verification job for s3tables package and s3api integration - Add format checking for S3 Tables code - Add go vet checks for code quality - Workflow runs on all pull requests - Includes test output logging and artifact upload on failure * s3tables: add handler_ prefix to operation handler files - Rename bucket_create.go → handler_bucket_create.go - Rename bucket_get_list_delete.go → handler_bucket_get_list_delete.go - Rename namespace.go → handler_namespace.go - Rename table.go → handler_table.go - Rename policy.go → handler_policy.go Improves file organization by clearly identifying handler implementations. No code changes, refactoring only. * s3tables test: refactor to eliminate duplicate definitions - Move all client methods to client.go - Remove duplicate types/constants from s3tables_integration_test.go - Keep setup.go for test infrastructure - Keep integration test logic in s3tables_integration_test.go - Clean up unused imports - Test compiles successfully * Delete client_methods.go * s3tables: add bucket name validation and fix error handling - Add isValidBucketName validation function for [a-z0-9_-] characters - Validate bucket name characters match ARN parsing regex - Fix error handling in WithFilerClient closure - properly check for lookup errors - Add error handling for json.Marshal calls (metadata and tags) - Improve error messages and logging * s3tables: add error handling for json.Marshal calls - Add error handling in handler_namespace.go (metadata marshaling) - Add error handling in handler_table.go (metadata and tags marshaling) - Add error handling in handler_policy.go (tag marshaling in TagResource and UntagResource) - Return proper errors with context instead of silently ignoring failures * s3tables: replace custom splitPath with stdlib functions - Remove custom splitPath implementation (23 lines) - Use filepath.Dir and filepath.Base from stdlib - More robust and handles edge cases correctly - Reduces code duplication * s3tables: improve error handling specificity in ListTableBuckets - Specifically check for 'not found' errors instead of catching all errors - Return empty list only when directory doesn't exist - Propagate other errors (network, permission) with context - Prevents masking real errors * s3api_tables: optimize action validation with map lookup - Replace O(n) slice iteration with O(1) map lookup - Move s3TablesActionsMap to package level - Avoid recreating the map on every function call - Improves performance for request validation * s3tables: implement permission checking and authorization - Add permissions.go with permission definitions and checks - Define permissions for all 21 S3 Tables operations - Add permission checking helper functions - Add getPrincipalFromRequest to extract caller identity - Implement access control in CreateTableBucket, GetTableBucket, DeleteTableBucket - Return 403 Forbidden for unauthorized operations - Only bucket owner can perform operations (extensible for future policies) - Add AuthError type for authorization failures * workflow: fix s3 tables tests path and working directory The workflow was failing because it was running inside 'weed' directory, but the tests are at the repository root. Removed working-directory default and updated relative paths to weed source. * workflow: remove emojis from echo statements * test: format s3tables client.go * workflow: fix go install path to ./weed * ci: fail s3 tables tests if any command in pipeline fails * s3tables: use path.Join for path construction and align namespace paths * s3tables: improve integration test stability and error reporting * s3tables: propagate request context to filer operations * s3tables: clean up unused code and improve error response formatting * Refine S3 Tables implementation to address code review feedback - Standardize namespace representation to []string - Improve listing logic with pagination and StartFromFileName - Enhance error handling with sentinel errors and robust checks - Add JSON encoding error logging - Fix CI workflow to use gofmt -l - Standardize timestamps in directory creation - Validate single-level namespaces * s3tables: further refinements to filer operations and utilities - Add multi-segment namespace support to ARN parsing - Refactor permission checking to use map lookup - Wrap lookup errors with ErrNotFound in filer operations - Standardize splitPath to use path package * test: improve S3 Tables client error handling and cleanup - Add detailed error reporting when decoding failure responses - Remove orphaned comments and unused sections * command: implement graceful shutdown for mini cluster - Introduce MiniClusterCtx to coordinate shutdown across mini services - Update Master, Volume, Filer, S3, and WebDAV servers to respect context cancellation - Ensure all resources are cleaned up properly during test teardown - Integrate MiniClusterCtx in s3tables integration tests * s3tables: fix pagination and enhance error handling in list/delete operations - Fix InclusiveStartFrom logic to ensure exclusive start on continued pages - Prevent duplicates in bucket, namespace, and table listings - Fail fast on listing errors during bucket and namespace deletion - Stop swallowing errors in handleListTables and return proper HTTP error responses * s3tables: align ARN formatting and optimize resource handling - Update generateTableARN to match AWS S3 Tables specification - Move defer r.Body.Close() to follow standard Go patterns - Remove unused generateNamespaceARN helper * command: fix stale error variable logging in filer serving goroutines - Use local 'err' variable instead of stale 'e' from outer scope - Applied to both TLS and non-TLS paths for local listener * s3tables: implement granular authorization and refine error responses - Remove mandatory ACTION_ADMIN at the router level - Enforce granular permissions in bucket and namespace handlers - Prioritize AccountID in ExtractPrincipalFromContext for ARN matching - Distinguish between 404 (NoSuchBucket) and 500 (InternalError) in metadata lookups - Clean up unused imports in s3api_tables.go * test: refactor S3 Tables client for DRYness and multi-segment namespaces - Implement doRequestAndDecode to eliminate HTTP boilerplate - Update client API to accept []string for namespaces to support hierarchy - Standardize error response decoding across all client methods * test: update integration tests to match refactored S3 Tables client - Pass namespaces as []string to support hierarchical structures - Adapt test calls to new client API signatures * s3tables: normalize filer errors and use standard helpers - Migrate from custom ErrNotFound to filer_pb.ErrNotFound - Use filer_pb.LookupEntry for automatic error normalization - Normalize entryExists and attribute lookups * s3tables: harden namespace validation and correct ARN parsing - Prohibit path traversal (".", "..") and "/" in namespaces - Restrict namespace characters to [a-z0-9_] for consistency - Switch to url.PathUnescape for correct decoding of ARN path components - Align ARN parsing regex with single-segment namespace validation * s3tables: improve robustness, security, and error propagation in handlers - Implement strict table name validation (prevention of path traversal and character enforcement) - Add nil checks for entry.Entry in all listing loops to prevent panics - Propagate backend errors instead of swallowing them or assuming 404 - Correctly map filer_pb.ErrNotFound to appropriate S3 error codes - Standardize existence checks across bucket, namespace, and table handlers * test: add miniClusterMutex to prevent race conditions - Introduce sync.Mutex to protect global state (os.Args, os.Chdir) - Ensure serialized initialization of the mini cluster runner - Fix intermittent race conditions during parallel test execution * s3tables: improve error handling and permission logic - Update handleGetNamespace to distinguish between 404 and 500 errors - Refactor CanManagePolicy to use CheckPermission for consistent enforcement - Ensure empty identities are correctly handled in policy management checks * s3tables: optimize regex usage and improve version token uniqueness - Pre-compile regex patterns as package-level variables to avoid re-compilation overhead on every call - Add a random component to version token generation to reduce collision probability under high concurrency * s3tables: harden auth and error handling - Add authorization checks to all S3 Tables handlers (policy, table ops) to enforce security - Improve error handling to distinguish between NotFound (404) and InternalError (500) - Fix directory FileMode usage in filer_ops - Improve test randomness for version tokens - Update permissions comments to acknowledge IAM gaps * S3 Tables: fix gRPC stream loop handling for list operations - Correctly handle io.EOF to terminate loops gracefully. - Propagate other errors to prevent silent failures. - Ensure all list results are processed effectively. * S3 Tables: validate ARN namespace to prevent path traversal - Enforce validation on decoded namespace in parseTableFromARN. - Ensures path components are safe after URL unescaping. * S3 Tables: secure API router with IAM authentication - Wrap S3 Tables handler with authenticateS3Tables. - Use AuthSignatureOnly to enforce valid credentials while delegating granular authorization to handlers. - Prevent anonymous access to all S3 Tables endpoints. * S3 Tables: fix gRPC stream loop handling in namespace handlers - Correctly handle io.EOF in handleListNamespaces and handleDeleteNamespace. - Propagate other errors to prevent silent failures or accidental data loss. - Added necessary io import. * S3 Tables: use os.ModeDir constant in filer_ops.go - Replace magic number 1<<31 with os.ModeDir for better readability. - Added necessary os import. * s3tables: improve principal extraction using identity context * s3tables: remove duplicate comment in permissions.go * s3tables test: improve error reporting on decoding failure * s3tables: implement validateTableName helper * s3tables: add table name validation and 404 propagation to policy handlers * s3tables: add table name validation and cleanup duplicated logic in table handlers * s3tables: ensure root tables directory exists before bucket creation * s3tables: implement token-based pagination for table buckets listing * s3tables: implement token-based pagination for namespace listing * s3tables: refine permission helpers to align with operation names * s3tables: return 404 in handleDeleteNamespace if namespace not found * s3tables: fix cross-namespace pagination in listTablesInAllNamespaces * s3tables test: expose pagination parameters in client list methods * s3tables test: update integration tests for new client API * s3tables: use crypto/rand for secure version token generation Replaced math/rand with crypto/rand to ensure version tokens are cryptographically secure and unpredictable for optimistic concurrency control. * s3tables: improve account ID handling and define missing error codes Updated getPrincipalFromRequest to prioritize X-Amz-Account-ID header and added getAccountID helper. Defined ErrVersionTokenMismatch and ErrCodeConflict for better optimistic concurrency support. * s3tables: update bucket handlers for multi-account support Ensured bucket ownership is correctly attributed to the authenticated account ID and updated ARNs to use the request-derived account ID. Added standard S3 existence checks for bucket deletion. * s3tables: update namespace handlers for multi-account support Updated namespace creation to use authenticated account ID for ownership and unified permission checks across all namespace operations to use the correct account principal. * s3tables: implement optimistic concurrency for table deletion Added VersionToken validation to handleDeleteTable. Refactored table listing to use request context for accurate ARN generation and fixed cross-namespace pagination issues. * s3tables: improve resource resolution and error mapping for policies and tagging Refactored resolveResourcePath to return resource type, enabling accurate NoSuchBucket vs NoSuchTable error codes. Added existence checks before deleting policies. * s3tables: enhance test robustness and resilience Updated random string generation to use crypto/rand in s3tables tests. Increased resilience of IAM distributed tests by adding "connection refused" to retryable errors. * s3tables: remove legacy principal fallback header Removed the fallback to X-Amz-Principal in getPrincipalFromRequest as S3 Tables is a new feature and does not require legacy header support. * s3tables: remove unused ExtractPrincipalFromContext function Removed the unused ExtractPrincipalFromContext utility and its accompanying iam/utils import to keep the new s3tables codebase clean. * s3tables: allow hyphens in namespace and table names Relaxed regex validation in utils.go to support hyphens in S3 Tables namespaces and table names, improving consistency with S3 bucket naming and allowing derived names from services like S3 Storage Lens. * s3tables: add isAuthError helper to handler.go * s3tables: refactor permission checks to use resource owner in bucket handlers * s3tables: refactor permission checks to use resource owner in namespace handlers * s3tables: refactor permission checks to use resource owner in table handlers * s3tables: refactor permission checks to use resource owner in policy and tagging handlers * ownerAccountID * s3tables: implement strict AWS-aligned name validation for buckets, namespaces, and tables * s3tables: enforce strict resource ownership and implement result filtering for buckets * s3tables: enforce strict resource ownership and implement result filtering for namespaces * s3tables: enforce strict resource ownership and implement result filtering for tables * s3tables: align getPrincipalFromRequest with account ID for IAM compatibility * s3tables: fix inconsistent permission check in handleCreateTableBucket * s3tables: improve pagination robustness and error handling in table listing handlers * s3tables: refactor handleDeleteTableBucket to use strongly typed AuthError * s3tables: align ARN regex patterns with S3 standards and refactor to constants * s3tables: standardize access denied errors using ErrAccessDenied constant * go fmt * s3tables: fix double-write issue in handleListTables Remove premature HTTP error writes from within WithFilerClient closure to prevent duplicate status code responses. Error handling is now consistently performed at the top level using isAuthError. * s3tables: update bucket name validation message Remove "underscores" from error message to accurately reflect that bucket names only allow lowercase letters, numbers, and hyphens. * s3tables: add table policy test coverage Add comprehensive test coverage for table policy operations: - Added PutTablePolicy, GetTablePolicy, DeleteTablePolicy methods to test client - Implemented testTablePolicy lifecycle test validating Put/Get/Delete operations - Verified error handling for missing policies * follow aws spec * s3tables: add request body size limiting Add request body size limiting (10MB) to readRequestBody method: - Define maxRequestBodySize constant to prevent unbounded reads - Use io.LimitReader to enforce size limit - Add explicit error handling for oversized requests - Prevents potential DoS attacks via large request bodies * S3 Tables API now properly enforces resource policies addressing the critical security gap where policies were created but never evaluated. * s3tables: Add upper bound validation for MaxTables parameter MaxTables is user-controlled and influences gRPC ListEntries limits via uint32(maxTables2). Without an upper bound, very large values can overflow uint32 or cause excessively large directory scans. Cap MaxTables to 1000 and return InvalidRequest for out-of-range values, consistent with S3 MaxKeys handling. s3tables: Add upper bound validation for MaxBuckets parameter MaxBuckets is user-controlled and used in uint32(maxBuckets2) for ListEntries. Very large values can overflow uint32 or trigger overly expensive scans. Cap MaxBuckets to 1000 and reject out-of-range values, consistent with MaxTables handling and S3 MaxKeys validation elsewhere in the codebase. s3tables: Validate bucket name in parseBucketNameFromARN() Enforce the same bucket name validation rules (length, characters, reserved prefixes/suffixes) when extracting from ARN. This prevents accepting ARNs that the system would never create and ensures consistency with CreateTableBucket validation. * s3tables: Fix parseTableFromARN() namespace and table name validation - Remove dead URL unescape for namespace (regex [a-z0-9_]+ cannot contain percent-escapes) - Add URL decoding and validation of extracted table name via validateTableName() to prevent callers from bypassing request validation done in other paths * s3tables: Rename tableMetadataInternal.Schema to Metadata The field name 'Schema' was confusing given it holds a TableMetadata struct and serializes as 'metadata' in JSON. Rename to 'Metadata' for clarity and consistency with the JSON tag and intended meaning. s3tables: Improve bucket name validation error message Replace misleading character-only error message with generic 'invalid bucket name'. The isValidBucketName() function checks multiple constraints beyond character set (length, reserved prefixes/suffixes, start/end rules), so a specific character message is inaccurate. * s3tables: Separate permission checks for tagging and untagging - Add CanTagResource() to check TagResource permission - Add CanUntagResource() to check UntagResource permission - Update CanManageTags() to check both operations (OR logic) This prevents UntagResource from incorrectly checking 'ManageTags' permission and ensures each operation validates the correct permission when per-operation permissions are enforced. * s3tables: Consolidate getPrincipalFromRequest and getAccountID into single method Both methods had identical implementations - they return the account ID from request header or fall back to handler's default. Remove the duplicate getPrincipalFromRequest and use getAccountID throughout, with updated comment explaining its dual role as both caller identity and principal for permission checks. * s3tables: Fetch bucket policy in handleListTagsForResource for permission evaluation Update handleListTagsForResource to fetch and pass bucket policy to CheckPermission, matching the behavior of handleTagResource/handleUntagResource. This enables bucket-policy-based permission grants to be evaluated for ListTagsForResource, not just ownership-based checks. * s3tables: Extract resource owner and bucket extraction into helper method Create extractResourceOwnerAndBucket() helper to consolidate the repeated pattern of unmarshaling metadata and extracting bucket name from resource path. This pattern was duplicated in handleTagResource, handleListTagsForResource, and handleUntagResource. Update all three handlers to use the helper. Also update remaining uses of getPrincipalFromRequest() (in handler_bucket_create, handler_bucket_get_list_delete, handler_namespace) to use getAccountID() after consolidating the two identical methods. * s3tables: Add log message when cluster shutdown times out The timeout path (2 second wait for graceful shutdown) was silent. Add a warning log message when it occurs to help diagnose flaky test issues and indicate when the mini cluster didn't shut down cleanly. * s3tables: Use policy_engine wildcard matcher for complete IAM compatibility Replace the custom suffix-only wildcard implementation in matchesActionPattern and matchesPrincipal with the policy_engine.MatchesWildcard function from PR #8052. This enables full wildcard support including: - Middle wildcards: s3tables:GetTable matches GetTable - Question mark wildcards: Get? matches any single character - Combined patterns: s3tables:Table* matches any action containing 'Table' Benefits: - Code reuse: eliminates duplicate wildcard logic - Complete IAM compatibility: supports all AWS wildcard patterns - Performance: uses efficient O(n) backtracking algorithm - Consistency: same wildcard behavior across S3 API and S3 Tables Add comprehensive unit tests covering exact matches, suffix wildcards, middle wildcards, question marks, and combined patterns for both action and principal matching. * go fmt * s3tables: Fix vet error - remove undefined c.t reference in Stop() The TestCluster.Stop() method doesn't have access to testing.T object. Remove the log statement and keep the timeout handling comment for clarity. The original intent (warning about shutdown timeout) is still captured in the code comment explaining potential issues. * clean up * s3tables: Add t field to TestCluster for logging Add testing.T field to TestCluster struct and initialize it in startMiniCluster. This allows Stop() to properly log warnings when cluster shutdown times out. Includes the t field in the test cluster initialization and restores the logging statement in Stop(). s3tables: Fix bucket policy error handling in permission checks Replace error-swallowing pattern where all errors from getExtendedAttribute were ignored for bucket policy reads. Now properly distinguish between: - ErrAttributeNotFound: Policy not found is expected; continue with empty policy - Other errors: Return internal server error and stop processing Applied fix to all bucket policy reads in: - handleDeleteTableBucketPolicy (line 220) - handleTagResource (line 313) - handleUntagResource (line 405) - handleListTagsForResource (line 488) - And additional occurrences in closures This prevents silent failures and ensures policy-related errors are surfaced to callers rather than being silently ignored. * s3tables: Pre-validate namespace to return 400 instead of 500 Move validateNamespace call outside of filerClient.WithFilerClient closure so that validation errors return HTTP 400 (InvalidRequest) instead of 500 (InternalError). Before: Validation error inside closure → treated as internal error → 500 After: Validation error before closure → handled as bad request → 400 This provides correct error semantics: namespace validation is an input validation issue, not a server error. * Update weed/s3api/s3tables/handler.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * s3tables: Normalize action names to include service prefix Add automatic normalization of operations to full IAM-style action names (e.g., 's3tables:CreateTableBucket') in CheckPermission(). This ensures policy statements using prefixed actions (s3tables:) correctly match operations evaluated by permission helpers. Also fixes incorrect r.Context() passed to GetIdentityNameFromContext which expects http.Request. Now passes r directly. * s3tables: Use policy framework for table creation authorization Replace strict ownership check in CreateTable with policy-based authorization. Now checks both namespace and bucket policies for CreateTable permission, allowing delegation via resource policies while still respecting owner bypass. Authorization logic: - Namespace policy grants CreateTable → allowed - Bucket policy grants CreateTable → allowed - Otherwise → denied (even if same owner) This enables cross-principal table creation via policies while maintaining security through explicit allow/deny semantics. * s3tables: Use policy framework for GetTable authorization Replace strict ownership check with policy-based authorization in GetTable. Now checks both table and bucket policies for GetTable permission, allowing authorized non-owners to read table metadata. Authorization logic: - Table policy grants GetTable → allowed - Bucket policy grants GetTable → allowed - Otherwise → 404 NotFound (no access disclosed) Maintains security through policy evaluation while enabling read delegation. * s3tables: Generate ARNs using resource owner account ID Change ARN generation to use resource OwnerAccountID instead of caller identity (h.getAccountID(r)). This ensures ARNs are stable and consistent regardless of which principal accesses the resource. Updated generateTableBucketARN and generateTableARN function signatures to accept ownerAccountID parameter. All call sites updated to pass the resource owner's account ID from metadata. This prevents ARN inconsistency issues when multiple principals have access to the same resource via policies. * s3tables: Fix remaining policy error handling in namespace and bucket handlers Replace silent error swallowing (err == nil) with proper error distinction for bucket policy reads. Now properly checks ErrAttributeNotFound and propagates other errors as internal server errors. Fixed 5 locations: - handleCreateNamespace (policy fetch) - handleDeleteNamespace (policy fetch) - handleListNamespaces (policy fetch) - handleGetNamespace (policy fetch) - handleGetTableBucket (policy fetch) This prevents masking of filer issues when policies cannot be read due to I/O errors or other transient failures. * ci: Pin GitHub Actions to commit SHAs for s3-tables-tests Update all action refs to use pinned commit SHAs instead of floating tags: - actions/checkout: @v6 → @8e8c483 (v4) - actions/setup-go: @v6 → @0c52d54 (v5) - actions/upload-artifact: @v6 → @65d8626 (v4) Pinned SHAs improve reproducibility and reduce supply chain risk by preventing accidental or malicious changes in action releases. Aligns with repository conventions used in other workflows (e.g., go.yml). * s3tables: Add resource ARN validation to policy evaluation Implement resource-specific policy validation to prevent over-broad permission grants. Add matchesResource and matchesResourcePattern functions to validate statement Resource fields against specific resource ARNs. Add new CheckPermissionWithResource function that includes resource ARN validation, while keeping CheckPermission unchanged for backward compatibility. This enables policies to grant access to specific resources only: - statements with Resource: "arn:aws:s3tables:...:bucket/specific-bucket/" will only match when accessing that specific bucket - statements without Resource field match all resources (implicit ) - resource patterns support wildcards (* for any sequence, ? for single char) For future use: Handlers can call CheckPermissionWithResource with the target resource ARN to enforce resource-level access control. * Revert "ci: Pin GitHub Actions to commit SHAs for s3-tables-tests" This reverts commit `01da26fbcb`. * s3tables: Remove duplicate bucket extraction logic in helper Move bucket name extraction outside the if/else block in extractResourceOwnerAndBucket since the logic is identical for both ResourceTypeTable and ResourceTypeBucket cases. This reduces code duplication and improves maintainability. The extraction pattern (parts[1] from /tables/{bucket}/...) works for both resource types, so it's now performed once before the type-specific metadata unmarshaling. * go fmt * s3tables: Fix ownership consistency across handlers Address three related ownership consistency issues: 1. CreateNamespace now sets OwnerAccountID to bucketMetadata.OwnerAccountID instead of request principal. This prevents namespaces created by delegated callers (via bucket policy) from becoming unmanageable, since ListNamespaces filters by bucket owner. 2. CreateTable now: - Fetches bucket metadata to use correct owner for bucket policy evaluation - Uses namespaceMetadata.OwnerAccountID for namespace policy checks - Uses bucketMetadata.OwnerAccountID for bucket policy checks - Sets table OwnerAccountID to namespaceMetadata.OwnerAccountID (inherited) 3. GetTable now: - Fetches bucket metadata to use correct owner for bucket policy evaluation - Uses metadata.OwnerAccountID for table policy checks - Uses bucketMetadata.OwnerAccountID for bucket policy checks This ensures: - Bucket owner retains implicit "owner always allowed" behavior even when evaluating bucket policies - Ownership hierarchy is consistent (namespace owned by bucket, table owned by namespace) - Cross-principal delegation via policies doesn't break ownership chains * s3tables: Fix ListTables authorization and policy parsing Make ListTables authorization consistent with GetTable/CreateTable: 1. ListTables authorization now evaluates policies instead of owner-only checks: - For namespace listing: checks namespace policy AND bucket policy - For bucket-wide listing: checks bucket policy - Uses CanListTables permission framework 2. Remove owner-only filter in listTablesWithClient that prevented policy-based sharing of tables. Authorization is now enforced at the handler level, so all tables in the namespace/bucket are returned to authorized callers (who have access either via ownership or policy). 3. Add flexible PolicyDocument.UnmarshalJSON to support both single-object and array forms of Statement field: - Handles: {"Statement": {...}} - Handles: {"Statement": [{...}, {...}]} - Improves AWS IAM compatibility This ensures cross-account table listing works when delegated via bucket/namespace policies, consistent with the authorization model for other operations. * go fmt * s3tables: Separate table name pattern constant for clarity Define a separate tableNamePatternStr constant for the table name component in the ARN regex, even though it currently has the same value as tableNamespacePatternStr. This improves code clarity and maintainability, making it easier to modify if the naming rules for tables and namespaces diverge in the future. * refactor --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	4 days ago
Ping Qiu	5c8de5e282	fix: close volumes and EC shards in tests for Windows compatibility (#8152 ) * fix: close volumes and EC shards in tests to prevent Windows cleanup failures On Windows, t.TempDir() cleanup fails when test files are still open because Windows enforces mandatory file locking. Add defer v.Close(), defer store.Close(), and EC volume cleanup to ensure all file handles are released before temp directory removal. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: extract closeEcVolumes helper to reduce duplication Address code review feedback by extracting the repeated EC volume cleanup loop into a closeEcVolumes() helper function. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	4 days ago
Emanuele Leopardi	51ef39fc76	Update Helm hook annotations for post-install and upgrade (#8150 ) * Update Helm hook annotations for post-install and upgrade I believe it makes sense to allow this job to run also after installation. Assuming weed shell is idempotent, and assuming someone wants to add a new bucket after the initial installation, it makes sense to trigger the job again. * Add check for existing buckets before creation * Enhances S3 bucket existence check Improves the reliability of checking for existing S3 buckets in the post-install hook. The previous `grep -w` command could lead to imprecise matches. This update extracts only the bucket name and performs an exact, whole-line match to ensure accurate detection of existing buckets. This prevents potential issues with redundant creation attempts or false negatives. * Currently Bucket Creation is ignored if filer.s3.enabled is disabled This commit enables bucket creation on both scenarios,i.e. if any of filer.s3.enabled or s3.enabled are used. --------- Co-authored-by: Emanuele <emanuele.leopardi@tset.com>	5 days ago
Chris Lu	6542d1e0aa	Enable weed fuse on FreeBSD (#8146 ) * Enable weed fuse on FreeBSD * Update weed/command/fuse_notsupported.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update weed/command/fuse_std.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	5 days ago
Chris Lu	e86e65e5ab	fix #8081 : build latest container is missing latest_large_disk (#8145 ) * fix #8081: build latest container is missing latest_large_disk * fix: simplify QEMU setup condition in container_latest.yml matrix	5 days ago
Chris Lu	8e9c12e2c7	mount: apply UID/GID mapping in lookupEntry for cache misses (#8144 ) * mount: apply UID/GID mapping in lookupEntry for cache misses This fixes issue #8134 where rsync would fail with "Operation not permitted" during chgrp. The issue was that entries fetched directly from the filer (on cache miss) were not being mapped to local UIDs/GIDs. * mount: add nil check for entry.Attributes in lookupEntry	5 days ago
Chris Lu	4f5f1f6be7	refactor(helm): Unified Naming Truncation and Bug Fixes (#8143 ) * refactor(helm): add componentName helper for truncation * fix(helm): unify ingress backend naming with truncation * fix(helm): unify statefulset/deployment naming with truncation * fix(helm): add missing labels to services for servicemonitor discovery * chore(helm): secure secrets and add upgrade notes * fix(helm): truncate context instead of suffix in componentName * revert(docs): remove upgrade notes per feedback * fix(helm): use componentName for COSI serviceAccountName * helm: update master -ip to use component name for correct truncation * helm: refactor masterServers helper to use truncated component names * helm: update volume -ip to use component name and cleanup redundant printf * helm: refine helpers with robustness check and updated docs	5 days ago
MorezMartin	20952aa514	Fix jwt error in admin UI (#8140 ) * add jwt token in weed admin headers requests * add jwt token to header for download * :s/upload/download * filer_signing.read despite of filer_signing key * finalize filer_browser_handlers.go * admin: add JWT authorization to file browser handlers * security: fix typos in JWT read validation descriptions * Move security.toml to example and secure keys * security: address PR feedback on JWT enforcement and example keys * security: refactor JWT logic and improve example keys readability * Update docker/Dockerfile.local Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	5 days ago
Chris Lu	c9c91ba568	Refactor Helm chart to use dynamic names for resources (#8142 ) * Refactor Helm chart to use dynamic names for resources * ensure name length	6 days ago
Alasdair Macmillan	41d079a316	Fix Javascript merge issue and UI worker detail display bug (#8135 ) * Fix previous merge issues in Javascript Signed-off-by: Alasdair Macmillan <aimmac23@gmail.com> * Fix issue where worker detail doesn't display without tasks --------- Signed-off-by: Alasdair Macmillan <aimmac23@gmail.com>	6 days ago
Andrei Kvapil	b1d63d0943	feat(shell): add Object Lock management commands (#8141 ) * feat(shell): add s3.bucket.lock command for Object Lock management Add new weed shell command to view and enable S3 Object Lock on existing buckets. This allows administrators to enable Object Lock without recreating buckets, which is useful when buckets already contain data. The command: - Shows current Object Lock and Versioning status - Enables Object Lock with -enable flag (irreversible, per AWS S3 spec) - Automatically enables Versioning if not already enabled (required for Object Lock) Usage: s3.bucket.lock -name <bucket> # view status s3.bucket.lock -name <bucket> -enable # enable Object Lock Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com> * feat(shell): add -withLock flag to s3.bucket.create command Add support for creating buckets with Object Lock enabled directly from weed shell. The flag automatically enables versioning as required by Object Lock. Usage: s3.bucket.create -name mybucket -withLock Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com> * Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Signed-off-by: Andrei Kvapil <kvapss@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	6 days ago
Chris Lu	92800c31a2	adjust logs and errors	6 days ago
Chris Lu	551a31e156	Implement IAM propagation to S3 servers (#8130 ) * Implement IAM propagation to S3 servers - Add PropagatingCredentialStore to propagate IAM changes to S3 servers via gRPC - Add Policy management RPCs to S3 proto and S3ApiServer - Update CredentialManager to use PropagatingCredentialStore when MasterClient is available - Wire FilerServer to enable propagation * Implement parallel IAM propagation and fix S3 cluster registration - Parallelized IAM change propagation with 10s timeout. - Refined context usage in PropagatingCredentialStore. - Added S3Type support to cluster node management. - Enabled S3 servers to register with gRPC address to the master. - Ensured IAM configuration reload after policy updates via gRPC. * Optimize IAM propagation with direct in-memory cache updates * Secure IAM propagation: Use metadata to skip persistence only on propagation * pb: refactor IAM and S3 services for unidirectional IAM propagation - Move SeaweedS3IamCache service from iam.proto to s3.proto. - Remove legacy IAM management RPCs and empty SeaweedS3 service from s3.proto. - Enforce that S3 servers only use the synchronization interface. * pb: regenerate Go code for IAM and S3 services Updated generated code following the proto refactoring of IAM synchronization services. * s3api: implement read-only mode for Embedded IAM API - Add readOnly flag to EmbeddedIamApi to reject write operations via HTTP. - Enable read-only mode by default in S3ApiServer. - Handle AccessDenied error in writeIamErrorResponse. - Embed SeaweedS3IamCacheServer in S3ApiServer. * credential: refactor PropagatingCredentialStore for unidirectional IAM flow - Update to use s3_pb.SeaweedS3IamCacheClient for propagation to S3 servers. - Propagate full Identity object via PutIdentity for consistency. - Remove redundant propagation of specific user/account/policy management RPCs. - Add timeout context for propagation calls. * s3api: implement SeaweedS3IamCacheServer for unidirectional sync - Update S3ApiServer to implement the cache synchronization gRPC interface. - Methods (PutIdentity, RemoveIdentity, etc.) now perform direct in-memory cache updates. - Register SeaweedS3IamCacheServer in command/s3.go. - Remove registration for the legacy and now empty SeaweedS3 service. * s3api: update tests for read-only IAM and propagation - Added TestEmbeddedIamReadOnly to verify rejection of write operations in read-only mode. - Update test setup to pass readOnly=false to NewEmbeddedIamApi in routing tests. - Updated EmbeddedIamApiForTest helper with read-only checks matching production behavior. * s3api: add back temporary debug logs for IAM updates Log IAM updates received via: - gRPC propagation (PutIdentity, PutPolicy, etc.) - Metadata configuration reloads (LoadS3ApiConfigurationFromCredentialManager) - Core identity management (UpsertIdentity, RemoveIdentity) * IAM: finalize propagation fix with reduced logging and clarified architecture * Allow configuring IAM read-only mode for S3 server integration tests * s3api: add defensive validation to UpsertIdentity * s3api: fix log message to reference correct IAM read-only flag * test/s3/iam: ensure WaitForS3Service checks for IAM write permissions * test: enable writable IAM in Makefile for integration tests * IAM: add GetPolicy/ListPolicies RPCs to s3.proto * S3: add GetBucketPolicy and ListBucketPolicies helpers * S3: support storing generic IAM policies in IdentityAccessManagement * S3: implement IAM policy RPCs using IdentityAccessManagement * IAM: fix stale user identity on rename propagation	6 days ago
dependabot[bot]	0a6b289025	chore(deps-dev): bump org.assertj:assertj-core from 3.24.2 to 3.27.7 in /other/java/s3copier (#8129 ) chore(deps-dev): bump org.assertj:assertj-core in /other/java/s3copier Bumps [org.assertj:assertj-core](https://github.com/assertj/assertj) from 3.24.2 to 3.27.7. - [Release notes](https://github.com/assertj/assertj/releases) - [Commits](https://github.com/assertj/assertj/compare/assertj-build-3.24.2...assertj-build-3.27.7) --- updated-dependencies: - dependency-name: org.assertj:assertj-core dependency-version: 3.27.7 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	6 days ago
Chris Lu	a72e269a2e	s3api: fix volume assignment failure in copy operations (#8128 ) Recent changes in the S3 unified copy strategy were constructing source and destination paths without the necessary BucketsPath prefix (typically /buckets). When these paths reached the Filer for volume assignment, it failed to resolve the correct collection and storage rules, defaulting to a disk type with no available capacity. This fix ensures all relevant paths in S3 copy handlers include the correct BucketsPath prefix for proper collection resolution. Fixes replication issue with Harbor upload via S3 API.	6 days ago
Chris Lu	43229b05ce	Explicit IAM gRPC APIs for S3 Server (#8126 ) * Update IAM and S3 protobuf definitions for explicit IAM gRPC APIs * Refactor s3api: Extract generic ExecuteAction method for IAM operations * Implement explicit IAM gRPC APIs in S3 server * iam: remove deprecated GetConfiguration and PutConfiguration RPCs * iamapi: refactor handlers to use CredentialManager directly * s3api: refactor embedded IAM to use CredentialManager directly * server: remove deprecated configuration gRPC handlers * credential/grpc: refactor configuration calls to return error * shell: update s3.configure to list users instead of full config * s3api: fix CreateServiceAccount gRPC handler to map required fields * s3api: fix UpdateServiceAccount gRPC handler to map fields and safe status * s3api: enforce UserName in embedded IAM ListAccessKeys * test: fix test_config.json structure to match proto definition * Revert "credential/grpc: refactor configuration calls to return error" This reverts commit `cde707dd8b`. * Revert "server: remove deprecated configuration gRPC handlers" This reverts commit `7307e205a0`. * Revert "s3api: enforce UserName in embedded IAM ListAccessKeys" This reverts commit `adf727ba52`. * Revert "s3api: fix UpdateServiceAccount gRPC handler to map fields and safe status" This reverts commit `6a4be3314d`. * Revert "s3api: fix CreateServiceAccount gRPC handler to map required fields" This reverts commit `9bb4425f07`. * Revert "shell: update s3.configure to list users instead of full config" This reverts commit `f3304ead53`. * Revert "s3api: refactor embedded IAM to use CredentialManager directly" This reverts commit `9012f27af8`. * Revert "iamapi: refactor handlers to use CredentialManager directly" This reverts commit `3a14821223`. * Revert "iam: remove deprecated GetConfiguration and PutConfiguration RPCs" This reverts commit `e16e08aa00`. * s3api: address IAM code review comments (error handling, logging, gRPC response mapping) * s3api: add robustness to startup by retrying KEK and IAM config loading from Filer * s3api: address IAM gRPC code review comments (safety, validation, status logic) * fix return	7 days ago
Chris Lu	c5b53397c6	templ	7 days ago
Chris Lu	5ba0db7af4	Merge branch 'origin/master' into master Resolved merge conflicts in: - weed/admin/static/js/modal-alerts.js: Adopted incoming improvements and HTML support. - weed/admin/view/app/collection_details.templ: Switched to showAlert info type. - weed/admin/view/app/file_browser.templ: Used descriptive delete message. - weed/admin/view/app/maintenance_workers.templ: Used encoding and headers in pauseWorker. - weed/admin/view/app/object_store_users.templ: Restored accidentally deleted delete functions and used encodeURIComponent. - weed/admin/view/app/policies.templ: Standardized on showAlert and descriptive confirmations. Regenerated all templ files.	7 days ago
Chris Lu	5a7c74feac	migrate IAM policies to multi-file storage (#8114 ) * Add IAM gRPC service definition - Add GetConfiguration/PutConfiguration for config management - Add CreateUser/GetUser/UpdateUser/DeleteUser/ListUsers for user management - Add CreateAccessKey/DeleteAccessKey/GetUserByAccessKey for access key management - Methods mirror existing IAM HTTP API functionality * Add IAM gRPC handlers on filer server - Implement IamGrpcServer with CredentialManager integration - Handle configuration get/put operations - Handle user CRUD operations - Handle access key create/delete operations - All methods delegate to CredentialManager for actual storage * Wire IAM gRPC service to filer server - Add CredentialManager field to FilerOption and FilerServer - Import credential store implementations in filer command - Initialize CredentialManager from credential.toml if available - Register IAM gRPC service on filer gRPC server - Enable credential management via gRPC alongside existing filer services * Regenerate IAM protobuf with gRPC service methods * fix: compilation error in DeleteUser * fix: address code review comments for IAM migration * feat: migrate policies to multi-file layout and fix identity duplicated content * refactor: remove configuration.json and migrate Service Accounts to multi-file layout * refactor: standardize Service Accounts as distinct store entities and fix Admin Server persistence * config: set ServiceAccountsDirectory to /etc/iam/service_accounts * Fix Chrome dialog auto-dismiss with Bootstrap modals - Add modal-alerts.js library with Bootstrap modal replacements - Replace all 15 confirm() calls with showConfirm/showDeleteConfirm - Auto-override window.alert() for all alert() calls - Fixes Chrome 132+ aggressively blocking native dialogs * Upgrade Bootstrap from 5.3.2 to 5.3.8 * Fix syntax error in object_store_users.templ - remove duplicate closing braces * create policy * display errors * migrate to multi-file policies * address PR feedback: use showDeleteConfirm and showErrorMessage in policies.templ, refine migration check * Update policies_templ.go * add service account to iam grpc * iam: fix potential path traversal in policy names by validating name pattern * iam: add GetServiceAccountByAccessKey to CredentialStore interface * iam: implement service account support for PostgresStore Includes full CRUD operations and efficient lookup by access key. * iam: implement GetServiceAccountByAccessKey for filer_etc, grpc, and memory stores Provides efficient lookup of service accounts by access key where possible, with linear scan fallbacks for file-based stores. * iam: remove filer_multiple support Deleted its implementation and references in imports, scaffold config, and core interface constants. Redundant with filer_etc. * clear comment * dash: robustify service account construction - Guard against nil sa.Credential when constructing responses - Fix Expiration logic to only set if > 0, avoiding Unix epoch 1970 - Ensure consistency across Get, Create, and Update handlers * credential/filer_etc: improve error propagation in configuration handlers - Return error from loadServiceAccountsFromMultiFile to callers - Ensure listEntries errors in SaveConfiguration (cleanup logic) are propagated unless they are "not found" failures. - Fixes potential silent failures during IAM configuration sync. * credential/filer_etc: add existence check to CreateServiceAccount Ensures consistency with other stores by preventing accidental overwrite of existing service accounts during creation. * credential/memory: improve store robustness and Reset logic - Enforce ID immutability in UpdateServiceAccount to prevent orphans - Update Reset() to also clear the policies map, ensuring full state cleanup for tests. * dash: improve service account robustness and policy docs - Wrap parent user lookup errors to preserve context - Strictly validate Status field in UpdateServiceAccount - Add deprecation comments to legacy policy management methods * credential/filer_etc: protect against path traversal in service accounts Implemented ID validation (alphanumeric, underscores, hyphens) and applied it to Get, Save, and Delete operations to ensure no directory traversal via saId.json filenames. * credential/postgres: improve robustness and cleanup comments - Removed brainstorming comments in GetServiceAccountByAccessKey - Added missing rows.Err() check during iteration - Properly propagate Scan and Unmarshal errors instead of swallowing them * admin: unify UI alerts and confirmations using Bootstrap modals - Updated modal-alerts.js with improved automated alert type detection - Replaced native alert() and confirm() with showAlert(), showConfirm(), and showDeleteConfirm() across various Templ components - Improved UX for delete operations by providing better context and styling - Ensured consistent error reporting across IAM and Maintenance views * admin: additional UI consistency fixes for alerts and confirmations - Replaced native alert() and confirm() with Bootstrap modals in: - EC volumes (repair flow) - Collection details (repair flow) - File browser (properties and delete) - Maintenance config schema (save and reset) - Improved delete confirmation in file browser with item context - Ensured consistent success/error/info styling for all feedbacks * make * iam: add GetServiceAccountByAccessKey RPC and update GetConfiguration * iam: implement GetServiceAccountByAccessKey on server and client * iam: centralize policy and service account validation * iam: optimize MemoryStore service account lookups with indexing * iam: fix postgres service_accounts table and optimize lookups * admin: refactor modal alerts and clean up dashboard logic * admin: fix EC shards table layout mismatch * admin: URL-encode IAM path parameters for safety * admin: implement pauseWorker logic in maintenance view * iam: add rows.Err() check to postgres ListServiceAccounts * iam: standardize ErrServiceAccountNotFound across credential stores * iam: map ErrServiceAccountNotFound to codes.NotFound in DeleteServiceAccount * iam: refine service account store logic, errors and schema * iam: add validation to GetServiceAccountByAccessKey * admin: refine modal titles and ensure URL safety * admin: address bot review comments for alerts and async usage * iam: fix syntax error by restoring missing function declaration * [FilerEtcStore] improve error handling in CreateServiceAccount Refine error handling to provide clearer messages when checking for existing service accounts. * [PostgresStore] add nil guards and validation to service account methods Ensure input parameters are not nil and required IDs are present to prevent runtime panics and ensure data integrity. * [JS] add shared IAM utility script Consolidate common IAM operations like deleteUser and deleteAccessKey into a shared utility script for better maintainability. * [View] include shared IAM utilities in layout Include iam-utils.js in the main layout to make IAM functions available across all administrative pages. * [View] refactor IAM logic and restore async in EC Shards view Remove redundant local IAM functions and ensure that delete confirmation callbacks are properly marked as async. * [View] consolidate IAM logic in Object Store Users view Remove redundant local definitions of deleteUser and deleteAccessKey, relying on the shared utilities instead. * [View] update generated templ files for UI consistency * credential/postgres: remove redundant name column from service_accounts table The id is already used as the unique identifier and was being copied to the name column. This removes the name column from the schema and updates the INSERT/UPDATE queries. * credential/filer_etc: improve logging for policy migration failures Added Errorf log if AtomicRenameEntry fails during migration to ensure visibility of common failure points. * credential: allow uppercase characters in service account ID username Updated ServiceAccountIdPattern to allow [A-Za-z0-9_-]+ for the username component, matching the actual service account creation logic which uses the parent user name directly. * Update object_store_users_templ.go * admin: fix ec_shards pagination to handle numeric page arguments Updated goToPage in cluster_ec_shards.templ to accept either an Event or a numeric page argument. This prevents errors when goToPage(1) is called directly. Corrected both the .templ source and generated Go code. * credential/filer_etc: improve service account storage robustness Added nil guard to saveServiceAccount, updated GetServiceAccount to return ErrServiceAccountNotFound for empty data, and improved deleteServiceAccount to handle response-level Filer errors.	7 days ago
dependabot[bot]	a29806d752	chore(deps): bump github.com/jackc/pgx/v5 from 5.7.6 to 5.8.0 (#8118 ) Bumps [github.com/jackc/pgx/v5](https://github.com/jackc/pgx) from 5.7.6 to 5.8.0. - [Changelog](https://github.com/jackc/pgx/blob/master/CHANGELOG.md) - [Commits](https://github.com/jackc/pgx/compare/v5.7.6...v5.8.0) --- updated-dependencies: - dependency-name: github.com/jackc/pgx/v5 dependency-version: 5.8.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	7 days ago
dependabot[bot]	9ca43c452b	chore(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.20.0 to 1.21.0 (#8119 ) chore(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.20.0 to 1.21.0. - [Release notes](https://github.com/Azure/azure-sdk-for-go/releases) - [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.20.0...sdk/azcore/v1.21.0) --- updated-dependencies: - dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore dependency-version: 1.21.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	7 days ago
dependabot[bot]	06920f79f9	chore(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.125.1 to 3.125.3 (#8120 ) chore(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.125.1 to 3.125.3. - [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases) - [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md) - [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.125.1...v3.125.3) --- updated-dependencies: - dependency-name: github.com/ydb-platform/ydb-go-sdk/v3 dependency-version: 3.125.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	7 days ago
dependabot[bot]	b502411884	chore(deps): bump actions/checkout from 4 to 6 (#8121 ) Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Commits](https://github.com/actions/checkout/compare/v4...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	7 days ago
dependabot[bot]	16dc90e3bd	chore(deps): bump actions/setup-go from 5 to 6 (#8124 ) Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5 to 6. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/setup-go dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	7 days ago
dependabot[bot]	6714973ffe	chore(deps): bump actions/upload-artifact from 4 to 6 (#8125 ) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 6. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v4...v6) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	7 days ago
dependabot[bot]	1e5a1871e5	chore(deps): bump modernc.org/sqlite from 1.44.2 to 1.44.3 (#8122 ) Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.44.2 to 1.44.3. - [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md) - [Commits](https://gitlab.com/cznic/sqlite/compare/v1.44.2...v1.44.3) --- updated-dependencies: - dependency-name: modernc.org/sqlite dependency-version: 1.44.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	7 days ago
dependabot[bot]	db6b4ab918	chore(deps): bump github.com/a-h/templ from 0.3.943 to 0.3.977 (#8123 ) Bumps [github.com/a-h/templ](https://github.com/a-h/templ) from 0.3.943 to 0.3.977. - [Release notes](https://github.com/a-h/templ/releases) - [Commits](https://github.com/a-h/templ/compare/v0.3.943...v0.3.977) --- updated-dependencies: - dependency-name: github.com/a-h/templ dependency-version: 0.3.977 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	7 days ago
Chris Lu	759a6cd345	Merge branch 'fix/chrome-dialog-modal-approach'	1 week ago
Chris Lu	7e3bb4016e	Fix syntax error in object_store_users.templ - remove duplicate closing braces	1 week ago
Chris Lu	1e09950ea7	Upgrade Bootstrap from 5.3.2 to 5.3.8	1 week ago
Chris Lu	74c7b10bc7	Fix Chrome dialog auto-dismiss with Bootstrap modals - Add modal-alerts.js library with Bootstrap modal replacements - Replace all 15 confirm() calls with showConfirm/showDeleteConfirm - Auto-override window.alert() for all alert() calls - Fixes Chrome 132+ aggressively blocking native dialogs	1 week ago
Chris Lu	31a1c87db0	fix compilation	1 week ago
Chris Lu	533650579d	credential/filer_etc: improve migration merge logic	1 week ago
Chris Lu	74b5c57dcf	credential/filer_etc: migrate to multi-file identity storage	1 week ago
Chris Lu	6bf088cec9	IAM Policy Management via gRPC (#8109 ) * Add IAM gRPC service definition - Add GetConfiguration/PutConfiguration for config management - Add CreateUser/GetUser/UpdateUser/DeleteUser/ListUsers for user management - Add CreateAccessKey/DeleteAccessKey/GetUserByAccessKey for access key management - Methods mirror existing IAM HTTP API functionality * Add IAM gRPC handlers on filer server - Implement IamGrpcServer with CredentialManager integration - Handle configuration get/put operations - Handle user CRUD operations - Handle access key create/delete operations - All methods delegate to CredentialManager for actual storage * Wire IAM gRPC service to filer server - Add CredentialManager field to FilerOption and FilerServer - Import credential store implementations in filer command - Initialize CredentialManager from credential.toml if available - Register IAM gRPC service on filer gRPC server - Enable credential management via gRPC alongside existing filer services * Regenerate IAM protobuf with gRPC service methods * iam_pb: add Policy Management to protobuf definitions * credential: implement PolicyManager in credential stores * filer: implement IAM Policy Management RPCs * shell: add s3.policy command * test: add integration test for s3.policy * test: fix compilation errors in policy_test * pb * fmt * test * weed shell: add -policies flag to s3.configure This allows linking/unlinking IAM policies to/from identities directly from the s3.configure command. * test: verify s3.configure policy linking and fix port allocation - Added test case for linking policies to users via s3.configure - Implemented findAvailablePortPair to ensure HTTP and gRPC ports are both available, avoiding conflicts with randomized port assignments. - Updated assertion to match jsonpb output (policyNames) * credential: add StoreTypeGrpc constant * credential: add IAM gRPC store boilerplate * credential: implement identity methods in gRPC store * credential: implement policy methods in gRPC store * admin: use gRPC credential store for AdminServer This ensures that all IAM and policy changes made through the Admin UI are persisted via the Filer's IAM gRPC service instead of direct file manipulation. * shell: s3.configure use granular IAM gRPC APIs instead of full config patching * shell: s3.configure use granular IAM gRPC APIs * shell: replace deprecated ioutil with os in s3.policy * filer: use gRPC FailedPrecondition for unconfigured credential manager * test: improve s3.policy integration tests and fix error checks * ci: add s3 policy shell integration tests to github workflow * filer: fix LoadCredentialConfiguration error handling * credential/grpc: propagate unmarshal errors in GetPolicies * filer/grpc: improve error handling and validation * shell: use gRPC status codes in s3.configure * credential: document PutPolicy as create-or-replace * credential/postgres: reuse CreatePolicy in PutPolicy to deduplicate logic * shell: add timeout context and strictly enforce flags in s3.policy * iam: standardize policy content field naming in gRPC and proto * shell: extract slice helper functions in s3.configure * filer: map credential store errors to gRPC status codes * filer: add input validation for UpdateUser and CreateAccessKey * iam: improve validation in policy and config handlers * filer: ensure IAM service registration by defaulting credential manager * credential: add GetStoreName method to manager * test: verify policy deletion in integration test	1 week ago
Lisandro Pin	59d40f7186	Return volume server state flags via `VolumeServerStatus()` RPCs. (#8016 )	1 week ago
Chris Lu	5a0c761976	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	1 week ago
Chris Lu	b40551f960	helm: seaweedfs admin should not allow setting multiple admin servers	1 week ago
Chris Lu	8814c2a07d	iam: support ForAnyValue and ForAllValues condition set operators (#8105 ) * iam: support ForAnyValue and ForAllValues condition set operators This implementation adds support for AWS-style IAM condition set operators `ForAnyValue:` and `ForAllValues:`. These are essential for trust policies that evaluate collection-based claims like `oidc:roles` or groups. - Updated EvaluateStringCondition to handle set operators. - Added set operator support to numeric, date, and boolean conditions. - ForAnyValue matches if any request value matches any condition value (default). - ForAllValues matches if every request value matches at least one condition value. * iam: add test suite for condition set operators * iam: ensure ForAllValues is vacuously true for all condition types Aligned Numeric, Date, and Boolean conditions with AWS IAM behavior where ForAllValues returns true when the request context values are empty. * iam: add Date vacuously true test case for ForAllValues * iam: expand policy variables in case-insensitive string conditions Added expandPolicyVariables support to evaluateStringConditionIgnoreCase to ensure consistency with case-sensitive counterparts. * iam: fix negation issues in string set operators Refactored EvaluateStringCondition and evaluateStringConditionIgnoreCase to evaluate operators (including negation) per context value before aggregating. This ensures StringNotEquals and StringNotLike work correctly with ForAllValues and ForAnyValue. * iam: add []string support for Date and Boolean context values Ensures consistency with Numeric conditions by allowing context values to be provided as slices of strings, which is common in JSON/OIDC claims. * iam: simplify redundant type check in policy engine The `evaluateStringConditionIgnoreCase` function had a redundant type check for `string` in the `default` block of a type switch that already handled the `string` case. * iam: remove outdated "currently fails" comment in negation tests * iam: add StringLikeIgnoreCase condition support * iam: explicitly handle empty context sets for ForAnyValue AWS IAM treats empty request sets as "no match" for ForAnyValue. Added an explicit check and comment to make this behavior clear. * iam: refactor EvaluateStringCondition to expand policy variables once Avoid redundant calls to expandPolicyVariables by expanding them once per condition value instead of inside awsIAMMatch or in the exact matching branch. * iam: fix StringLike case sensitivity to match AWS IAM specs StringLike and StringNotLike condition operators are case-sensitive in AWS IAM. Changed the implementation to use filepath.Match for case-sensitive wildcard matching instead of the case-insensitive awsIAMMatch. * iam: integrate StringLike case-sensitivity test into suite Integrated the case-sensitivity verification into condition_set_test.go and updated the consistency test to use StringLikeIgnoreCase to maintain its case-insensitive matching verification. * iam: fix NumericNotEquals logic to follow "not equal to any" semantics Updated evaluateNumericCondition to correctly handle NumericNotEquals by ensuring a context value matches only if it is not equal to ANY of the provided expected values. Also added support for []string expected values. * iam: fix DateNotEquals logic and integrate tests Updated evaluateDateCondition to correctly handle DateNotEquals logic. Integrated the new test cases for NumericNotEquals and DateNotEquals into condition_set_test.go. * iam: fix validation error in integrated NotEquals tests Added missing Resource field to IAM policy statements in condition_set_test.go to satisfy validation requirements. * iam: add set operator support for IP and Null conditions Implemented ForAllValues and ForAnyValue support for IpAddress, NotIpAddress, and Null condition operators. Also added test coverage for ForAnyValue with an empty context to ensure correct behavior. * iam: refine IP condition evaluation to handle multiple policy value types Updated evaluateIPCondition to correctly handle string, []string, and []interface{} values for IP address conditions in policy documents. Added IpAddress:SingleStringValue test case to verify consistency. * iam: refine Null and case-insensitive string conditions - Reverted evaluateNullCondition to standard AWS behavior (no set operators). - Refactored evaluateStringConditionIgnoreCase to use idiomatic helpers (strings.EqualFold and AwsWildcardMatch). - Cleaned up tests in condition_set_test.go. * iam: normalize policy value handling across condition evaluators - Implemented normalizeRanges helper for consistent IP range extraction. - Expanded type switches in IP, Bool, and String condition evaluators to support string, []string, and []interface{} policy values. - Fixed ForAnyValue bool matching to support string slices. - Added targeted tests for []string policy values in condition_set_test.go. * iam: refactor IP condition to support arbitrary context keys Refactored evaluateIPCondition to iterate through all keys in the condition block instead of hardcoding aws:SourceIp. This ensures consistency with other condition types and allows custom context keys. Added IpAddress:CustomContextKey test case to verify the change.	1 week ago
Chris Lu	d3f79d4c38	Update detection.go	1 week ago
Chris Lu	6394e2f6a5	Fix IAM OIDC role mapping and OIDC claims in trust policy (#8104 ) * Fix IAM OIDC role mapping and OIDC claims in trust policy * Address PR review: Add config safety checks and refactor tests	1 week ago
Chris Lu	57a16b0b87	Improve error handling in GetObjectStoreUsers per PR review	1 week ago
Chris Lu	e559b8df37	Refactor Admin UI to use unified IAM storage and add Shutdown hook	1 week ago
Chris Lu	81009c1a81	Refactor IAM Storage: Multi-File Backend & Unified Interface (#8102 ) Refactor IAM Shutdown to use sync.Once for thread safety	1 week ago
Chris Lu	f6318edbc9	Refactor Admin UI to use unified IAM storage and add MultipleFileStore (#8101 ) * Refactor Admin UI to use unified IAM storage and add MultipleFileStore * Address PR feedback: fix renames, error handling, and sync logic in FilerMultipleStore * Address refined PR feedback: safe rename order, rollback logic, and structural sync refinement * Optimize LoadConfiguration: use streaming callback for memory efficiency * Refactor UpdateUser: log rollback failures during rename * Implement PolicyManager for FilerMultipleStore * include the filer_multiple backend configuration * Implement cross-S3 synchronization and proper shutdown for all IAM backends * Extract Admin UI refactoring to a separate PR	1 week ago
Chris Lu	535be3096b	Add AWS IAM integration tests and refactor admin authorization (#8098 ) * Add AWS IAM integration tests and refactor admin authorization - Added AWS IAM management integration tests (User, AccessKey, Policy) - Updated test framework to support IAM client creation with JWT/OIDC - Refactored s3api authorization to be policy-driven for IAM actions - Removed hardcoded role name checks for admin privileges - Added new tests to GitHub Actions basic test matrix * test(s3/iam): add UpdateUser and UpdateAccessKey tests and fix nil pointer dereference * feat(s3api): add DeletePolicy and update tests with cleanup logic * test(s3/iam): use t.Cleanup for managed policy deletion in CreatePolicy test	1 week ago
Chris Lu	25a4691135	Update store_ec_recovery_test.go	1 week ago
Chris Lu	d664ca5ed3	fix: IAM authentication with AWS Signature V4 and environment credentials (#8099 ) * fix: IAM authentication with AWS Signature V4 and environment credentials Three key fixes for authenticated IAM requests to work: 1. Fix request body consumption before signature verification - iamMatcher was calling r.ParseForm() which consumed POST body - This broke AWS Signature V4 verification on subsequent reads - Now only check query string in matcher, preserving body for verification - File: weed/s3api/s3api_server.go 2. Preserve environment variable credentials across config reloads - After IAM mutations, config reload overwrote env var credentials - Extract env var loading into loadEnvironmentVariableCredentials() - Call after every config reload to persist credentials - File: weed/s3api/auth_credentials.go 3. Add authenticated IAM tests and test infrastructure - New TestIAMAuthenticated suite with AWS SDK + Signature V4 - Dynamic port allocation for independent test execution - Flag reset to prevent state leakage between tests - CI workflow to run S3 and IAM tests separately - Files: test/s3/example/, .github/workflows/s3-example-integration-tests.yml All tests pass: - TestIAMCreateUser (unauthenticated) - TestIAMAuthenticated (with AWS Signature V4) - S3 integration tests fmt * chore: rename test/s3/example to test/s3/normal * simplify: CI runs all integration tests in single job * Update s3-example-integration-tests.yml * ci: run each test group separately to avoid raft registry conflicts	1 week ago
Chris Lu	afbe52f262	cleanup	1 week ago
Chris Lu	67b35757d7	Delete reproduce_issue_s3.sh	1 week ago

1 2 3 4 5 ...

12607 Commits (2f155ee5eefab9cdfcb6d21e50c740508b653fea) All Branches Search

12607 Commits (2f155ee5eefab9cdfcb6d21e50c740508b653fea)

All Branches