seaweedfs

Commit Graph

Author	SHA1	Message	Date
Lisandro Pin	2af293ce60	Boostrap persistent state for volume servers. (#7984 ) This PR implements logic load/save persistent state information for storages associated with volume servers, and reporting state changes back to masters via heartbeat messages. More work ensues! See https://github.com/seaweedfs/seaweedfs/issues/7977 for details.	1 day ago
Chris Lu	06391701ed	Add AssumeRole and AssumeRoleWithLDAPIdentity STS actions (#8003 ) * test: add integration tests for AssumeRole and AssumeRoleWithLDAPIdentity STS actions - Add s3_sts_assume_role_test.go with comprehensive tests for AssumeRole: * Parameter validation (missing RoleArn, RoleSessionName, invalid duration) * AWS SigV4 authentication with valid/invalid credentials * Temporary credential generation and usage - Add s3_sts_ldap_test.go with tests for AssumeRoleWithLDAPIdentity: * Parameter validation (missing LDAP credentials, RoleArn) * LDAP authentication scenarios (valid/invalid credentials) * Integration with LDAP server (when configured) - Update Makefile with new test targets: * test-sts: run all STS tests * test-sts-assume-role: run AssumeRole tests only * test-sts-ldap: run LDAP STS tests only * test-sts-suite: run tests with full service lifecycle - Enhance setup_all_tests.sh: * Add OpenLDAP container setup for LDAP testing * Create test LDAP users (testuser, ldapadmin) * Set LDAP environment variables for tests * Update cleanup to remove LDAP container - Fix setup_keycloak.sh: * Enable verbose error logging for realm creation * Improve error diagnostics Tests use fail-fast approach (t.Fatal) when server not configured, ensuring clear feedback when infrastructure is missing. * feat: implement AssumeRole and AssumeRoleWithLDAPIdentity STS actions Implement two new STS actions to match MinIO's STS feature set: AssumeRole Implementation: - Add handleAssumeRole with full AWS SigV4 authentication - Integrate with existing IAM infrastructure via verifyV4Signature - Validate required parameters (RoleArn, RoleSessionName) - Validate DurationSeconds (900-43200 seconds range) - Generate temporary credentials with expiration - Return AWS-compatible XML response AssumeRoleWithLDAPIdentity Implementation: - Add handleAssumeRoleWithLDAPIdentity handler (stub) - Validate LDAP-specific parameters (LDAPUsername, LDAPPassword) - Validate common STS parameters (RoleArn, RoleSessionName, DurationSeconds) - Return proper error messages for missing LDAP provider - Ready for LDAP provider integration Routing Fixes: - Add explicit routes for AssumeRole and AssumeRoleWithLDAPIdentity - Prevent IAM handler from intercepting authenticated STS requests - Ensure proper request routing priority Handler Infrastructure: - Add IAM field to STSHandlers for SigV4 verification - Update NewSTSHandlers to accept IAM reference - Add STS-specific error codes and response types - Implement writeSTSErrorResponse for AWS-compatible errors The AssumeRole action is fully functional and tested. AssumeRoleWithLDAPIdentity requires LDAP provider implementation. * fix: update IAM matcher to exclude STS actions from interception Update the IAM handler matcher to check for STS actions (AssumeRole, AssumeRoleWithWebIdentity, AssumeRoleWithLDAPIdentity) and exclude them from IAM handler processing. This allows STS requests to be handled by the STS fallback handler even when they include AWS SigV4 authentication. The matcher now parses the form data to check the Action parameter and returns false for STS actions, ensuring they are routed to the correct handler. Note: This is a work-in-progress fix. Tests are still showing some routing issues that need further investigation. * fix: address PR review security issues for STS handlers This commit addresses all critical security issues from PR review: Security Fixes: - Use crypto/rand for cryptographically secure credential generation instead of time.Now().UnixNano() (fixes predictable credentials) - Add sts:AssumeRole permission check via VerifyActionPermission to prevent unauthorized role assumption - Generate proper session tokens using crypto/rand instead of placeholder strings Code Quality Improvements: - Refactor DurationSeconds parsing into reusable parseDurationSeconds() helper function used by all three STS handlers - Create generateSecureCredentials() helper for consistent and secure temporary credential generation - Fix iamMatcher to check query string as fallback when Action not found in form data LDAP Provider Implementation: - Add go-ldap/ldap/v3 dependency - Create LDAPProvider implementing IdentityProvider interface with full LDAP authentication support (connect, bind, search, groups) - Update ProviderFactory to create real LDAP providers - Wire LDAP provider into AssumeRoleWithLDAPIdentity handler Test Infrastructure: - Add LDAP user creation verification step in setup_all_tests.sh * fix: address PR feedback (Round 2) - config validation & provider improvements - Implement `validateLDAPConfig` in `ProviderFactory` - Improve `LDAPProvider.Initialize`: - Support `connectionTimeout` parsing (string/int/float) from config map - Warn if `BindDN` is present but `BindPassword` is empty - Improve `LDAPProvider.GetUserInfo`: - Add fallback to `searchUserGroups` if `memberOf` returns no groups (consistent with Authenticate) * fix: address PR feedback (Round 3) - LDAP connection improvements & build fix - Improve `LDAPProvider` connection handling: - Use `net.Dialer` with configured timeout for connection establishment - Enforce TLS 1.2+ (`MinVersion: tls.VersionTLS12`) for both LDAPS and StartTLS - Fix build error in `s3api_sts.go` (format verb for ErrorCode) * fix: address PR feedback (Round 4) - LDAP hardening, Authz check & Routing fix - LDAP Provider Hardening: - Prevent re-initialization - Enforce single user match in `GetUserInfo` (was explicit only in Authenticate) - Ensure connection closure if StartTLS fails - STS Handlers: - Add robust provider detection using type assertion - Security: Implement authorization check (`VerifyActionPermission`) after LDAP authentication - Routing: - Update tests to reflect that STS actions are handled by STS handler, not generic IAM * fix: address PR feedback (Round 5) - JWT tokens, ARN formatting, PrincipalArn CRITICAL FIXES: - Replace standalone credential generation with STS service JWT tokens - handleAssumeRole now generates proper JWT session tokens - handleAssumeRoleWithLDAPIdentity now generates proper JWT session tokens - Session tokens can be validated across distributed instances - Fix ARN formatting in responses - Extract role name from ARN using utils.ExtractRoleNameFromArn() - Prevents malformed ARNs like "arn:aws:sts::assumed-role/arn:aws:iam::..." - Add configurable AccountId for federated users - Add AccountId field to STSConfig (defaults to "111122223333") - PrincipalArn now uses configured account ID instead of hardcoded "aws" - Enables proper trust policy validation IMPROVEMENTS: - Sanitize LDAP authentication error messages (don't leak internal details) - Remove duplicate comment in provider detection - Add utils import for ARN parsing utilities * feat: implement LDAP connection pooling to prevent resource exhaustion PERFORMANCE IMPROVEMENT: - Add connection pool to LDAPProvider (default size: 10 connections) - Reuse LDAP connections across authentication requests - Prevent file descriptor exhaustion under high load IMPLEMENTATION: - connectionPool struct with channel-based connection management - getConnection(): retrieves from pool or creates new connection - returnConnection(): returns healthy connections to pool - createConnection(): establishes new LDAP connection with TLS support - Close(): cleanup method to close all pooled connections - Connection health checking (IsClosing()) before reuse BENEFITS: - Reduced connection overhead (no TCP handshake per request) - Better resource utilization under load - Prevents "too many open files" errors - Non-blocking pool operations (creates new conn if pool empty) * fix: correct TokenGenerator access in STS handlers CRITICAL FIX: - Make TokenGenerator public in STSService (was private tokenGenerator) - Update all references from Config.TokenGenerator to TokenGenerator - Remove TokenGenerator from STSConfig (it belongs in STSService) This fixes the "NotImplemented" errors in distributed and Keycloak tests. The issue was that Round 5 changes tried to access Config.TokenGenerator which didn't exist - TokenGenerator is a field in STSService, not STSConfig. The TokenGenerator is properly initialized in STSService.Initialize() and is now accessible for JWT token generation in AssumeRole handlers. * fix: update tests to use public TokenGenerator field Following the change to make TokenGenerator public in STSService, this commit updates the test files to reference the correct public field name. This resolves compilation errors in the IAM STS test suite. * fix: update distributed tests to use valid Keycloak users Updated s3_iam_distributed_test.go to use 'admin-user' and 'read-user' which exist in the standard Keycloak setup provided by setup_keycloak.sh. This resolves 'unknown test user' errors in distributed integration tests. * fix: ensure iam_config.json exists in setup target for CI The GitHub Actions workflow calls 'make setup' which was not creating iam_config.json, causing the server to start without IAM integration enabled (iamIntegration = nil), resulting in NotImplemented errors. Now 'make setup' copies iam_config.local.json to iam_config.json if it doesn't exist, ensuring IAM is properly configured in CI. * fix(iam/ldap): fix connection pool race and rebind corruption - Add atomic 'closed' flag to connection pool to prevent racing on Close() - Rebind authenticated user connections back to service account before returning to pool - Close connections on error instead of returning potentially corrupted state to pool * fix(iam/ldap): populate standard TokenClaims fields in ValidateToken - Set Subject, Issuer, Audience, IssuedAt, and ExpiresAt to satisfy the interface - Use time.Time for timestamps as required by TokenClaims struct - Default to 1 hour TTL for LDAP tokens * fix(s3api): include account ID in STS AssumedRoleUser ARN - Consistent with AWS, include the account ID in the assumed-role ARN - Use the configured account ID from STS service if available, otherwise default to '111122223333' - Apply to both AssumeRole and AssumeRoleWithLDAPIdentity handlers - Also update .gitignore to ignore IAM test environment files * refactor(s3api): extract shared STS credential generation logic - Move common logic for session claims and credential generation to prepareSTSCredentials - Update handleAssumeRole and handleAssumeRoleWithLDAPIdentity to use the helper - Remove stale comments referencing outdated line numbers * feat(iam/ldap): make pool size configurable and add audience support - Add PoolSize to LDAPConfig (default 10) - Add Audience to LDAPConfig to align with OIDC validation - Update initialization and ValidateToken to use new fields * update tests * debug * chore(iam): cleanup debug prints and fix test config port * refactor(iam): use mapstructure for LDAP config parsing * feat(sts): implement strict trust policy validation for AssumeRole * test(iam): refactor STS tests to use AWS SDK signer * test(s3api): implement ValidateTrustPolicyForPrincipal in MockIAMIntegration * fix(s3api): ensure IAM matcher checks query string on ParseForm error * fix(sts): use crypto/rand for secure credentials and extract constants * fix(iam): fix ldap connection leaks and add insecure warning * chore(iam): improved error wrapping and test parameterization * feat(sts): add support for LDAPProviderName parameter * Update weed/iam/ldap/ldap_provider.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_sts.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix(sts): use STSErrSTSNotReady when LDAP provider is missing * fix(sts): encapsulate TokenGenerator in STSService and add getter --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	1 day ago
Chris Lu	d7c30fdb2b	fix: admin does not show all master servers #7999 (#8002 )	2 days ago
Sheya Bernstein	844859de7f	fix: add filer fallback after consecutive connection failures (#8000 )	2 days ago
Sheya Bernstein	8740a087b9	fix: apply tpl function to all component extraEnvironmentVars (#8001 )	2 days ago
Chris Lu	2b5e951390	use context.WithoutCancel to avoid context cancellation when the client connection is closed	2 days ago
Chris Lu	ce6e9be66b	4.06	3 days ago
Chris Lu	379c032868	Fix chown Input/output error on large file sets (#7996 ) * Fix chown Input/output error on large file sets (Fixes #7911) Implemented retry logic for MySQL/MariaDB backend to handle transient errors like deadlocks and timeouts. * Fix syntax error: missing closing brace * Refactor: Use %w for error wrapping and errors.As for extraction * Fix: Disable retry logic inside transactions	4 days ago
Nicholas Boyd Isacsson	88e9e2c471	fix: Invalid volume mount conditional in filer template (#7992 ) There is a mistmatch in the conditionals for the definition and mounting of the `config-users` volume in the filer's template. Volume definition: ``` {{- if and .Values.filer.s3.enabled .Values.filer.s3.enableAuth }} ``` Mount: ``` {{- if .Values.filer.s3.enableAuth }} ``` This leads to an invalid specification in the case where s3 is disabled but the enableAuth value is set to true, as it tries to mount in an undefined volume. I've fixed it here by adding the extra check to the latter conditional.	4 days ago
Chris Lu	ad76487e9d	Fix special characters in admin-generated secret keys (#7994 ) Fixes #7990 The issue was that the Charset constant used for generating secret keys included the '/' character, which is URL-unsafe. When secret keys containing '/' were used in HTTP requests, they would be URL-encoded, causing a mismatch during signature verification. Changes: - Removed '/' from the Charset constant in weed/iam/constants.go - Added TestGenerateSecretAccessKey_URLSafe to verify generated keys don't contain URL-unsafe characters like '/' or '+' This ensures all newly generated secret keys are URL-safe and will work correctly with S3 authentication. Existing keys continue to work.	4 days ago
Chris Lu	1ea6b0c0d9	cleanup: deduplicate environment variable credential loading Previously, `weed mini` logic duplicated the credential loading process by creating a temporary IAM config file from environment variables. `auth_credentials.go` also had fallback logic to load these variables. This change: 1. Updates `auth_credentials.go` to always check for and merge AWS environment variable credentials (`AWS_ACCESS_KEY_ID`, etc.) into the identity list. This ensures they are available regardless of whether other configurations (static file or filer) are loaded. 2. Removes the redundant file creation logic from `weed/command/mini.go`. 3. Updates `weed mini` user messages to accurately reflect that credentials are loaded from environment variables in-memory. This results in a cleaner implementation where `weed/s3api` manages all credential loading logic, and `weed mini` simply relies on it.	5 days ago
Chris Lu	7f1182472a	fix: enable dual loading of static and dynamic IAM configuration Refactored `NewIdentityAccessManagementWithStore` to remove mutual exclusivity between static (file-based) and dynamic (filer-based) configuration loading. Previously, if a static config configuration was present (including the legacy `IamConfig` option used by `weed mini`), it prevented loading users from the filer. Now, the system loads the static configuration first (if present), and then always attempts to merge in the dynamic configuration from the filer. This ensures that: 1. Static users (e.g. from `weed mini` env vars or `-s3.config`) are loaded and protected. 2. Dynamic users (e.g. created via Admin UI and stored in Filer) are also loaded and available.	5 days ago
Chris Lu	451b897d56	fix: support loading static config from IamConfig option for mini mode `weed mini` sets the `-s3.iam.config` flag instead of `-s3.config`, which populates `S3ApiServerOption.IamConfig`. Previously, `NewIdentityAccessManagementWithStore` only checked `option.Config`. This caused `weed mini` generated credentials (written to a temp file passed via IamConfig) to be ignored, breaking S3 access in mini mode even when environment variables were provided. This change ensures we try to load the configuration from `IamConfig` if `Config` is empty, restoring functionality for `weed mini`.	5 days ago
Chris Lu	48ded6b965	fix: allow environment variable fallback when filer config is empty Fixed regression where AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables were not being loaded as fallback credentials. The issue was that configLoaded was set to true when filer call succeeded, even if it returned an empty configuration. This blocked the environment variable fallback logic. Now only set configLoaded = true when we actually have loaded identities, allowing env vars to work correctly in mini mode and other scenarios where filer config is empty.	5 days ago
Chris Lu	4e835a1d81	fix(s3api): ensure S3 configuration persistence and refactor authorization tests (#7989 ) * fix(s3api): ensure static config file takes precedence over dynamic updates When a static S3 configuration file is provided, avoid overwriting the configuration from dynamic filer updates. This ensures the documented "Highest Priority" for the configuration file is respected. * refactor(s3api): implement merge-based static config with immutable identities Static identities from config file are now immutable and protected from dynamic updates. Dynamic identities (from admin panel) can be added and updated without affecting static entries. - Track identity names loaded from static config file - Implement merge logic that preserves static identities - Allow dynamic identities to be added or updated - Remove blanket block on config file updates * fix: address PR review comments for static config merge logic Critical Bugs: - Fix existingIdx always-false condition causing duplicate identities - Fix race condition in static config initialization (move useStaticConfig inside mutex) Security & Robustness: - Add nil identity check in VerifyActionPermission to fail closed - Mask access keys in STS validation logs to avoid exposing credentials - Add nil guard for s3a.iam in subscription handler Test Improvements: - Add authCalled tracking to MockIAMIntegration for explicit verification - Lower log level for static config messages to reduce noise * fix: prevent duplicates and race conditions in merge logic Data Integrity: - Prevent service account credential duplicates on repeated merges - Clean up stale accessKeyIdent entries when replacing identities - Check existing credentials before appending Concurrency Safety: - Add synchronization to IsStaticConfig method Test Improvements: - Add mux route vars for proper GetBucketAndObject extraction - Add STS session token header to trigger correct auth path	5 days ago
Chris Lu	6bf0c16862	fix admin copy text functions	5 days ago
Chris Lu	abfa64456b	Fix STS authorization in streaming/chunked uploads (#7988 ) * Fix STS authorization in streaming/chunked uploads During streaming/chunked uploads (SigV4 streaming), authorization happens twice: 1. Initial authorization in authRequestWithAuthType() - works correctly 2. Second authorization in verifyV4Signature() - was failing for STS The issue was that verifyV4Signature() only used identity.canDo() for permission checks, which always denies STS identities (they have empty Actions). This bypassed IAM authorization completely. This commit makes verifyV4Signature() IAM-aware by adding the same fallback logic used in authRequestWithAuthType(): - Traditional identities (with Actions) use legacy canDo() check - STS/JWT identities (empty Actions) fall back to IAM authorization Fixes: https://github.com/seaweedfs/seaweedfs/pull/7986#issuecomment-3723196038 * Add comprehensive unit tests for STS authorization in streaming uploads Created test suite to verify that verifyV4Signature properly handles STS identities by falling back to IAM authorization when shouldCheckPermissions is true. Tests cover: - STS identities with IAM integration (allow and deny cases) - STS identities without IAM integration (should deny) - Traditional identities with Actions (canDo check) - Permission check bypass when shouldCheckPermissions=false - Specific streaming upload scenario from bug report - Action determination based on HTTP method All tests pass successfully. * Refactor authorization logic to avoid duplication Centralized the authorization logic into IdentityAccessManagement.VerifyActionPermission. Updated auth_signature_v4.go and auth_credentials.go to use this new helper. Updated tests to clarify that they mirror the centralized logic. * Refactor tests to use VerifyActionPermission directly Introduced IAMIntegration interface to facilitate mocking of internal IAM integration logic. Updated IdentityAccessManagement to use the interface. Updated tests to directy call VerifyActionPermission using a mocked IAM integration, eliminating duplicated logic in tests. * fix(s3api): ensure static config file takes precedence and refactor tests - Track if configuration was loaded from a static file using `useStaticConfig`. - Ignore filer-based IAM updates when a static configuration is in use to respect "Highest Priority" rule. - Refactor `TestVerifyV4SignatureWithSTSIdentity` to use `VerifyActionPermission` directly. - Fix typed nil interface panic in authorization test.	5 days ago
Chris Lu	217d8b9e0e	Fix: ListObjectVersions delimiter support (#7987 ) * Fix: Add delimiter support to ListObjectVersions with proper truncation - Implemented delimiter support to group keys into CommonPrefixes - Fixed critical truncation bug: now merges versions and common prefixes into single sorted list before truncation - Ensures total items never exceed MaxKeys (prevents infinite pagination loops) - Properly sets NextKeyMarker and NextVersionIdMarker for pagination - Added integration tests in test/s3/versioning/s3_versioning_delimiter_test.go - Verified behavior matches S3 API specification * Fix: Add delimiter support to ListObjectVersions with proper truncation - Implemented delimiter support to group keys into CommonPrefixes - Fixed critical truncation bug: now merges versions and common prefixes before truncation - Added safety guard for maxKeys=0 to prevent panics - Condensed verbose comments for better readability - Added robust Go integration tests with nil checks for AWS SDK pointers - Verified behavior matches S3 API specification - Resolved compilation error in integration tests - Refined pagination comments and ensured exclusive KeyMarker behavior - Refactored listObjectVersions into helper methods for better maintainability	5 days ago
Chris Lu	4ba89bf73b	adjust log level	5 days ago
Chris Lu	bd237999bb	weed mini can optionally skip s3	5 days ago
Chris Lu	5a3aade445	less logs	5 days ago
Chris Lu	f02e283ad2	add a nginx with ssl for easier testing	5 days ago
promalert	9012069bd7	chore: execute goimports to format the code (#7983 ) * chore: execute goimports to format the code Signed-off-by: promalert <promalert@outlook.com> * goimports -w . --------- Signed-off-by: promalert <promalert@outlook.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	6 days ago
Chris Lu	6432019d08	Fix STS identity authorization by populating PolicyNames (#7985 ) (#7986 ) * Fix STS identity authorization by populating PolicyNames (#7985) This commit fixes GitHub issue #7985 where STS-assumed identities received empty identity.Actions, causing all S3 operations to be denied even when the role had valid IAM policies attached. Changes: 1. Populate PolicyNames field from sessionInfo.Policies in validateSTSSessionToken() to enable IAM-based authorization for STS identities 2. Fix bucket+objectKey path construction in canDo() method to include proper slash separator between bucket and object key 3. Add comprehensive test suite to validate the fix and prevent regression The fix ensures that STS-assumed identities are properly authorized through the IAM path when iamIntegration is available, allowing roles with valid IAM policies to perform S3 operations as expected. * Update STS identity tests to be more rigorous and use actual implementation path * Fix regression in canDo() path concatenation The previous fix blindly added a slash separator, which caused double slashes when objectKey already started with a slash (common in existing tests and some code paths). This broke TestCanDo and TestObjectLevelListPermissions. This commit updates the logic to only add the slash separator if objectKey is not empty and does not already start with a slash. This fixes the regressions while maintaining the fix for issue #7985. * Refactor STS identity tests: extract helpers and simplify redundant logic - Extracted setupTestSTSService and newTestIdentity helper functions - Removed redundant if-else verification blocks that were already covered by assertions - Cleaned up test cases to improve maintainability as suggested in code review. * Add canDo() verification to STS identity tests Address code review suggestion: verify that identities with empty Actions correctly return false for canDo() checks, which confirms the behavior that forces authorization to fall back to the IAM path. * Simplify TestCanDoPathConstruction variable names Rename expectedPath to fullPath and simplify logging/assertion logic based on code review feedback. * Refactor path construction and logging in canDo() - Compute fullPath early and use it for logging to prevent double slashes - Update TestCanDoPathConstruction to use robust path verification - Add test case for objectKey with leading slash to ensure correct handling	6 days ago
Chris Lu	e67973dc53	Support Policy Attachment for Object Store Users (#7981 ) * Implement Policy Attachment support for Object Store Users - Added policy_names field to iam.proto and regenerated protos. - Updated S3 API and IAM integration to support direct policy evaluation for users. - Enhanced Admin UI to allow attaching policies to users via modals. - Renamed 'policies' to 'policy_names' to clarify that it stores identifiers. - Fixed syntax error in user_management.go. * Fix policy dropdown not populating The API returns {policies: [...]} but JavaScript was treating response as direct array. Updated loadPolicies() to correctly access data.policies property. * Add null safety checks for policy dropdowns Added checks to prevent "undefined" errors when: - Policy select elements don't exist - Policy dropdowns haven't loaded yet - User is being edited before policies are loaded * Fix policy dropdown by using correct JSON field name JSON response has lowercase 'name' field but JavaScript was accessing 'Name'. Changed policy.Name to policy.name to match the IAMPolicy JSON structure. * Fix policy names not being saved on user update Changed condition from len(req.PolicyNames) > 0 to req.PolicyNames != nil to ensure policy names are always updated when present in the request, even if it's an empty array (to allow clearing policies). * Add debug logging for policy names update flow Added console.log in frontend and glog in backend to trace policy_names data through the update process. * Temporarily disable auto-reload for debugging Commented out window.location.reload() so console logs are visible when updating a user. * Add detailed debug logging and alert for policy selection Added console.log for each step and an alert to show policy_names value to help diagnose why it's not being included in the request. * Regenerate templ files for object_store_users Ran templ generate to ensure _templ.go files are up to date with the latest .templ changes including debug logging. * Remove debug logging and restore normal functionality Cleaned up temporary debug code (console.log and alert statements) and re-enabled automatic page reload after user update. * Add step-by-step alert debugging for policy update Added 5 alert checkpoints to trace policy data through the update flow: 1. Check if policiesSelect element exists 2. Show selected policy values 3. Show userData.policy_names 4. Show full request body 5. Confirm server response Temporarily disabled auto-reload to see alerts. * Add version check alert on page load Added alert on DOMContentLoaded to verify new JavaScript is being executed and not cached by the browser. * Compile templates using make Ran make to compile all template files and install the weed binary. * Add button click detection and make handleUpdateUser global - Added inline alert on button click to verify click is detected - Made handleUpdateUser a window-level function to ensure it's accessible - Added alert at start of handleUpdateUser function * Fix handleUpdateUser scope issue - remove duplicate definition Removed duplicate function definition that was inside DOMContentLoaded. Now handleUpdateUser is defined only once in global scope (line 383) making it accessible when button onclick fires. * Remove all duplicate handleUpdateUser definitions Now handleUpdateUser is defined only once at the very top of the script block (line 352), before DOMContentLoaded, ensuring it's available when the button onclick fires. * Add function existence check and error catching Added alerts to check if handleUpdateUser is defined and wrapped the function call in try-catch to capture any JavaScript errors. Also added console.log statements to verify function definition. * Simplify handleUpdateUser to non-async for testing Removed async/await and added early return to test if function can be called at all. This will help identify if async is causing the issue. * Add cache-control headers to prevent browser caching Added no-cache headers to ShowObjectStoreUsers handler to prevent aggressive browser caching of inline JavaScript in the HTML page. * Fix syntax error - make handleUpdateUser async Changed function back to async to fix 'await is only valid in async functions' error. The cache-control headers are working - browser is now loading new code. * Update version check to v3 to verify cache busting Changed version alert to 'v3 - WITH EARLY RETURN' to confirm the new code with early return statement is being loaded. * Remove all debug code - clean implementation Removed all alerts, console.logs, and test code. Implemented clean policy update functionality with proper error handling. * Add ETag header for cache-busting and update walkthrough * Fix policy pre-selection in Edit User modal - Updated admin.js editUser function to pre-select policies - Root cause: duplicate editUser in admin.js overwrote inline version - Added policy pre-selection logic to match inline template - Verified working in browser: policies now pre-select correctly * Fix policy persistence in handleUpdateUser - Added policy_names field to userData payload in handleUpdateUser - Policies were being lost because handleUpdateUser only sent email and actions - Now collects selected policies from editPolicies dropdown - Verified working: policies persist correctly across updates * Fix XSS vulnerability in access keys display - Escape HTML in access key display using escapeHtml utility - Replace inline onclick handlers with data attributes - Add event delegation for delete access key buttons - Prevents script injection via malicious access key values * Fix additional XSS vulnerabilities in user details display - Escape HTML in actions badges (line 626) - Escape HTML in policy_names badges (line 636) - Prevents script injection via malicious action or policy names * Fix XSS vulnerability in loadPolicies function - Replace innerHTML string concatenation with DOM API - Use createElement and textContent for safe policy name insertion - Prevents script injection via malicious policy names - Apply same pattern to both create and edit select elements * Remove debug logging from UpdateObjectStoreUser - Removed glog.V(0) debug statements - Clean up temporary debugging code before production * Remove duplicate handleUpdateUser function - Removed inline handleUpdateUser that duplicated admin.js logic - Removed debug console.log statement - admin.js version is now the single source of truth - Eliminates maintenance burden of keeping two versions in sync * Refine user management and address code review feedback - Preserve PolicyNames in UpdateUserPolicies - Allow clearing actions in UpdateObjectStoreUser by checking for nil - Remove version comment from object_store_users.templ - Refactor loadPolicies for DRYness using cloneNode while keeping DOM API security * IAM Authorization for Static Access Keys * verified XSS Fixes in Templates * fix div	7 days ago
Chris Lu	d4ecfaeda7	Enable writeback_cache and async_dio FUSE options (#7980 ) * Enable writeback_cache and async_dio FUSE options Fixes #7978 - Update mount_std.go to use EnableWriteback and EnableAsyncDio from go-fuse - Add go.mod replace directive to use local go-fuse with capability support - Remove temporary workaround that disabled these options This enables proper FUSE kernel capability negotiation for writeback cache and async direct I/O, improving performance for small writes and concurrent direct I/O operations. * Address PR review comments - Remove redundant nil checks for writebackCache and asyncDio flags - Update go.mod replace directive to use seaweedfs/go-fuse fork instead of local path * Add TODO comment for go.mod replace directive The replace directive must use a local path until seaweedfs/go-fuse#1 is merged. After merge, this should be updated to use the proper version. * Use seaweedfs/go-fuse v2.9.0 instead of local repository Replace local path with seaweedfs/go-fuse v2.9.0 fork which includes the writeback_cache and async_dio capability support. * Use github.com/seaweedfs/go-fuse/v2 directly without replace directive - Updated all imports to use github.com/seaweedfs/go-fuse/v2 - Removed replace directive from go.mod - Using seaweedfs/go-fuse v2.0.0-20260106181308-87f90219ce09 which includes: * writeback_cache and async_dio support * Corrected module path * Update to seaweedfs/go-fuse v2.9.1 Use v2.9.1 tag which includes the corrected module path (github.com/seaweedfs/go-fuse/v2) along with writeback_cache and async_dio support.	1 week ago
Chris Lu	e10f11b480	opt: reduce ShardsInfo memory usage with bitmap and sorted slice (#7974 ) * opt: reduce ShardsInfo memory usage with bitmap and sorted slice - Replace map[ShardId]ShardInfo with sorted []ShardInfo slice - Add ShardBits (uint32) bitmap for O(1) existence checks - Use binary search for O(log n) lookups by shard ID - Maintain sorted order for efficient iteration - Add comprehensive unit tests and benchmarks Memory savings: - Map overhead: ~48 bytes per entry eliminated - Pointers: 8 bytes per entry eliminated - Total: ~56 bytes per shard saved Performance improvements: - Has(): O(1) using bitmap - Size(): O(log n) using binary search (was O(1), acceptable tradeoff) - Count(): O(1) using popcount on bitmap - Iteration: Faster due to cache locality refactor: add methods to ShardBits type - Add Has(), Set(), Clear(), and Count() methods to ShardBits - Simplify ShardsInfo methods by using ShardBits methods - Improves code readability and encapsulation * opt: use ShardBits directly in ShardsCountFromVolumeEcShardInformationMessage Avoid creating a full ShardsInfo object just to count shards. Directly cast vi.EcIndexBits to ShardBits and use Count() method. * opt: use strings.Builder in ShardsInfo.String() for efficiency * refactor: change AsSlice to return []ShardInfo (values instead of pointers) This completes the memory optimization by avoiding unnecessary pointer slices and potential allocations. * refactor: rename ShardsCountFromVolumeEcShardInformationMessage to GetShardCount * fix: prevent deadlock in Add and Subtract methods Copy shards data from 'other' before releasing its lock to avoid potential deadlock when a.Add(b) and b.Add(a) are called concurrently. The previous implementation held other's lock while calling si.Set/Delete, which acquires si's lock. This could deadlock if two goroutines tried to add/subtract each other concurrently. * opt: avoid unnecessary locking in constructor functions ShardsInfoFromVolume and ShardsInfoFromVolumeEcShardInformationMessage now build shards slice and bitmap directly without calling Set(), which acquires a lock on every call. Since the object is local and not yet shared, locking is unnecessary and adds overhead. This improves performance during object construction. * fix: rename 'copy' variable to avoid shadowing built-in function The variable name 'copy' in TestShardsInfo_Copy shadowed the built-in copy() function, which is confusing and bad practice. Renamed to 'siCopy'. * opt: use math/bits.OnesCount32 and reorganize types 1. Replace manual popcount loop with math/bits.OnesCount32 for better performance and idiomatic Go code 2. Move ShardSize type definition to ec_shards_info.go for better code organization since it's primarily used there * refactor: Set() now accepts ShardInfo for future extensibility Changed Set(id ShardId, size ShardSize) to Set(shard ShardInfo) to support future additions to ShardInfo without changing the API. This makes the code more extensible as new fields can be added to ShardInfo (e.g., checksum, location, etc.) without breaking the Set API. * refactor: move ShardInfo and ShardSize to separate file Created ec_shard_info.go to hold the basic shard types (ShardInfo and ShardSize) for better code organization and separation of concerns. * refactor: add ShardInfo constructor and helper functions Added NewShardInfo() constructor and IsValid() method to better encapsulate ShardInfo creation and validation. Updated code to use the constructor for cleaner, more maintainable code. * fix: update remaining Set() calls to use NewShardInfo constructor Fixed compilation errors in storage and shell packages where Set() calls were not updated to use the new NewShardInfo() constructor. * fix: remove unreachable code in filer backup commands Removed unreachable return statements after infinite loops in filer_backup.go and filer_meta_backup.go to fix compilation errors. * fix: rename 'new' variable to avoid shadowing built-in Renamed 'new' to 'result' in MinusParityShards, Plus, and Minus methods to avoid shadowing Go's built-in new() function. * fix: update remaining test files to use NewShardInfo constructor Fixed Set() calls in command_volume_list_test.go and ec_rebalance_slots_test.go to use NewShardInfo() constructor.	1 week ago
Chris Lu	a75cc09cdf	test: fix EC integration test needle blob mismatch (#7972 ) fix needle blob	1 week ago
Chris Lu	ec1c27a4b3	storage/needle: add bounds check for WriteNeedleBlob buffer (#7973 ) * storage/needle: add bounds check for WriteNeedleBlob buffer * storage/needle: use int offsets when checking/writing Version3 timestamp * Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	1 week ago
Chris Lu	d2f0d6c03b	go mod	1 week ago
Chris Lu	d75162370c	Fix trust policy wildcard principal handling (#7970 ) * Fix trust policy wildcard principal handling This change fixes the trust policy validation to properly support AWS-standard wildcard principals like {"Federated": ""}. Previously, the evaluatePrincipalValue() function would check for context existence before evaluating wildcards, causing wildcard principals to fail when the context key didn't exist. This forced users to use the plain "" workaround instead of the more specific {"Federated": ""} format. Changes: - Modified evaluatePrincipalValue() to check for "" FIRST before validating against context - Added support for wildcards in principal arrays - Added comprehensive tests for wildcard principal handling - All existing tests continue to pass (no regressions) This matches AWS IAM behavior where "" in a principal field means "allow any value" without requiring context validation. Fixes: https://github.com/seaweedfs/seaweedfs/issues/7917 Refactor: Move Principal matching to PolicyEngine This refactoring consolidates all policy evaluation logic into the PolicyEngine, improving code organization and eliminating duplication. Changes: - Added matchesPrincipal() and evaluatePrincipalValue() to PolicyEngine - Added EvaluateTrustPolicy() method for direct trust policy evaluation - Updated statementMatches() to check Principal field when present - Made resource matching optional (trust policies don't have Resources) - Simplified evaluateTrustPolicy() in iam_manager.go to delegate to PolicyEngine - Removed ~170 lines of duplicate code from iam_manager.go Benefits: - Single source of truth for all policy evaluation - Better code reusability and maintainability - Consistent evaluation rules for all policy types - Easier to test and debug All tests pass with no regressions. * Make PolicyEngine AWS-compatible and add unit tests Changes: 1. AWS-Compatible Context Keys: - Changed "seaweed:FederatedProvider" -> "aws:FederatedProvider" - Changed "seaweed:AWSPrincipal" -> "aws:PrincipalArn" - Changed "seaweed:ServicePrincipal" -> "aws:PrincipalServiceName" - This ensures 100% AWS compatibility for trust policies 2. Added Comprehensive Unit Tests: - TestPrincipalMatching: 8 test cases for Principal matching - TestEvaluatePrincipalValue: 7 test cases for value evaluation - TestTrustPolicyEvaluation: 6 test cases for trust policy evaluation - TestGetPrincipalContextKey: 4 test cases for context key mapping - Total: 25 new unit tests for PolicyEngine All tests pass: - Policy engine tests: 54 passed - Integration tests: 9 passed - Total: 63 tests passing * Update context keys to standard AWS/OIDC formats Replaced remaining seaweed: context keys with standard AWS and OIDC keys to ensure 100% compatibility with AWS IAM policies. Mappings: - seaweed:TokenIssuer -> oidc:iss - seaweed:Issuer -> oidc:iss - seaweed:Subject -> oidc:sub - seaweed:SourceIP -> aws:SourceIp Also updated unit tests to reflect these changes. All 63 tests pass successfully. * Add advanced policy tests for variable substitution and conditions Added comprehensive tests inspired by AWS IAM patterns: - TestPolicyVariableSubstitution: Tests ${oidc:sub} variable in resources - TestConditionWithNumericComparison: Tests sts:DurationSeconds condition - TestMultipleConditionOperators: Tests combining StringEquals and StringLike Results: - TestMultipleConditionOperators: ✅ All 3 subtests pass - Other tests reveal need for sts:DurationSeconds context population These tests validate the PolicyEngine's ability to handle complex AWS-compatible policy scenarios. * Fix federated provider context and add DurationSeconds support Changes: - Use iss claim as aws:FederatedProvider (AWS standard) - Add sts:DurationSeconds to trust policy evaluation context - TestPolicyVariableSubstitution now passes ✅ Remaining work: - TestConditionWithNumericComparison partially works (1/3 pass) - Need to investigate NumericLessThanEquals evaluation * Update trust policies to use issuer URL for AWS compatibility Changed trust policy from using provider name ("test-oidc") to using the issuer URL ("https://test-issuer.com") to match AWS standard behavior where aws:FederatedProvider contains the OIDC issuer URL. Test Results: - 10/12 test suites passing - TestFullOIDCWorkflow: ✅ All subtests pass - TestPolicyEnforcement: ✅ All subtests pass - TestSessionExpiration: ✅ Pass - TestPolicyVariableSubstitution: ✅ Pass - TestMultipleConditionOperators: ✅ All subtests pass Remaining work: - TestConditionWithNumericComparison needs investigation - One subtest in TestTrustPolicyValidation needs fix * Fix S3 API tests for AWS compatibility Updated all S3 API tests to use AWS-compatible context keys and trust policy principals: Changes: - seaweed:SourceIP → aws:SourceIp (IP-based conditions) - Federated: "test-oidc" → "https://test-issuer.com" (trust policies) Test Results: - TestS3EndToEndWithJWT: ✅ All 13 subtests pass - TestIPBasedPolicyEnforcement: ✅ All 3 subtests pass This ensures policies are 100% AWS-compatible and portable. * Fix ValidateTrustPolicy for AWS compatibility Updated ValidateTrustPolicy method to check for: - OIDC: issuer URL ("https://test-issuer.com") - LDAP: provider name ("test-ldap") - Wildcard: "" Test Results: - TestTrustPolicyValidation: ✅ All 3 subtests pass This ensures trust policy validation uses the same AWS-compatible principals as the PolicyEngine. Fix multipart and presigned URL tests for AWS compatibility Updated trust policies in: - s3_multipart_iam_test.go - s3_presigned_url_iam_test.go Changed "Federated": "test-oidc" → "https://test-issuer.com" Test Results: - TestMultipartIAMValidation: ✅ All 7 subtests pass - TestPresignedURLIAMValidation: ✅ All 4 subtests pass - TestPresignedURLGeneration: ✅ All 4 subtests pass - TestPresignedURLExpiration: ✅ All 4 subtests pass - TestPresignedURLSecurityPolicy: ✅ All 4 subtests pass All S3 API tests now use AWS-compatible trust policies. * Fix numeric condition evaluation and trust policy validation interface Major updates to ensure robust AWS-compatible policy evaluation: 1. Policy Engine: Added support for `int` and `int64` types in `evaluateNumericCondition`, fixing issues where raw numbers in policy documents caused evaluation failures. 2. Trust Policy Validation: Updated `TrustPolicyValidator` interface and `STSService` to propagate `DurationSeconds` correctly during the double-validation flow (Validation -> STS -> Validation callback). 3. IAM Manager: Updated implementation to match the new interface and correctly pass `sts:DurationSeconds` context key. Test Results: - TestConditionWithNumericComparison: ✅ All 3 subtests pass - All IAM and S3 integration tests pass (100%) This resolves the final edge case with DurationSeconds numeric conditions. * Fix MockTrustPolicyValidator interface and unreachable code warnings Updates: 1. Updated MockTrustPolicyValidator.ValidateTrustPolicyForWebIdentity to match new interface signature with durationSeconds parameter 2. Removed unreachable code after infinite loops in filer_backup.go and filer_meta_backup.go to satisfy linter Test Results: - All STS tests pass ✅ - Build warnings resolved ✅ * Refactor matchesPrincipal to consolidate array handling logic Consolidated duplicated logic for []interface{} and []string types by converting them to a unified []interface{} upfront. * Fix malformed AWS docs URL in iam_manager.go comment * dup * Enhance IAM integration tests with negative cases and interface array support Added test cases to TestTrustPolicyWildcardPrincipal to: 1. Verify rejection of roles when principal context does not match (negative test) 2. Verify support for principal arrays as []interface{} (simulating JSON unmarshaled roles) * Fix syntax errors in filer_backup and filer_meta_backup Restored missing closing braces for for-loops and re-added return statements. The previous attempt to remove unreachable code accidentally broke the function structure. Build now passes successfully.	1 week ago
Chris Lu	d15f32ae46	feat: add flags to disable WebDAV and Admin UI in weed mini (#7971 ) * feat: add flags to disable WebDAV and Admin UI in weed mini - Add -webdav flag (default: true) to optionally disable WebDAV server - Add -admin.ui flag (default: true) to optionally disable Admin UI only (server still runs) - Conditionally skip WebDAV service startup based on flag - Pass disableUI flag to SetupRoutes to skip UI route registration - Admin server still runs for gRPC and API access when UI is disabled Addresses issue from https://github.com/seaweedfs/seaweedfs/pull/7833#issuecomment-3711924150 * refactor: use positive enableUI parameter instead of disableUI across admin server and handlers * docs: update mini welcome message to list enabled components * chore: remove unused welcomeMessageTemplate constant * docs: split S3 credential message into separate sb.WriteString calls	1 week ago
dependabot[bot]	021d9fdab5	chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.93.0 to 1.95.0 (#7964 ) chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.93.0 to 1.95.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.93.0...service/s3/v1.95.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.95.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	1 week ago
dependabot[bot]	91b8cb0733	chore(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.122.0 to 3.125.1 (#7966 ) chore(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.122.0 to 3.125.1. - [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases) - [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md) - [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.122.0...v3.125.1) --- updated-dependencies: - dependency-name: github.com/ydb-platform/ydb-go-sdk/v3 dependency-version: 3.125.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	1 week ago
Chris Lu	15ca301e43	Fix flaky EC integration tests by collecting server logs on failure (#7969 ) * Fix flaky EC integration tests by collecting server logs on failure The EC Integration Tests were experiencing flaky timeouts with errors like "error reading from server: EOF" and master client reconnection attempts. When tests failed, server logs were not collected, making debugging difficult. Changes: - Updated all test functions to use t.TempDir() instead of os.MkdirTemp() and manual cleanup. t.TempDir() automatically preserves directories when tests fail, ensuring logs are available for debugging. - Modified GitHub Actions workflow to collect server logs from temp directories when tests fail, including master.log and volume.log files. - Added explicit log collection step that searches for test temp directories and copies them to artifacts for upload. This will make debugging flaky test failures much easier by providing access to the actual server logs showing what went wrong. Fix find command precedence in log collection The -type d flag only applied to the first -name predicate because -o has lower precedence than the implicit AND. Grouped the -name predicates with escaped parentheses so -type d applies to all directory name patterns.	1 week ago
dependabot[bot]	fd1cac8123	chore(deps): bump github.com/getsentry/sentry-go from 0.38.0 to 0.40.0 (#7968 ) Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go) from 0.38.0 to 0.40.0. - [Release notes](https://github.com/getsentry/sentry-go/releases) - [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md) - [Commits](https://github.com/getsentry/sentry-go/compare/v0.38.0...v0.40.0) --- updated-dependencies: - dependency-name: github.com/getsentry/sentry-go dependency-version: 0.40.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	1 week ago
dependabot[bot]	d5fcdc345a	chore(deps): bump gocloud.dev/pubsub/rabbitpubsub from 0.43.0 to 0.44.0 (#7967 ) Bumps [gocloud.dev/pubsub/rabbitpubsub](https://github.com/google/go-cloud) from 0.43.0 to 0.44.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](https://github.com/google/go-cloud/compare/v0.43.0...v0.44.0) --- updated-dependencies: - dependency-name: gocloud.dev/pubsub/rabbitpubsub dependency-version: 0.44.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	1 week ago
dependabot[bot]	22fda45ccb	chore(deps): bump go.etcd.io/etcd/client/pkg/v3 from 3.6.6 to 3.6.7 (#7965 ) Bumps [go.etcd.io/etcd/client/pkg/v3](https://github.com/etcd-io/etcd) from 3.6.6 to 3.6.7. - [Release notes](https://github.com/etcd-io/etcd/releases) - [Commits](https://github.com/etcd-io/etcd/compare/v3.6.6...v3.6.7) --- updated-dependencies: - dependency-name: go.etcd.io/etcd/client/pkg/v3 dependency-version: 3.6.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	1 week ago
Chris Lu	383c2e3b41	fix: handle range requests on empty objects (size=0) (#7963 ) * fix: handle range requests on empty objects (size=0) Range requests on empty objects were incorrectly being rejected with: 'invalid range start for ...: 0 >= 0' The validation logic used 'startOffset >= totalSize' which failed when both were 0, incorrectly rejecting valid range requests like bytes=0-1535 on 0-byte files. Fix: Added special case handling before validation to properly return 416 Range Not Satisfiable for any range request on an empty object, per RFC 7233. Fixed at two locations (lines 873 and 1154) in s3api_object_handlers.go * refactor: return 404 for directory objects, not 416 Per S3 semantics, GET requests on directory paths (without trailing "/") should return 404 Not Found, not try to serve them as objects. Updated fix to: 1. Check if entry.IsDirectory and return 404 (S3-compliant) 2. Only return 416 for true empty files (size=0, not directory) This matches AWS S3 behavior where directories don't exist as objects unless they're explicit directory markers ending with "/". * reduce repeated info * refactor: move directory check before range branching This ensures that any Range header (including suffix ranges like bytes=-N) on a directory path (without trailing slash) returns 404 (ErrNoSuchKey) instead of potentially returning 416 or attempting to serve as an object. Applied to both streamFromVolumeServers and streamFromVolumeServersWithSSE. * refactoring	1 week ago
MorezMartin	629d9479a1	Fix jwt error in Filer pod (k8s) (#7960 ) * Avoid JWT error on liveprobeness * fix jwt error * address comments * lint --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	1 week ago
Chris Lu	21a871f3ad	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	1 week ago
Chris Lu	0e9f433ec4	refactoring	1 week ago
Alasdair Macmillan	9778b9589e	Fix unaligned 64-bit atomic operation on ARM32 (#7958 ) (#7959 )	1 week ago
Chris Lu	8269dc136d	simplify	1 week ago
Robert Schade	de3df211d7	store S3 storage class in extended atrributes #7961 (#7962 ) * store S3 storage class in extended atrributes #7961 * canonical * remove issue reference --------- Co-authored-by: Robert Schade <robert.schade@uni-paderborn.de> Co-authored-by: Chris Lu <chris.lu@gmail.com>	1 week ago
LeeXN	c909724bf1	Fix: prevent panic when swap file creation fails (#7957 ) * Fix: prevent panic when swap file creation fails * weed mount: fix race condition in swap file initialization Ensure thread-safe access to sf.file and other state in NewSwapFileChunk and FreeResource by using sf.chunkTrackingLock consistently. Also set sf.file to nil after closing to prevent reuse. * weed mount: improve swap directory creation logic - Check error for os.MkdirAll and log it if it fails. - Use 0700 permissions for the swap directory for better security. - Improve error logging context. * weed mount: add unit tests for swap file creation Add tests to verify: - Concurrent initialization of the swap file. - Correct directory permissions (0700). - Automatic directory recreation if deleted. * weed mount: fix thread-safety in swap file unit tests Use atomic.Uint32 to track failures within goroutines in TestSwapFile_NewSwapFileChunk_Concurrent to avoid unsafe calls to t.Errorf from multiple goroutines. * weed mount: simplify swap file creation logic Refactor the directory check and retry logic for better readability and to avoid re-using the main error variable for directory creation errors. Remove redundant error logging. * weed mount: improve error checking in swap file tests Explicitly check if NewSwapFileChunk returns nil to provide more informative failures. * weed mount: update DirtyPages interface to return error Propagate errors from SaveDataAt when swap file creation fails. This prevents potential panics in the write path. * weed mount: handle AddPage errors in write paths Update ChunkedDirtyPages and PageWriter to propagate errors and update WFS.Write and WFS.CopyFileRange to return fuse.EIO on failure. * weed mount: update swap directory creation error message Change "recreate" to "create/recreate" to better reflect that this path is also taken during the initial creation of the swap directory. --------- Co-authored-by: lixiang58 <lixiang58@lenovo.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	1 week ago
Chris Lu	0b5a65e00b	avoid extra missing shard warning fix https://github.com/seaweedfs/seaweedfs/issues/7956	1 week ago
Chris Lu	63b2fe0d76	fix: EC UI template error when viewing shard details (#7955 ) * fix: EC UI template error when viewing shard details Fixed field name mismatch in volume.html where it was using .ShardDetails instead of .Shards. Added a robust type conversion wrapper in templates.go to handle int64 to uint64 conversion for bytesToHumanReadable. Added regression test to ensure future stability. * refactor: improve bytesToHumanReadable and test robustness - Handled more integer types (uint32, int32, uint) in bytesToHumanReadable. - Improved volume_test.go to verify both shards are formatted correctly. * refactor: add bounds checking to bytesToHumanReadable Added checks for negative values in signed integer types to avoid incorrect formatting when converting to uint64. Addressed feedback from coderabbitai.	1 week ago
Chris Lu	0647bc24d5	s3api: fix authentication bypass and potential SIGSEGV (Issue #7912 ) (#7954 ) * s3api: fix authentication bypass and potential SIGSEGV * s3api: improve security tests with positive cases and nil identity guards * s3api: fix secondary authentication bypass in AuthSignatureOnly * s3api: refactor account loading and refine security tests based on review feedback * s3api: refine security tests with realistic signature failures * Update weed/s3api/auth_security_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	1 week ago
Chris Lu	54de32f207	Support AWS standard IAM role ARN formats (issue #7946 ) (#7948 ) * fix(iam): support both AWS standard and legacy IAM role ARN formats Fix issue #7946 where SeaweedFS only recognized legacy IAM role ARN format (arn:aws:iam::role/RoleName) but not the standard AWS format with account ID (arn:aws:iam::ACCOUNT:role/RoleName). This was breaking EKS pod identity integration which expects the standard format. Changes: - Update ExtractRoleNameFromArn() to handle both formats by searching for 'role/' marker instead of matching a fixed prefix - Update ExtractRoleNameFromPrincipal() to clearly document both STS and IAM formats it supports with or without account ID - Simplify role ARN validation in validateRoleAssumptionForWebIdentity() and validateRoleAssumptionForCredentials() to use the extraction function - Add comprehensive test coverage with 25 test cases covering both formats The fix maintains backward compatibility with legacy format while adding support for standard AWS format with account ID. Fixes: https://github.com/seaweedfs/seaweedfs/issues/7946 * docs: improve docstring coverage for ARN utility functions - Add comprehensive package-level documentation - Enhance ExtractRoleNameFromPrincipal docstring with parameter and return descriptions - Enhance ExtractRoleNameFromArn docstring with detailed format documentation - Add docstrings to test functions explaining test coverage - Update all docstrings to 80%+ coverage for code review compliance * refactor: improve ARN parsing code maintainability and error messages - Define constants for ARN prefixes and markers (stsPrefix, stsAssumedRoleMarker, iamPrefix, iamRoleMarker) - Replace hardcoded magic strings with named constants in ExtractRoleNameFromPrincipal and ExtractRoleNameFromArn - Enhance error messages in sts_service.go to show expected ARN format when validation fails - Error message now shows: 'arn:aws:iam::[ACCOUNT_ID:]role/ROLE_NAME' format - Improves code readability and maintainability - Facilitates future ARN format changes and debugging * feat: add structured ARN type for better debugging and extensibility Implements Option 2 (Structured ARN Type) from ARN handling comparison: New Features: - ARNInfo struct with Original, RoleName, AccountID, and Format fields - ARNFormat enum (Legacy, Standard, Invalid) for type-safe format tracking - ParseRoleARN() function for structured IAM role ARN parsing - ParsePrincipalARN() function for structured STS/IAM principal parsing Benefits: - Better debugging: Can see original ARN, extracted components, and format type - Extensible: Easy to add more fields (Region, Service, etc.) in future - Type-safe: Format is an enum, not a string - Backward compatible: Kept original string-based functions STS Service Updates: - Uses ParseRoleARN() for structured validation - Logs ARN components at V(4) level for debugging (role, account, format) - Better error context when validation fails Test Coverage: - 7 new tests for ParseRoleARN (legacy, standard, invalid formats) - 7 new tests for ParsePrincipalARN (STS/IAM, legacy/standard) - All 39 existing tests still pass - Total: 53 ARN-related tests Comparison with MinIO: - More flexible: Supports both AWS formats (MinIO only supports MinIO format) - Better tested: 53 tests vs MinIO's 8 tests - Structured like MinIO but more practical for AWS use cases * security: fix ARN parsing to prevent malicious ARN acceptance Fix critical security vulnerability where malicious ARNs could bypass validation: - ARNs like 'arn:aws:iam::123456789012:user/role/malicious' were incorrectly accepted - The previous implementation used strings.Index to find 'role/' anywhere in the ARN - This allowed non-role resource types to be accepted if they contained 'role/' in their path Changes: 1. Updated ExtractRoleNameFromArn() to validate resource type is exactly 'role/' 2. Updated ExtractRoleNameFromPrincipal() to validate resource type is exactly 'assumed-role/' 3. Updated ParseRoleARN() to validate structure before extracting fields 4. Updated ParsePrincipalARN() to validate structure before extracting fields 5. Added 6 security test cases to prevent regression The fix validates ARN structure by: - Splitting on ':' to separate account ID from resource type - Verifying resource type starts with exact marker ('role/' or 'assumed-role/') - Only then extracting role name, account ID, and format All 59 tests pass, including new security tests that verify malicious ARNs are rejected. Fixes: GitHub Copilot review #3624499048 * test: add test cases for empty role names and improve validation Address review feedback to improve edge case coverage: 1. Added test case for standard format with empty role name - TestExtractRoleNameFromArn: arn:aws:iam::123456789012:role/ - TestParseRoleARN: arn:aws:iam::123456789012:role/ 2. Added empty role name validation for STS ARNs in ParsePrincipalARN - Now matches ParseRoleARN behavior - Prevents ARNs like arn:aws:sts::assumed-role/ from having valid Format 3. Added test cases for empty STS role names - TestParsePrincipalARN: arn:aws:sts::assumed-role/ - TestParsePrincipalARN: arn:aws:sts::123456789012:assumed-role/ All 65 tests pass (15 for ExtractRoleNameFromArn, 10 for ExtractRoleNameFromPrincipal, 8 for ParseRoleARN, 9 for ParsePrincipalARN, 4 security user ARNs, 2 security STS, plus existing tests). * refactor: simplify ARNInfo by removing Format enum Remove ARNFormat enum (ARNFormatLegacy, ARNFormatStandard, ARNFormatInvalid) as it's not needed for backward compatibility. Simplifications: 1. Removed ARNFormat type and all format constants 2. Removed Format field from ARNInfo struct 3. Validation now checks if RoleName is empty (simpler and clearer) 4. AccountID presence already distinguishes legacy (empty) from standard (non-empty) formats 5. Updated STS service to check RoleName emptiness instead of Format field 6. Improved debug logging to explicitly show "(legacy format)" or "(standard format)" Benefits: - Simpler code with fewer concepts - AccountID field already provides format information - Validation is clearer: empty RoleName = invalid ARN - All 65 tests still pass This change maintains the same functionality while reducing code complexity. No backward compatibility concerns as the structured ARN parsing is new. * test: add comprehensive edge case tests for ARN parsing Add 4 new test functions covering: - Multiple role markers in paths (e.g., role/role/name) - Consecutive slashes in role paths (preserved as valid components) - Special characters valid in AWS role names (+=,.@-_) - Extremely long role names near AWS limits These tests verify the parser's resilience to edge cases and ensure proper handling of various valid role name formats and special characters.	1 week ago

1 2 3 4 5 ...

12481 Commits (2af293ce60354d79906350109e14131a61332a1f) All Branches Search

12481 Commits (2af293ce60354d79906350109e14131a61332a1f)

All Branches