Tree: d7c30fdb2b

add-ec-vacuum

add_fasthttp_client

add_remote_storage

adding-message-queue-integration-tests

adjust-fsck-cutoff-default

also-delete-parent-directory-if-empty

avoid_releasing_temp_file_on_write

changing-to-zap

collect-public-metrics

copilot/fix-helm-chart-installation

copilot/fix-s3-object-tagging-issue

copilot/make-renew-interval-configurable

copilot/make-renew-interval-configurable-again

copilot/sub-pr-7677

create-table-snapshot-api-design

data_query_pushdown

dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5

dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0

detect-and-plan-ec-tasks

do-not-retry-if-error-is-NotFound

ec-disk-type-support

enhance-erasure-coding

fasthttp

feature/mini-port-detection

feature/modernize-s3-tests

filer1_maintenance_branch

fix-GetObjectLockConfigurationHandler

fix-bucket-name-case-7910

fix-mount-http-parallelism

fix-mount-read-throughput-7504

fix-pr-7909

fix-s3-object-tagging-issue-7589

fix-sts-session-token-7941

fix-versioning-listing-only

ftp

gh-pages

improve-fuse-mount

improve-fuse-mount2

logrus

master

message_send

mount2

mq-subscribe

mq2

nfs-cookie-prefix-list-fixes

optimize-delete-lookups

original_weed_mount

pr-7412

pr/7984

raft-dual-write

random_access_file

refactor-needle-read-operations

refactor-volume-write

remote_overlay

remove-implicit-directory-handling

revert-5134-patch-1

revert-5819-patch-1

revert-6434-bugfix-missing-s3-audit

s3-remote-cache-singleflight

s3-select

sub

tcp_read

test-reverting-lock-table

test_udp

testing

testing-sdx-generation

tikv

track-mount-e2e

upgrade-versions-to-4.00

volume_buffered_writes

worker-execute-ec-tasks

0.72

0.72.release

0.73

0.74

0.75

0.76

0.77

0.90

0.91

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1.00

1.01

1.02

1.03

1.04

1.05

1.06

1.07

1.08

1.09

1.10

1.11

1.12

1.14

1.15

1.16

1.17

1.18

1.19

1.20

1.21

1.22

1.23

1.24

1.25

1.26

1.27

1.28

1.29

1.30

1.31

1.32

1.33

1.34

1.35

1.36

1.37

1.38

1.40

1.41

1.42

1.43

1.44

1.45

1.46

1.47

1.48

1.49

1.50

1.51

1.52

1.53

1.54

1.55

1.56

1.57

1.58

1.59

1.60

1.61

1.61RC

1.62

1.63

1.64

1.65

1.66

1.67

1.68

1.69

1.70

1.71

1.72

1.73

1.74

1.75

1.76

1.77

1.78

1.79

1.80

1.81

1.82

1.83

1.84

1.85

1.86

1.87

1.88

1.90

1.91

1.92

1.93

1.94

1.95

1.96

1.97

1.98

1.99

1;70

2.00

2.01

2.02

2.03

2.04

2.05

2.06

2.07

2.08

2.09

2.10

2.11

2.12

2.13

2.14

2.15

2.16

2.17

2.18

2.19

2.20

2.21

2.22

2.23

2.24

2.25

2.26

2.27

2.28

2.29

2.30

2.31

2.32

2.33

2.34

2.35

2.36

2.37

2.38

2.39

2.40

2.41

2.42

2.43

2.47

2.48

2.49

2.50

2.51

2.52

2.53

2.54

2.55

2.56

2.57

2.58

2.59

2.60

2.61

2.62

2.63

2.64

2.65

2.66

2.67

2.68

2.69

2.70

2.71

2.72

2.73

2.74

2.75

2.76

2.77

2.78

2.79

2.80

2.81

2.82

2.83

2.84

2.85

2.86

2.87

2.88

2.89

2.90

2.91

2.92

2.93

2.94

2.95

2.96

2.97

2.98

2.99

3.00

3.01

3.02

3.03

3.04

3.05

3.06

3.07

3.08

3.09

3.10

3.11

3.12

3.13

3.14

3.15

3.16

3.18

3.19

3.20

3.21

3.22

3.23

3.24

3.25

3.26

3.27

3.28

3.29

3.30

3.31

3.32

3.33

3.34

3.35

3.36

3.37

3.38

3.39

3.40

3.41

3.42

3.43

3.44

3.45

3.46

3.47

3.48

3.50

3.51

3.52

3.53

3.54

3.55

3.56

3.57

3.58

3.59

3.60

3.61

3.62

3.63

3.64

3.65

3.66

3.67

3.68

3.69

3.71

3.72

3.73

3.74

3.75

3.76

3.77

3.78

3.79

3.80

3.81

3.82

3.83

3.84

3.85

3.86

3.87

3.88

3.89

3.90

3.91

3.92

3.93

3.94

3.95

3.96

3.97

3.98

3.99

4.00

4.01

4.02

4.03

4.04

4.05

4.06

dev

helm-3.65.1

v0.69

v0.70beta

v3.33

Branches Tags

${ item.name }

Create tag ${ searchTerm }

Create branch ${ searchTerm }

from 'd7c30fdb2b'

${ noResults }

Commit Graph

7977 Commits (d7c30fdb2bc41dbd1eba45bddc4b4424aa1a1bcb)

Author	SHA1	Message	Date
Chris Lu	d7c30fdb2b	fix: admin does not show all master servers #7999 (#8002)	2 days ago
Sheya Bernstein	844859de7f	fix: add filer fallback after consecutive connection failures (#8000)	2 days ago
Chris Lu	2b5e951390	use context.WithoutCancel to avoid context cancellation when the client connection is closed	2 days ago
Chris Lu	ce6e9be66b	4.06	3 days ago
Chris Lu	379c032868	Fix chown Input/output error on large file sets (#7996) * Fix chown Input/output error on large file sets (Fixes #7911) Implemented retry logic for MySQL/MariaDB backend to handle transient errors like deadlocks and timeouts. * Fix syntax error: missing closing brace * Refactor: Use %w for error wrapping and errors.As for extraction * Fix: Disable retry logic inside transactions	3 days ago
Chris Lu	ad76487e9d	Fix special characters in admin-generated secret keys (#7994) Fixes #7990 The issue was that the Charset constant used for generating secret keys included the '/' character, which is URL-unsafe. When secret keys containing '/' were used in HTTP requests, they would be URL-encoded, causing a mismatch during signature verification. Changes: - Removed '/' from the Charset constant in weed/iam/constants.go - Added TestGenerateSecretAccessKey_URLSafe to verify generated keys don't contain URL-unsafe characters like '/' or '+' This ensures all newly generated secret keys are URL-safe and will work correctly with S3 authentication. Existing keys continue to work.	4 days ago
Chris Lu	1ea6b0c0d9	cleanup: deduplicate environment variable credential loading Previously, `weed mini` logic duplicated the credential loading process by creating a temporary IAM config file from environment variables. `auth_credentials.go` also had fallback logic to load these variables. This change: 1. Updates `auth_credentials.go` to *always* check for and merge AWS environment variable credentials (`AWS_ACCESS_KEY_ID`, etc.) into the identity list. This ensures they are available regardless of whether other configurations (static file or filer) are loaded. 2. Removes the redundant file creation logic from `weed/command/mini.go`. 3. Updates `weed mini` user messages to accurately reflect that credentials are loaded from environment variables in-memory. This results in a cleaner implementation where `weed/s3api` manages all credential loading logic, and `weed mini` simply relies on it.	4 days ago
Chris Lu	7f1182472a	fix: enable dual loading of static and dynamic IAM configuration Refactored `NewIdentityAccessManagementWithStore` to remove mutual exclusivity between static (file-based) and dynamic (filer-based) configuration loading. Previously, if a static config configuration was present (including the legacy `IamConfig` option used by `weed mini`), it prevented loading users from the filer. Now, the system loads the static configuration first (if present), and then *always* attempts to merge in the dynamic configuration from the filer. This ensures that: 1. Static users (e.g. from `weed mini` env vars or `-s3.config`) are loaded and protected. 2. Dynamic users (e.g. created via Admin UI and stored in Filer) are also loaded and available.	4 days ago
Chris Lu	451b897d56	fix: support loading static config from IamConfig option for mini mode `weed mini` sets the `-s3.iam.config` flag instead of `-s3.config`, which populates `S3ApiServerOption.IamConfig`. Previously, `NewIdentityAccessManagementWithStore` only checked `option.Config`. This caused `weed mini` generated credentials (written to a temp file passed via IamConfig) to be ignored, breaking S3 access in mini mode even when environment variables were provided. This change ensures we try to load the configuration from `IamConfig` if `Config` is empty, restoring functionality for `weed mini`.	4 days ago
Chris Lu	48ded6b965	fix: allow environment variable fallback when filer config is empty Fixed regression where AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables were not being loaded as fallback credentials. The issue was that configLoaded was set to true when filer call succeeded, even if it returned an empty configuration. This blocked the environment variable fallback logic. Now only set configLoaded = true when we actually have loaded identities, allowing env vars to work correctly in mini mode and other scenarios where filer config is empty.	4 days ago
Chris Lu	4e835a1d81	fix(s3api): ensure S3 configuration persistence and refactor authorization tests (#7989) * fix(s3api): ensure static config file takes precedence over dynamic updates When a static S3 configuration file is provided, avoid overwriting the configuration from dynamic filer updates. This ensures the documented "Highest Priority" for the configuration file is respected. * refactor(s3api): implement merge-based static config with immutable identities Static identities from config file are now immutable and protected from dynamic updates. Dynamic identities (from admin panel) can be added and updated without affecting static entries. - Track identity names loaded from static config file - Implement merge logic that preserves static identities - Allow dynamic identities to be added or updated - Remove blanket block on config file updates * fix: address PR review comments for static config merge logic Critical Bugs: - Fix existingIdx always-false condition causing duplicate identities - Fix race condition in static config initialization (move useStaticConfig inside mutex) Security & Robustness: - Add nil identity check in VerifyActionPermission to fail closed - Mask access keys in STS validation logs to avoid exposing credentials - Add nil guard for s3a.iam in subscription handler Test Improvements: - Add authCalled tracking to MockIAMIntegration for explicit verification - Lower log level for static config messages to reduce noise * fix: prevent duplicates and race conditions in merge logic Data Integrity: - Prevent service account credential duplicates on repeated merges - Clean up stale accessKeyIdent entries when replacing identities - Check existing credentials before appending Concurrency Safety: - Add synchronization to IsStaticConfig method Test Improvements: - Add mux route vars for proper GetBucketAndObject extraction - Add STS session token header to trigger correct auth path	4 days ago
Chris Lu	6bf0c16862	fix admin copy text functions	4 days ago
Chris Lu	abfa64456b	Fix STS authorization in streaming/chunked uploads (#7988) * Fix STS authorization in streaming/chunked uploads During streaming/chunked uploads (SigV4 streaming), authorization happens twice: 1. Initial authorization in authRequestWithAuthType() - works correctly 2. Second authorization in verifyV4Signature() - was failing for STS The issue was that verifyV4Signature() only used identity.canDo() for permission checks, which always denies STS identities (they have empty Actions). This bypassed IAM authorization completely. This commit makes verifyV4Signature() IAM-aware by adding the same fallback logic used in authRequestWithAuthType(): - Traditional identities (with Actions) use legacy canDo() check - STS/JWT identities (empty Actions) fall back to IAM authorization Fixes: https://github.com/seaweedfs/seaweedfs/pull/7986#issuecomment-3723196038 * Add comprehensive unit tests for STS authorization in streaming uploads Created test suite to verify that verifyV4Signature properly handles STS identities by falling back to IAM authorization when shouldCheckPermissions is true. Tests cover: - STS identities with IAM integration (allow and deny cases) - STS identities without IAM integration (should deny) - Traditional identities with Actions (canDo check) - Permission check bypass when shouldCheckPermissions=false - Specific streaming upload scenario from bug report - Action determination based on HTTP method All tests pass successfully. * Refactor authorization logic to avoid duplication Centralized the authorization logic into IdentityAccessManagement.VerifyActionPermission. Updated auth_signature_v4.go and auth_credentials.go to use this new helper. Updated tests to clarify that they mirror the centralized logic. * Refactor tests to use VerifyActionPermission directly Introduced IAMIntegration interface to facilitate mocking of internal IAM integration logic. Updated IdentityAccessManagement to use the interface. Updated tests to directy call VerifyActionPermission using a mocked IAM integration, eliminating duplicated logic in tests. * fix(s3api): ensure static config file takes precedence and refactor tests - Track if configuration was loaded from a static file using `useStaticConfig`. - Ignore filer-based IAM updates when a static configuration is in use to respect "Highest Priority" rule. - Refactor `TestVerifyV4SignatureWithSTSIdentity` to use `VerifyActionPermission` directly. - Fix typed nil interface panic in authorization test.	4 days ago
Chris Lu	217d8b9e0e	Fix: ListObjectVersions delimiter support (#7987) * Fix: Add delimiter support to ListObjectVersions with proper truncation - Implemented delimiter support to group keys into CommonPrefixes - Fixed critical truncation bug: now merges versions and common prefixes into single sorted list before truncation - Ensures total items never exceed MaxKeys (prevents infinite pagination loops) - Properly sets NextKeyMarker and NextVersionIdMarker for pagination - Added integration tests in test/s3/versioning/s3_versioning_delimiter_test.go - Verified behavior matches S3 API specification * Fix: Add delimiter support to ListObjectVersions with proper truncation - Implemented delimiter support to group keys into CommonPrefixes - Fixed critical truncation bug: now merges versions and common prefixes before truncation - Added safety guard for maxKeys=0 to prevent panics - Condensed verbose comments for better readability - Added robust Go integration tests with nil checks for AWS SDK pointers - Verified behavior matches S3 API specification - Resolved compilation error in integration tests - Refined pagination comments and ensured exclusive KeyMarker behavior - Refactored listObjectVersions into helper methods for better maintainability	5 days ago
Chris Lu	4ba89bf73b	adjust log level	5 days ago
Chris Lu	bd237999bb	weed mini can optionally skip s3	5 days ago
Chris Lu	5a3aade445	less logs	5 days ago
promalert	9012069bd7	chore: execute goimports to format the code (#7983) * chore: execute goimports to format the code Signed-off-by: promalert <promalert@outlook.com> * goimports -w . --------- Signed-off-by: promalert <promalert@outlook.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	6 days ago
Chris Lu	6432019d08	Fix STS identity authorization by populating PolicyNames (#7985) (#7986) * Fix STS identity authorization by populating PolicyNames (#7985) This commit fixes GitHub issue #7985 where STS-assumed identities received empty identity.Actions, causing all S3 operations to be denied even when the role had valid IAM policies attached. Changes: 1. Populate PolicyNames field from sessionInfo.Policies in validateSTSSessionToken() to enable IAM-based authorization for STS identities 2. Fix bucket+objectKey path construction in canDo() method to include proper slash separator between bucket and object key 3. Add comprehensive test suite to validate the fix and prevent regression The fix ensures that STS-assumed identities are properly authorized through the IAM path when iamIntegration is available, allowing roles with valid IAM policies to perform S3 operations as expected. * Update STS identity tests to be more rigorous and use actual implementation path * Fix regression in canDo() path concatenation The previous fix blindly added a slash separator, which caused double slashes when objectKey already started with a slash (common in existing tests and some code paths). This broke TestCanDo and TestObjectLevelListPermissions. This commit updates the logic to only add the slash separator if objectKey is not empty and does not already start with a slash. This fixes the regressions while maintaining the fix for issue #7985. * Refactor STS identity tests: extract helpers and simplify redundant logic - Extracted setupTestSTSService and newTestIdentity helper functions - Removed redundant if-else verification blocks that were already covered by assertions - Cleaned up test cases to improve maintainability as suggested in code review. * Add canDo() verification to STS identity tests Address code review suggestion: verify that identities with empty Actions correctly return false for canDo() checks, which confirms the behavior that forces authorization to fall back to the IAM path. * Simplify TestCanDoPathConstruction variable names Rename expectedPath to fullPath and simplify logging/assertion logic based on code review feedback. * Refactor path construction and logging in canDo() - Compute fullPath early and use it for logging to prevent double slashes - Update TestCanDoPathConstruction to use robust path verification - Add test case for objectKey with leading slash to ensure correct handling	6 days ago
Chris Lu	e67973dc53	Support Policy Attachment for Object Store Users (#7981) * Implement Policy Attachment support for Object Store Users - Added policy_names field to iam.proto and regenerated protos. - Updated S3 API and IAM integration to support direct policy evaluation for users. - Enhanced Admin UI to allow attaching policies to users via modals. - Renamed 'policies' to 'policy_names' to clarify that it stores identifiers. - Fixed syntax error in user_management.go. * Fix policy dropdown not populating The API returns {policies: [...]} but JavaScript was treating response as direct array. Updated loadPolicies() to correctly access data.policies property. * Add null safety checks for policy dropdowns Added checks to prevent "undefined" errors when: - Policy select elements don't exist - Policy dropdowns haven't loaded yet - User is being edited before policies are loaded * Fix policy dropdown by using correct JSON field name JSON response has lowercase 'name' field but JavaScript was accessing 'Name'. Changed policy.Name to policy.name to match the IAMPolicy JSON structure. * Fix policy names not being saved on user update Changed condition from len(req.PolicyNames) > 0 to req.PolicyNames != nil to ensure policy names are always updated when present in the request, even if it's an empty array (to allow clearing policies). * Add debug logging for policy names update flow Added console.log in frontend and glog in backend to trace policy_names data through the update process. * Temporarily disable auto-reload for debugging Commented out window.location.reload() so console logs are visible when updating a user. * Add detailed debug logging and alert for policy selection Added console.log for each step and an alert to show policy_names value to help diagnose why it's not being included in the request. * Regenerate templ files for object_store_users Ran templ generate to ensure _templ.go files are up to date with the latest .templ changes including debug logging. * Remove debug logging and restore normal functionality Cleaned up temporary debug code (console.log and alert statements) and re-enabled automatic page reload after user update. * Add step-by-step alert debugging for policy update Added 5 alert checkpoints to trace policy data through the update flow: 1. Check if policiesSelect element exists 2. Show selected policy values 3. Show userData.policy_names 4. Show full request body 5. Confirm server response Temporarily disabled auto-reload to see alerts. * Add version check alert on page load Added alert on DOMContentLoaded to verify new JavaScript is being executed and not cached by the browser. * Compile templates using make Ran make to compile all template files and install the weed binary. * Add button click detection and make handleUpdateUser global - Added inline alert on button click to verify click is detected - Made handleUpdateUser a window-level function to ensure it's accessible - Added alert at start of handleUpdateUser function * Fix handleUpdateUser scope issue - remove duplicate definition Removed duplicate function definition that was inside DOMContentLoaded. Now handleUpdateUser is defined only once in global scope (line 383) making it accessible when button onclick fires. * Remove all duplicate handleUpdateUser definitions Now handleUpdateUser is defined only once at the very top of the script block (line 352), before DOMContentLoaded, ensuring it's available when the button onclick fires. * Add function existence check and error catching Added alerts to check if handleUpdateUser is defined and wrapped the function call in try-catch to capture any JavaScript errors. Also added console.log statements to verify function definition. * Simplify handleUpdateUser to non-async for testing Removed async/await and added early return to test if function can be called at all. This will help identify if async is causing the issue. * Add cache-control headers to prevent browser caching Added no-cache headers to ShowObjectStoreUsers handler to prevent aggressive browser caching of inline JavaScript in the HTML page. * Fix syntax error - make handleUpdateUser async Changed function back to async to fix 'await is only valid in async functions' error. The cache-control headers are working - browser is now loading new code. * Update version check to v3 to verify cache busting Changed version alert to 'v3 - WITH EARLY RETURN' to confirm the new code with early return statement is being loaded. * Remove all debug code - clean implementation Removed all alerts, console.logs, and test code. Implemented clean policy update functionality with proper error handling. * Add ETag header for cache-busting and update walkthrough * Fix policy pre-selection in Edit User modal - Updated admin.js editUser function to pre-select policies - Root cause: duplicate editUser in admin.js overwrote inline version - Added policy pre-selection logic to match inline template - Verified working in browser: policies now pre-select correctly * Fix policy persistence in handleUpdateUser - Added policy_names field to userData payload in handleUpdateUser - Policies were being lost because handleUpdateUser only sent email and actions - Now collects selected policies from editPolicies dropdown - Verified working: policies persist correctly across updates * Fix XSS vulnerability in access keys display - Escape HTML in access key display using escapeHtml utility - Replace inline onclick handlers with data attributes - Add event delegation for delete access key buttons - Prevents script injection via malicious access key values * Fix additional XSS vulnerabilities in user details display - Escape HTML in actions badges (line 626) - Escape HTML in policy_names badges (line 636) - Prevents script injection via malicious action or policy names * Fix XSS vulnerability in loadPolicies function - Replace innerHTML string concatenation with DOM API - Use createElement and textContent for safe policy name insertion - Prevents script injection via malicious policy names - Apply same pattern to both create and edit select elements * Remove debug logging from UpdateObjectStoreUser - Removed glog.V(0) debug statements - Clean up temporary debugging code before production * Remove duplicate handleUpdateUser function - Removed inline handleUpdateUser that duplicated admin.js logic - Removed debug console.log statement - admin.js version is now the single source of truth - Eliminates maintenance burden of keeping two versions in sync * Refine user management and address code review feedback - Preserve PolicyNames in UpdateUserPolicies - Allow clearing actions in UpdateObjectStoreUser by checking for nil - Remove version comment from object_store_users.templ - Refactor loadPolicies for DRYness using cloneNode while keeping DOM API security * IAM Authorization for Static Access Keys * verified XSS Fixes in Templates * fix div	6 days ago
Chris Lu	d4ecfaeda7	Enable writeback_cache and async_dio FUSE options (#7980) * Enable writeback_cache and async_dio FUSE options Fixes #7978 - Update mount_std.go to use EnableWriteback and EnableAsyncDio from go-fuse - Add go.mod replace directive to use local go-fuse with capability support - Remove temporary workaround that disabled these options This enables proper FUSE kernel capability negotiation for writeback cache and async direct I/O, improving performance for small writes and concurrent direct I/O operations. * Address PR review comments - Remove redundant nil checks for writebackCache and asyncDio flags - Update go.mod replace directive to use seaweedfs/go-fuse fork instead of local path * Add TODO comment for go.mod replace directive The replace directive must use a local path until seaweedfs/go-fuse#1 is merged. After merge, this should be updated to use the proper version. * Use seaweedfs/go-fuse v2.9.0 instead of local repository Replace local path with seaweedfs/go-fuse v2.9.0 fork which includes the writeback_cache and async_dio capability support. * Use github.com/seaweedfs/go-fuse/v2 directly without replace directive - Updated all imports to use github.com/seaweedfs/go-fuse/v2 - Removed replace directive from go.mod - Using seaweedfs/go-fuse v2.0.0-20260106181308-87f90219ce09 which includes: * writeback_cache and async_dio support * Corrected module path * Update to seaweedfs/go-fuse v2.9.1 Use v2.9.1 tag which includes the corrected module path (github.com/seaweedfs/go-fuse/v2) along with writeback_cache and async_dio support.	7 days ago
Chris Lu	e10f11b480	opt: reduce ShardsInfo memory usage with bitmap and sorted slice (#7974) * opt: reduce ShardsInfo memory usage with bitmap and sorted slice - Replace map[ShardId]*ShardInfo with sorted []ShardInfo slice - Add ShardBits (uint32) bitmap for O(1) existence checks - Use binary search for O(log n) lookups by shard ID - Maintain sorted order for efficient iteration - Add comprehensive unit tests and benchmarks Memory savings: - Map overhead: ~48 bytes per entry eliminated - Pointers: 8 bytes per entry eliminated - Total: ~56 bytes per shard saved Performance improvements: - Has(): O(1) using bitmap - Size(): O(log n) using binary search (was O(1), acceptable tradeoff) - Count(): O(1) using popcount on bitmap - Iteration: Faster due to cache locality * refactor: add methods to ShardBits type - Add Has(), Set(), Clear(), and Count() methods to ShardBits - Simplify ShardsInfo methods by using ShardBits methods - Improves code readability and encapsulation * opt: use ShardBits directly in ShardsCountFromVolumeEcShardInformationMessage Avoid creating a full ShardsInfo object just to count shards. Directly cast vi.EcIndexBits to ShardBits and use Count() method. * opt: use strings.Builder in ShardsInfo.String() for efficiency * refactor: change AsSlice to return []ShardInfo (values instead of pointers) This completes the memory optimization by avoiding unnecessary pointer slices and potential allocations. * refactor: rename ShardsCountFromVolumeEcShardInformationMessage to GetShardCount * fix: prevent deadlock in Add and Subtract methods Copy shards data from 'other' before releasing its lock to avoid potential deadlock when a.Add(b) and b.Add(a) are called concurrently. The previous implementation held other's lock while calling si.Set/Delete, which acquires si's lock. This could deadlock if two goroutines tried to add/subtract each other concurrently. * opt: avoid unnecessary locking in constructor functions ShardsInfoFromVolume and ShardsInfoFromVolumeEcShardInformationMessage now build shards slice and bitmap directly without calling Set(), which acquires a lock on every call. Since the object is local and not yet shared, locking is unnecessary and adds overhead. This improves performance during object construction. * fix: rename 'copy' variable to avoid shadowing built-in function The variable name 'copy' in TestShardsInfo_Copy shadowed the built-in copy() function, which is confusing and bad practice. Renamed to 'siCopy'. * opt: use math/bits.OnesCount32 and reorganize types 1. Replace manual popcount loop with math/bits.OnesCount32 for better performance and idiomatic Go code 2. Move ShardSize type definition to ec_shards_info.go for better code organization since it's primarily used there * refactor: Set() now accepts ShardInfo for future extensibility Changed Set(id ShardId, size ShardSize) to Set(shard ShardInfo) to support future additions to ShardInfo without changing the API. This makes the code more extensible as new fields can be added to ShardInfo (e.g., checksum, location, etc.) without breaking the Set API. * refactor: move ShardInfo and ShardSize to separate file Created ec_shard_info.go to hold the basic shard types (ShardInfo and ShardSize) for better code organization and separation of concerns. * refactor: add ShardInfo constructor and helper functions Added NewShardInfo() constructor and IsValid() method to better encapsulate ShardInfo creation and validation. Updated code to use the constructor for cleaner, more maintainable code. * fix: update remaining Set() calls to use NewShardInfo constructor Fixed compilation errors in storage and shell packages where Set() calls were not updated to use the new NewShardInfo() constructor. * fix: remove unreachable code in filer backup commands Removed unreachable return statements after infinite loops in filer_backup.go and filer_meta_backup.go to fix compilation errors. * fix: rename 'new' variable to avoid shadowing built-in Renamed 'new' to 'result' in MinusParityShards, Plus, and Minus methods to avoid shadowing Go's built-in new() function. * fix: update remaining test files to use NewShardInfo constructor Fixed Set() calls in command_volume_list_test.go and ec_rebalance_slots_test.go to use NewShardInfo() constructor.	1 week ago
Chris Lu	ec1c27a4b3	storage/needle: add bounds check for WriteNeedleBlob buffer (#7973) * storage/needle: add bounds check for WriteNeedleBlob buffer * storage/needle: use int offsets when checking/writing Version3 timestamp * Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>	1 week ago
Chris Lu	d75162370c	Fix trust policy wildcard principal handling (#7970) * Fix trust policy wildcard principal handling This change fixes the trust policy validation to properly support AWS-standard wildcard principals like {"Federated": "*"}. Previously, the evaluatePrincipalValue() function would check for context existence before evaluating wildcards, causing wildcard principals to fail when the context key didn't exist. This forced users to use the plain "*" workaround instead of the more specific {"Federated": "*"} format. Changes: - Modified evaluatePrincipalValue() to check for "*" FIRST before validating against context - Added support for wildcards in principal arrays - Added comprehensive tests for wildcard principal handling - All existing tests continue to pass (no regressions) This matches AWS IAM behavior where "*" in a principal field means "allow any value" without requiring context validation. Fixes: https://github.com/seaweedfs/seaweedfs/issues/7917 * Refactor: Move Principal matching to PolicyEngine This refactoring consolidates all policy evaluation logic into the PolicyEngine, improving code organization and eliminating duplication. Changes: - Added matchesPrincipal() and evaluatePrincipalValue() to PolicyEngine - Added EvaluateTrustPolicy() method for direct trust policy evaluation - Updated statementMatches() to check Principal field when present - Made resource matching optional (trust policies don't have Resources) - Simplified evaluateTrustPolicy() in iam_manager.go to delegate to PolicyEngine - Removed ~170 lines of duplicate code from iam_manager.go Benefits: - Single source of truth for all policy evaluation - Better code reusability and maintainability - Consistent evaluation rules for all policy types - Easier to test and debug All tests pass with no regressions. * Make PolicyEngine AWS-compatible and add unit tests Changes: 1. AWS-Compatible Context Keys: - Changed "seaweed:FederatedProvider" -> "aws:FederatedProvider" - Changed "seaweed:AWSPrincipal" -> "aws:PrincipalArn" - Changed "seaweed:ServicePrincipal" -> "aws:PrincipalServiceName" - This ensures 100% AWS compatibility for trust policies 2. Added Comprehensive Unit Tests: - TestPrincipalMatching: 8 test cases for Principal matching - TestEvaluatePrincipalValue: 7 test cases for value evaluation - TestTrustPolicyEvaluation: 6 test cases for trust policy evaluation - TestGetPrincipalContextKey: 4 test cases for context key mapping - Total: 25 new unit tests for PolicyEngine All tests pass: - Policy engine tests: 54 passed - Integration tests: 9 passed - Total: 63 tests passing * Update context keys to standard AWS/OIDC formats Replaced remaining seaweed: context keys with standard AWS and OIDC keys to ensure 100% compatibility with AWS IAM policies. Mappings: - seaweed:TokenIssuer -> oidc:iss - seaweed:Issuer -> oidc:iss - seaweed:Subject -> oidc:sub - seaweed:SourceIP -> aws:SourceIp Also updated unit tests to reflect these changes. All 63 tests pass successfully. * Add advanced policy tests for variable substitution and conditions Added comprehensive tests inspired by AWS IAM patterns: - TestPolicyVariableSubstitution: Tests ${oidc:sub} variable in resources - TestConditionWithNumericComparison: Tests sts:DurationSeconds condition - TestMultipleConditionOperators: Tests combining StringEquals and StringLike Results: - TestMultipleConditionOperators: ✅ All 3 subtests pass - Other tests reveal need for sts:DurationSeconds context population These tests validate the PolicyEngine's ability to handle complex AWS-compatible policy scenarios. * Fix federated provider context and add DurationSeconds support Changes: - Use iss claim as aws:FederatedProvider (AWS standard) - Add sts:DurationSeconds to trust policy evaluation context - TestPolicyVariableSubstitution now passes ✅ Remaining work: - TestConditionWithNumericComparison partially works (1/3 pass) - Need to investigate NumericLessThanEquals evaluation * Update trust policies to use issuer URL for AWS compatibility Changed trust policy from using provider name ("test-oidc") to using the issuer URL ("https://test-issuer.com") to match AWS standard behavior where aws:FederatedProvider contains the OIDC issuer URL. Test Results: - 10/12 test suites passing - TestFullOIDCWorkflow: ✅ All subtests pass - TestPolicyEnforcement: ✅ All subtests pass - TestSessionExpiration: ✅ Pass - TestPolicyVariableSubstitution: ✅ Pass - TestMultipleConditionOperators: ✅ All subtests pass Remaining work: - TestConditionWithNumericComparison needs investigation - One subtest in TestTrustPolicyValidation needs fix * Fix S3 API tests for AWS compatibility Updated all S3 API tests to use AWS-compatible context keys and trust policy principals: Changes: - seaweed:SourceIP → aws:SourceIp (IP-based conditions) - Federated: "test-oidc" → "https://test-issuer.com" (trust policies) Test Results: - TestS3EndToEndWithJWT: ✅ All 13 subtests pass - TestIPBasedPolicyEnforcement: ✅ All 3 subtests pass This ensures policies are 100% AWS-compatible and portable. * Fix ValidateTrustPolicy for AWS compatibility Updated ValidateTrustPolicy method to check for: - OIDC: issuer URL ("https://test-issuer.com") - LDAP: provider name ("test-ldap") - Wildcard: "*" Test Results: - TestTrustPolicyValidation: ✅ All 3 subtests pass This ensures trust policy validation uses the same AWS-compatible principals as the PolicyEngine. * Fix multipart and presigned URL tests for AWS compatibility Updated trust policies in: - s3_multipart_iam_test.go - s3_presigned_url_iam_test.go Changed "Federated": "test-oidc" → "https://test-issuer.com" Test Results: - TestMultipartIAMValidation: ✅ All 7 subtests pass - TestPresignedURLIAMValidation: ✅ All 4 subtests pass - TestPresignedURLGeneration: ✅ All 4 subtests pass - TestPresignedURLExpiration: ✅ All 4 subtests pass - TestPresignedURLSecurityPolicy: ✅ All 4 subtests pass All S3 API tests now use AWS-compatible trust policies. * Fix numeric condition evaluation and trust policy validation interface Major updates to ensure robust AWS-compatible policy evaluation: 1. **Policy Engine**: Added support for `int` and `int64` types in `evaluateNumericCondition`, fixing issues where raw numbers in policy documents caused evaluation failures. 2. **Trust Policy Validation**: Updated `TrustPolicyValidator` interface and `STSService` to propagate `DurationSeconds` correctly during the double-validation flow (Validation -> STS -> Validation callback). 3. **IAM Manager**: Updated implementation to match the new interface and correctly pass `sts:DurationSeconds` context key. Test Results: - TestConditionWithNumericComparison: ✅ All 3 subtests pass - All IAM and S3 integration tests pass (100%) This resolves the final edge case with DurationSeconds numeric conditions. * Fix MockTrustPolicyValidator interface and unreachable code warnings Updates: 1. Updated MockTrustPolicyValidator.ValidateTrustPolicyForWebIdentity to match new interface signature with durationSeconds parameter 2. Removed unreachable code after infinite loops in filer_backup.go and filer_meta_backup.go to satisfy linter Test Results: - All STS tests pass ✅ - Build warnings resolved ✅ * Refactor matchesPrincipal to consolidate array handling logic Consolidated duplicated logic for []interface{} and []string types by converting them to a unified []interface{} upfront. * Fix malformed AWS docs URL in iam_manager.go comment * dup * Enhance IAM integration tests with negative cases and interface array support Added test cases to TestTrustPolicyWildcardPrincipal to: 1. Verify rejection of roles when principal context does not match (negative test) 2. Verify support for principal arrays as []interface{} (simulating JSON unmarshaled roles) * Fix syntax errors in filer_backup and filer_meta_backup Restored missing closing braces for for-loops and re-added return statements. The previous attempt to remove unreachable code accidentally broke the function structure. Build now passes successfully.	1 week ago
Chris Lu	d15f32ae46	feat: add flags to disable WebDAV and Admin UI in weed mini (#7971) * feat: add flags to disable WebDAV and Admin UI in weed mini - Add -webdav flag (default: true) to optionally disable WebDAV server - Add -admin.ui flag (default: true) to optionally disable Admin UI only (server still runs) - Conditionally skip WebDAV service startup based on flag - Pass disableUI flag to SetupRoutes to skip UI route registration - Admin server still runs for gRPC and API access when UI is disabled Addresses issue from https://github.com/seaweedfs/seaweedfs/pull/7833#issuecomment-3711924150 * refactor: use positive enableUI parameter instead of disableUI across admin server and handlers * docs: update mini welcome message to list enabled components * chore: remove unused welcomeMessageTemplate constant * docs: split S3 credential message into separate sb.WriteString calls	1 week ago
Chris Lu	383c2e3b41	fix: handle range requests on empty objects (size=0) (#7963) * fix: handle range requests on empty objects (size=0) Range requests on empty objects were incorrectly being rejected with: 'invalid range start for ...: 0 >= 0' The validation logic used 'startOffset >= totalSize' which failed when both were 0, incorrectly rejecting valid range requests like bytes=0-1535 on 0-byte files. Fix: Added special case handling before validation to properly return 416 Range Not Satisfiable for any range request on an empty object, per RFC 7233. Fixed at two locations (lines 873 and 1154) in s3api_object_handlers.go * refactor: return 404 for directory objects, not 416 Per S3 semantics, GET requests on directory paths (without trailing "/") should return 404 Not Found, not try to serve them as objects. Updated fix to: 1. Check if entry.IsDirectory and return 404 (S3-compliant) 2. Only return 416 for true empty files (size=0, not directory) This matches AWS S3 behavior where directories don't exist as objects unless they're explicit directory markers ending with "/". * reduce repeated info * refactor: move directory check before range branching This ensures that any Range header (including suffix ranges like bytes=-N) on a directory path (without trailing slash) returns 404 (ErrNoSuchKey) instead of potentially returning 416 or attempting to serve as an object. Applied to both streamFromVolumeServers and streamFromVolumeServersWithSSE. * refactoring	1 week ago
Chris Lu	0e9f433ec4	refactoring	1 week ago
Alasdair Macmillan	9778b9589e	Fix unaligned 64-bit atomic operation on ARM32 (#7958) (#7959)	1 week ago
Chris Lu	8269dc136d	simplify	1 week ago
Robert Schade	de3df211d7	store S3 storage class in extended atrributes #7961 (#7962) * store S3 storage class in extended atrributes #7961 * canonical * remove issue reference --------- Co-authored-by: Robert Schade <robert.schade@uni-paderborn.de> Co-authored-by: Chris Lu <chris.lu@gmail.com>	1 week ago
LeeXN	c909724bf1	Fix: prevent panic when swap file creation fails (#7957) * Fix: prevent panic when swap file creation fails * weed mount: fix race condition in swap file initialization Ensure thread-safe access to sf.file and other state in NewSwapFileChunk and FreeResource by using sf.chunkTrackingLock consistently. Also set sf.file to nil after closing to prevent reuse. * weed mount: improve swap directory creation logic - Check error for os.MkdirAll and log it if it fails. - Use 0700 permissions for the swap directory for better security. - Improve error logging context. * weed mount: add unit tests for swap file creation Add tests to verify: - Concurrent initialization of the swap file. - Correct directory permissions (0700). - Automatic directory recreation if deleted. * weed mount: fix thread-safety in swap file unit tests Use atomic.Uint32 to track failures within goroutines in TestSwapFile_NewSwapFileChunk_Concurrent to avoid unsafe calls to t.Errorf from multiple goroutines. * weed mount: simplify swap file creation logic Refactor the directory check and retry logic for better readability and to avoid re-using the main error variable for directory creation errors. Remove redundant error logging. * weed mount: improve error checking in swap file tests Explicitly check if NewSwapFileChunk returns nil to provide more informative failures. * weed mount: update DirtyPages interface to return error Propagate errors from SaveDataAt when swap file creation fails. This prevents potential panics in the write path. * weed mount: handle AddPage errors in write paths Update ChunkedDirtyPages and PageWriter to propagate errors and update WFS.Write and WFS.CopyFileRange to return fuse.EIO on failure. * weed mount: update swap directory creation error message Change "recreate" to "create/recreate" to better reflect that this path is also taken during the initial creation of the swap directory. --------- Co-authored-by: lixiang58 <lixiang58@lenovo.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	1 week ago
Chris Lu	0b5a65e00b	avoid extra missing shard warning fix https://github.com/seaweedfs/seaweedfs/issues/7956	1 week ago
Chris Lu	63b2fe0d76	fix: EC UI template error when viewing shard details (#7955) * fix: EC UI template error when viewing shard details Fixed field name mismatch in volume.html where it was using .ShardDetails instead of .Shards. Added a robust type conversion wrapper in templates.go to handle int64 to uint64 conversion for bytesToHumanReadable. Added regression test to ensure future stability. * refactor: improve bytesToHumanReadable and test robustness - Handled more integer types (uint32, int32, uint) in bytesToHumanReadable. - Improved volume_test.go to verify both shards are formatted correctly. * refactor: add bounds checking to bytesToHumanReadable Added checks for negative values in signed integer types to avoid incorrect formatting when converting to uint64. Addressed feedback from coderabbitai.	1 week ago
Chris Lu	0647bc24d5	s3api: fix authentication bypass and potential SIGSEGV (Issue #7912) (#7954) * s3api: fix authentication bypass and potential SIGSEGV * s3api: improve security tests with positive cases and nil identity guards * s3api: fix secondary authentication bypass in AuthSignatureOnly * s3api: refactor account loading and refine security tests based on review feedback * s3api: refine security tests with realistic signature failures * Update weed/s3api/auth_security_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	1 week ago
Chris Lu	54de32f207	Support AWS standard IAM role ARN formats (issue #7946) (#7948) * fix(iam): support both AWS standard and legacy IAM role ARN formats Fix issue #7946 where SeaweedFS only recognized legacy IAM role ARN format (arn:aws:iam::role/RoleName) but not the standard AWS format with account ID (arn:aws:iam::ACCOUNT:role/RoleName). This was breaking EKS pod identity integration which expects the standard format. Changes: - Update ExtractRoleNameFromArn() to handle both formats by searching for 'role/' marker instead of matching a fixed prefix - Update ExtractRoleNameFromPrincipal() to clearly document both STS and IAM formats it supports with or without account ID - Simplify role ARN validation in validateRoleAssumptionForWebIdentity() and validateRoleAssumptionForCredentials() to use the extraction function - Add comprehensive test coverage with 25 test cases covering both formats The fix maintains backward compatibility with legacy format while adding support for standard AWS format with account ID. Fixes: https://github.com/seaweedfs/seaweedfs/issues/7946 * docs: improve docstring coverage for ARN utility functions - Add comprehensive package-level documentation - Enhance ExtractRoleNameFromPrincipal docstring with parameter and return descriptions - Enhance ExtractRoleNameFromArn docstring with detailed format documentation - Add docstrings to test functions explaining test coverage - Update all docstrings to 80%+ coverage for code review compliance * refactor: improve ARN parsing code maintainability and error messages - Define constants for ARN prefixes and markers (stsPrefix, stsAssumedRoleMarker, iamPrefix, iamRoleMarker) - Replace hardcoded magic strings with named constants in ExtractRoleNameFromPrincipal and ExtractRoleNameFromArn - Enhance error messages in sts_service.go to show expected ARN format when validation fails - Error message now shows: 'arn:aws:iam::[ACCOUNT_ID:]role/ROLE_NAME' format - Improves code readability and maintainability - Facilitates future ARN format changes and debugging * feat: add structured ARN type for better debugging and extensibility Implements Option 2 (Structured ARN Type) from ARN handling comparison: New Features: - ARNInfo struct with Original, RoleName, AccountID, and Format fields - ARNFormat enum (Legacy, Standard, Invalid) for type-safe format tracking - ParseRoleARN() function for structured IAM role ARN parsing - ParsePrincipalARN() function for structured STS/IAM principal parsing Benefits: - Better debugging: Can see original ARN, extracted components, and format type - Extensible: Easy to add more fields (Region, Service, etc.) in future - Type-safe: Format is an enum, not a string - Backward compatible: Kept original string-based functions STS Service Updates: - Uses ParseRoleARN() for structured validation - Logs ARN components at V(4) level for debugging (role, account, format) - Better error context when validation fails Test Coverage: - 7 new tests for ParseRoleARN (legacy, standard, invalid formats) - 7 new tests for ParsePrincipalARN (STS/IAM, legacy/standard) - All 39 existing tests still pass - Total: 53 ARN-related tests Comparison with MinIO: - More flexible: Supports both AWS formats (MinIO only supports MinIO format) - Better tested: 53 tests vs MinIO's 8 tests - Structured like MinIO but more practical for AWS use cases * security: fix ARN parsing to prevent malicious ARN acceptance Fix critical security vulnerability where malicious ARNs could bypass validation: - ARNs like 'arn:aws:iam::123456789012:user/role/malicious' were incorrectly accepted - The previous implementation used strings.Index to find 'role/' anywhere in the ARN - This allowed non-role resource types to be accepted if they contained 'role/' in their path Changes: 1. Updated ExtractRoleNameFromArn() to validate resource type is exactly 'role/' 2. Updated ExtractRoleNameFromPrincipal() to validate resource type is exactly 'assumed-role/' 3. Updated ParseRoleARN() to validate structure before extracting fields 4. Updated ParsePrincipalARN() to validate structure before extracting fields 5. Added 6 security test cases to prevent regression The fix validates ARN structure by: - Splitting on ':' to separate account ID from resource type - Verifying resource type starts with exact marker ('role/' or 'assumed-role/') - Only then extracting role name, account ID, and format All 59 tests pass, including new security tests that verify malicious ARNs are rejected. Fixes: GitHub Copilot review #3624499048 * test: add test cases for empty role names and improve validation Address review feedback to improve edge case coverage: 1. Added test case for standard format with empty role name - TestExtractRoleNameFromArn: arn:aws:iam::123456789012:role/ - TestParseRoleARN: arn:aws:iam::123456789012:role/ 2. Added empty role name validation for STS ARNs in ParsePrincipalARN - Now matches ParseRoleARN behavior - Prevents ARNs like arn:aws:sts::assumed-role/ from having valid Format 3. Added test cases for empty STS role names - TestParsePrincipalARN: arn:aws:sts::assumed-role/ - TestParsePrincipalARN: arn:aws:sts::123456789012:assumed-role/ All 65 tests pass (15 for ExtractRoleNameFromArn, 10 for ExtractRoleNameFromPrincipal, 8 for ParseRoleARN, 9 for ParsePrincipalARN, 4 security user ARNs, 2 security STS, plus existing tests). * refactor: simplify ARNInfo by removing Format enum Remove ARNFormat enum (ARNFormatLegacy, ARNFormatStandard, ARNFormatInvalid) as it's not needed for backward compatibility. Simplifications: 1. Removed ARNFormat type and all format constants 2. Removed Format field from ARNInfo struct 3. Validation now checks if RoleName is empty (simpler and clearer) 4. AccountID presence already distinguishes legacy (empty) from standard (non-empty) formats 5. Updated STS service to check RoleName emptiness instead of Format field 6. Improved debug logging to explicitly show "(legacy format)" or "(standard format)" Benefits: - Simpler code with fewer concepts - AccountID field already provides format information - Validation is clearer: empty RoleName = invalid ARN - All 65 tests still pass This change maintains the same functionality while reducing code complexity. No backward compatibility concerns as the structured ARN parsing is new. * test: add comprehensive edge case tests for ARN parsing Add 4 new test functions covering: - Multiple role markers in paths (e.g., role/role/name) - Consecutive slashes in role paths (preserved as valid components) - Special characters valid in AWS role names (+=,.@-_) - Extremely long role names near AWS limits These tests verify the parser's resilience to edge cases and ensure proper handling of various valid role name formats and special characters.	1 week ago
Taylor Jasko	6a9860098f	fix: correcting S3 nil cipher dereference in filer init (#7952) Resolves the following error reported in #7949: ``` I0103 21:38:30.230662 s3.go:275 Starting S3 API Server with standard IAM panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x38ca961] goroutine 102 [running]: github.com/seaweedfs/seaweedfs/weed/command.(*S3Options).startS3Server(0x7caf840) /go/src/github.com/seaweedfs/seaweedfs/weed/command/s3.go:295 +0x741 github.com/seaweedfs/seaweedfs/weed/command.runFiler.func1(...) /go/src/github.com/seaweedfs/seaweedfs/weed/command/filer.go:244 created by github.com/seaweedfs/seaweedfs/weed/command.runFiler in goroutine 1 /go/src/github.com/seaweedfs/seaweedfs/weed/command/filer.go:242 +0x353 ```	1 week ago
Chris Lu	25975bacfb	fix(gcs): resolve credential conflict and improve backup logging (#7951) * fix(gcs): resolve credential conflict and improve backup logging - Workaround GCS SDK's "multiple credential options" error by manually constructing an authenticated HTTP client. - Include source entry path in filer backup error logs for better visibility on missing volumes/404s. * fix: address PR review feedback - Add nil check for EventNotification in getSourceKey - Avoid reassigning google_application_credentials parameter in gcs_sink.go * fix(gcs): return errors instead of calling glog.Fatalf in initialize Adheres to Go best practices and allows for more graceful failure handling by callers. * read from bind ip	1 week ago
Chris Lu	23fc3f2621	Fix AWS SDK Signature V4 with STS credentials (issue #7941) (#7944) * Add documentation for issue #7941 fix * ensure auth * rm FIX_ISSUE_7941.md * Integrate STS session token validation into V4 signature verification - Check for X-Amz-Security-Token header in verifyV4Signature - Call validateSTSSessionToken for STS requests - Skip regular access key lookup and expiration check for STS sessions * Fix variable scoping in verifyV4Signature for STS session token validation * Add ErrExpiredToken error for better AWS S3 compatibility with STS session tokens * Support STS session token in query parameters for presigned URLs * Fix nil pointer dereference in validateSTSSessionToken * Enhance STS token validation with detailed error diagnostics and logging * Fix missing credentials in STSSessionClaims.ToSessionInfo() * test: Add comprehensive STS session claims validation tests - TestSTSSessionClaimsToSessionInfo: Validates basic claims conversion - TestSTSSessionClaimsToSessionInfoCredentialGeneration: Verifies credential generation - TestSTSSessionClaimsToSessionInfoPreservesAllFields: Ensures all fields are preserved - TestSTSSessionClaimsToSessionInfoEmptyFields: Tests handling of empty/nil fields - TestSTSSessionClaimsToSessionInfoCredentialExpiration: Validates expiration handling All tests pass with proper timing tolerance for credential generation. * perf: Reuse CredentialGenerator instance for STS session claims Optimize ToSessionInfo() to reuse a package-level defaultCredentialGenerator instead of allocating a new CredentialGenerator on every call. This reduces allocation overhead since this method is called frequently during signature verification (potentially once per request). The CredentialGenerator is stateless and deterministic, making it safe to reuse across concurrent calls without synchronization. * refactor: Surface credential generation errors and remove sensitive logging Two improvements to error handling and security: 1. weed/iam/sts/session_claims.go: - Add logging for credential generation failures in ToSessionInfo() - Wrap errors with context (session ID) to aid debugging - Use glog.Warningf() to surface errors instead of silently swallowing them - Add fmt import for error wrapping 2. weed/s3api/auth_signature_v4.go: - Remove debug logging of actual access key IDs (glog.V(2) call) - Security improvement: avoid exposing sensitive access keys even at debug level - Keep warning-level logging that shows only count of available keys This ensures credential generation failures are observable while protecting sensitive authentication material from logs. * test: Verify deterministic credential generation in session claims tests Update TestSTSSessionClaimsToSessionInfoCredentialGeneration to properly verify deterministic credential generation: - Remove misleading comment about 'randomness' - parts of credentials ARE deterministic - Add assertions that AccessKeyId is identical for same SessionId (hash-based, deterministic) - Add assertions that SessionToken is identical for same SessionId (hash-based, deterministic) - Verify Expiration matches when SessionId is identical - Document that SecretAccessKey is NOT deterministic (uses random.Read) - Truncate expiresAt to second precision to avoid timing issues This test now properly verifies that the deterministic components of credential generation work correctly while acknowledging the cryptographic randomness of the secret access key. * test(sts): Assert credentials expiration relative to now in credential expiration tests Replace wallclock assertions comparing tc.expiresAt to time.Now() (which only verified test setup) with assertions that check sessionInfo.Credentials.Expiration relative to time.Now(), thus exercising the code under test. Include clarifying comment for intent. * feat(sts): Add IsExpired helpers and use them in expiration tests - Add Credentials.IsExpired() and SessionInfo.IsExpired() in new file session_helpers.go. - Update TestSTSSessionClaimsToSessionInfoCredentialExpiration to use helpers for clearer intent. * test: revert test-only IsExpired helpers; restore direct expiration assertions Remove session_helpers.go and update TestSTSSessionClaimsToSessionInfoCredentialExpiration to assert against sessionInfo.Credentials.Expiration directly as requested by reviewer., * fix(s3api): restore error return when access key not found Critical fix: The previous cleanup of sensitive logging inadvertently removed the error return statement when access key lookup fails. This caused the code to continue and call isCredentialExpired() on nil pointer, crashing the server. This explains EOF errors in CORS tests - server was panicking on requests with invalid keys. * fix(sts): make secret access key deterministic based on sessionId CRITICAL FIX: The secret access key was being randomly generated, causing signature verification failures when the same session token was used twice: 1. AssumeRoleWithWebIdentity generates random secret key X 2. Client signs request using secret key X 3. Server validates token, regenerates credentials via ToSessionInfo() 4. ToSessionInfo() calls generateSecretAccessKey(), which generates random key Y 5. Server tries to verify signature using key Y, but signature was made with X 6. Signature verification fails (SignatureDoesNotMatch) Solution: Make generateSecretAccessKey() deterministic by using SHA256 hash of 'secret-key:' + sessionId, just like generateAccessKeyId() already does. This ensures: - AssumeRoleWithWebIdentity generates deterministic secret key from sessionId - ToSessionInfo() regenerates the same secret key from the same sessionId - Client signature verification succeeds because keys match Fixes: AWS SDK v2 CORS tests failing with 'ExpiredToken' errors Affected files: - weed/iam/sts/token_utils.go: Updated generateSecretAccessKey() signature and implementation to be deterministic - Updated GenerateTemporaryCredentials() to pass sessionId parameter Tests: All 54 STS tests pass with this fix * test(sts): add comprehensive secret key determinism test coverage Updated tests to verify that secret access keys are now deterministic: 1. Updated TestSTSSessionClaimsToSessionInfoCredentialGeneration: - Changed comment from 'NOT deterministic' to 'NOW deterministic' - Added assertion that same sessionId produces identical secret key - Explains why this is critical for signature verification 2. Added TestSecretAccessKeyDeterminism (new dedicated test): - Verifies secret key is identical across multiple calls with same sessionId - Verifies access key ID and session token are also identical - Verifies different sessionIds produce different credentials - Includes detailed comments explaining why determinism is critical These tests ensure that the STS implementation correctly regenerates deterministic credentials during signature verification. Without determinism, signature verification would always fail because the server would use different secret keys than the client used to sign. * refactor(sts): add explicit zero-time expiration handling Improved defensive programming in IsExpired() methods: 1. Credentials.IsExpired(): - Added explicit check for zero-time expiration (time.Time{}) - Treats uninitialized credentials as expired - Prevents accidentally treating uninitialized creds as valid 2. SessionInfo.IsExpired(): - Added same explicit zero-time check - Treats uninitialized sessions as expired - Protects against bugs where sessions might not be properly initialized This is important because time.Now().After(time.Time{}) returns true, but explicitly checking for zero time makes the intent clear and helps catch initialization bugs during code review and debugging. * refactor(sts): remove unused IsExpired() helper functions The session_helpers.go file contained two unused IsExpired() methods: - Credentials.IsExpired() - SessionInfo.IsExpired() These were never called anywhere in the codebase. The actual expiration checks use: - isCredentialExpired() in weed/s3api/auth_credentials.go (S3 auth) - Direct time.Now().After() checks Removing unused code improves code clarity and reduces maintenance burden. * fix(auth): pass STS session token to IAM authorization for V4 signature auth CRITICAL FIX: Session tokens were not being passed to the authorization check when using AWS Signature V4 authentication with STS credentials. The bug: 1. AWS SDK sends request with X-Amz-Security-Token header (V4 signature) 2. validateSTSSessionToken validates the token, creates Identity with PrincipalArn 3. authorizeWithIAM only checked X-SeaweedFS-Session-Token (JWT auth header) 4. Since it was empty, fell into 'static V4' branch which set SessionToken = '' 5. AuthorizeAction returned ErrAccessDenied because SessionToken was empty The fix (in authorizeWithIAM): - Check X-SeaweedFS-Session-Token first (JWT auth) - If empty, fallback to X-Amz-Security-Token header (V4 STS auth) - If still empty, check X-Amz-Security-Token query param (presigned URLs) - When session token is found with PrincipalArn, use 'STS V4 signature' path - Only use 'static V4' path when there's no session token This ensures: - JWT Bearer auth with session tokens works (existing path) - STS V4 signature auth with session tokens works (new path) - Static V4 signature auth without session tokens works (existing path) Logging updated to distinguish: - 'JWT-based IAM authorization' - 'STS V4 signature IAM authorization' (new) - 'static V4 signature IAM authorization' (clarified) * test(s3api): add comprehensive STS session token authorization test coverage Added new test file auth_sts_v4_test.go with comprehensive tests for the STS session token authorization fix: 1. TestAuthorizeWithIAMSessionTokenExtraction: - Verifies X-SeaweedFS-Session-Token is extracted from JWT auth headers - Verifies X-Amz-Security-Token is extracted from V4 STS auth headers - Verifies X-Amz-Security-Token is extracted from query parameters (presigned URLs) - Verifies JWT tokens take precedence when both are present - Regression test for the bug where V4 STS tokens were not being passed to authorization 2. TestSTSSessionTokenIntoCredentials: - Verifies STS credentials have all required fields (AccessKeyId, SecretAccessKey, SessionToken) - Verifies deterministic generation from sessionId (same sessionId = same credentials) - Verifies different sessionIds produce different credentials - Critical for signature verification: same session must regenerate same secret key 3. TestActionConstantsForV4Auth: - Verifies S3 action constants are available for authorization checks - Ensures ACTION_READ, ACTION_WRITE, etc. are properly defined These tests ensure that: - V4 Signature auth with STS tokens properly extracts and uses session tokens - Session tokens are prioritized correctly (JWT > X-Amz-Security-Token header > query param) - STS credentials are deterministically generated for signature verification - The fix for passing STS session tokens to authorization is properly covered All 3 test functions pass (6 test cases total). * refactor(s3api): improve code quality and performance - Rename authorization path constants to avoid conflict with existing authType enum - Replace nested if/else with clean switch statement in authorizeWithIAM() - Add determineIAMAuthPath() helper for clearer intent and testability - Optimize key counting in auth_signature_v4.go: remove unnecessary slice allocation - Fix timing assertion in session_claims_test.go: use WithinDuration for symmetric tolerance These changes improve code readability, maintainability, and performance while maintaining full backward compatibility and test coverage. * refactor(s3api): use typed iamAuthPath for authorization path constants - Define iamAuthPath as a named string type (similar to existing authType enum) - Update constants to use explicit type: iamAuthPathJWT, iamAuthPathSTS_V4, etc. - Update determineIAMAuthPath() to return typed iamAuthPath - Improves type safety and prevents accidental string value misuse	1 week ago
Chris Lu	24556ebdcc	Refine Bucket Size Metrics: Logical and Physical Size (#7943) * refactor: implement logical size calculation with replication factor using dedicated helper * ui: update bucket list to show logical/physical size	1 week ago
Chris Lu	b97d17f79f	Standardize -ip.bind flags to default to empty and fall back to -ip (#7945) * Add documentation for issue #7941 fix * rm FIX_ISSUE_7941.md * Standardize -ip.bind flags to default to empty string and fall back to -ip option - Change s3 command -ip.bind default logic to use -ip instead of localhost - Change sftp command -ip.bind default to empty and fall back to 0.0.0.0 - Update help text for consistency * Fix compilation error: add -ip flag to s3 command and update bindIp fallback * Revert -ip flag addition for s3 command, set bindIp fallback to 0.0.0.0 * Update s3 command -ip.bind help text to reflect correct default behavior	1 week ago
Chris Lu	4d4b2e2d4a	add debug messages	1 week ago
Chris Lu	f2373f9e8d	fix: directory incorrectly listed as object in S3 ListObjects (#7939) * fix: directory incorrectly listed as object in S3 ListObjects Regular directories (without MIME type) were only added to CommonPrefixes when delimiter was exactly '/'. This caused directories to be silently skipped for other delimiter values. Changed the condition from 'delimiter == "/"' to 'delimiter != ""' to ensure directories are correctly added to CommonPrefixes for any delimiter. Fixes issue where directories like 'data/file.vhd' were being returned as objects instead of prefixes in ListObjects responses. * fix: complete the directory listing fix for all delimiters Address reviewer feedback: - Changed doListFilerEntries line 549 from 'delimiter != "/"' to 'delimiter == ""' This ensures directories are yielded to the callback for ANY delimiter, not just "/" - Parameterized test to verify fix works with multiple delimiters (/, _, :) The previous fix only addressed line 260 but line 549 was still causing recursion for non-"/" delimiters, preventing directories from being added to CommonPrefixes. * docs: update test comment to reflect multiple delimiters Address reviewer feedback - clarify that the test verifies behavior for any non-empty delimiter, not just '/'. * docs: clarify test comment with delimiter examples Add specific examples of delimiters ('/', '_', ':') to make it clear that the test verifies behavior with multiple delimiter types. * fix: revert line 549 to original logic, only line 260 needed changing The fix for directories being listed as objects only required changing line 260 from 'delimiter == "/"' to 'delimiter != ""'. Line 549 should remain as 'delimiter != "/"' to allow recursion for delimiters that don't exist in paths (e.g., delimiter=z for paths like b/a/c). This is correct S3 behavior. Updated test to only verify delimiter="/" since other delimiters should recurse into directories to find actual files. * docs: clarify test scope in directory listing test	1 week ago
Chris Lu	0f786cf0d2	Fix S3 list objects marker adjustment for delimiters (#7938)	2 weeks ago
Chris Lu	fca0a38435	Update s3api_object_handlers.go	2 weeks ago
Chris Lu	87b71029f7	4.05	2 weeks ago
Chris Lu	ade2e58cf4	refactoring	2 weeks ago
Chris Lu	4e2af080df	optimize: enable immediate EC shard reporting during startup (#7933) * optimize: enable immediate EC shard reporting during startup Ported the immediate EC shard reporting feature from Enterprise to Community version. This allows the master to be notified about EC shards immediately during volume server startup, instead of waiting for the first heartbeat. Changes: 1. Updated NewStore to initialize notification channels BEFORE loading volumes (fixes potential nil panic). 2. Added ecShardNotifyHandler to report EC shards to NewEcShardsChan during startup. 3. Implemented non-blocking channel send for EC reporting to prevent deadlock when loading many EC shards (fixing the enterprise bug 17ac1290c). 4. Updated DiskLocation and EC loading logic to support the callback. This optimization improves cluster state consistency and startup speed for EC-heavy clusters. * optimize: report actual EC shard size during startup * optimize: increase notification channel buffer size to 1024 * optimize: fix variable shadowing in store.go	2 weeks ago
Chris Lu	c405ff1374	feat(iam): add TLS configuration support for OIDC provider (#7929) * feat(iam): add TLS configuration support for OIDC provider Adds tlsCaCert and tlsInsecureSkipVerify options to OIDC provider configuration to allow using custom CA certificates and skipping verification in development environments. * fix: use SystemCertPool for custom CA and add security warning - Use x509.SystemCertPool() to preserve trust in public CAs - Add warning log when TLSInsecureSkipVerify is enabled - Addresses code review feedback from gemini-code-assist * docs: enhance TLS configuration field documentation - Add explicit warning about TLSInsecureSkipVerify production usage - Clarify TLSCACert is for custom/self-signed certificates * security: enforce TLS 1.2 minimum version - Set MinVersion to TLS 1.2 to prevent downgrade attacks - Ensures secure communication with OIDC providers * security: validate CA cert path is absolute - Add filepath.IsAbs check before reading CA certificate - Prevents reading unintended files from relative paths - Fail fast on misconfigured paths	2 weeks ago
Chris Lu	998bcf2b3f	classify grpc errors	2 weeks ago
Chris Lu	f7f133166a	adjust fuse logs	2 weeks ago