The workflow was failing because it was running inside 'weed' directory,
but the tests are at the repository root. Removed working-directory
default and updated relative paths to weed source.
- Add permissions.go with permission definitions and checks
- Define permissions for all 21 S3 Tables operations
- Add permission checking helper functions
- Add getPrincipalFromRequest to extract caller identity
- Implement access control in CreateTableBucket, GetTableBucket, DeleteTableBucket
- Return 403 Forbidden for unauthorized operations
- Only bucket owner can perform operations (extensible for future policies)
- Add AuthError type for authorization failures
- Replace O(n) slice iteration with O(1) map lookup
- Move s3TablesActionsMap to package level
- Avoid recreating the map on every function call
- Improves performance for request validation
- Specifically check for 'not found' errors instead of catching all errors
- Return empty list only when directory doesn't exist
- Propagate other errors (network, permission) with context
- Prevents masking real errors
- Move all client methods to client.go
- Remove duplicate types/constants from s3tables_integration_test.go
- Keep setup.go for test infrastructure
- Keep integration test logic in s3tables_integration_test.go
- Clean up unused imports
- Test compiles successfully
- Create new workflow for S3 Tables integration testing
- Add build verification job for s3tables package and s3api integration
- Add format checking for S3 Tables code
- Add go vet checks for code quality
- Workflow runs on all pull requests
- Includes test output logging and artifact upload on failure
- Create setup.go with TestCluster and S3TablesClient definitions
- Create client.go with HTTP client methods for all operations
- Test utilities and client methods organized for reusability
- Foundation for S3 Tables integration tests
- Add S3 Tables route registration in s3api_server.go registerRouter method
- Enable S3 Tables API operations to be routed through S3 API server
- Routes handled by s3api_tables.go integration layer
- Minimal changes to existing S3 API structure
- Create s3api_tables.go to integrate S3 Tables with S3 API server
- Implement S3 Tables route matcher for X-Amz-Target header
- Register S3 Tables routes with API router
- Provide gRPC filer client interface for S3 Tables handlers
- All S3 Tables operations accessible via S3 API endpoint
- Reduce handler.go from 370 to 195 lines (47% reduction)
- Remove duplicate ARN parsing and path helper functions
- Remove filer operation methods moved to filer_ops.go
- Remove metadata structure definitions moved to utils.go
- Keep handler focused on request routing and response formatting
- Maintains all functionality with improved code organization
- Create bucket_create.go for CreateTableBucket operation
- Create bucket_get_list_delete.go for Get, List, Delete operations
- Related operations grouped for better maintainability
- Each file has a single, clear responsibility
- Improves code clarity and makes it easier to test
- Move ARN parsing, path helpers, and metadata structures to utils.go
- Extract all extended attribute and filer operations to filer_ops.go
- Reduces code duplication and improves modularity
- Improves code organization and maintainability
* mount: apply UID/GID mapping in lookupEntry for cache misses
This fixes issue #8134 where rsync would fail with "Operation not permitted"
during chgrp. The issue was that entries fetched directly from the filer
(on cache miss) were not being mapped to local UIDs/GIDs.
* mount: add nil check for entry.Attributes in lookupEntry
* refactor(helm): add componentName helper for truncation
* fix(helm): unify ingress backend naming with truncation
* fix(helm): unify statefulset/deployment naming with truncation
* fix(helm): add missing labels to services for servicemonitor discovery
* chore(helm): secure secrets and add upgrade notes
* fix(helm): truncate context instead of suffix in componentName
* revert(docs): remove upgrade notes per feedback
* fix(helm): use componentName for COSI serviceAccountName
* helm: update master -ip to use component name for correct truncation
* helm: refactor masterServers helper to use truncated component names
* helm: update volume -ip to use component name and cleanup redundant printf
* helm: refine helpers with robustness check and updated docs
* feat(shell): add s3.bucket.lock command for Object Lock management
Add new weed shell command to view and enable S3 Object Lock on existing
buckets. This allows administrators to enable Object Lock without
recreating buckets, which is useful when buckets already contain data.
The command:
- Shows current Object Lock and Versioning status
- Enables Object Lock with -enable flag (irreversible, per AWS S3 spec)
- Automatically enables Versioning if not already enabled (required for Object Lock)
Usage:
s3.bucket.lock -name <bucket> # view status
s3.bucket.lock -name <bucket> -enable # enable Object Lock
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
* feat(shell): add -withLock flag to s3.bucket.create command
Add support for creating buckets with Object Lock enabled directly from
weed shell. The flag automatically enables versioning as required by
Object Lock.
Usage:
s3.bucket.create -name mybucket -withLock
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
* Apply suggestion from @gemini-code-assist[bot]
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Implement IAM propagation to S3 servers
- Add PropagatingCredentialStore to propagate IAM changes to S3 servers via gRPC
- Add Policy management RPCs to S3 proto and S3ApiServer
- Update CredentialManager to use PropagatingCredentialStore when MasterClient is available
- Wire FilerServer to enable propagation
* Implement parallel IAM propagation and fix S3 cluster registration
- Parallelized IAM change propagation with 10s timeout.
- Refined context usage in PropagatingCredentialStore.
- Added S3Type support to cluster node management.
- Enabled S3 servers to register with gRPC address to the master.
- Ensured IAM configuration reload after policy updates via gRPC.
* Optimize IAM propagation with direct in-memory cache updates
* Secure IAM propagation: Use metadata to skip persistence only on propagation
* pb: refactor IAM and S3 services for unidirectional IAM propagation
- Move SeaweedS3IamCache service from iam.proto to s3.proto.
- Remove legacy IAM management RPCs and empty SeaweedS3 service from s3.proto.
- Enforce that S3 servers only use the synchronization interface.
* pb: regenerate Go code for IAM and S3 services
Updated generated code following the proto refactoring of IAM synchronization services.
* s3api: implement read-only mode for Embedded IAM API
- Add readOnly flag to EmbeddedIamApi to reject write operations via HTTP.
- Enable read-only mode by default in S3ApiServer.
- Handle AccessDenied error in writeIamErrorResponse.
- Embed SeaweedS3IamCacheServer in S3ApiServer.
* credential: refactor PropagatingCredentialStore for unidirectional IAM flow
- Update to use s3_pb.SeaweedS3IamCacheClient for propagation to S3 servers.
- Propagate full Identity object via PutIdentity for consistency.
- Remove redundant propagation of specific user/account/policy management RPCs.
- Add timeout context for propagation calls.
* s3api: implement SeaweedS3IamCacheServer for unidirectional sync
- Update S3ApiServer to implement the cache synchronization gRPC interface.
- Methods (PutIdentity, RemoveIdentity, etc.) now perform direct in-memory cache updates.
- Register SeaweedS3IamCacheServer in command/s3.go.
- Remove registration for the legacy and now empty SeaweedS3 service.
* s3api: update tests for read-only IAM and propagation
- Added TestEmbeddedIamReadOnly to verify rejection of write operations in read-only mode.
- Update test setup to pass readOnly=false to NewEmbeddedIamApi in routing tests.
- Updated EmbeddedIamApiForTest helper with read-only checks matching production behavior.
* s3api: add back temporary debug logs for IAM updates
Log IAM updates received via:
- gRPC propagation (PutIdentity, PutPolicy, etc.)
- Metadata configuration reloads (LoadS3ApiConfigurationFromCredentialManager)
- Core identity management (UpsertIdentity, RemoveIdentity)
* IAM: finalize propagation fix with reduced logging and clarified architecture
* Allow configuring IAM read-only mode for S3 server integration tests
* s3api: add defensive validation to UpsertIdentity
* s3api: fix log message to reference correct IAM read-only flag
* test/s3/iam: ensure WaitForS3Service checks for IAM write permissions
* test: enable writable IAM in Makefile for integration tests
* IAM: add GetPolicy/ListPolicies RPCs to s3.proto
* S3: add GetBucketPolicy and ListBucketPolicies helpers
* S3: support storing generic IAM policies in IdentityAccessManagement
* S3: implement IAM policy RPCs using IdentityAccessManagement
* IAM: fix stale user identity on rename propagation
Recent changes in the S3 unified copy strategy were constructing source
and destination paths without the necessary BucketsPath prefix (typically
/buckets). When these paths reached the Filer for volume assignment, it
failed to resolve the correct collection and storage rules, defaulting to
a disk type with no available capacity.
This fix ensures all relevant paths in S3 copy handlers include the
correct BucketsPath prefix for proper collection resolution.
Fixes replication issue with Harbor upload via S3 API.
* Update IAM and S3 protobuf definitions for explicit IAM gRPC APIs
* Refactor s3api: Extract generic ExecuteAction method for IAM operations
* Implement explicit IAM gRPC APIs in S3 server
* iam: remove deprecated GetConfiguration and PutConfiguration RPCs
* iamapi: refactor handlers to use CredentialManager directly
* s3api: refactor embedded IAM to use CredentialManager directly
* server: remove deprecated configuration gRPC handlers
* credential/grpc: refactor configuration calls to return error
* shell: update s3.configure to list users instead of full config
* s3api: fix CreateServiceAccount gRPC handler to map required fields
* s3api: fix UpdateServiceAccount gRPC handler to map fields and safe status
* s3api: enforce UserName in embedded IAM ListAccessKeys
* test: fix test_config.json structure to match proto definition
* Revert "credential/grpc: refactor configuration calls to return error"
This reverts commit cde707dd8b.
* Revert "server: remove deprecated configuration gRPC handlers"
This reverts commit 7307e205a0.
* Revert "s3api: enforce UserName in embedded IAM ListAccessKeys"
This reverts commit adf727ba52.
* Revert "s3api: fix UpdateServiceAccount gRPC handler to map fields and safe status"
This reverts commit 6a4be3314d.
* Revert "s3api: fix CreateServiceAccount gRPC handler to map required fields"
This reverts commit 9bb4425f07.
* Revert "shell: update s3.configure to list users instead of full config"
This reverts commit f3304ead53.
* Revert "s3api: refactor embedded IAM to use CredentialManager directly"
This reverts commit 9012f27af8.
* Revert "iamapi: refactor handlers to use CredentialManager directly"
This reverts commit 3a14821223.
* Revert "iam: remove deprecated GetConfiguration and PutConfiguration RPCs"
This reverts commit e16e08aa00.
* s3api: address IAM code review comments (error handling, logging, gRPC response mapping)
* s3api: add robustness to startup by retrying KEK and IAM config loading from Filer
* s3api: address IAM gRPC code review comments (safety, validation, status logic)
* fix return
Resolved merge conflicts in:
- weed/admin/static/js/modal-alerts.js: Adopted incoming improvements and HTML support.
- weed/admin/view/app/collection_details.templ: Switched to showAlert info type.
- weed/admin/view/app/file_browser.templ: Used descriptive delete message.
- weed/admin/view/app/maintenance_workers.templ: Used encoding and headers in pauseWorker.
- weed/admin/view/app/object_store_users.templ: Restored accidentally deleted delete functions and used encodeURIComponent.
- weed/admin/view/app/policies.templ: Standardized on showAlert and descriptive confirmations.
Regenerated all templ files.
* Add IAM gRPC service definition
- Add GetConfiguration/PutConfiguration for config management
- Add CreateUser/GetUser/UpdateUser/DeleteUser/ListUsers for user management
- Add CreateAccessKey/DeleteAccessKey/GetUserByAccessKey for access key management
- Methods mirror existing IAM HTTP API functionality
* Add IAM gRPC handlers on filer server
- Implement IamGrpcServer with CredentialManager integration
- Handle configuration get/put operations
- Handle user CRUD operations
- Handle access key create/delete operations
- All methods delegate to CredentialManager for actual storage
* Wire IAM gRPC service to filer server
- Add CredentialManager field to FilerOption and FilerServer
- Import credential store implementations in filer command
- Initialize CredentialManager from credential.toml if available
- Register IAM gRPC service on filer gRPC server
- Enable credential management via gRPC alongside existing filer services
* Regenerate IAM protobuf with gRPC service methods
* fix: compilation error in DeleteUser
* fix: address code review comments for IAM migration
* feat: migrate policies to multi-file layout and fix identity duplicated content
* refactor: remove configuration.json and migrate Service Accounts to multi-file layout
* refactor: standardize Service Accounts as distinct store entities and fix Admin Server persistence
* config: set ServiceAccountsDirectory to /etc/iam/service_accounts
* Fix Chrome dialog auto-dismiss with Bootstrap modals
- Add modal-alerts.js library with Bootstrap modal replacements
- Replace all 15 confirm() calls with showConfirm/showDeleteConfirm
- Auto-override window.alert() for all alert() calls
- Fixes Chrome 132+ aggressively blocking native dialogs
* Upgrade Bootstrap from 5.3.2 to 5.3.8
* Fix syntax error in object_store_users.templ - remove duplicate closing braces
* create policy
* display errors
* migrate to multi-file policies
* address PR feedback: use showDeleteConfirm and showErrorMessage in policies.templ, refine migration check
* Update policies_templ.go
* add service account to iam grpc
* iam: fix potential path traversal in policy names by validating name pattern
* iam: add GetServiceAccountByAccessKey to CredentialStore interface
* iam: implement service account support for PostgresStore
Includes full CRUD operations and efficient lookup by access key.
* iam: implement GetServiceAccountByAccessKey for filer_etc, grpc, and memory stores
Provides efficient lookup of service accounts by access key where possible,
with linear scan fallbacks for file-based stores.
* iam: remove filer_multiple support
Deleted its implementation and references in imports, scaffold config,
and core interface constants. Redundant with filer_etc.
* clear comment
* dash: robustify service account construction
- Guard against nil sa.Credential when constructing responses
- Fix Expiration logic to only set if > 0, avoiding Unix epoch 1970
- Ensure consistency across Get, Create, and Update handlers
* credential/filer_etc: improve error propagation in configuration handlers
- Return error from loadServiceAccountsFromMultiFile to callers
- Ensure listEntries errors in SaveConfiguration (cleanup logic) are
propagated unless they are "not found" failures.
- Fixes potential silent failures during IAM configuration sync.
* credential/filer_etc: add existence check to CreateServiceAccount
Ensures consistency with other stores by preventing accidental overwrite
of existing service accounts during creation.
* credential/memory: improve store robustness and Reset logic
- Enforce ID immutability in UpdateServiceAccount to prevent orphans
- Update Reset() to also clear the policies map, ensuring full state
cleanup for tests.
* dash: improve service account robustness and policy docs
- Wrap parent user lookup errors to preserve context
- Strictly validate Status field in UpdateServiceAccount
- Add deprecation comments to legacy policy management methods
* credential/filer_etc: protect against path traversal in service accounts
Implemented ID validation (alphanumeric, underscores, hyphens) and applied
it to Get, Save, and Delete operations to ensure no directory traversal
via saId.json filenames.
* credential/postgres: improve robustness and cleanup comments
- Removed brainstorming comments in GetServiceAccountByAccessKey
- Added missing rows.Err() check during iteration
- Properly propagate Scan and Unmarshal errors instead of swallowing them
* admin: unify UI alerts and confirmations using Bootstrap modals
- Updated modal-alerts.js with improved automated alert type detection
- Replaced native alert() and confirm() with showAlert(), showConfirm(),
and showDeleteConfirm() across various Templ components
- Improved UX for delete operations by providing better context and styling
- Ensured consistent error reporting across IAM and Maintenance views
* admin: additional UI consistency fixes for alerts and confirmations
- Replaced native alert() and confirm() with Bootstrap modals in:
- EC volumes (repair flow)
- Collection details (repair flow)
- File browser (properties and delete)
- Maintenance config schema (save and reset)
- Improved delete confirmation in file browser with item context
- Ensured consistent success/error/info styling for all feedbacks
* make
* iam: add GetServiceAccountByAccessKey RPC and update GetConfiguration
* iam: implement GetServiceAccountByAccessKey on server and client
* iam: centralize policy and service account validation
* iam: optimize MemoryStore service account lookups with indexing
* iam: fix postgres service_accounts table and optimize lookups
* admin: refactor modal alerts and clean up dashboard logic
* admin: fix EC shards table layout mismatch
* admin: URL-encode IAM path parameters for safety
* admin: implement pauseWorker logic in maintenance view
* iam: add rows.Err() check to postgres ListServiceAccounts
* iam: standardize ErrServiceAccountNotFound across credential stores
* iam: map ErrServiceAccountNotFound to codes.NotFound in DeleteServiceAccount
* iam: refine service account store logic, errors and schema
* iam: add validation to GetServiceAccountByAccessKey
* admin: refine modal titles and ensure URL safety
* admin: address bot review comments for alerts and async usage
* iam: fix syntax error by restoring missing function declaration
* [FilerEtcStore] improve error handling in CreateServiceAccount
Refine error handling to provide clearer messages when checking for
existing service accounts.
* [PostgresStore] add nil guards and validation to service account methods
Ensure input parameters are not nil and required IDs are present
to prevent runtime panics and ensure data integrity.
* [JS] add shared IAM utility script
Consolidate common IAM operations like deleteUser and deleteAccessKey
into a shared utility script for better maintainability.
* [View] include shared IAM utilities in layout
Include iam-utils.js in the main layout to make IAM functions
available across all administrative pages.
* [View] refactor IAM logic and restore async in EC Shards view
Remove redundant local IAM functions and ensure that delete
confirmation callbacks are properly marked as async.
* [View] consolidate IAM logic in Object Store Users view
Remove redundant local definitions of deleteUser and deleteAccessKey,
relying on the shared utilities instead.
* [View] update generated templ files for UI consistency
* credential/postgres: remove redundant name column from service_accounts table
The id is already used as the unique identifier and was being copied to the name column.
This removes the name column from the schema and updates the INSERT/UPDATE queries.
* credential/filer_etc: improve logging for policy migration failures
Added Errorf log if AtomicRenameEntry fails during migration to ensure visibility of common failure points.
* credential: allow uppercase characters in service account ID username
Updated ServiceAccountIdPattern to allow [A-Za-z0-9_-]+ for the username component,
matching the actual service account creation logic which uses the parent user name directly.
* Update object_store_users_templ.go
* admin: fix ec_shards pagination to handle numeric page arguments
Updated goToPage in cluster_ec_shards.templ to accept either an Event
or a numeric page argument. This prevents errors when goToPage(1)
is called directly. Corrected both the .templ source and generated Go code.
* credential/filer_etc: improve service account storage robustness
Added nil guard to saveServiceAccount, updated GetServiceAccount
to return ErrServiceAccountNotFound for empty data, and improved
deleteServiceAccount to handle response-level Filer errors.
Chris Lu
Chris Lu
MorezMartin
Alasdair Macmillan
Andrei Kvapil
dependabot[bot]