Browse Source

S3: Implement IAM defaults and STS signing key fallback (#8348)

* S3: Implement IAM defaults and STS signing key fallback logic

* S3: Refactor startup order to init SSE-S3 key manager before IAM

* S3: Derive STS signing key from KEK using HKDF for security isolation

* S3: Document STS signing key fallback in security.toml

* fix(s3api): refine anonymous access logic and secure-by-default behavior

- Initialize anonymous identity by default in `NewIdentityAccessManagement` to prevent nil pointer exceptions.
- Ensure `ReplaceS3ApiConfiguration` preserves the anonymous identity if not present in the new configuration.
- Update `NewIdentityAccessManagement` signature to accept `filerClient`.
- In legacy mode (no policy engine), anonymous defaults to Deny (no actions), preserving secure-by-default behavior.
- Use specific `LookupAnonymous` method instead of generic map lookup.
- Update tests to accommodate signature changes and verify improved anonymous handling.

* feat(s3api): make IAM configuration optional

- Start S3 API server without a configuration file if `EnableIam` option is set.
- Default to `Allow` effect for policy engine when no configuration is provided (Zero-Config mode).
- Handle empty configuration path gracefully in `loadIAMManagerFromConfig`.
- Add integration test `iam_optional_test.go` to verify empty config behavior.

* fix(iamapi): fix signature mismatch in NewIdentityAccessManagementWithStore

* fix(iamapi): properly initialize FilerClient instead of passing nil

* fix(iamapi): properly initialize filer client for IAM management

- Instead of passing `nil`, construct a `wdclient.FilerClient` using the provided `Filers` addresses.
- Ensure `NewIdentityAccessManagementWithStore` receives a valid `filerClient` to avoid potential nil pointer dereferences or limited functionality.

* clean: remove dead code in s3api_server.go

* refactor(s3api): improve IAM initialization, safety and anonymous access security

* fix(s3api): ensure IAM config loads from filer after client init

* fix(s3): resolve test failures in integration, CORS, and tagging tests

- Fix CORS tests by providing explicit anonymous permissions config
- Fix S3 integration tests by setting admin credentials in init
- Align tagging test credentials in CI with IAM defaults
- Added goroutine to retry IAM config load in iamapi server

* fix(s3): allow anonymous access to health targets and S3 Tables when identities are present

* fix(ci): use /healthz for Caddy health check in awscli tests

* iam, s3api: expose DefaultAllow from IAM and Policy Engine

This allows checking the global "Open by Default" configuration from
other components like S3 Tables.

* s3api/s3tables: support DefaultAllow in permission logic and handler

Updated CheckPermissionWithContext to respect the DefaultAllow flag
in PolicyContext. This enables "Open by Default" behavior for
unauthenticated access in zero-config environments. Added a targeted
unit test to verify the logic.

* s3api/s3tables: propagate DefaultAllow through handlers

Propagated the DefaultAllow flag to individual handlers for
namespaces, buckets, tables, policies, and tagging. This ensures
consistent "Open by Default" behavior across all S3 Tables API
endpoints.

* s3api: wire up DefaultAllow for S3 Tables API initialization

Updated registerS3TablesRoutes to query the global IAM configuration
and set the DefaultAllow flag on the S3 Tables API server. This
completes the end-to-end propagation required for anonymous access in
zero-config environments. Added a SetDefaultAllow method to
S3TablesApiServer to facilitate this.

* s3api: fix tests by adding DefaultAllow to mock IAM integrations

The IAMIntegration interface was updated to include DefaultAllow(),
breaking several mock implementations in tests. This commit fixes
the build errors by adding the missing method to the mocks.

* env

* ensure ports

* env

* env

* fix default allow

* add one more test using non-anonymous user

* debug

* add more debug

* less logs
pull/8359/head
Chris Lu 4 weeks ago
committed by GitHub
parent
commit
0d8588e3ae
No known key found for this signature in database GPG Key ID: B5690EEEBB952194
  1. 2
      .github/workflows/java_integration_tests.yml
  2. 2
      .github/workflows/s3-go-tests.yml
  3. 2
      .github/workflows/s3-keycloak-tests.yml
  4. 4
      .github/workflows/test-s3-over-https-using-awscli.yml
  5. 11
      test/s3/cors/Makefile
  6. 27
      test/s3/cors/s3_test_config.json
  7. 7
      test/s3/normal/iam_test.go
  8. 8
      test/s3/normal/s3_integration_test.go
  9. 8
      test/s3/policy/policy_test.go
  10. 4
      test/s3tables/catalog/iceberg_catalog_test.go
  11. 68
      test/s3tables/catalog/pyiceberg_test.go
  12. 36
      test/s3tables/catalog/pyiceberg_test_helpers.go
  13. 1
      test/s3tables/catalog/test_rest_catalog.py
  14. 223
      test/s3tables/catalog/test_rest_catalog_auth.py
  15. 8
      test/s3tables/table-buckets/s3tables_integration_test.go
  16. 18
      weed/command/mini.go
  17. 1
      weed/command/scaffold/security.toml
  18. 8
      weed/iam/integration/iam_manager.go
  19. 8
      weed/iam/policy/policy_engine.go
  20. 18
      weed/iam/sts/sts_service.go
  21. 23
      weed/iam/sts/sts_service_test.go
  22. 11
      weed/iamapi/iamapi_server.go
  23. 79
      weed/s3api/auth_credentials.go
  24. 6
      weed/s3api/auth_credentials_test.go
  25. 2
      weed/s3api/auth_security_test.go
  26. 4
      weed/s3api/auth_signature_v4_sts_test.go
  27. 6
      weed/s3api/auth_sts_identity_test.go
  28. 156
      weed/s3api/iam_defaults_test.go
  29. 50
      weed/s3api/iam_optional_test.go
  30. 32
      weed/s3api/iceberg/server.go
  31. 9
      weed/s3api/s3_iam_middleware.go
  32. 52
      weed/s3api/s3_sse_s3.go
  33. 6
      weed/s3api/s3api_put_object_helper_test.go
  34. 134
      weed/s3api/s3api_server.go
  35. 2
      weed/s3api/s3api_server_routing_test.go
  36. 23
      weed/s3api/s3api_tables.go
  37. 15
      weed/s3api/s3tables/handler.go
  38. 4
      weed/s3api/s3tables/handler_bucket_create.go
  39. 4
      weed/s3api/s3tables/handler_bucket_get_list_delete.go
  40. 4
      weed/s3api/s3tables/handler_namespace.go
  41. 9
      weed/s3api/s3tables/handler_policy.go
  42. 11
      weed/s3api/s3tables/handler_table.go
  43. 10
      weed/s3api/s3tables/manager.go
  44. 27
      weed/s3api/s3tables/permissions.go
  45. 45
      weed/s3api/s3tables/permissions_test.go
  46. 3
      weed/s3api/sts_params_test.go

2
.github/workflows/java_integration_tests.yml

@ -111,7 +111,7 @@ jobs:
# Wait for S3 API
for i in {1..30}; do
if curl -s http://localhost:8333/ > /dev/null 2>&1; then
if curl -s http://localhost:8333/healthz > /dev/null 2>&1; then
echo "✓ S3 API is ready"
break
fi

2
.github/workflows/s3-go-tests.yml

@ -461,6 +461,8 @@ jobs:
export S3_ENDPOINT="http://localhost:8006"
export S3_ACCESS_KEY="0555b35654ad1656d804"
export S3_SECRET_KEY="h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=="
export AWS_ACCESS_KEY_ID="$S3_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="$S3_SECRET_KEY"
# Run the specific test that is equivalent to AWS S3 tagging behavior
make test-with-server || {

2
.github/workflows/s3-keycloak-tests.yml

@ -97,7 +97,7 @@ jobs:
# Verify service accessibility
echo "=== Verifying Service Accessibility ==="
curl -f http://localhost:8080/realms/master
curl -s http://localhost:8333
curl -s http://localhost:8333/healthz
echo "✅ SeaweedFS S3 API is responding (IAM-protected endpoint)"
# Run Keycloak-specific tests

4
.github/workflows/test-s3-over-https-using-awscli.yml

@ -35,7 +35,7 @@ jobs:
set -e
mkdir -p /tmp/data
./weed -v=3 server -s3 -dir=/tmp/data -s3.config=../docker/compose/s3.json -master.peers=none > weed.log 2>&1 &
until curl -s http://localhost:8333/ > /dev/null; do sleep 1; done
until curl -s http://localhost:8333/healthz > /dev/null; do sleep 1; done
- name: Setup Caddy
run: |
@ -54,7 +54,7 @@ jobs:
- name: Start Caddy
run: |
./caddy start
until curl -fsS --insecure https://localhost:8443 > /dev/null; do sleep 1; done
until curl -fsS --insecure https://localhost:8443/healthz > /dev/null; do sleep 1; done
- name: Create Bucket
run: |

11
test/s3/cors/Makefile

@ -12,6 +12,7 @@ FILER_PORT := 8888
TEST_TIMEOUT := 10m
TEST_PATTERN := TestCORS
SERVER_DIR := test-mini-data
S3_CONFIG := s3_test_config.json
# Default target
help:
@ -80,13 +81,15 @@ start-server: check-deps
@echo "🔍 DEBUG: Creating volume directory..."
@mkdir -p $(SERVER_DIR)
@echo "🔍 DEBUG: Launching SeaweedFS S3 server in background..."
@echo "🔍 DEBUG: Command: $(WEED_BINARY) mini -dir=$(SERVER_DIR) -s3.port=$(S3_PORT) -s3.config=$(S3_CONFIG)"
@$(WEED_BINARY) mini \
@echo "🔍 DEBUG: Command: AWS_ACCESS_KEY_ID=some_access_key1 AWS_SECRET_ACCESS_KEY=some_secret_key1 $(WEED_BINARY) mini -dir=$(SERVER_DIR) -s3.port=$(S3_PORT) -s3.config=$(S3_CONFIG)"
@env AWS_ACCESS_KEY_ID=some_access_key1 \
AWS_SECRET_ACCESS_KEY=some_secret_key1 \
$(WEED_BINARY) mini \
-dir=$(SERVER_DIR) \
-s3.port=$(S3_PORT) \
-s3.config=$(S3_CONFIG) \
> weed-test.log 2>&1 & \
echo $$! > weed-test.pid
echo $$! > weed-server.pid
@echo "Waiting for S3 server to be ready..."
@for i in $$(seq 1 30); do \
@ -97,7 +100,7 @@ start-server: check-deps
sleep 1; \
done; \
echo "S3 server failed to start"; \
exit 1 > weed-server.pid
exit 1
@echo "🔍 DEBUG: Server PID: $$(cat weed-server.pid 2>/dev/null || echo 'PID file not found')"
@echo "🔍 DEBUG: Checking if PID is still running..."
@sleep 2

27
test/s3/cors/s3_test_config.json

@ -0,0 +1,27 @@
{
"identities": [
{
"name": "anonymous",
"actions": [
"Read",
"List"
]
},
{
"name": "admin",
"credentials": [
{
"accessKey": "some_access_key1",
"secretKey": "some_secret_key1"
}
],
"actions": [
"Admin",
"Read",
"List",
"Tagging",
"Write"
]
}
]
}

7
test/s3/normal/iam_test.go

@ -1,7 +1,6 @@
package example
import (
"os"
"testing"
"time"
@ -22,10 +21,8 @@ func TestIAMOperations(t *testing.T) {
// Set credentials before starting cluster
accessKey := "testkey123"
secretKey := "testsecret456"
os.Setenv("AWS_ACCESS_KEY_ID", accessKey)
os.Setenv("AWS_SECRET_ACCESS_KEY", secretKey)
defer os.Unsetenv("AWS_ACCESS_KEY_ID")
defer os.Unsetenv("AWS_SECRET_ACCESS_KEY")
t.Setenv("AWS_ACCESS_KEY_ID", accessKey)
t.Setenv("AWS_SECRET_ACCESS_KEY", secretKey)
// Create and start test cluster
cluster, err := startMiniCluster(t)

8
test/s3/normal/s3_integration_test.go

@ -146,6 +146,14 @@ func startMiniCluster(t *testing.T) (*TestCluster, error) {
return nil, fmt.Errorf("failed to create security.toml: %v", err)
}
// Set environment variables for admin credentials safely for this test
if os.Getenv("AWS_ACCESS_KEY_ID") == "" {
t.Setenv("AWS_ACCESS_KEY_ID", "admin")
}
if os.Getenv("AWS_SECRET_ACCESS_KEY") == "" {
t.Setenv("AWS_SECRET_ACCESS_KEY", "admin")
}
// Start weed mini in a goroutine by calling the command directly
cluster.wg.Add(1)
go func() {

8
test/s3/policy/policy_test.go

@ -216,6 +216,14 @@ enabled = true
err = os.WriteFile(credentialToml, []byte(credentialConfig), 0644)
require.NoError(t, err)
// Set environment variables for admin credentials safely for this test
if os.Getenv("AWS_ACCESS_KEY_ID") == "" {
t.Setenv("AWS_ACCESS_KEY_ID", "admin")
}
if os.Getenv("AWS_SECRET_ACCESS_KEY") == "" {
t.Setenv("AWS_SECRET_ACCESS_KEY", "admin")
}
cluster.wg.Add(1)
go func() {
defer cluster.wg.Done()

4
test/s3tables/catalog/iceberg_catalog_test.go

@ -511,8 +511,10 @@ func createTableBucket(t *testing.T, env *TestEnvironment, bucketName string) {
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
t.Logf("Create table bucket %s response: status=%d, body=%s", bucketName, resp.StatusCode, string(body))
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusConflict {
body, _ := io.ReadAll(resp.Body)
t.Fatalf("Failed to create table bucket %s, status %d: %s", bucketName, resp.StatusCode, body)
}
t.Logf("Created table bucket %s", bucketName)

68
test/s3tables/catalog/pyiceberg_test.go

@ -56,8 +56,6 @@ func TestPyIcebergRestCatalog(t *testing.T) {
cmd := exec.Command("docker", "run", "--rm",
"--add-host", "host.docker.internal:host-gateway",
"-e", fmt.Sprintf("AWS_ACCESS_KEY_ID=%s", "test"),
"-e", fmt.Sprintf("AWS_SECRET_ACCESS_KEY=%s", "test"),
"-e", fmt.Sprintf("AWS_ENDPOINT_URL=%s", s3Endpoint),
"-v", fmt.Sprintf("%s:/app:ro", testDir),
"iceberg-rest-test",
@ -78,3 +76,69 @@ func TestPyIcebergRestCatalog(t *testing.T) {
t.Errorf("PyIceberg test failed: %v", err)
}
}
// TestPyIcebergRestCatalogAuthenticated tests the Iceberg REST Catalog using PyIceberg with authentication.
// This test uses the default admin credentials that SeaweedFS creates on startup.
func TestPyIcebergRestCatalogAuthenticated(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
env := NewTestEnvironment(t)
defer env.Cleanup(t)
if !env.dockerAvailable {
t.Skip("Docker not available, skipping PyIceberg integration test")
}
// Use default admin credentials
testAccessKey := "admin"
testSecretKey := "admin"
// Start SeaweedFS (it will use default admin credentials from environment if set)
env.StartSeaweedFS(t)
// Create the test bucket first (using unauthenticated request, which works with DefaultAllow)
bucketName := "pyiceberg-auth-test"
createTableBucket(t, env, bucketName)
// Build the test working directory path
testDir := filepath.Join(env.seaweedDir, "test", "s3tables", "catalog")
// Run PyIceberg test using Docker with authentication
catalogURL := fmt.Sprintf("http://host.docker.internal:%d", env.icebergPort)
s3Endpoint := fmt.Sprintf("http://host.docker.internal:%d", env.s3Port)
warehouse := fmt.Sprintf("s3://%s/", bucketName)
// Build the test image first for faster repeated runs
buildCmd := exec.Command("docker", "build", "-t", "iceberg-rest-test", "-f", "Dockerfile.pyiceberg", ".")
buildCmd.Dir = testDir
if out, err := buildCmd.CombinedOutput(); err != nil {
t.Fatalf("Failed to build test image: %v\n%s", err, string(out))
}
cmd := exec.Command("docker", "run", "--rm",
"--add-host", "host.docker.internal:host-gateway",
"-e", fmt.Sprintf("AWS_ENDPOINT_URL=%s", s3Endpoint),
"-v", fmt.Sprintf("%s:/app:ro", testDir),
"iceberg-rest-test",
"python3", "/app/test_rest_catalog_auth.py",
"--catalog-url", catalogURL,
"--warehouse", warehouse,
"--prefix", bucketName,
"--access-key", testAccessKey,
"--secret-key", testSecretKey,
)
cmd.Dir = testDir
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
t.Logf("Running PyIceberg REST catalog test with authentication...")
t.Logf(" Catalog URL: %s", catalogURL)
t.Logf(" Warehouse: %s", warehouse)
t.Logf(" Access Key: %s", testAccessKey)
if err := cmd.Run(); err != nil {
t.Errorf("PyIceberg authenticated test failed: %v", err)
}
}

36
test/s3tables/catalog/pyiceberg_test_helpers.go

@ -0,0 +1,36 @@
package catalog
import (
"fmt"
"io"
"net/http"
"testing"
)
// verifyTableBucketMetadata verifies that a table bucket was created with proper metadata
func verifyTableBucketMetadata(t *testing.T, env *TestEnvironment, bucketName string) {
t.Helper()
// Use S3Tables REST API to get the bucket
endpoint := fmt.Sprintf("http://localhost:%d/buckets/%s", env.s3Port, bucketName)
req, err := http.NewRequest(http.MethodGet, endpoint, nil)
if err != nil {
t.Fatalf("Failed to create request: %v", err)
}
req.Header.Set("Content-Type", "application/x-amz-json-1.1")
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("Failed to get table bucket %s: %v", bucketName, err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
t.Logf("Get table bucket %s response: status=%d, body=%s", bucketName, resp.StatusCode, string(body))
if resp.StatusCode != http.StatusOK {
t.Fatalf("Failed to get table bucket %s, status %d: %s", bucketName, resp.StatusCode, body)
}
t.Logf("Verified table bucket %s exists with metadata", bucketName)
}

1
test/s3tables/catalog/test_rest_catalog.py

@ -201,6 +201,7 @@ def main():
"uri": args.catalog_url,
"warehouse": args.warehouse,
"prefix": args.prefix,
"s3.anonymous": "true", # Disable AWS request signing for unauthenticated access
}
)
print(f"Successfully connected to catalog on attempt {attempt + 1}")

223
test/s3tables/catalog/test_rest_catalog_auth.py

@ -0,0 +1,223 @@
#!/usr/bin/env python3
"""
Iceberg REST Catalog Compatibility Test for SeaweedFS (Authenticated)
This script tests the Iceberg REST Catalog API compatibility with authentication.
Usage:
python3 test_rest_catalog_auth.py --catalog-url http://localhost:8182 \\
--access-key admin --secret-key admin
Requirements:
pip install pyiceberg[s3fs]
"""
import argparse
import sys
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import (
IntegerType,
LongType,
StringType,
NestedField,
)
from pyiceberg.exceptions import (
NamespaceAlreadyExistsError,
NoSuchNamespaceError,
TableAlreadyExistsError,
NoSuchTableError,
)
def test_config_endpoint(catalog):
"""Test that the catalog config endpoint returns valid configuration."""
print("Testing /v1/config endpoint...")
# The catalog is already loaded which means config endpoint worked
print(" /v1/config endpoint working")
return True
def test_namespace_operations(catalog, prefix):
"""Test namespace CRUD operations."""
print("Testing namespace operations...")
namespace = (f"{prefix.replace('-', '_')}_auth_test_ns",)
# List initial namespaces
namespaces = catalog.list_namespaces()
print(f" Initial namespaces: {namespaces}")
# Create namespace
try:
catalog.create_namespace(namespace)
print(f" Created namespace: {namespace}")
except NamespaceAlreadyExistsError:
print(f" ! Namespace already exists: {namespace}")
# List namespaces (should include our new one)
namespaces = catalog.list_namespaces()
if namespace in namespaces:
print(" Namespace appears in list")
else:
print(f" Namespace not found in list: {namespaces}")
return False
# Get namespace properties
try:
props = catalog.load_namespace_properties(namespace)
print(f" Loaded namespace properties: {props}")
except NoSuchNamespaceError:
print(f" Failed to load namespace properties")
return False
return True
def test_table_operations(catalog, prefix):
"""Test table CRUD operations."""
print("Testing table operations...")
namespace = (f"{prefix.replace('-', '_')}_auth_test_ns",)
table_name = "auth_test_table"
table_id = namespace + (table_name,)
# Define a simple schema
schema = Schema(
NestedField(field_id=1, name="id", field_type=LongType(), required=True),
NestedField(field_id=2, name="name", field_type=StringType(), required=False),
NestedField(field_id=3, name="age", field_type=IntegerType(), required=False),
)
# Create table
try:
table = catalog.create_table(
identifier=table_id,
schema=schema,
)
print(f" Created table: {table_id}")
except TableAlreadyExistsError:
print(f" ! Table already exists: {table_id}")
_ = catalog.load_table(table_id)
# List tables
tables = catalog.list_tables(namespace)
if table_name in [t[1] for t in tables]:
print(" Table appears in list")
else:
print(f" Table not found in list: {tables}")
return False
# Load table
try:
loaded_table = catalog.load_table(table_id)
print(f" Loaded table: {loaded_table.name()}")
print(f" Schema: {loaded_table.schema()}")
print(f" Location: {loaded_table.location()}")
except NoSuchTableError:
print(f" Failed to load table")
return False
return True
def test_cleanup(catalog, prefix):
"""Test table and namespace deletion."""
print("Testing cleanup operations...")
namespace = (f"{prefix.replace('-', '_')}_auth_test_ns",)
table_id = namespace + ("auth_test_table",)
# Drop table
try:
catalog.drop_table(table_id)
print(f" Dropped table: {table_id}")
except NoSuchTableError:
print(f" ! Table already deleted: {table_id}")
# Drop namespace
try:
catalog.drop_namespace(namespace)
print(f" Dropped namespace: {namespace}")
except NoSuchNamespaceError:
print(f" ! Namespace already deleted: {namespace}")
except Exception as e:
print(f" ? Namespace drop error (may be expected): {e}")
return True
def main():
parser = argparse.ArgumentParser(description="Test Iceberg REST Catalog with authentication")
parser.add_argument("--catalog-url", required=True, help="Iceberg REST Catalog URL")
parser.add_argument("--warehouse", default="s3://iceberg-test/", help="Warehouse location")
parser.add_argument("--prefix", required=True, help="Table bucket prefix")
parser.add_argument("--access-key", required=True, help="AWS Access Key ID")
parser.add_argument("--secret-key", required=True, help="AWS Secret Access Key")
parser.add_argument("--skip-cleanup", action="store_true", help="Skip cleanup at the end")
args = parser.parse_args()
print(f"Connecting to Iceberg REST Catalog at: {args.catalog_url}")
print(f"Warehouse: {args.warehouse}")
print(f"Prefix: {args.prefix}")
print(f"Using authenticated access with key: {args.access_key}")
print()
# Load the REST catalog with authentication
import time
max_retries = 10
catalog = None
for attempt in range(max_retries):
try:
catalog = load_catalog(
"rest",
**{
"type": "rest",
"uri": args.catalog_url,
"warehouse": args.warehouse,
"prefix": args.prefix,
"s3.access-key-id": args.access_key,
"s3.secret-access-key": args.secret_key,
}
)
print(f"Successfully connected to catalog on attempt {attempt + 1}")
break
except Exception as e:
if attempt < max_retries - 1:
print(f" Attempt {attempt + 1} failed, retrying in 2s... ({e})")
time.sleep(2)
else:
print(f" All {max_retries} attempts failed.")
raise e
# Run tests
tests = [
("Config Endpoint", lambda: test_config_endpoint(catalog)),
("Namespace Operations", lambda: test_namespace_operations(catalog, args.prefix)),
("Table Operations", lambda: test_table_operations(catalog, args.prefix)),
]
if not args.skip_cleanup:
tests.append(("Cleanup", lambda: test_cleanup(catalog, args.prefix)))
passed = 0
failed = 0
for name, test_fn in tests:
print(f"\n{'='*50}")
try:
if test_fn():
passed += 1
print(f"PASSED: {name}")
else:
failed += 1
print(f"FAILED: {name}")
except Exception as e:
failed += 1
print(f"ERROR in {name}: {e}")
print(f"\n{'='*50}")
print(f"Results: {passed} passed, {failed} failed")
return 0 if failed == 0 else 1
if __name__ == "__main__":
sys.exit(main())

8
test/s3tables/table-buckets/s3tables_integration_test.go

@ -556,6 +556,14 @@ func startMiniCluster(t *testing.T) (*TestCluster, error) {
return nil, fmt.Errorf("failed to create security.toml: %v", err)
}
// Set environment variables for admin credentials safely for this test
if os.Getenv("AWS_ACCESS_KEY_ID") == "" {
t.Setenv("AWS_ACCESS_KEY_ID", "admin")
}
if os.Getenv("AWS_SECRET_ACCESS_KEY") == "" {
t.Setenv("AWS_SECRET_ACCESS_KEY", "admin")
}
// Start weed mini in a goroutine by calling the command directly
cluster.wg.Add(1)
go func() {

18
weed/command/mini.go

@ -355,6 +355,9 @@ func isFlagPassed(name string) bool {
// isPortOpenOnIP checks if a port is available for binding on a specific IP address
func isPortOpenOnIP(ip string, port int) bool {
if port <= 0 || port > 65535 {
return false
}
listener, err := net.Listen("tcp", fmt.Sprintf("%s:%d", ip, port))
if err != nil {
return false
@ -366,6 +369,9 @@ func isPortOpenOnIP(ip string, port int) bool {
// isPortAvailable checks if a port is available on any interface
// This is more comprehensive than checking a single IP
func isPortAvailable(port int) bool {
if port <= 0 || port > 65535 {
return false
}
// Try to listen on all interfaces (0.0.0.0)
listener, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
if err != nil {
@ -381,6 +387,10 @@ func isPortAvailable(port int) bool {
func findAvailablePortOnIP(ip string, startPort int, maxAttempts int, reservedPorts map[int]bool) int {
for i := 0; i < maxAttempts; i++ {
port := startPort + i
if port > 65535 {
// Wrap around to a lower range if we exceed 65535
port = 10000 + (port % 65535)
}
// Skip ports reserved for gRPC calculation
if reservedPorts[port] {
continue
@ -398,15 +408,15 @@ func findAvailablePortOnIP(ip string, startPort int, maxAttempts int, reservedPo
// If the port is not available, it finds the next available port and updates the pointer
// The reservedPorts map contains ports that should not be allocated (for gRPC collision avoidance)
func ensurePortAvailableOnIP(portPtr *int, serviceName string, ip string, reservedPorts map[int]bool, flagName string) error {
if portPtr == nil {
// Check if this port was explicitly specified by the user (from CLI, before config file was applied)
isExplicitPort := explicitPortFlags[flagName]
if *portPtr == 0 {
return nil
}
original := *portPtr
// Check if this port was explicitly specified by the user (from CLI, before config file was applied)
isExplicitPort := explicitPortFlags[flagName]
// Skip if this port is reserved for gRPC calculation
if reservedPorts[original] {
if isExplicitPort {

1
weed/command/scaffold/security.toml

@ -43,6 +43,7 @@ expires_after_seconds = 10 # seconds
# If this JWT key is configured, Filer only accepts writes over HTTP if they are signed with this JWT:
# - f.e. the S3 API Shim generates the JWT
# - the Filer server validates the JWT on writing
# NOTE: This key is ALSO used as a fallback signing key for S3 STS if s3.iam.config does not specify a signingKey.
# the jwt defaults to expire after 10 seconds.
[jwt.filer_signing]
key = ""

8
weed/iam/integration/iam_manager.go

@ -654,6 +654,14 @@ func (m *IAMManager) GetSTSService() *sts.STSService {
return m.stsService
}
// DefaultAllow returns whether the default effect is Allow
func (m *IAMManager) DefaultAllow() bool {
if !m.initialized || m.policyEngine == nil {
return true // Default to true if not initialized
}
return m.policyEngine.DefaultAllow()
}
// parseJWTTokenForTrustPolicy parses a JWT token to extract claims for trust policy evaluation
func parseJWTTokenForTrustPolicy(tokenString string) (map[string]interface{}, error) {
// Simple JWT parsing without verification (for trust policy context only)

8
weed/iam/policy/policy_engine.go

@ -324,6 +324,14 @@ func (e *PolicyEngine) IsInitialized() bool {
return e.initialized
}
// DefaultAllow returns whether the default effect is Allow
func (e *PolicyEngine) DefaultAllow() bool {
if e.config == nil {
return true // Default to Allow if not configured
}
return e.config.DefaultEffect == string(EffectAllow)
}
// AddPolicy adds a policy to the engine (filerAddress ignored for memory stores)
func (e *PolicyEngine) AddPolicy(filerAddress string, name string, policy *PolicyDocument) error {
if !e.initialized {

18
weed/iam/sts/sts_service.go

@ -270,6 +270,9 @@ func (s *STSService) Initialize(config *STSConfig) error {
return fmt.Errorf(ErrConfigCannotBeNil)
}
// Apply defaults before validation
config.ApplyDefaults()
if err := s.validateConfig(config); err != nil {
return fmt.Errorf("invalid STS configuration: %w", err)
}
@ -288,6 +291,21 @@ func (s *STSService) Initialize(config *STSConfig) error {
return nil
}
// ApplyDefaults applies default values to the STS configuration
func (c *STSConfig) ApplyDefaults() {
if c.TokenDuration.Duration <= 0 {
c.TokenDuration.Duration = time.Duration(DefaultTokenDuration) * time.Second
}
if c.MaxSessionLength.Duration <= 0 {
c.MaxSessionLength.Duration = time.Duration(DefaultMaxSessionLength) * time.Second
}
if c.Issuer == "" {
c.Issuer = DefaultIssuer
}
}
// validateConfig validates the STS configuration
func (s *STSService) validateConfig(config *STSConfig) error {
if config.TokenDuration.Duration <= 0 {

23
weed/iam/sts/sts_service_test.go

@ -75,11 +75,34 @@ func TestSTSServiceInitialization(t *testing.T) {
} else {
assert.NoError(t, err)
assert.True(t, service.IsInitialized())
// Verify defaults if applicable
if tt.config.Issuer == "" {
assert.Equal(t, DefaultIssuer, service.Config.Issuer)
}
if tt.config.TokenDuration.Duration == 0 {
assert.Equal(t, time.Duration(DefaultTokenDuration)*time.Second, service.Config.TokenDuration.Duration)
}
}
})
}
}
func TestSTSServiceDefaults(t *testing.T) {
service := NewSTSService()
config := &STSConfig{
SigningKey: []byte("test-signing-key"),
// Missing duration and issuer
}
err := service.Initialize(config)
assert.NoError(t, err)
assert.Equal(t, DefaultIssuer, config.Issuer)
assert.Equal(t, time.Duration(DefaultTokenDuration)*time.Second, config.TokenDuration.Duration)
assert.Equal(t, time.Duration(DefaultMaxSessionLength)*time.Second, config.MaxSessionLength.Duration)
}
// TestAssumeRoleWithWebIdentity tests role assumption with OIDC tokens
func TestAssumeRoleWithWebIdentity(t *testing.T) {
service := setupTestSTSService(t)

11
weed/iamapi/iamapi_server.go

@ -89,7 +89,9 @@ func NewIamApiServerWithStore(router *mux.Router, option *IamServerOption, expli
GrpcDialOption: option.GrpcDialOption,
}
iam := s3api.NewIdentityAccessManagementWithStore(&s3Option, explicitStore)
// Initialize FilerClient for IAM - explicit filers only (no discovery as FilerGroup unspecified)
filerClient := wdclient.NewFilerClient(option.Filers, option.GrpcDialOption, "")
iam := s3api.NewIdentityAccessManagementWithStore(&s3Option, filerClient, explicitStore)
configure.credentialManager = iam.GetCredentialManager()
iamApiServer = &IamApiServer{
@ -100,6 +102,13 @@ func NewIamApiServerWithStore(router *mux.Router, option *IamServerOption, expli
masterClient: masterClient,
}
// Keep attempting to load configuration from filer now that we have a client
go func() {
if err := iam.LoadS3ApiConfigurationFromCredentialManager(); err != nil {
glog.Warningf("Failed to load IAM config from credential manager after client update: %v", err)
}
}()
iamApiServer.registerRouter(router)
return iamApiServer, nil

79
weed/s3api/auth_credentials.go

@ -22,6 +22,7 @@ import (
"github.com/seaweedfs/seaweedfs/weed/s3api/policy_engine"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
"github.com/seaweedfs/seaweedfs/weed/wdclient"
// Import KMS providers to register them
_ "github.com/seaweedfs/seaweedfs/weed/kms/aws"
@ -54,7 +55,7 @@ type IdentityAccessManagement struct {
domain string
isAuthEnabled bool
credentialManager *credential.CredentialManager
filerClient filer_pb.SeaweedFilerClient
filerClient *wdclient.FilerClient
grpcDialOption grpc.DialOption
// IAM Integration for advanced features
@ -132,15 +133,37 @@ func (c *Credential) isCredentialExpired() bool {
return c.Expiration > 0 && c.Expiration < time.Now().Unix()
}
func NewIdentityAccessManagement(option *S3ApiServerOption) *IdentityAccessManagement {
return NewIdentityAccessManagementWithStore(option, "")
// NewIdentityAccessManagement creates a new IAM manager
// SetFilerClient updates the filer client and its associated credential store
func (iam *IdentityAccessManagement) SetFilerClient(filerClient *wdclient.FilerClient) {
iam.m.Lock()
iam.filerClient = filerClient
iam.m.Unlock()
if iam.credentialManager == nil || filerClient == nil {
return
}
// Update credential store to use FilerClient's current filer for HA
if store := iam.credentialManager.GetStore(); store != nil {
if filerFuncSetter, ok := store.(interface {
SetFilerAddressFunc(func() pb.ServerAddress, grpc.DialOption)
}); ok {
filerFuncSetter.SetFilerAddressFunc(filerClient.GetCurrentFiler, iam.grpcDialOption)
}
}
}
func NewIdentityAccessManagement(option *S3ApiServerOption, filerClient *wdclient.FilerClient) *IdentityAccessManagement {
return NewIdentityAccessManagementWithStore(option, filerClient, "")
}
func NewIdentityAccessManagementWithStore(option *S3ApiServerOption, explicitStore string) *IdentityAccessManagement {
func NewIdentityAccessManagementWithStore(option *S3ApiServerOption, filerClient *wdclient.FilerClient, explicitStore string) *IdentityAccessManagement {
iam := &IdentityAccessManagement{
domain: option.DomainName,
hashes: make(map[string]*sync.Pool),
hashCounters: make(map[string]*int32),
filerClient: filerClient,
}
// Always initialize credential manager with fallback to defaults
@ -172,6 +195,25 @@ func NewIdentityAccessManagementWithStore(option *S3ApiServerOption, explicitSto
iam.credentialManager = credentialManager
iam.stopChan = make(chan struct{})
iam.grpcDialOption = option.GrpcDialOption
// Initialize default anonymous identity
// This ensures consistent behavior for anonymous access:
// 1. In simple auth mode (no IAM integration):
// - lookupAnonymous returns this identity
// - VerifyActionPermission checks actions (which are empty) -> Denies access
// - This preserves the secure-by-default behavior for simple auth
// 2. In advanced IAM mode (with Policy Engine):
// - lookupAnonymous returns this identity
// - VerifyActionPermission proceeds to Policy Engine
// - Policy Engine evaluates against policies (DefaultEffect=Allow if no config)
// - This enables the flexible "Open by Default" for zero-config startup
iam.identityAnonymous = &Identity{
Name: "anonymous",
Account: &AccountAnonymous,
Actions: []Action{},
IsStatic: true,
}
// First, try to load configurations from file or filer
startConfigFile := option.Config
@ -552,6 +594,16 @@ func (iam *IdentityAccessManagement) ReplaceS3ApiConfiguration(config *iam_pb.S3
}
}
// Ensure anonymous identity exists
if identityAnonymous == nil {
identityAnonymous = &Identity{
Name: "anonymous",
Account: accounts[AccountAnonymous.Id],
Actions: []Action{},
IsStatic: true,
}
}
// atomically switch
iam.identities = identities
iam.identityAnonymous = identityAnonymous
@ -572,6 +624,9 @@ func (iam *IdentityAccessManagement) ReplaceS3ApiConfiguration(config *iam_pb.S3
}
}
if !exists {
if len(envIdent.Credentials) == 0 {
continue
}
iam.identities = append(iam.identities, envIdent)
iam.accessKeyIdent[envIdent.Credentials[0].AccessKey] = envIdent
iam.nameToIdentity[envIdent.Name] = envIdent
@ -992,7 +1047,8 @@ func (iam *IdentityAccessManagement) LookupByAccessKey(accessKey string) (identi
return iam.lookupByAccessKey(accessKey)
}
func (iam *IdentityAccessManagement) lookupAnonymous() (identity *Identity, found bool) {
// LookupAnonymous returns the anonymous identity if it exists
func (iam *IdentityAccessManagement) LookupAnonymous() (identity *Identity, found bool) {
iam.m.RLock()
defer iam.m.RUnlock()
if iam.identityAnonymous != nil {
@ -1112,6 +1168,9 @@ func (iam *IdentityAccessManagement) handleAuthResult(w http.ResponseWriter, r *
// Wrapper to maintain backward compatibility
func (iam *IdentityAccessManagement) authRequest(r *http.Request, action Action) (*Identity, s3err.ErrorCode) {
identity, err, _ := iam.authRequestWithAuthType(r, action)
if err != s3err.ErrNone {
return nil, err
}
return identity, err
}
@ -1173,7 +1232,7 @@ func (iam *IdentityAccessManagement) authenticateRequestInternal(r *http.Request
}
case authTypeAnonymous:
amzAuthType = "Anonymous"
if identity, found = iam.lookupAnonymous(); !found {
if identity, found = iam.LookupAnonymous(); !found {
r.Header.Set(s3_constants.AmzAuthType, amzAuthType)
return identity, s3err.ErrAccessDenied, reqAuthType
}
@ -1212,8 +1271,8 @@ func (iam *IdentityAccessManagement) authRequestWithAuthType(r *http.Request, ac
// through buckets and checking permissions for each. Skip the global check here.
policyAllows := false
if action == s3_constants.ACTION_LIST && bucket == "" {
// ListBuckets operation - authorization handled per-bucket in the handler
if action == s3_constants.ACTION_LIST && bucket == "" && identity.Name != s3_constants.AccountAnonymousId {
// ListBuckets operation for authenticated users - authorization handled per-bucket in the handler
} else {
// First check bucket policy if one exists
// Bucket policies can grant or deny access to specific users/principals
@ -1307,8 +1366,8 @@ func (iam *IdentityAccessManagement) AuthSignatureOnly(r *http.Request) (*Identi
return identity, s3err.ErrNotImplemented
}
case authTypeAnonymous:
// Anonymous users cannot use IAM API
return identity, s3err.ErrAccessDenied
// Anonymous users can be authenticated, but authorization is handled separately
return iam.identityAnonymous, s3err.ErrNone
default:
return identity, s3err.ErrNotImplemented
}

6
weed/s3api/auth_credentials_test.go

@ -450,7 +450,7 @@ func TestNewIdentityAccessManagementWithStoreEnvVars(t *testing.T) {
option := &S3ApiServerOption{
Config: "", // No config file - this should trigger environment variable fallback
}
iam := NewIdentityAccessManagementWithStore(option, string(credential.StoreTypeMemory))
iam := NewIdentityAccessManagementWithStore(option, nil, string(credential.StoreTypeMemory))
if tt.expectEnvIdentity {
// Should have exactly one identity from environment variables
@ -510,7 +510,7 @@ func TestConfigFileWithNoIdentitiesAllowsEnvVars(t *testing.T) {
option := &S3ApiServerOption{
Config: tmpFile.Name(),
}
iam := NewIdentityAccessManagementWithStore(option, string(credential.StoreTypeMemory))
iam := NewIdentityAccessManagementWithStore(option, nil, string(credential.StoreTypeMemory))
// Should have exactly one identity from environment variables
assert.Len(t, iam.identities, 1, "Should have exactly one identity from environment variables even when config file exists with no identities")
@ -762,7 +762,7 @@ func TestSignatureVerificationDoesNotCheckPermissions(t *testing.T) {
}
func TestStaticIdentityProtection(t *testing.T) {
iam := NewIdentityAccessManagement(&S3ApiServerOption{})
iam := NewIdentityAccessManagement(&S3ApiServerOption{}, nil)
// Add a static identity
staticIdent := &Identity{

2
weed/s3api/auth_security_test.go

@ -66,7 +66,7 @@ func TestReproIssue7912(t *testing.T) {
option := &S3ApiServerOption{
Config: tmpFile.Name(),
}
iam := NewIdentityAccessManagementWithStore(option, "memory")
iam := NewIdentityAccessManagementWithStore(option, nil, "memory")
assert.True(t, iam.isEnabled(), "Auth should be enabled")

4
weed/s3api/auth_signature_v4_sts_test.go

@ -44,6 +44,10 @@ func (m *MockIAMIntegration) ValidateTrustPolicyForPrincipal(ctx context.Context
return nil
}
func (m *MockIAMIntegration) DefaultAllow() bool {
return true
}
// TestVerifyV4SignatureWithSTSIdentity tests that verifyV4Signature properly handles STS identities
// by falling back to IAM authorization when shouldCheckPermissions is true
func TestVerifyV4SignatureWithSTSIdentity(t *testing.T) {

6
weed/s3api/auth_sts_identity_test.go

@ -22,7 +22,7 @@ func TestSTSIdentityPolicyNamesPopulation(t *testing.T) {
stsService, config := setupTestSTSService(t)
// Create IAM with STS integration
iam := NewIdentityAccessManagementWithStore(&S3ApiServerOption{}, "memory")
iam := NewIdentityAccessManagementWithStore(&S3ApiServerOption{}, nil, "memory")
s3iam := &S3IAMIntegration{
stsService: stsService,
}
@ -264,7 +264,7 @@ func TestValidateSTSSessionTokenIntegration(t *testing.T) {
stsService, config := setupTestSTSService(t)
// Create IAM with STS integration
iam := NewIdentityAccessManagementWithStore(&S3ApiServerOption{}, "memory")
iam := NewIdentityAccessManagementWithStore(&S3ApiServerOption{}, nil, "memory")
s3iam := &S3IAMIntegration{
stsService: stsService,
}
@ -311,7 +311,7 @@ func TestSTSIdentityClaimsPopulation(t *testing.T) {
stsService, config := setupTestSTSService(t)
// Create IAM with STS integration
iam := NewIdentityAccessManagementWithStore(&S3ApiServerOption{}, "memory")
iam := NewIdentityAccessManagementWithStore(&S3ApiServerOption{}, nil, "memory")
s3iam := &S3IAMIntegration{
stsService: stsService,
}

156
weed/s3api/iam_defaults_test.go

@ -0,0 +1,156 @@
package s3api
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
)
func TestLoadIAMManagerFromConfig_Defaults(t *testing.T) {
// Create a temporary config file with minimal content (just policy)
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "iam_config.json")
configContent := `{
"sts": {
"providers": []
},
"policy": {
"storeType": "memory",
"defaultEffect": "Allow"
}
}`
err := os.WriteFile(configPath, []byte(configContent), 0644)
assert.NoError(t, err)
// dummy filer address provider
filerProvider := func() string { return "localhost:8888" }
defaultSigningKeyProvider := func() string { return "default-secure-signing-key" }
// Load the manager
manager, err := loadIAMManagerFromConfig(configPath, filerProvider, defaultSigningKeyProvider)
assert.NoError(t, err)
assert.NotNil(t, manager)
}
func TestLoadIAMManagerFromConfig_Overrides(t *testing.T) {
// Create a temporary config file with EXPLICIT values
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "iam_config_explicit.json")
configContent := `{
"sts": {
"tokenDuration": "2h",
"maxSessionLength": "24h",
"issuer": "custom-issuer",
"signingKey": "ZXhwbGljaXQtc2lnbmluZy1rZXktMTIzNDU="
},
"policy": {
"storeType": "memory",
"defaultEffect": "Allow"
}
}`
// Base64 encoded "explicit-signing-key-12345" is "ZXhwbGljaXQtc2lnbmluZy1rZXktMTIzNDU="
err := os.WriteFile(configPath, []byte(configContent), 0644)
assert.NoError(t, err)
filerProvider := func() string { return "localhost:8888" }
defaultSigningKeyProvider := func() string { return "default-secure-signing-key" }
// Load
manager, err := loadIAMManagerFromConfig(configPath, filerProvider, defaultSigningKeyProvider)
assert.NoError(t, err)
assert.NotNil(t, manager)
}
func TestLoadIAMManagerFromConfig_PartialDefaults(t *testing.T) {
// Test that partial configs (e.g. providing SigningKey but not Duration) work
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "iam_config_partial.json")
// Signing key provided in JSON, others missing
configContent := `{
"sts": {
"signingKey": "anNvbi1wcm92aWRlZC1rZXktMTIzNDU="
},
"policy": {
"storeType": "memory",
"defaultEffect": "Allow"
}
}`
err := os.WriteFile(configPath, []byte(configContent), 0644)
assert.NoError(t, err)
filerProvider := func() string { return "localhost:8888" }
// Default signing key provided but should be IGNORED because JSON has one
defaultSigningKeyProvider := func() string { return "server-default-key-should-be-ignored" }
manager, err := loadIAMManagerFromConfig(configPath, filerProvider, defaultSigningKeyProvider)
assert.NoError(t, err)
assert.NotNil(t, manager)
}
func TestLoadIAMManagerFromConfig_ExplicitEmptyKey(t *testing.T) {
// Test that if JSON has empty signing key string, it still falls back
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "iam_config_empty_key.json")
// Signing key explicitly empty
configContent := `{
"sts": {
"signingKey": ""
},
"policy": {
"storeType": "memory",
"defaultEffect": "Allow"
}
}`
err := os.WriteFile(configPath, []byte(configContent), 0644)
assert.NoError(t, err)
filerProvider := func() string { return "localhost:8888" }
defaultSigningKeyProvider := func() string { return "fallback-key-should-be-used" }
manager, err := loadIAMManagerFromConfig(configPath, filerProvider, defaultSigningKeyProvider)
assert.NoError(t, err)
assert.NotNil(t, manager)
}
func TestLoadIAMManagerFromConfig_MissingKeyError(t *testing.T) {
// Test that if BOTH keys are empty, it fails with a clear error
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "iam_config_all_empty.json")
// Signing key explicitly empty in JSON
configContent := `{
"sts": {
"signingKey": ""
},
"policy": {
"storeType": "memory",
"defaultEffect": "Allow"
}
}`
err := os.WriteFile(configPath, []byte(configContent), 0644)
assert.NoError(t, err)
filerProvider := func() string { return "localhost:8888" }
defaultSigningKeyProvider := func() string { return "" } // Empty default too
// Ensure no SSE-S3 key interferes (global state in tests is tricky, but let's assume clean state or no mock)
// Ideally we would mock GetSSES3KeyManager().GetMasterKey() but it's a global singleton.
// For this unit test, if the global key manager has no key, it should fail.
_, err = loadIAMManagerFromConfig(configPath, filerProvider, defaultSigningKeyProvider)
// Should return a clear error
assert.Error(t, err)
assert.Contains(t, err.Error(), "no signing key found for STS service")
}

50
weed/s3api/iam_optional_test.go

@ -0,0 +1,50 @@
package s3api
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestLoadIAMManagerFromConfig_OptionalConfig(t *testing.T) {
// Mock dependencies
filerAddressProvider := func() string { return "localhost:8888" }
getFilerSigningKey := func() string { return "test-signing-key" }
// Test Case 1: Empty config path should load defaults
iamManager, err := loadIAMManagerFromConfig("", filerAddressProvider, getFilerSigningKey)
require.NoError(t, err)
require.NotNil(t, iamManager)
// Verify STS Service is initialized with defaults
stsService := iamManager.GetSTSService()
assert.NotNil(t, stsService)
// Verify defaults are applied
// Since we can't easily access the internal config of stsService,
// we rely on the fact that initialization succeeded without error.
// We can also verify that the policy engine uses memory store by default.
// Verify Policy Engine is initialized with defaults (Memory store, Deny effect)
// Again, internal state might be hard to access directly, but successful init implies defaults worked.
}
func TestLoadIAMManagerFromConfig_EmptyConfigWithFallbackKey(t *testing.T) {
// Mock dependencies where getFilerSigningKey returns empty, forcing fallback logic
// Initialize IAM with empty config (should trigger defaults)
// We pass empty string for config file path
option := &S3ApiServerOption{
Config: "",
IamConfig: "",
EnableIam: true,
}
iamManager := NewIdentityAccessManagementWithStore(option, nil, "memory")
// Verify identityAnonymous is initialized
// This confirms the fix for anonymous access in zero-config mode
anonIdentity, found := iamManager.LookupAnonymous()
assert.True(t, found, "Anonymous identity should be found by default")
assert.NotNil(t, anonIdentity, "Anonymous identity should not be nil")
assert.Equal(t, "anonymous", anonIdentity.Name)
}

32
weed/s3api/iceberg/server.go

@ -18,6 +18,7 @@ type FilerClient interface {
type S3Authenticator interface {
AuthenticateRequest(r *http.Request) (string, interface{}, s3err.ErrorCode)
DefaultAllow() bool
}
// Server implements the Iceberg REST Catalog API.
@ -128,20 +129,25 @@ func (s *Server) Auth(handler http.HandlerFunc) http.HandlerFunc {
identityName, identity, errCode := s.authenticator.AuthenticateRequest(r)
if errCode != s3err.ErrNone {
apiErr := s3err.GetAPIError(errCode)
errorType := "RESTException"
switch apiErr.HTTPStatusCode {
case http.StatusForbidden:
errorType = "ForbiddenException"
case http.StatusUnauthorized:
errorType = "NotAuthorizedException"
case http.StatusBadRequest:
errorType = "BadRequestException"
case http.StatusInternalServerError:
errorType = "InternalServerError"
// If authentication failed but DefaultAllow is enabled, proceed without identity
if s.authenticator.DefaultAllow() {
glog.V(2).Infof("Iceberg: AuthenticateRequest failed (%v), but DefaultAllow is true, proceeding", errCode)
} else {
apiErr := s3err.GetAPIError(errCode)
errorType := "RESTException"
switch apiErr.HTTPStatusCode {
case http.StatusForbidden:
errorType = "ForbiddenException"
case http.StatusUnauthorized:
errorType = "NotAuthorizedException"
case http.StatusBadRequest:
errorType = "BadRequestException"
case http.StatusInternalServerError:
errorType = "InternalServerError"
}
writeError(w, apiErr.HTTPStatusCode, errorType, apiErr.Description)
return
}
writeError(w, apiErr.HTTPStatusCode, errorType, apiErr.Description)
return
}
if identityName != "" || identity != nil {

9
weed/s3api/s3_iam_middleware.go

@ -44,6 +44,7 @@ type IAMIntegration interface {
AuthorizeAction(ctx context.Context, identity *IAMIdentity, action Action, bucket string, objectKey string, r *http.Request) s3err.ErrorCode
ValidateSessionToken(ctx context.Context, token string) (*sts.SessionInfo, error)
ValidateTrustPolicyForPrincipal(ctx context.Context, roleArn, principalArn string) error
DefaultAllow() bool
}
// S3IAMIntegration provides IAM integration for S3 API
@ -310,6 +311,14 @@ func (s3iam *S3IAMIntegration) ValidateTrustPolicyForPrincipal(ctx context.Conte
return s3iam.iamManager.ValidateTrustPolicyForPrincipal(ctx, roleArn, principalArn)
}
// DefaultAllow returns whether access is allowed by default when no policy is found
func (s3iam *S3IAMIntegration) DefaultAllow() bool {
if s3iam.iamManager == nil {
return true // Default to true if IAM is not enabled
}
return s3iam.iamManager.DefaultAllow()
}
// IAMIdentity represents an authenticated identity with session information
type IAMIdentity struct {
Name string

52
weed/s3api/s3_sse_s3.go

@ -5,6 +5,7 @@ import (
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
@ -19,9 +20,13 @@ import (
"time"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/util"
"github.com/seaweedfs/seaweedfs/weed/wdclient"
"golang.org/x/crypto/hkdf"
"google.golang.org/grpc"
)
// SSE-S3 uses AES-256 encryption with server-managed keys
@ -452,6 +457,27 @@ func (km *SSES3KeyManager) GetKey(keyID string) (*SSES3Key, bool) {
return nil, false
}
// GetMasterKey returns a derived key from the master KEK for STS signing
// This uses HKDF to isolate the STS security domain from the SSE-S3 domain
func (km *SSES3KeyManager) GetMasterKey() []byte {
km.mu.RLock()
defer km.mu.RUnlock()
if len(km.superKey) == 0 {
return nil
}
// Derive a separate key for STS to isolate security domains
// We use the KEK as the secret, and "seaweedfs-sts-signing-key" as the info
hkdfReader := hkdf.New(sha256.New, km.superKey, nil, []byte("seaweedfs-sts-signing-key"))
derived := make([]byte, 32) // 256-bit derived key
if _, err := io.ReadFull(hkdfReader, derived); err != nil {
glog.Errorf("Failed to derive STS key: %v", err)
return nil
}
return derived
}
// Global SSE-S3 key manager instance
var globalSSES3KeyManager = NewSSES3KeyManager()
@ -460,9 +486,31 @@ func GetSSES3KeyManager() *SSES3KeyManager {
return globalSSES3KeyManager
}
// KeyManagerFilerClient wraps wdclient.FilerClient to satisfy filer_pb.FilerClient interface
type KeyManagerFilerClient struct {
*wdclient.FilerClient
grpcDialOption grpc.DialOption
}
func (k *KeyManagerFilerClient) AdjustedUrl(location *filer_pb.Location) string {
return location.Url
}
func (k *KeyManagerFilerClient) WithFilerClient(streamingMode bool, fn func(filer_pb.SeaweedFilerClient) error) error {
filerAddress := k.GetCurrentFiler()
if filerAddress == "" {
return fmt.Errorf("no filer available")
}
return pb.WithGrpcFilerClient(streamingMode, 0, filerAddress, k.grpcDialOption, fn)
}
// InitializeGlobalSSES3KeyManager initializes the global key manager with filer access
func InitializeGlobalSSES3KeyManager(s3ApiServer *S3ApiServer) error {
return globalSSES3KeyManager.InitializeWithFiler(s3ApiServer)
func InitializeGlobalSSES3KeyManager(filerClient *wdclient.FilerClient, grpcDialOption grpc.DialOption) error {
wrapper := &KeyManagerFilerClient{
FilerClient: filerClient,
grpcDialOption: grpcDialOption,
}
return globalSSES3KeyManager.InitializeWithFiler(wrapper)
}
// ProcessSSES3Request processes an SSE-S3 request and returns encryption metadata

6
weed/s3api/s3api_put_object_helper_test.go

@ -13,7 +13,7 @@ import (
func TestGetRequestDataReader_ChunkedEncodingWithoutIAM(t *testing.T) {
// Create an S3ApiServer with IAM disabled
s3a := &S3ApiServer{
iam: NewIdentityAccessManagementWithStore(&S3ApiServerOption{}, string(credential.StoreTypeMemory)),
iam: NewIdentityAccessManagementWithStore(&S3ApiServerOption{}, nil, string(credential.StoreTypeMemory)),
}
// Ensure IAM is disabled for this test
s3a.iam.isAuthEnabled = false
@ -87,7 +87,7 @@ func TestGetRequestDataReader_ChunkedEncodingWithoutIAM(t *testing.T) {
func TestGetRequestDataReader_AuthTypeDetection(t *testing.T) {
// Create an S3ApiServer with IAM disabled
s3a := &S3ApiServer{
iam: NewIdentityAccessManagementWithStore(&S3ApiServerOption{}, string(credential.StoreTypeMemory)),
iam: NewIdentityAccessManagementWithStore(&S3ApiServerOption{}, nil, string(credential.StoreTypeMemory)),
}
s3a.iam.isAuthEnabled = false
@ -122,7 +122,7 @@ func TestGetRequestDataReader_AuthTypeDetection(t *testing.T) {
func TestGetRequestDataReader_IAMEnabled(t *testing.T) {
// Create an S3ApiServer with IAM enabled
s3a := &S3ApiServer{
iam: NewIdentityAccessManagementWithStore(&S3ApiServerOption{}, string(credential.StoreTypeMemory)),
iam: NewIdentityAccessManagementWithStore(&S3ApiServerOption{}, nil, string(credential.StoreTypeMemory)),
}
s3a.iam.isAuthEnabled = true

134
weed/s3api/s3api_server.go

@ -110,7 +110,8 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
option.AllowedOrigins = domains
}
iam := NewIdentityAccessManagementWithStore(option, explicitStore)
// Initialize basic/legacy IAM - filerClient not available yet, passed as nil
iam := NewIdentityAccessManagementWithStore(option, nil, explicitStore)
// Initialize bucket policy engine first
policyEngine := NewBucketPolicyEngine()
@ -146,16 +147,24 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
glog.V(1).Infof("S3 API initialized FilerClient with %d filer(s) (no discovery)", len(option.Filers))
}
// Initialize Global SSE-S3 Key Manager early so it's available for IAM fallback
// This ensures we can access the KEK for STS signing key if needed
if err := InitializeGlobalSSES3KeyManager(filerClient, option.GrpcDialOption); err != nil {
glog.Errorf("Failed to initialize SSE-S3 Key Manager: %v", err)
// We continue, as this might be a transient failure or non-critical for some setups,
// but IAM fallback to KEK will fail if this didn't succeed.
}
// Update credential store to use FilerClient's current filer for HA
if store := iam.credentialManager.GetStore(); store != nil {
if filerFuncSetter, ok := store.(interface {
SetFilerAddressFunc(func() pb.ServerAddress, grpc.DialOption)
}); ok {
// Use FilerClient's GetCurrentFiler for true HA
filerFuncSetter.SetFilerAddressFunc(filerClient.GetCurrentFiler, option.GrpcDialOption)
glog.V(1).Infof("Updated credential store to use FilerClient's current active filer (HA-aware)")
iam.SetFilerClient(filerClient)
// Keep attempting to load configuration from filer now that we have a client
// The initial load in NewIdentityAccessManagementWithStore might have failed if client was nil
go func() {
if err := iam.loadS3ApiConfigurationFromFiler(option); err != nil {
glog.Warningf("Failed to load IAM config from filer after client update: %v", err)
}
}
}()
s3ApiServer = &S3ApiServer{
option: option,
@ -178,19 +187,25 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
// This avoids circular dependency by not passing the entire S3ApiServer
iam.policyEngine = policyEngine
// Initialize advanced IAM system if config is provided
if option.IamConfig != "" {
glog.V(1).Infof("Loading advanced IAM configuration from: %s", option.IamConfig)
// Initialize advanced IAM system if config is provided or explicitly enabled
if option.IamConfig != "" || option.EnableIam {
configSource := "defaults"
if option.IamConfig != "" {
configSource = option.IamConfig
}
glog.V(1).Infof("Loading advanced IAM configuration from: %s", configSource)
// Use FilerClient's GetCurrentFiler for HA-aware filer selection
iamManager, err := loadIAMManagerFromConfig(option.IamConfig, func() string {
return string(filerClient.GetCurrentFiler())
}, func() string {
return signingKey
})
if err != nil {
glog.Errorf("Failed to load IAM configuration: %v", err)
} else {
if iam.credentialManager != nil {
iamManager.SetUserStore(iam.credentialManager)
if s3ApiServer.iam.credentialManager != nil {
iamManager.SetUserStore(s3ApiServer.iam.credentialManager)
}
glog.V(1).Infof("IAM Manager loaded, creating integration")
// Create S3 IAM integration with the loaded IAM manager
@ -233,6 +248,10 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
})
}
s3ApiServer.bucketRegistry = NewBucketRegistry(s3ApiServer)
// Update IAM with the final filer client (already handled by SetFilerClient above,
// but this reinforces it if we ever change the flow)
s3ApiServer.iam.SetFilerClient(s3ApiServer.filerClient)
if option.LocalFilerSocket == "" {
if s3ApiServer.client, err = util_http.NewGlobalHttpClient(); err != nil {
return nil, err
@ -249,11 +268,6 @@ func NewS3ApiServerWithStore(router *mux.Router, option *S3ApiServerOption, expl
s3ApiServer.registerRouter(router)
// Initialize the global SSE-S3 key manager with filer access
if err := InitializeGlobalSSES3KeyManager(s3ApiServer); err != nil {
return nil, fmt.Errorf("failed to initialize SSE-S3 key manager: %w", err)
}
go s3ApiServer.subscribeMetaEvents("s3", startTsNs, filer.DirectoryEtcRoot, []string{
option.BucketsPath,
filer.IamConfigDirectory,
@ -830,14 +844,7 @@ func (s3a *S3ApiServer) registerRouter(router *mux.Router) {
}
// loadIAMManagerFromConfig loads the advanced IAM manager from configuration file
func loadIAMManagerFromConfig(configPath string, filerAddressProvider func() string) (*integration.IAMManager, error) {
// Read configuration file
configData, err := os.ReadFile(configPath)
if err != nil {
return nil, fmt.Errorf("failed to read config file: %w", err)
}
// Parse configuration structure
func loadIAMManagerFromConfig(configPath string, filerAddressProvider func() string, getFilerSigningKey func() string) (*integration.IAMManager, error) {
var configRoot struct {
STS *sts.STSConfig `json:"sts"`
Policy *policy.PolicyEngineConfig `json:"policy"`
@ -849,24 +856,43 @@ func loadIAMManagerFromConfig(configPath string, filerAddressProvider func() str
} `json:"policies"`
}
if err := json.Unmarshal(configData, &configRoot); err != nil {
return nil, fmt.Errorf("failed to parse config: %w", err)
if configPath != "" {
// Read configuration file
configData, err := os.ReadFile(configPath)
if err != nil {
return nil, fmt.Errorf("failed to read config file: %w", err)
}
if err := json.Unmarshal(configData, &configRoot); err != nil {
return nil, fmt.Errorf("failed to parse config: %w", err)
}
} else {
glog.V(1).Infof("No IAM config file provided; using defaults")
// Initialize with empty config which will trigger defaults below
}
// Ensure STS config exists so we can apply defaults later
if configRoot.STS == nil {
configRoot.STS = &sts.STSConfig{}
}
// Ensure a valid policy engine config exists
if configRoot.Policy == nil {
// Provide a secure default if not specified in the config file
// Default to Deny with in-memory store so that JSON-defined policies work without filer
glog.V(1).Infof("No policy engine config provided; using defaults (DefaultEffect=%s, StoreType=%s)", sts.EffectDeny, sts.StoreTypeMemory)
configRoot.Policy = &policy.PolicyEngineConfig{
DefaultEffect: sts.EffectDeny,
StoreType: sts.StoreTypeMemory,
}
} else if configRoot.Policy.StoreType == "" {
// If policy config exists but storeType is not specified, use memory store
// This ensures JSON-defined policies are stored in memory and work correctly
configRoot.Policy = &policy.PolicyEngineConfig{}
}
if configRoot.Policy.StoreType == "" {
configRoot.Policy.StoreType = sts.StoreTypeMemory
glog.V(1).Infof("Policy storeType not specified; using memory store for JSON config-based setup")
}
if configRoot.Policy.DefaultEffect == "" {
// Default to Allow (open) with in-memory store so that
// users can start using STS without locking themselves out immediately.
// For other stores (e.g. filer), default to Deny (closed) for security.
if configRoot.Policy.StoreType == sts.StoreTypeMemory {
configRoot.Policy.DefaultEffect = sts.EffectAllow
} else {
configRoot.Policy.DefaultEffect = sts.EffectDeny
}
glog.V(1).Infof("Using policy defaults: DefaultEffect=%s, StoreType=%s", configRoot.Policy.DefaultEffect, configRoot.Policy.StoreType)
}
// Create IAM configuration
@ -878,6 +904,26 @@ func loadIAMManagerFromConfig(configPath string, filerAddressProvider func() str
},
}
// Apply default signing key if not present in config
if iamConfig.STS != nil && len(iamConfig.STS.SigningKey) == 0 {
// 1. Try server-configured signing key (security.toml / CLI)
if key := getFilerSigningKey(); key != "" {
iamConfig.STS.SigningKey = []byte(key)
glog.V(1).Infof("Using default filer signing key for STS service")
} else {
// 2. Try cluster-wide SSE-S3 Master Key (KEK) from Filer
// This ensures zero-config consistency across the cluster
if kek := GetSSES3KeyManager().GetMasterKey(); len(kek) > 0 {
iamConfig.STS.SigningKey = kek
glog.V(1).Infof("Using SSE-S3 Master Key (KEK) for STS service")
} else {
// 3. Fail if no signing key is available
// This ensures consistency across multiple S3 servers and secure operation
return nil, fmt.Errorf("no signing key found for STS service; please provide 'signingKey' in IAM config, configure 'jwt.filer_signing.key' in security.toml, or ensure SSE-S3 is initialized")
}
}
}
// Initialize IAM manager
iamManager := integration.NewIAMManager()
if err := iamManager.Initialize(iamConfig, filerAddressProvider); err != nil {
@ -960,3 +1006,11 @@ func (s3a *S3ApiServer) AuthenticateRequest(r *http.Request) (string, interface{
}
return "", nil, err
}
// DefaultAllow returns whether access is allowed by default when no policy is found
func (s3a *S3ApiServer) DefaultAllow() bool {
if s3a.iam == nil || s3a.iam.iamIntegration == nil {
return false
}
return s3a.iam.iamIntegration.DefaultAllow()
}

2
weed/s3api/s3api_server_routing_test.go

@ -16,7 +16,7 @@ import (
// setupRoutingTestServer creates a minimal S3ApiServer for routing tests
func setupRoutingTestServer(t *testing.T) *S3ApiServer {
opt := &S3ApiServerOption{EnableIam: true}
iam := NewIdentityAccessManagementWithStore(opt, "memory")
iam := NewIdentityAccessManagementWithStore(opt, nil, "memory")
iam.isAuthEnabled = true
if iam.credentialManager == nil {

23
weed/s3api/s3api_tables.go

@ -43,6 +43,11 @@ func (st *S3TablesApiServer) SetAccountID(accountID string) {
st.handler.SetAccountID(accountID)
}
// SetDefaultAllow sets whether to allow access by default
func (st *S3TablesApiServer) SetDefaultAllow(allow bool) {
st.handler.SetDefaultAllow(allow)
}
// S3TablesHandler handles S3 Tables API requests
func (st *S3TablesApiServer) S3TablesHandler(w http.ResponseWriter, r *http.Request) {
st.handler.HandleRequest(w, r, st)
@ -57,6 +62,12 @@ func (st *S3TablesApiServer) WithFilerClient(streamingMode bool, fn func(filer_p
func (s3a *S3ApiServer) registerS3TablesRoutes(router *mux.Router) {
// Create S3 Tables handler
s3TablesApi := NewS3TablesApiServer(s3a)
if s3a.iam != nil && s3a.iam.iamIntegration != nil {
s3TablesApi.SetDefaultAllow(s3a.iam.iamIntegration.DefaultAllow())
} else {
// If IAM is not configured, allow all access by default
s3TablesApi.SetDefaultAllow(true)
}
// Regex for S3 Tables Bucket ARN
const tableBucketARNRegex = "arn:aws:s3tables:[^/:]*:[^/:]*:bucket/[^/]+"
@ -618,9 +629,15 @@ func (s3a *S3ApiServer) authenticateS3Tables(f http.HandlerFunc) http.HandlerFun
// Use AuthSignatureOnly to authenticate the request without authorizing specific actions
identity, errCode := s3a.iam.AuthSignatureOnly(r)
if errCode != s3err.ErrNone {
glog.Errorf("S3Tables: AuthSignatureOnly failed: %v", errCode)
s3err.WriteErrorResponse(w, r, errCode)
return
// If IAM is enabled but DefaultAllow is true, we can proceed even if unauthenticated
// authorization checks in handlers will then use DefaultAllow logic.
if s3a.iam.iamIntegration != nil && s3a.iam.iamIntegration.DefaultAllow() {
glog.V(2).Infof("S3Tables: AuthSignatureOnly failed (%v), but DefaultAllow is true, proceeding", errCode)
} else {
glog.Errorf("S3Tables: AuthSignatureOnly failed: %v", errCode)
s3err.WriteErrorResponse(w, r, errCode)
return
}
}
// Store the authenticated identity in request context

15
weed/s3api/s3tables/handler.go

@ -44,15 +44,17 @@ const (
// S3TablesHandler handles S3 Tables API requests
type S3TablesHandler struct {
region string
accountID string
region string
accountID string
defaultAllow bool // Whether to allow access by default (for zero-config IAM)
}
// NewS3TablesHandler creates a new S3 Tables handler
func NewS3TablesHandler() *S3TablesHandler {
return &S3TablesHandler{
region: DefaultRegion,
accountID: DefaultAccountID,
region: DefaultRegion,
accountID: DefaultAccountID,
defaultAllow: false,
}
}
@ -70,6 +72,11 @@ func (h *S3TablesHandler) SetAccountID(accountID string) {
}
}
// SetDefaultAllow sets whether to allow access by default
func (h *S3TablesHandler) SetDefaultAllow(allow bool) {
h.defaultAllow = allow
}
// FilerClient interface for filer operations
type FilerClient interface {
WithFilerClient(streamingMode bool, fn func(client filer_pb.SeaweedFilerClient) error) error

4
weed/s3api/s3tables/handler_bucket_create.go

@ -16,7 +16,9 @@ import (
func (h *S3TablesHandler) handleCreateTableBucket(w http.ResponseWriter, r *http.Request, filerClient FilerClient) error {
// Check permission
principal := h.getAccountID(r)
if !CanCreateTableBucket(principal, principal, "") {
if !CheckPermissionWithContext("CreateTableBucket", principal, principal, "", "", &PolicyContext{
DefaultAllow: h.defaultAllow,
}) {
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to create table buckets")
return NewAuthError("CreateTableBucket", principal, "not authorized to create table buckets")
}

4
weed/s3api/s3tables/handler_bucket_get_list_delete.go

@ -72,6 +72,7 @@ func (h *S3TablesHandler) handleGetTableBucket(w http.ResponseWriter, r *http.Re
if !CheckPermissionWithContext("GetTableBucket", principal, metadata.OwnerAccountID, bucketPolicy, bucketARN, &PolicyContext{
TableBucketName: bucketName,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to get table bucket details")
return ErrAccessDenied
@ -101,6 +102,7 @@ func (h *S3TablesHandler) handleListTableBuckets(w http.ResponseWriter, r *http.
identityActions := getIdentityActions(r)
if !CheckPermissionWithContext("ListTableBuckets", principal, accountID, "", "", &PolicyContext{
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to list table buckets")
return NewAuthError("ListTableBuckets", principal, "not authorized to list table buckets")
@ -198,6 +200,7 @@ func (h *S3TablesHandler) handleListTableBuckets(w http.ResponseWriter, r *http.
if !CheckPermissionWithContext("GetTableBucket", accountID, metadata.OwnerAccountID, bucketPolicy, bucketARN, &PolicyContext{
TableBucketName: entry.Entry.Name,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
continue
}
@ -300,6 +303,7 @@ func (h *S3TablesHandler) handleDeleteTableBucket(w http.ResponseWriter, r *http
if !CheckPermissionWithContext("DeleteTableBucket", principal, metadata.OwnerAccountID, bucketPolicy, bucketARN, &PolicyContext{
TableBucketName: bucketName,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
return NewAuthError("DeleteTableBucket", principal, fmt.Sprintf("not authorized to delete bucket %s", bucketName))
}

4
weed/s3api/s3tables/handler_namespace.go

@ -118,6 +118,7 @@ func (h *S3TablesHandler) handleCreateNamespace(w http.ResponseWriter, r *http.R
Namespace: namespaceName,
TableBucketTags: bucketTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
glog.Infof("S3Tables: Permission denied for CreateNamespace - principal=%s, owner=%s", principal, bucketMetadata.OwnerAccountID)
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to create namespace in this bucket")
@ -258,6 +259,7 @@ func (h *S3TablesHandler) handleGetNamespace(w http.ResponseWriter, r *http.Requ
Namespace: namespaceName,
TableBucketTags: bucketTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchNamespace, "namespace not found")
return ErrAccessDenied
@ -344,6 +346,7 @@ func (h *S3TablesHandler) handleListNamespaces(w http.ResponseWriter, r *http.Re
TableBucketName: bucketName,
TableBucketTags: bucketTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchBucket, fmt.Sprintf("table bucket %s not found", bucketName))
return ErrAccessDenied
@ -528,6 +531,7 @@ func (h *S3TablesHandler) handleDeleteNamespace(w http.ResponseWriter, r *http.R
Namespace: namespaceName,
TableBucketTags: bucketTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
h.writeError(w, http.StatusNotFound, ErrCodeNoSuchNamespace, "namespace not found")
return ErrAccessDenied

9
weed/s3api/s3tables/handler_policy.go

@ -94,6 +94,7 @@ func (h *S3TablesHandler) handlePutTableBucketPolicy(w http.ResponseWriter, r *h
if !CheckPermissionWithContext("PutTableBucketPolicy", principal, bucketMetadata.OwnerAccountID, "", bucketARN, &PolicyContext{
TableBucketName: bucketName,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to put table bucket policy")
return NewAuthError("PutTableBucketPolicy", principal, "not authorized to put table bucket policy")
@ -171,6 +172,7 @@ func (h *S3TablesHandler) handleGetTableBucketPolicy(w http.ResponseWriter, r *h
if !CheckPermissionWithContext("GetTableBucketPolicy", principal, bucketMetadata.OwnerAccountID, string(policy), bucketARN, &PolicyContext{
TableBucketName: bucketName,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to get table bucket policy")
return NewAuthError("GetTableBucketPolicy", principal, "not authorized to get table bucket policy")
@ -246,6 +248,7 @@ func (h *S3TablesHandler) handleDeleteTableBucketPolicy(w http.ResponseWriter, r
if !CheckPermissionWithContext("DeleteTableBucketPolicy", principal, bucketMetadata.OwnerAccountID, bucketPolicy, bucketARN, &PolicyContext{
TableBucketName: bucketName,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to delete table bucket policy")
return NewAuthError("DeleteTableBucketPolicy", principal, "not authorized to delete table bucket policy")
@ -346,6 +349,7 @@ func (h *S3TablesHandler) handlePutTablePolicy(w http.ResponseWriter, r *http.Re
Namespace: namespaceName,
TableName: tableName,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to put table policy")
return NewAuthError("PutTablePolicy", principal, "not authorized to put table policy")
@ -453,6 +457,7 @@ func (h *S3TablesHandler) handleGetTablePolicy(w http.ResponseWriter, r *http.Re
Namespace: namespaceName,
TableName: tableName,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to get table policy")
return NewAuthError("GetTablePolicy", principal, "not authorized to get table policy")
@ -542,6 +547,7 @@ func (h *S3TablesHandler) handleDeleteTablePolicy(w http.ResponseWriter, r *http
Namespace: namespaceName,
TableName: tableName,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to delete table policy")
return NewAuthError("DeleteTablePolicy", principal, "not authorized to delete table policy")
@ -640,6 +646,7 @@ func (h *S3TablesHandler) handleTagResource(w http.ResponseWriter, r *http.Reque
TagKeys: requestTagKeys,
ResourceTags: existingTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
return NewAuthError("TagResource", principal, "not authorized to tag resource")
}
@ -757,6 +764,7 @@ func (h *S3TablesHandler) handleListTagsForResource(w http.ResponseWriter, r *ht
TableBucketTags: bucketTags,
ResourceTags: tags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
return NewAuthError("ListTagsForResource", principal, "not authorized to list tags for resource")
}
@ -864,6 +872,7 @@ func (h *S3TablesHandler) handleUntagResource(w http.ResponseWriter, r *http.Req
TagKeys: req.TagKeys,
ResourceTags: tags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
return NewAuthError("UntagResource", principal, "not authorized to untag resource")
}

11
weed/s3api/s3tables/handler_table.go

@ -145,6 +145,7 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque
TagKeys: mapKeys(req.Tags),
TableBucketTags: bucketTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
})
bucketAllowed := CheckPermissionWithContext("CreateTable", accountID, bucketMetadata.OwnerAccountID, bucketPolicy, bucketARN, &PolicyContext{
TableBucketName: bucketName,
@ -154,6 +155,7 @@ func (h *S3TablesHandler) handleCreateTable(w http.ResponseWriter, r *http.Reque
TagKeys: mapKeys(req.Tags),
TableBucketTags: bucketTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
})
if !nsAllowed && !bucketAllowed {
@ -390,6 +392,7 @@ func (h *S3TablesHandler) handleGetTable(w http.ResponseWriter, r *http.Request,
TableBucketTags: bucketTags,
ResourceTags: tableTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
})
bucketAllowed := CheckPermissionWithContext("GetTable", accountID, bucketMetadata.OwnerAccountID, bucketPolicy, bucketARN, &PolicyContext{
TableBucketName: bucketName,
@ -398,6 +401,7 @@ func (h *S3TablesHandler) handleGetTable(w http.ResponseWriter, r *http.Request,
TableBucketTags: bucketTags,
ResourceTags: tableTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
})
if !tableAllowed && !bucketAllowed {
@ -527,12 +531,14 @@ func (h *S3TablesHandler) handleListTables(w http.ResponseWriter, r *http.Reques
Namespace: namespaceName,
TableBucketTags: bucketTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
})
bucketAllowed := CheckPermissionWithContext("ListTables", accountID, bucketMeta.OwnerAccountID, bucketPolicy, bucketARN, &PolicyContext{
TableBucketName: bucketName,
Namespace: namespaceName,
TableBucketTags: bucketTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
})
if !nsAllowed && !bucketAllowed {
return ErrAccessDenied
@ -574,6 +580,7 @@ func (h *S3TablesHandler) handleListTables(w http.ResponseWriter, r *http.Reques
TableBucketName: bucketName,
TableBucketTags: bucketTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
}) {
return ErrAccessDenied
}
@ -910,6 +917,7 @@ func (h *S3TablesHandler) handleDeleteTable(w http.ResponseWriter, r *http.Reque
TableBucketTags: bucketTags,
ResourceTags: tableTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
})
bucketAllowed := CheckPermissionWithContext("DeleteTable", principal, bucketMetadata.OwnerAccountID, bucketPolicy, bucketARN, &PolicyContext{
TableBucketName: bucketName,
@ -918,6 +926,7 @@ func (h *S3TablesHandler) handleDeleteTable(w http.ResponseWriter, r *http.Reque
TableBucketTags: bucketTags,
ResourceTags: tableTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
})
if !tableAllowed && !bucketAllowed {
h.writeError(w, http.StatusForbidden, ErrCodeAccessDenied, "not authorized to delete table")
@ -1053,6 +1062,7 @@ func (h *S3TablesHandler) handleUpdateTable(w http.ResponseWriter, r *http.Reque
TableBucketTags: bucketTags,
ResourceTags: tableTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
})
bucketAllowed := CheckPermissionWithContext("UpdateTable", principal, bucketMetadata.OwnerAccountID, bucketPolicy, bucketARN, &PolicyContext{
TableBucketName: bucketName,
@ -1061,6 +1071,7 @@ func (h *S3TablesHandler) handleUpdateTable(w http.ResponseWriter, r *http.Reque
TableBucketTags: bucketTags,
ResourceTags: tableTags,
IdentityActions: identityActions,
DefaultAllow: h.defaultAllow,
})
if !tableAllowed && !bucketAllowed {

10
weed/s3api/s3tables/manager.go

@ -20,7 +20,10 @@ type Manager struct {
// NewManager creates a new Manager.
func NewManager() *Manager {
return &Manager{handler: NewS3TablesHandler()}
m := &Manager{handler: NewS3TablesHandler()}
// Default to allowing access when IAM is not configured
m.handler.SetDefaultAllow(true)
return m
}
// SetRegion sets the AWS region for ARN generation.
@ -33,6 +36,11 @@ func (m *Manager) SetAccountID(accountID string) {
m.handler.SetAccountID(accountID)
}
// SetDefaultAllow sets whether to allow access by default.
func (m *Manager) SetDefaultAllow(allow bool) {
m.handler.SetDefaultAllow(allow)
}
// Execute runs an S3 Tables operation and decodes the response into resp (if provided).
func (m *Manager) Execute(ctx context.Context, filerClient FilerClient, operation string, req interface{}, resp interface{}, identity string) error {
body, err := json.Marshal(req)

27
weed/s3api/s3tables/permissions.go

@ -86,6 +86,7 @@ type PolicyContext struct {
SSEAlgorithm string
KMSKeyArn string
StorageClass string
DefaultAllow bool
}
// CheckPermissionWithResource checks if a principal has permission to perform an operation on a specific resource
@ -117,17 +118,30 @@ func CheckPermissionWithContext(operation, principal, owner, resourcePolicy, res
}
func checkPermission(operation, principal, owner, resourcePolicy, resourceARN string, ctx *PolicyContext) bool {
fmt.Printf("DEBUG: checkPermission op=%s princ=%s owner=%s policyLen=%d defaultAllow=%v\n",
operation, principal, owner, len(resourcePolicy), ctx != nil && ctx.DefaultAllow)
if resourcePolicy != "" {
fmt.Printf("DEBUG: policy content: %s\n", resourcePolicy)
}
// Owner always has permission
if principal == owner {
fmt.Printf("DEBUG: Allowed by Owner check\n")
return true
}
if hasIdentityPermission(operation, ctx) {
fmt.Printf("DEBUG: Allowed by Identity check\n")
return true
}
// If no policy is provided, deny access (default deny)
// If no policy is provided, use default allow if enabled
if resourcePolicy == "" {
if ctx != nil && ctx.DefaultAllow {
fmt.Printf("DEBUG: Allowed by DefaultAllow\n")
return true
}
fmt.Printf("DEBUG: Denied by DefaultAllow=false (no policy)\n")
return false
}
@ -177,7 +191,16 @@ func checkPermission(operation, principal, owner, resourcePolicy, resourceARN st
}
}
return hasAllow
if hasAllow {
return true
}
// If no statement matched, use default allow if enabled
if ctx != nil && ctx.DefaultAllow {
return true
}
return false
}
func hasIdentityPermission(operation string, ctx *PolicyContext) bool {

45
weed/s3api/s3tables/permissions_test.go

@ -206,3 +206,48 @@ func TestEvaluatePolicyWithConditions(t *testing.T) {
})
}
}
func TestCheckPermissionWithDefaultAllow(t *testing.T) {
tests := []struct {
name string
defaultAllow bool
policy string
expected bool
}{
{
"default deny (no policy, DefaultAllow=false)",
false,
"",
false,
},
{
"default allow (no policy, DefaultAllow=true)",
true,
"",
true,
},
{
"explicit deny overrides DefaultAllow=true",
true,
`{"Statement":[{"Effect":"Deny","Principal":"*","Action":"s3tables:GetTable"}]}`,
false,
},
{
"explicit allow works with DefaultAllow=false",
false,
`{"Statement":[{"Effect":"Allow","Principal":"*","Action":"s3tables:GetTable"}]}`,
true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
ctx := &PolicyContext{
DefaultAllow: tt.defaultAllow,
}
result := CheckPermissionWithContext("s3tables:GetTable", "user123", "owner123", tt.policy, "", ctx)
if result != tt.expected {
t.Errorf("CheckPermissionWithContext() = %v, want %v (DefaultAllow=%v, Policy=%s)", result, tt.expected, tt.defaultAllow, tt.policy)
}
})
}
}

3
weed/s3api/sts_params_test.go

@ -41,6 +41,9 @@ func (m *mockIAMIntegration) ValidateTrustPolicyForPrincipal(ctx context.Context
func (m *mockIAMIntegration) ValidateSessionToken(ctx context.Context, token string) (*sts.SessionInfo, error) {
return nil, nil
}
func (m *mockIAMIntegration) DefaultAllow() bool {
return true
}
func TestSTSAssumeRolePostBody(t *testing.T) {
// Setup S3ApiServer with IAM enabled

Loading…
Cancel
Save