seaweedfs/weed/iam/sts/RUNTIME_FILER_ADDRESS.md

Runtime Filer Address Implementation

This document describes the implementation of runtime filer address passing for the STSService, addressing the requirement that filer addresses should be passed at call-time rather than initialization time.

Problem Statement

The user identified a critical issue with the original STS implementation:

"the filer address should be passed when called, not during init time, since the filer may change."

This is important because:

1. Filer Failover: Filer addresses can change during runtime due to failover scenarios
2. Load Balancing: Different requests may need to hit different filer instances
3. Environment Agnostic: Configuration files should work across dev/staging/prod without hardcoded addresses
4. SeaweedFS Patterns: Follows existing SeaweedFS patterns used throughout the codebase

Implementation Changes

1. SessionStore Interface Refactoring

Before:

type SessionStore interface {
    StoreSession(ctx context.Context, sessionId string, session *SessionInfo) error
    GetSession(ctx context.Context, sessionId string) (*SessionInfo, error)
    RevokeSession(ctx context.Context, sessionId string) error
    CleanupExpiredSessions(ctx context.Context) error
}

After:

type SessionStore interface {
    // filerAddress ignored for memory stores, required for filer stores
    StoreSession(ctx context.Context, filerAddress string, sessionId string, session *SessionInfo) error
    GetSession(ctx context.Context, filerAddress string, sessionId string) (*SessionInfo, error)
    RevokeSession(ctx context.Context, filerAddress string, sessionId string) error
    CleanupExpiredSessions(ctx context.Context, filerAddress string) error
}

2. FilerSessionStore Changes

Before:

type FilerSessionStore struct {
    filerGrpcAddress string  // ❌ Fixed at init time
    grpcDialOption   grpc.DialOption
    basePath         string
}

func NewFilerSessionStore(filerAddress string, config map[string]interface{}) (*FilerSessionStore, error) {
    store := &FilerSessionStore{
        filerGrpcAddress: filerAddress,  // ❌ Locked in during init
        basePath:         DefaultSessionBasePath,
    }
    // ...
}

After:

type FilerSessionStore struct {
    grpcDialOption grpc.DialOption  // ✅ No fixed filer address
    basePath       string
}

func NewFilerSessionStore(config map[string]interface{}) (*FilerSessionStore, error) {
    store := &FilerSessionStore{
        basePath: DefaultSessionBasePath,  // ✅ Only path configuration
    }
    // ✅ filerAddress passed at call time
}

func (f *FilerSessionStore) StoreSession(ctx context.Context, filerAddress string, sessionId string, session *SessionInfo) error {
    // ✅ filerAddress provided per call
    return f.withFilerClient(filerAddress, func(client filer_pb.SeaweedFilerClient) error {
        // ... store logic
    })
}

3. STS Service Method Signatures

Before:

func (s *STSService) AssumeRoleWithWebIdentity(ctx context.Context, request *AssumeRoleWithWebIdentityRequest) (*AssumeRoleResponse, error)
func (s *STSService) ValidateSessionToken(ctx context.Context, sessionToken string) (*SessionInfo, error)
func (s *STSService) RevokeSession(ctx context.Context, sessionToken string) error

After:

func (s *STSService) AssumeRoleWithWebIdentity(ctx context.Context, filerAddress string, request *AssumeRoleWithWebIdentityRequest) (*AssumeRoleResponse, error)
func (s *STSService) ValidateSessionToken(ctx context.Context, filerAddress string, sessionToken string) (*SessionInfo, error)
func (s *STSService) RevokeSession(ctx context.Context, filerAddress string, sessionToken string) error

4. Configuration Cleanup

Before (iam_config_distributed.json):

{
  "sts": {
    "sessionStoreConfig": {
      "filerAddress": "localhost:8888",  // ❌ Environment-specific
      "basePath": "/etc/iam/sessions"
    }
  },
  "policy": {
    "storeConfig": {
      "filerAddress": "localhost:8888",  // ❌ Environment-specific
      "basePath": "/etc/iam/policies"
    }
  }
}

After (iam_config_distributed.json):

{
  "sts": {
    "sessionStoreConfig": {
      "basePath": "/etc/iam/sessions"    // ✅ Environment-agnostic
    }
  },
  "policy": {
    "storeConfig": {
      "basePath": "/etc/iam/policies"    // ✅ Environment-agnostic
    }
  }
}

Usage Examples

Caller Perspective (S3 API Server)

Before:

// STS service locked to specific filer during init
stsService.Initialize(&STSConfig{
    SessionStoreConfig: map[string]interface{}{
        "filerAddress": "filer-1:8888",  // ❌ Fixed choice
        "basePath": "/etc/iam/sessions",
    },
})

// All calls go to filer-1, no failover possible
response, err := stsService.AssumeRoleWithWebIdentity(ctx, request)

After:

// STS service configured without specific filer
stsService.Initialize(&STSConfig{
    SessionStoreConfig: map[string]interface{}{
        "basePath": "/etc/iam/sessions",  // ✅ Just the path
    },
})

// Caller determines filer address per request
currentFiler := s.getCurrentFilerAddress()  // ✅ Dynamic selection
response, err := stsService.AssumeRoleWithWebIdentity(ctx, currentFiler, request)

Dynamic Filer Selection

type S3ApiServer struct {
    stsService   *sts.STSService
    filerClient  *filer.Client
}

func (s *S3ApiServer) getCurrentFilerAddress() string {
    // ✅ Can implement any strategy:
    // - Load balancing across multiple filers
    // - Health checking and failover
    // - Geographic routing
    // - Round-robin selection
    return s.filerClient.GetAvailableFiler()
}

func (s *S3ApiServer) handleAssumeRole(ctx context.Context, request *AssumeRoleRequest) {
    // ✅ Filer address determined at request time
    filerAddr := s.getCurrentFilerAddress()
    
    response, err := s.stsService.AssumeRoleWithWebIdentity(ctx, filerAddr, request)
    if err != nil && isNetworkError(err) {
        // ✅ Retry with different filer
        filerAddr = s.getBackupFilerAddress()
        response, err = s.stsService.AssumeRoleWithWebIdentity(ctx, filerAddr, request)
    }
}

Memory Store Compatibility

The MemorySessionStore accepts the filerAddress parameter but ignores it, maintaining interface consistency:

func (m *MemorySessionStore) StoreSession(ctx context.Context, filerAddress string, sessionId string, session *SessionInfo) error {
    // filerAddress ignored for memory store - maintains interface compatibility
    if sessionId == "" {
        return fmt.Errorf(ErrSessionIDCannotBeEmpty)
    }
    // ... in-memory storage logic
}

Benefits Achieved

1. Dynamic Filer Selection

// Load balancing
filerAddr := loadBalancer.GetNextFiler()

// Failover support
filerAddr := failoverManager.GetHealthyFiler()

// Geographic routing
filerAddr := geoRouter.GetClosestFiler(clientIP)

2. Environment Portability

# Same config works everywhere
dev:     STSService.method(ctx, "dev-filer:8888", ...)
staging: STSService.method(ctx, "staging-filer:8888", ...)
prod:    STSService.method(ctx, "prod-filer-lb:8888", ...)

3. Operational Flexibility

• Hot filer replacement: Switch filers without restarting STS
• A/B testing: Route different requests to different filers
• Disaster recovery: Automatic failover to backup filers
• Performance optimization: Route to least loaded filer

4. SeaweedFS Consistency

Follows the same pattern used throughout SeaweedFS codebase where filer addresses are passed to methods, not stored in structs.

Migration Guide

For Code Calling STS Methods

Before:

response, err := stsService.AssumeRoleWithWebIdentity(ctx, request)
session, err := stsService.ValidateSessionToken(ctx, token)
err := stsService.RevokeSession(ctx, token)

After:

filerAddr := getCurrentFilerAddress()  // Implement your strategy
response, err := stsService.AssumeRoleWithWebIdentity(ctx, filerAddr, request)
session, err := stsService.ValidateSessionToken(ctx, filerAddr, token)
err := stsService.RevokeSession(ctx, filerAddr, token)

For Configuration Files

Remove filerAddress from all store configurations:

# Update all iam_config*.json files
sed -i 's|"filerAddress": ".*",||g' iam_config*.json

Testing

All tests have been updated to pass a test filer address:

func TestAssumeRoleWithWebIdentity(t *testing.T) {
    service := setupTestSTSService(t)
    testFilerAddress := "localhost:8888" // Test filer address
    
    response, err := service.AssumeRoleWithWebIdentity(ctx, testFilerAddress, request)
    // ... test logic
}

Production Deployment

High Availability Setup

type FilerManager struct {
    primaryFilers   []string
    backupFilers    []string
    healthChecker   *HealthChecker
}

func (fm *FilerManager) GetAvailableFiler() string {
    // Check primary filers first
    for _, filer := range fm.primaryFilers {
        if fm.healthChecker.IsHealthy(filer) {
            return filer
        }
    }
    
    // Fallback to backup filers
    for _, filer := range fm.backupFilers {
        if fm.healthChecker.IsHealthy(filer) {
            return filer
        }
    }
    
    // Return first primary as last resort
    return fm.primaryFilers[0]
}

Load Balanced Configuration

{
  "sts": {
    "sessionStoreType": "filer",
    "sessionStoreConfig": {
      "basePath": "/etc/iam/sessions"
    }
  }
}

// Runtime filer selection
filerLoadBalancer := &RoundRobinBalancer{
    Filers: []string{
        "filer-1.prod:8888",
        "filer-2.prod:8888", 
        "filer-3.prod:8888",
    },
}

response, err := stsService.AssumeRoleWithWebIdentity(
    ctx, 
    filerLoadBalancer.Next(),  // ✅ Dynamic selection
    request,
)

Conclusion

This refactoring successfully addresses the requirement for runtime filer address passing, enabling:

• ✅ Dynamic filer selection per request
• ✅ Automatic failover capabilities
• ✅ Environment-agnostic configurations
• ✅ Load balancing support
• ✅ SeaweedFS pattern compliance
• ✅ Operational flexibility for production deployments

The implementation maintains backward compatibility for memory stores while enabling powerful distributed deployment scenarios for filer-backed stores.