Browse Source
docs(sts): add comprehensive runtime filer address documentation
docs(sts): add comprehensive runtime filer address documentation
- Document the complete refactoring rationale and implementation - Provide before/after code examples and usage patterns - Include migration guide for existing code - Detail production deployment strategies - Show dynamic filer selection, failover, and load balancing examples - Explain memory store compatibility and interface consistency - Demonstrate environment-agnostic configuration benefitspull/7160/head
1 changed files with 356 additions and 0 deletions
@ -0,0 +1,356 @@ |
|||||
|
# Runtime Filer Address Implementation |
||||
|
|
||||
|
This document describes the implementation of runtime filer address passing for the STSService, addressing the requirement that filer addresses should be passed at call-time rather than initialization time. |
||||
|
|
||||
|
## Problem Statement |
||||
|
|
||||
|
The user identified a critical issue with the original STS implementation: |
||||
|
|
||||
|
> "the filer address should be passed when called, not during init time, since the filer may change." |
||||
|
|
||||
|
This is important because: |
||||
|
1. **Filer Failover**: Filer addresses can change during runtime due to failover scenarios |
||||
|
2. **Load Balancing**: Different requests may need to hit different filer instances |
||||
|
3. **Environment Agnostic**: Configuration files should work across dev/staging/prod without hardcoded addresses |
||||
|
4. **SeaweedFS Patterns**: Follows existing SeaweedFS patterns used throughout the codebase |
||||
|
|
||||
|
## Implementation Changes |
||||
|
|
||||
|
### 1. SessionStore Interface Refactoring |
||||
|
|
||||
|
**Before:** |
||||
|
```go |
||||
|
type SessionStore interface { |
||||
|
StoreSession(ctx context.Context, sessionId string, session *SessionInfo) error |
||||
|
GetSession(ctx context.Context, sessionId string) (*SessionInfo, error) |
||||
|
RevokeSession(ctx context.Context, sessionId string) error |
||||
|
CleanupExpiredSessions(ctx context.Context) error |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
**After:** |
||||
|
```go |
||||
|
type SessionStore interface { |
||||
|
// filerAddress ignored for memory stores, required for filer stores |
||||
|
StoreSession(ctx context.Context, filerAddress string, sessionId string, session *SessionInfo) error |
||||
|
GetSession(ctx context.Context, filerAddress string, sessionId string) (*SessionInfo, error) |
||||
|
RevokeSession(ctx context.Context, filerAddress string, sessionId string) error |
||||
|
CleanupExpiredSessions(ctx context.Context, filerAddress string) error |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
### 2. FilerSessionStore Changes |
||||
|
|
||||
|
**Before:** |
||||
|
```go |
||||
|
type FilerSessionStore struct { |
||||
|
filerGrpcAddress string // ❌ Fixed at init time |
||||
|
grpcDialOption grpc.DialOption |
||||
|
basePath string |
||||
|
} |
||||
|
|
||||
|
func NewFilerSessionStore(filerAddress string, config map[string]interface{}) (*FilerSessionStore, error) { |
||||
|
store := &FilerSessionStore{ |
||||
|
filerGrpcAddress: filerAddress, // ❌ Locked in during init |
||||
|
basePath: DefaultSessionBasePath, |
||||
|
} |
||||
|
// ... |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
**After:** |
||||
|
```go |
||||
|
type FilerSessionStore struct { |
||||
|
grpcDialOption grpc.DialOption // ✅ No fixed filer address |
||||
|
basePath string |
||||
|
} |
||||
|
|
||||
|
func NewFilerSessionStore(config map[string]interface{}) (*FilerSessionStore, error) { |
||||
|
store := &FilerSessionStore{ |
||||
|
basePath: DefaultSessionBasePath, // ✅ Only path configuration |
||||
|
} |
||||
|
// ✅ filerAddress passed at call time |
||||
|
} |
||||
|
|
||||
|
func (f *FilerSessionStore) StoreSession(ctx context.Context, filerAddress string, sessionId string, session *SessionInfo) error { |
||||
|
// ✅ filerAddress provided per call |
||||
|
return f.withFilerClient(filerAddress, func(client filer_pb.SeaweedFilerClient) error { |
||||
|
// ... store logic |
||||
|
}) |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
### 3. STS Service Method Signatures |
||||
|
|
||||
|
**Before:** |
||||
|
```go |
||||
|
func (s *STSService) AssumeRoleWithWebIdentity(ctx context.Context, request *AssumeRoleWithWebIdentityRequest) (*AssumeRoleResponse, error) |
||||
|
func (s *STSService) ValidateSessionToken(ctx context.Context, sessionToken string) (*SessionInfo, error) |
||||
|
func (s *STSService) RevokeSession(ctx context.Context, sessionToken string) error |
||||
|
``` |
||||
|
|
||||
|
**After:** |
||||
|
```go |
||||
|
func (s *STSService) AssumeRoleWithWebIdentity(ctx context.Context, filerAddress string, request *AssumeRoleWithWebIdentityRequest) (*AssumeRoleResponse, error) |
||||
|
func (s *STSService) ValidateSessionToken(ctx context.Context, filerAddress string, sessionToken string) (*SessionInfo, error) |
||||
|
func (s *STSService) RevokeSession(ctx context.Context, filerAddress string, sessionToken string) error |
||||
|
``` |
||||
|
|
||||
|
### 4. Configuration Cleanup |
||||
|
|
||||
|
**Before (iam_config_distributed.json):** |
||||
|
```json |
||||
|
{ |
||||
|
"sts": { |
||||
|
"sessionStoreConfig": { |
||||
|
"filerAddress": "localhost:8888", // ❌ Environment-specific |
||||
|
"basePath": "/etc/iam/sessions" |
||||
|
} |
||||
|
}, |
||||
|
"policy": { |
||||
|
"storeConfig": { |
||||
|
"filerAddress": "localhost:8888", // ❌ Environment-specific |
||||
|
"basePath": "/etc/iam/policies" |
||||
|
} |
||||
|
} |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
**After (iam_config_distributed.json):** |
||||
|
```json |
||||
|
{ |
||||
|
"sts": { |
||||
|
"sessionStoreConfig": { |
||||
|
"basePath": "/etc/iam/sessions" // ✅ Environment-agnostic |
||||
|
} |
||||
|
}, |
||||
|
"policy": { |
||||
|
"storeConfig": { |
||||
|
"basePath": "/etc/iam/policies" // ✅ Environment-agnostic |
||||
|
} |
||||
|
} |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
## Usage Examples |
||||
|
|
||||
|
### Caller Perspective (S3 API Server) |
||||
|
|
||||
|
**Before:** |
||||
|
```go |
||||
|
// STS service locked to specific filer during init |
||||
|
stsService.Initialize(&STSConfig{ |
||||
|
SessionStoreConfig: map[string]interface{}{ |
||||
|
"filerAddress": "filer-1:8888", // ❌ Fixed choice |
||||
|
"basePath": "/etc/iam/sessions", |
||||
|
}, |
||||
|
}) |
||||
|
|
||||
|
// All calls go to filer-1, no failover possible |
||||
|
response, err := stsService.AssumeRoleWithWebIdentity(ctx, request) |
||||
|
``` |
||||
|
|
||||
|
**After:** |
||||
|
```go |
||||
|
// STS service configured without specific filer |
||||
|
stsService.Initialize(&STSConfig{ |
||||
|
SessionStoreConfig: map[string]interface{}{ |
||||
|
"basePath": "/etc/iam/sessions", // ✅ Just the path |
||||
|
}, |
||||
|
}) |
||||
|
|
||||
|
// Caller determines filer address per request |
||||
|
currentFiler := s.getCurrentFilerAddress() // ✅ Dynamic selection |
||||
|
response, err := stsService.AssumeRoleWithWebIdentity(ctx, currentFiler, request) |
||||
|
``` |
||||
|
|
||||
|
### Dynamic Filer Selection |
||||
|
|
||||
|
```go |
||||
|
type S3ApiServer struct { |
||||
|
stsService *sts.STSService |
||||
|
filerClient *filer.Client |
||||
|
} |
||||
|
|
||||
|
func (s *S3ApiServer) getCurrentFilerAddress() string { |
||||
|
// ✅ Can implement any strategy: |
||||
|
// - Load balancing across multiple filers |
||||
|
// - Health checking and failover |
||||
|
// - Geographic routing |
||||
|
// - Round-robin selection |
||||
|
return s.filerClient.GetAvailableFiler() |
||||
|
} |
||||
|
|
||||
|
func (s *S3ApiServer) handleAssumeRole(ctx context.Context, request *AssumeRoleRequest) { |
||||
|
// ✅ Filer address determined at request time |
||||
|
filerAddr := s.getCurrentFilerAddress() |
||||
|
|
||||
|
response, err := s.stsService.AssumeRoleWithWebIdentity(ctx, filerAddr, request) |
||||
|
if err != nil && isNetworkError(err) { |
||||
|
// ✅ Retry with different filer |
||||
|
filerAddr = s.getBackupFilerAddress() |
||||
|
response, err = s.stsService.AssumeRoleWithWebIdentity(ctx, filerAddr, request) |
||||
|
} |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
## Memory Store Compatibility |
||||
|
|
||||
|
The `MemorySessionStore` accepts the `filerAddress` parameter but ignores it, maintaining interface consistency: |
||||
|
|
||||
|
```go |
||||
|
func (m *MemorySessionStore) StoreSession(ctx context.Context, filerAddress string, sessionId string, session *SessionInfo) error { |
||||
|
// filerAddress ignored for memory store - maintains interface compatibility |
||||
|
if sessionId == "" { |
||||
|
return fmt.Errorf(ErrSessionIDCannotBeEmpty) |
||||
|
} |
||||
|
// ... in-memory storage logic |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
## Benefits Achieved |
||||
|
|
||||
|
### 1. **Dynamic Filer Selection** |
||||
|
```go |
||||
|
// Load balancing |
||||
|
filerAddr := loadBalancer.GetNextFiler() |
||||
|
|
||||
|
// Failover support |
||||
|
filerAddr := failoverManager.GetHealthyFiler() |
||||
|
|
||||
|
// Geographic routing |
||||
|
filerAddr := geoRouter.GetClosestFiler(clientIP) |
||||
|
``` |
||||
|
|
||||
|
### 2. **Environment Portability** |
||||
|
```bash |
||||
|
# Same config works everywhere |
||||
|
dev: STSService.method(ctx, "dev-filer:8888", ...) |
||||
|
staging: STSService.method(ctx, "staging-filer:8888", ...) |
||||
|
prod: STSService.method(ctx, "prod-filer-lb:8888", ...) |
||||
|
``` |
||||
|
|
||||
|
### 3. **Operational Flexibility** |
||||
|
- **Hot filer replacement**: Switch filers without restarting STS |
||||
|
- **A/B testing**: Route different requests to different filers |
||||
|
- **Disaster recovery**: Automatic failover to backup filers |
||||
|
- **Performance optimization**: Route to least loaded filer |
||||
|
|
||||
|
### 4. **SeaweedFS Consistency** |
||||
|
Follows the same pattern used throughout SeaweedFS codebase where filer addresses are passed to methods, not stored in structs. |
||||
|
|
||||
|
## Migration Guide |
||||
|
|
||||
|
### For Code Calling STS Methods |
||||
|
|
||||
|
**Before:** |
||||
|
```go |
||||
|
response, err := stsService.AssumeRoleWithWebIdentity(ctx, request) |
||||
|
session, err := stsService.ValidateSessionToken(ctx, token) |
||||
|
err := stsService.RevokeSession(ctx, token) |
||||
|
``` |
||||
|
|
||||
|
**After:** |
||||
|
```go |
||||
|
filerAddr := getCurrentFilerAddress() // Implement your strategy |
||||
|
response, err := stsService.AssumeRoleWithWebIdentity(ctx, filerAddr, request) |
||||
|
session, err := stsService.ValidateSessionToken(ctx, filerAddr, token) |
||||
|
err := stsService.RevokeSession(ctx, filerAddr, token) |
||||
|
``` |
||||
|
|
||||
|
### For Configuration Files |
||||
|
|
||||
|
Remove `filerAddress` from all store configurations: |
||||
|
|
||||
|
```bash |
||||
|
# Update all iam_config*.json files |
||||
|
sed -i 's|"filerAddress": ".*",||g' iam_config*.json |
||||
|
``` |
||||
|
|
||||
|
## Testing |
||||
|
|
||||
|
All tests have been updated to pass a test filer address: |
||||
|
|
||||
|
```go |
||||
|
func TestAssumeRoleWithWebIdentity(t *testing.T) { |
||||
|
service := setupTestSTSService(t) |
||||
|
testFilerAddress := "localhost:8888" // Test filer address |
||||
|
|
||||
|
response, err := service.AssumeRoleWithWebIdentity(ctx, testFilerAddress, request) |
||||
|
// ... test logic |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
## Production Deployment |
||||
|
|
||||
|
### High Availability Setup |
||||
|
|
||||
|
```go |
||||
|
type FilerManager struct { |
||||
|
primaryFilers []string |
||||
|
backupFilers []string |
||||
|
healthChecker *HealthChecker |
||||
|
} |
||||
|
|
||||
|
func (fm *FilerManager) GetAvailableFiler() string { |
||||
|
// Check primary filers first |
||||
|
for _, filer := range fm.primaryFilers { |
||||
|
if fm.healthChecker.IsHealthy(filer) { |
||||
|
return filer |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// Fallback to backup filers |
||||
|
for _, filer := range fm.backupFilers { |
||||
|
if fm.healthChecker.IsHealthy(filer) { |
||||
|
return filer |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// Return first primary as last resort |
||||
|
return fm.primaryFilers[0] |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
### Load Balanced Configuration |
||||
|
|
||||
|
```json |
||||
|
{ |
||||
|
"sts": { |
||||
|
"sessionStoreType": "filer", |
||||
|
"sessionStoreConfig": { |
||||
|
"basePath": "/etc/iam/sessions" |
||||
|
} |
||||
|
} |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
```go |
||||
|
// Runtime filer selection |
||||
|
filerLoadBalancer := &RoundRobinBalancer{ |
||||
|
Filers: []string{ |
||||
|
"filer-1.prod:8888", |
||||
|
"filer-2.prod:8888", |
||||
|
"filer-3.prod:8888", |
||||
|
}, |
||||
|
} |
||||
|
|
||||
|
response, err := stsService.AssumeRoleWithWebIdentity( |
||||
|
ctx, |
||||
|
filerLoadBalancer.Next(), // ✅ Dynamic selection |
||||
|
request, |
||||
|
) |
||||
|
``` |
||||
|
|
||||
|
## Conclusion |
||||
|
|
||||
|
This refactoring successfully addresses the requirement for runtime filer address passing, enabling: |
||||
|
|
||||
|
- ✅ **Dynamic filer selection** per request |
||||
|
- ✅ **Automatic failover** capabilities |
||||
|
- ✅ **Environment-agnostic** configurations |
||||
|
- ✅ **Load balancing** support |
||||
|
- ✅ **SeaweedFS pattern** compliance |
||||
|
- ✅ **Operational flexibility** for production deployments |
||||
|
|
||||
|
The implementation maintains backward compatibility for memory stores while enabling powerful distributed deployment scenarios for filer-backed stores. |
||||
Write
Preview
Loading…
Cancel
Save
Reference in new issue