This adds a new configuration option ConcurrentFileUploadLimit that limits
the number of concurrent file uploads based on file count, complementing
the existing ConcurrentUploadLimit which limits based on total data size.
This addresses an OOM vulnerability where requests with missing/zero
Content-Length headers could bypass the size-based rate limiter.
Changes:
- Add ConcurrentUploadLimit and ConcurrentFileUploadLimit fields to S3ApiServerOption
- Add inFlightDataSize, inFlightUploads, and inFlightDataLimitCond to S3ApiServer
- Add s3a reference to CircuitBreaker for upload limiting
- Enhance CircuitBreaker.Limit() to apply upload limiting for write actions
- Add -concurrentUploadLimitMB and -concurrentFileUploadLimit command line flags
- Add s3.concurrentUploadLimitMB and s3.concurrentFileUploadLimit flags to filer command
The upload limiting is integrated into the existing CircuitBreaker.Limit()
function, avoiding creation of new wrapper functions and reusing the existing
handler registration pattern.
Fixes#7529
Critical fixes based on code review feedback:
1. **IAM API Failover (Critical)**:
- Replace pb.WithGrpcFilerClient with pb.WithOneOfGrpcFilerClients in:
* GetS3ApiConfigurationFromFiler()
* PutS3ApiConfigurationToFiler()
* GetPolicies()
* PutPolicies()
- Now all IAM operations support automatic failover across multiple filers
2. **Validation Improvements**:
- Add validation in NewIamApiServerWithStore() to require at least one filer
- Add validation in NewS3ApiServerWithStore() to require at least one filer
- Add warning log when no filers configured for credential store
3. **Error Logging**:
- Circuit breaker now logs when config load fails instead of silently ignoring
- Helps operators understand why circuit breaker limits aren't applied
4. **Code Quality**:
- Use ToGrpcAddress() for filer address in credential store setup
- More consistent with rest of codebase and future-proof
These changes ensure IAM operations have the same high availability guarantees
as S3 operations, completing the multi-filer failover implementation.
This change adds support for multiple filer addresses in the 'weed s3' and 'weed iam' commands, enabling high availability through automatic failover.
Key changes:
- Updated S3ApiServerOption.Filer to Filers ([]pb.ServerAddress)
- Updated IamServerOption.Filer to Filers ([]pb.ServerAddress)
- Modified -filer flag to accept comma-separated addresses
- Added getFilerAddress() helper methods for backward compatibility
- Updated all filer client calls to support multiple addresses
- Uses pb.WithOneOfGrpcFilerClients for automatic failover
Usage:
weed s3 -filer=localhost:8888,localhost:8889
weed iam -filer=localhost:8888,localhost:8889
The underlying FilerClient already supported multiple filers with health
tracking and automatic failover - this change exposes that capability
through the command-line interface.
Chris Lu
Chris Lu
石昌林