Critical fix for multi-filer deployments:
**Problem:**
- S3ApiServer.WithFilerClient() was creating direct connections to ONE filer
- Used pb.WithGrpcClient() with single filer address
- No failover - if that filer failed, ALL operations failed
- Caused test failures: "bucket directory not found"
- IAM Integration Tests failing with 500 Internal Error
**Root Cause:**
- WithFilerClient bypassed filerClient connection management
- Always connected to getFilerAddress() (current filer only)
- Didn't retry other filers on failure
- All getEntry(), updateEntry(), etc. operations failed if current filer down
**Solution:**
1. Added FilerClient.GetAllFilers() method
- Returns snapshot of all filer addresses
- Thread-safe copy to avoid races
2. Implemented withFilerClientFailover()
- Try current filer first (fast path)
- On failure, try all other filers
- Log successful failover
- Return error only if ALL filers fail
3. Updated WithFilerClient()
- Use filerClient for failover when available
- Fallback to direct connection for testing/init
**Impact:**
✅ All S3 operations now support multi-filer failover
✅ Bucket metadata reads work with any available filer
✅ Entry operations (getEntry, updateEntry) failover automatically
✅ IAM tests should pass now
✅ Production-ready HA support
**Files Changed:**
- wdclient/filer_client.go: Add GetAllFilers() method
- s3api/s3api_handlers.go: Implement failover logic
This fixes the test failure where bucket operations failed when
the primary filer was temporarily unavailable during cleanup.
This change adds support for multiple filer addresses in the 'weed s3' and 'weed iam' commands, enabling high availability through automatic failover.
Key changes:
- Updated S3ApiServerOption.Filer to Filers ([]pb.ServerAddress)
- Updated IamServerOption.Filer to Filers ([]pb.ServerAddress)
- Modified -filer flag to accept comma-separated addresses
- Added getFilerAddress() helper methods for backward compatibility
- Updated all filer client calls to support multiple addresses
- Uses pb.WithOneOfGrpcFilerClients for automatic failover
Usage:
weed s3 -filer=localhost:8888,localhost:8889
weed iam -filer=localhost:8888,localhost:8889
The underlying FilerClient already supported multiple filers with health
tracking and automatic failover - this change exposes that capability
through the command-line interface.
* Add a way to use a JWT in an HTTP only cookie
If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie.
* Added a way to specify allowed origins header from config
* Removed unecessary log
* Check list of domains from config or command flag
* Handle default wildcard and change name of config value to cors
* remove old raft servers if they don't answer to pings for too long
add ping durations as options
rename ping fields
fix some todos
get masters through masterclient
raft remove server from leader
use raft servers to ping them
CheckMastersAlive for hashicorp raft only
* prepare blocking ping
* pass waitForReady as param
* pass waitForReady through all functions
* waitForReady works
* refactor
* remove unneeded params
* rollback unneeded changes
* fix
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
1. option for "weed s3 -filer.dir.buckets" to choose a folder for buckets
2. create a bucket
3. delete a bucket, recursively delete all metadata on filer
Chris Lu
jerebear12
askeipx
Konstantin Lebedev
Konstantin Lebedev
Rinat Shigapov
j.laycock
Chris Lu