Fix HIGH issues:
- Fix empty disk discovery: Now discovers all disks from VolumeInfos,
not just from EC shards. This ensures disks without EC shards are
still considered for placement.
- Fix EC shard count calculation in detection.go: Now correctly filters
by DiskId and sums actual shard counts using ShardBits.ShardIdCount()
instead of just counting EcShardInfo entries.
Fix MEDIUM issues:
- Add disk ID to evacuation log messages for consistency with other logging
- Remove unused serverToDisks variable in placement.go
- Fix comment that incorrectly said 'ascending' when sorting is 'descending'
- Modified test/s3/tagging/s3_tagging_test.go to use environment variables for configurable endpoint and credentials
- Added s3-tagging-tests job to .github/workflows/s3-go-tests.yml to run tagging tests in CI
- Tests will now run automatically on pull requests
- Add X-Amz-Tagging header parsing in putToFiler function for PUT object operations
- Store tags with X-Amz-Tagging- prefix in entry.Extended metadata
- Add comprehensive test suite for S3 object tagging functionality
- Tests cover upload tagging, API operations, special characters, and edge cases
* filer: add username and keyPrefix support for Redis stores
Addresses https://github.com/seaweedfs/seaweedfs/issues/7299
- Add username config option to redis2, redis_cluster2, redis_lua, and
redis_lua_cluster stores (sentinel stores already had it)
- Add keyPrefix config option to all Redis stores to prefix all keys,
useful for Envoy Redis Proxy or multi-tenant Redis setups
* refactor: reduce duplication in redis.NewClient creation
Address code review feedback by defining redis.Options once and
conditionally setting TLSConfig instead of duplicating the entire
NewClient call.
* filer.toml: add username and keyPrefix to redis2.tmp example
* Fix#7575: Correct interface check for filer address function in admin server
Problem:
User creation in object store was failing with error:
'filer_etc: filer address function not configured'
Root Cause:
In admin_server.go, the code checked for incorrect interface method
SetFilerClient(string, grpc.DialOption) instead of the actual
SetFilerAddressFunc(func() pb.ServerAddress, grpc.DialOption)
This interface mismatch prevented the filer address function from being
configured, causing user creation operations to fail.
Solution:
- Fixed interface check to use SetFilerAddressFunc
- Updated function call to properly configure filer address function
- Function now dynamically returns current active filer address
Tests Added:
- Unit tests in weed/admin/dash/user_management_test.go
- Integration tests in test/admin/user_creation_integration_test.go
- Documentation in test/admin/README.md
All tests pass successfully.
* Fix#7575: Correct interface check for filer address function in admin UI
Problem:
User creation in Admin UI was failing with error:
'filer_etc: filer address function not configured'
Root Cause:
In admin_server.go, the code checked for incorrect interface method
SetFilerClient(string, grpc.DialOption) instead of the actual
SetFilerAddressFunc(func() pb.ServerAddress, grpc.DialOption)
This interface mismatch prevented the filer address function from being
configured, causing user creation operations to fail in the Admin UI.
Note: This bug only affects the Admin UI. The S3 API and weed shell
commands (s3.configure) were unaffected as they use the correct interface
or bypass the credential manager entirely.
Solution:
- Fixed interface check in admin_server.go to use SetFilerAddressFunc
- Updated function call to properly configure filer address function
- Function now dynamically returns current active filer (HA-aware)
- Cleaned up redundant comments in the code
Tests Added:
- Unit tests in weed/admin/dash/user_management_test.go
* TestFilerAddressFunctionInterface - verifies correct interface
* TestGenerateAccessKey - tests key generation
* TestGenerateSecretKey - tests secret generation
* TestGenerateAccountId - tests account ID generation
All tests pass and will run automatically in CI.
* Fix#7575: Correct interface check for filer address function in admin UI
Problem:
User creation in Admin UI was failing with error:
'filer_etc: filer address function not configured'
Root Cause:
1. In admin_server.go, the code checked for incorrect interface method
SetFilerClient(string, grpc.DialOption) instead of the actual
SetFilerAddressFunc(func() pb.ServerAddress, grpc.DialOption)
2. The admin command was missing the filer_etc import, so the store
was never registered
This interface mismatch prevented the filer address function from being
configured, causing user creation operations to fail in the Admin UI.
Note: This bug only affects the Admin UI. The S3 API and weed shell
commands (s3.configure) were unaffected as they use the correct interface
or bypass the credential manager entirely.
Solution:
- Added filer_etc import to weed/command/admin.go to register the store
- Fixed interface check in admin_server.go to use SetFilerAddressFunc
- Updated function call to properly configure filer address function
- Function now dynamically returns current active filer (HA-aware)
- Hoisted credentialManager assignment to reduce code duplication
Tests Added:
- Unit tests in weed/admin/dash/user_management_test.go
* TestFilerAddressFunctionInterface - verifies correct interface
* TestGenerateAccessKey - tests key generation
* TestGenerateSecretKey - tests secret generation
* TestGenerateAccountId - tests account ID generation
All tests pass and will run automatically in CI.
* Enable FIPS 140-3 compliant crypto by default
Addresses #6889
- Enable GOEXPERIMENT=systemcrypto by default in all Makefiles
- Enable GOEXPERIMENT=systemcrypto by default in all Dockerfiles
- Go 1.24+ has native FIPS 140-3 support via this setting
- Users can disable by setting GOEXPERIMENT= (empty)
Algorithms used (all FIPS approved):
- AES-256-GCM for data encryption
- AES-256-CTR for SSE-C
- HMAC-SHA256 for S3 signatures
- TLS 1.2/1.3 for transport encryption
* Fix: Remove invalid GOEXPERIMENT=systemcrypto
Go 1.24 uses GODEBUG=fips140=on at runtime, not GOEXPERIMENT at build time.
- Remove GOEXPERIMENT=systemcrypto from all Makefiles
- Remove GOEXPERIMENT=systemcrypto from all Dockerfiles
FIPS 140-3 mode can be enabled at runtime:
GODEBUG=fips140=on ./weed server ...
* Add FIPS 140-3 support enabled by default
Addresses #6889
- FIPS 140-3 mode is ON by default in Docker containers
- Sets GODEBUG=fips140=on via entrypoint.sh
- To disable: docker run -e GODEBUG=fips140=off ...
Have `volume.check.disk` select a random (heathly) source volume when repairing read-only volumes.
This ensures uniform load across the topology when the command is run. Also remove a lingering
TODO about ignoring full volumes; not only there's no way to discern read-only volumes from
being full vs. being damaged, we ultimately want to check the former anyway.
- Use pickBestDiskOnNode() when selecting evacuation target disk
- Specify target disk in evacuation RPC requests
- Maintains balanced disk distribution during server evacuations
- Replace custom EC disk selection logic with shared placement package
- Convert topology DiskInfo to placement.DiskCandidate format
- Use SelectDestinations() for multi-rack/server/disk spreading
- Convert placement results back to topology DiskInfo for task creation
- Ensures EC detection uses same placement logic as shell commands
- Add EcDisk struct to track individual disks on volume servers
- Update EcNode to maintain per-disk shard distribution
- Parse disk_id from EC shard information during topology collection
- Implement pickBestDiskOnNode() for selecting best disk per shard
- Add diskDistributionScore() for tie-breaking node selection
- Update all move operations to specify target disk in RPC calls
- Improves shard balance within multi-disk servers, not just across servers
- Consolidate EC shard placement algorithm for reuse across shell and worker tasks
- Support multi-pass selection: racks, then servers, then disks
- Include proper spread verification and scoring functions
- Comprehensive test coverage for various cluster topologies
* Add link to wiki installation page in README
* Add building for docker in weed/Makefile
Building without `CGO_ENABLED=0` and using the executable in docker can result in a docker container exiting with an error
* Update README.md
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Add GOOS=linux to build_docker target for cross-compilation
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* mount: improve read throughput with parallel chunk fetching
This addresses issue #7504 where a single weed mount FUSE instance
does not fully utilize node network bandwidth when reading large files.
Changes:
- Add -concurrentReaders mount option (default: 16) to control the
maximum number of parallel chunk fetches during read operations
- Implement parallel section reading in ChunkGroup.ReadDataAt() using
errgroup for better throughput when reading across multiple sections
- Enhance ReaderCache with MaybeCacheMany() to prefetch multiple chunks
ahead in parallel during sequential reads (now prefetches 4 chunks)
- Increase ReaderCache limit dynamically based on concurrentReaders
to support higher read parallelism
The bottleneck was that chunks were being read sequentially even when
they reside on different volume servers. By introducing parallel chunk
fetching, a single mount instance can now better saturate available
network bandwidth.
Fixes: #7504
* fmt
* Address review comments: make prefetch configurable, improve error handling
Changes:
1. Add DefaultPrefetchCount constant (4) to reader_at.go
2. Add GetPrefetchCount() method to ChunkGroup that derives prefetch count
from concurrentReaders (1/4 ratio, min 1, max 8)
3. Pass prefetch count through NewChunkReaderAtFromClient
4. Fix error handling in readDataAtParallel to prioritize errgroup error
5. Update all callers to use DefaultPrefetchCount constant
For mount operations, prefetch scales with -concurrentReaders:
- concurrentReaders=16 (default) -> prefetch=4
- concurrentReaders=32 -> prefetch=8 (capped)
- concurrentReaders=4 -> prefetch=1
For non-mount paths (WebDAV, query engine, MQ), uses DefaultPrefetchCount.
* fmt
* Refactor: use variadic parameter instead of new function name
Use NewChunkGroup with optional concurrentReaders parameter instead of
creating a separate NewChunkGroupWithConcurrency function.
This maintains backward compatibility - existing callers without the
parameter get the default of 16 concurrent readers.
* Use explicit concurrentReaders parameter instead of variadic
* Refactor: use MaybeCache with count parameter instead of new MaybeCacheMany function
* Address nitpick review comments
- Add upper bound (128) on concurrentReaders to prevent excessive goroutine fan-out
- Cap readerCacheLimit at 256 accordingly
- Fix SetChunks: use Lock() instead of RLock() since we are writing to group.sections
* filer use context without cancellation
* pass along context
* fix: copy to bucket with default SSE-S3 encryption fails (#7562)
When copying an object from an encrypted bucket to a temporary unencrypted
bucket, then to another bucket with default SSE-S3 encryption, the operation
fails with 'invalid SSE-S3 source key type' error.
Root cause:
When objects are copied from an SSE-S3 encrypted bucket to an unencrypted
bucket, the 'X-Amz-Server-Side-Encryption: AES256' header is preserved but
the actual encryption key (SeaweedFSSSES3Key) is stripped. This creates an
'orphaned' SSE-S3 header that causes IsSSES3EncryptedInternal() to return
true, triggering decryption logic with a nil key.
Fix:
1. Modified IsSSES3EncryptedInternal() to require BOTH the AES256 header
AND the SeaweedFSSSES3Key to be present before returning true
2. Added isOrphanedSSES3Header() to detect orphaned SSE-S3 headers
3. Updated copy handler to strip orphaned headers during copy operations
Fixes#7562
* fmt
* refactor: simplify isOrphanedSSES3Header function logic
Remove redundant existence check since the caller iterates through
metadata map, making the check unnecessary. Improves readability
while maintaining the same functionality.
Chris Lu
Chris Lu
copilot-swe-agent[bot]
dependabot[bot]
Lisandro Pin
Krzysztof Osiniak