6bd6bba594
Fix inconsistent admin argument in worker pods ( #8316 )
* Fix inconsistent admin argument in worker pods
* Use seaweedfs.componentName for admin service naming
1 week ago
b8ef48c8f1
Add RisingWave catalog tests ( #8308 )
* Add RisingWave catalog tests for S3 tables
* Add RisingWave catalog integration tests to CI workflow
* Refactor RisingWave catalog tests based on PR feedback
* Address PR feedback: optimize checks, cleanup logs
* fix tests
* consistent
1 week ago
75faf826d4
Fix LevelDB panic on lazy reload ( #8269 ) ( #8307 )
* fix LevelDB panic on lazy reload
Implemented a thread-safe reload mechanism using double-checked
locking and a retry loop in Get, Put, and Delete. Added a concurrency
test to verify the fix and prevent regressions.
Fixes #8269
* refactor: use helper for leveldb fix and remove deprecated ioutil
* fix: prevent deadlock by using getFromDb helper
Extracted DB lookup to internal helper to avoid recursive RLock in Put/Delete methods.
Updated Get to use the helper as well.
* fix: resolve syntax error and commit deadlock prevention
Fixed a duplicate function declaration syntax error.
Verified that getFromDb helper correctly prevents recursive RLock scenarios.
* refactor: remove redundant timeout checks
Removed nested `if m.ldbTimeout > 0` checks in Get, Put, and Delete
methods as suggested in PR review.
1 week ago
221bd237c4
Fix file stat collection metric bug for the `cluster.status` command. ( #8302 )
When the `--files` flag is present, `cluster.status` will scrape file metrics
from volume servers to provide detailed stats on those. The progress indicator
was not being updated properly though, so the command would complete before
it read 100%.
1 week ago
a3136c523f
Fix volume.fsck 401 Unauthorized by adding JWT to HTTP delete requests ( #8306 )
* Fix volume.fsck 401 Unauthorized by adding JWT to HTTP delete requests
* Additionally, for performance, consider fetching the jwt.filer_signing.key once before any loops that call httpDelete, rather than inside httpDelete itself, to avoid repeated configuration lookups.
1 week ago
ac242d04ee
one time manual run
1 week ago
21543134c8
fix manual build process
1 week ago
8b5d31e5eb
s3api/policy_engine: use forwarded client IP for aws:SourceIp ( #8304 )
* s3api: honor forwarded source IP for policy conditions
Prefer X-Forwarded-For/X-Real-Ip before RemoteAddr when populating aws:SourceIp in policy condition evaluation. Also avoid noisy parsing behavior for unix socket markers and add coverage for precedence/fallback paths.\n\nFixes #8301 .
* s3api: simplify remote addr parsing
* s3api: guard aws:SourceIp against DNS hosts
* s3api: simplify remote addr fallback
* s3api: simplify remote addr parsing
* Update weed/s3api/policy_engine/engine.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Fix TestExtractConditionValuesFromRequestSourceIPPrecedence using trusted private IP
* Refactor extractSourceIP to use R-to-L XFF parsing and net.IP.IsPrivate
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 week ago
7151181d54
fix flaky tests
1 week ago
e657e7d827
Implement local scrubbing for EC volumes. ( #8283 )
1 week ago
2a73219397
Add weed shell command `volumeServer.state` to query/update volume server state settings. ( #8271 )
Add weed shell command `volumeServer.state` to query/update volume server states.
1 week ago
7fcbffed7f
filer.sync: support manifest chunks ( #8299 )
* filer.sync support manifest chunks
* filersink: address manifest sync review feedback
1 week ago
be0379f6fd
Fix filer.sync retry on stale chunk ( #8298 )
* Fix filer.sync stale chunk uploads
* Tweak filersink stale logging
1 week ago
b57429ef2e
Switch empty-folder cleanup to bucket policy ( #8292 )
* Fix Spark _temporary cleanup and add issue #8285 regression test
* Generalize empty folder cleanup for Spark temp artifacts
* Revert synchronous folder pruning and add cleanup diagnostics
* Add actionable empty-folder cleanup diagnostics
* Fix Spark temp marker cleanup in async folder cleaner
* Fix Spark temp cleanup with implicit directory markers
* Keep explicit directory markers non-implicit
* logging
* more logs
* Switch empty-folder cleanup to bucket policy
* Seaweed-X-Amz-Allow-Empty-Folders
* less logs
* go vet
* less logs
* refactoring
1 week ago
5c365e7090
s3api: return 400 for invalid namespace query in REST table routes ( #8296 )
* s3api: reject invalid namespace query in REST table routes
* s3api: expand namespace validation REST tests
1 week ago
822dbed552
s3api: fix ListObjectsV2 NextContinuationToken duplication for nested prefix ( #8294 )
* s3api: fix duplicate ListObjectsV2 continuation token for nested prefix
* s3api: include prefix in common-prefix continuation token
1 week ago
2d97685390
ci: fix container_release_unified manual dispatch and workflow parsing ( #8293 )
ci: fix unified release workflow dispatch matrix filtering
1 week ago
1b2f719d7c
admin: fix file browser items-per-page selector ( #8291 )
* admin: fix file browser page size selector
Fix file browser pagination page-size selectors to use explicit select IDs instead of this.value in templ-generated handlers, which could resolve to undefined and produce limit=undefined in requests.
Add a focused template render regression test to prevent this from recurring.
Fixes #8284
* revert file browser template regression test
1 week ago
b73bd08470
ci: move manual container builds to unified release workflow ( #8290 )
* ci: move manual dev container build into unified release workflow
* ci: make unified manual container build release-tag based
1 week ago
17f85361e9
Remove unsupported iceberg rest signing-region from tests and docs ( #8289 )
1 week ago
b261c89675
Fix RocksDB container build compatibility and add manual rocksdb dispatch ( #8288 )
* Fix RocksDB build compatibility and add manual rocksdb trigger
* Upgrade RocksDB defaults and keep grocksdb v1.10.7
* Add manual latest-image trigger inputs for ref and variant
* Allow manual latest build to set image tag and source ref
* Fix manual variant selection using setup job matrix output
1 week ago
0385acba02
s3tables: fix shared table-location bucket mapping collisions ( #8286 )
* s3tables: prevent shared table-location bucket mapping overwrite
* Update weed/s3api/bucket_paths.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
1 week ago
d6825ffce2
Iceberg: implement stage-create finalize flow (phase 1) ( #8279 )
* iceberg: implement stage-create and create-on-commit finalize
* iceberg: add create validation error typing and stage-create integration test
* tests: merge stage-create integration check into catalog suite
* tests: cover stage-create finalize lifecycle in catalog integration
* iceberg: persist and cleanup stage-create markers
* iceberg: add stage-create rollout flag and marker pruning
* docs: add stage-create support design and rollout plan
* docs: drop stage-create design draft from PR
* iceberg: use conservative 72h stage-marker retention
* iceberg: address review comments on create-on-commit and tests
* iceberg: keep stage-create metadata out of table location
* refactor(iceberg): split iceberg.go into focused files
1 week ago
d88f6ed0af
Iceberg commit reliability: preserve statistics updates and return 409 conflicts ( #8277 )
* iceberg: harden table commit updates and conflict handling
* iceberg: refine commit retry and statistics patching
* iceberg: cleanup metadata on non-conflict commit errors
1 week ago
5ae3be44d1
iceberg: persist namespace properties for create/get ( #8276 )
* iceberg: persist namespace properties via s3tables metadata
* iceberg: simplify namespace properties normalization
* s3tables: broaden namespace properties round-trip test
* adjust logs
* adjust logs
1 week ago
1c62808c0e
iceberg: wire pagination for list namespaces/tables REST APIs ( #8275 )
* s3api/iceberg: wire list pagination tokens and page size
* fmt
* Update weed/s3api/iceberg/iceberg.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
1 week ago
db76eb26e7
compile
1 week ago
4ccc7668ce
admin: resolve merge conflicts
1 week ago
aef2de3109
s3tables: support multi-level namespaces in parser/admin paths ( #8273 )
* s3tables: support multi-level namespace normalization
* admin: handle namespace parsing errors centrally
* admin: clean namespace validation duplication
1 week ago
be26ce74ce
s3tables: support multi-level namespace normalization
1 week ago
0b80f055c2
Merge branch 'fix/8270-leader-not-elected'
1 week ago
af8273386d
4.12
1 week ago
ba8e2aaae9
Fix master leader election when grpc ports change ( #8272 )
* Fix master leader detection when grpc ports change
* Canonicalize self peer entry to avoid raft self-alias panic
* Normalize and deduplicate master peer addresses
1 week ago
15d0a46679
Normalize and deduplicate master peer addresses
1 week ago
ae27e17e6f
Canonicalize self peer entry to avoid raft self-alias panic
1 week ago
02dac23119
Fix master leader detection when grpc ports change
1 week ago
f400fb44a0
Update `cluster.status` to resolve file details on EC volumes. ( #8268 )
Also parallelizes queries for file metrics collections when the `--files`
flag is specified, and improves the command's output for readability:
```
> cluster.status --files
collecting file stats: 100%
cluster:
id: topo
status: LOCKED
nodes: 10
topology: 1 DC, 10 disks on 1 rack
volumes:
total: 3 volumes, 1 collection
max size: 32 GB
regular: 1/80 volume on 3 replicas, 3 writable (100%), 0 read-only (0%)
EC: 2 EC volumes on 28 shards (14 shards/volume)
storage:
total: 269 MB (522 MB raw, 193.95%)
regular volumes: 91 MB (272 MB raw, 300%)
EC volumes: 178 MB (250 MB raw, 140%)
files:
total: 363 files, 300 readable (82.64%), 63 deleted (17.35%), avg 522 kB per file
regular: 168 files, 105 readable (62.5%), 63 deleted (37.5%), avg 540 kB per file
EC: 195 files, 195 readable (100%), 0 deleted (0%), avg 506 kB per file
```
1 week ago
30812b85f3
fix ec.encode skipping volumes when one replica is on a full disk ( #8227 )
* fix ec.encode skipping volumes when one replica is on a full disk
This fixes issue #8218 . Previously, ec.encode would skip a volume if ANY
of its replicas resided on a disk with low free volume count. Now it
accepts the volume if AT LEAST ONE replica is on a healthy disk.
* refine noFreeDisk counter logic in ec.encode
Ensure noFreeDisk is decremented if a volume initially marked as bad
is later found to have a healthy replica. This ensures accurate
summary statistics.
* defer noFreeDisk counting and refine logging in ec.encode
Updated logging to be replica-scoped and deferred noFreeDisk counting to
the final pass over vidMap. This ensures that the counter only reflects
volumes that are definitively excluded because all replicas are on full
disks.
* filter replicas by free space during ec.encode
Updated doEcEncode to filter out replicas on disks with
FreeVolumeCount < 2 before selecting the best replica for encoding.
This ensures that EC shards are not generated on healthy source
replicas that happen to be on disks with low free space.
2 weeks ago
6a61037333
fix issue #8230 : volume.fsck deletion logic to respect purgeAbsent flag ( #8266 )
* fix issue #8230 : volume.fsck deletion logic to respect purgeAbsent flag
This commit fixes two issues in volume.fsck:
1. Missing chunks in existing volumes are now deleted if -reallyDeleteFilerEntries is set.
2. Missing volumes are now properly handled when a -volumeId filter is specified, allowing deletion of filer entries for those volumes.
* address PR feedback for issue #8230
- Ensure volume filter is applied before reporting missing volumes
- Fix potential nil-pointer dereferences in httpDelete method
- Use proper error checking throughout httpDelete
* address second round PR feedback for issue #8230
- Use fmt.Fprintf(c.writer, ...) instead of fmt.Printf
- Add missing newline in "deleting path" log message
2 weeks ago
839028b2e0
Fix EC rebuild shard detection ( #8265 )
Fix EC rebuild shard counting
2 weeks ago
1a5679a5eb
Implement a `VolumeEcStatus()` RPC for volume servers. ( #8006 )
Just like `VolumeStatus()`, this call allows inspecting details for
a given EC volume - including number of files and their total size.
2 weeks ago
818a1ff8b1
build(deps): bump golang.org/x/crypto from 0.47.0 to 0.48.0 ( #8258 )
Bumps [golang.org/x/crypto](https://github.com/golang/crypto ) from 0.47.0 to 0.48.0.
- [Commits](https://github.com/golang/crypto/compare/v0.47.0...v0.48.0 )
---
updated-dependencies:
- dependency-name: golang.org/x/crypto
dependency-version: 0.48.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 weeks ago
59b02e0cba
s3api: fix multipart Complete ETag matching and lower empty-upload log noise ( #8264 )
s3api: fix multipart part etag validation and reduce empty upload warning noise
2 weeks ago
d9987669cb
build(deps): bump github.com/jhump/protoreflect from 1.17.0 to 1.18.0 ( #8261 )
Bumps [github.com/jhump/protoreflect](https://github.com/jhump/protoreflect ) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/jhump/protoreflect/releases )
- [Commits](https://github.com/jhump/protoreflect/compare/v1.17.0...v1.18.0 )
---
updated-dependencies:
- dependency-name: github.com/jhump/protoreflect
dependency-version: 1.18.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 weeks ago
0c16663014
build(deps): bump golang.org/x/sys from 0.40.0 to 0.41.0 ( #8260 )
Bumps [golang.org/x/sys](https://github.com/golang/sys ) from 0.40.0 to 0.41.0.
- [Commits](https://github.com/golang/sys/compare/v0.40.0...v0.41.0 )
---
updated-dependencies:
- dependency-name: golang.org/x/sys
dependency-version: 0.41.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 weeks ago
ccf35459be
Explicitly disable signing for public buckets. ( #8263 )
2 weeks ago
88b1768814
build(deps): bump go.etcd.io/etcd/client/v3 from 3.6.6 to 3.6.7 ( #8257 )
Bumps [go.etcd.io/etcd/client/v3](https://github.com/etcd-io/etcd ) from 3.6.6 to 3.6.7.
- [Release notes](https://github.com/etcd-io/etcd/releases )
- [Commits](https://github.com/etcd-io/etcd/compare/v3.6.6...v3.6.7 )
---
updated-dependencies:
- dependency-name: go.etcd.io/etcd/client/v3
dependency-version: 3.6.7
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 weeks ago
62063de1ca
build(deps): bump github.com/linxGnu/grocksdb from 1.10.3 to 1.10.7 ( #8259 )
Bumps [github.com/linxGnu/grocksdb](https://github.com/linxGnu/grocksdb ) from 1.10.3 to 1.10.7.
- [Release notes](https://github.com/linxGnu/grocksdb/releases )
- [Commits](https://github.com/linxGnu/grocksdb/compare/v1.10.3...v1.10.7 )
---
updated-dependencies:
- dependency-name: github.com/linxGnu/grocksdb
dependency-version: 1.10.7
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 weeks ago
63b846b73b
Parallelize operations for the `volume.scrub` and `ec.scrub` commands ( #8247 )
Parallelize operations for the `volume.scrub` and `ec.scrub` commands.
2 weeks ago
c9428c2c0f
Modify AI review comments checklist in PR template
Updated AI code review checklist to include potential additional reviews.
2 weeks ago
Chris Lu
Lisandro Pin
Chris Lu
dependabot[bot]