9009e38f7b
docs(volume_server): log ping volume-server unreachable coverage
4 weeks ago
b9fbb85af2
test(volume_server/grpc): add ping unreachable volume-server target case
4 weeks ago
47d3001572
docs(volume_server): log csv query payload parity coverage
4 weeks ago
a12dd5f8d3
test(volume_server/grpc): cover csv-query payload no-output parity
4 weeks ago
8e614486a3
docs(volume_server): log tail-receiver interruption coverage
4 weeks ago
a5864c3eb6
test(volume_server/grpc): cover tail-receiver source-unavailable branch
4 weeks ago
6302809442
docs(volume_server): log tail sender cancellation coverage
4 weeks ago
27a80f7607
test(volume_server/grpc): add tail-sender cancellation interruption coverage
4 weeks ago
ec429e0361
docs(volume_server): log framework port-range hardening and rerun
4 weeks ago
90e82b15ce
test(volume_server/framework): allocate volume ports within safe grpc-offset range
4 weeks ago
a3e1ee1653
docs(volume_server): log mkcol method parity coverage
4 weeks ago
2ab30900d4
test(volume_server/http): add mkcol unsupported-method parity
4 weeks ago
62ee14fa61
docs(volume_server): log read-all-needles multi-volume coverage
4 weeks ago
ab95a6ef15
test(volume_server/grpc): cover read-all-needles multi-volume success
4 weeks ago
24965fd489
docs(volume_server): log head conditional precedence coverage
4 weeks ago
ed23e290fc
test(volume_server/http): expand head conditional precedence coverage
4 weeks ago
9b57fb6961
docs(volume_server): log ec batch delete success coverage
4 weeks ago
1bb40b6bc5
test(volume_server/grpc): add ec batch delete success coverage
4 weeks ago
34e342da63
docs(volume_server): log replicated write failure coverage
4 weeks ago
4835d34438
test(volume_server/http): cover replicated write failure when replication unmet
4 weeks ago
5814729def
docs(volume_server): log ec-only read meta coverage
4 weeks ago
37bf9b5ebf
test(volume_server/grpc): cover ec-only read needle meta unsupported path
4 weeks ago
19201df6d7
docs(volume_server): log oversized upload limit coverage
4 weeks ago
4d61cbdeed
test(volume_server/http): cover oversized upload file-size limit rejection
4 weeks ago
3ce883624e
docs(volume_server): log jwt ui access override coverage
4 weeks ago
de974c05d5
test(volume_server/http): cover jwt ui access override behavior
4 weeks ago
7768fda023
docs(volume_server): record proxy-mode validation and CI matrix
4 weeks ago
548b3d9a38
ci(volume_server): run rust smoke tests in exec and proxy modes
4 weeks ago
a7f50d23b5
feat(rust/volume_server): add proxy supervision mode for integration parity
4 weeks ago
6ce4d7eded
docs(volume_server): record rust-mode full-suite validation
4 weeks ago
3bd20e6a10
chore(rust/volume_server): add Cargo.lock
4 weeks ago
d402573ea8
docs(volume_server): document rust-mode harness and tracking
4 weeks ago
63d08e8a91
ci(volume_server): add rust-mode integration smoke job
4 weeks ago
880c2e1dab
feat(rust/volume_server): add compatibility launcher and migration plan
4 weeks ago
7beab85c21
test(volume_server/framework): support selectable volume server binary
4 weeks ago
703d5e27b3
Fix S3 ListObjectsV2 recursion issue ( #8347 )
* Fix S3 ListObjectsV2 recursion issue (#8346 )
Removed aggressive Limit=1 optimization in doListFilerEntries that caused missed directory entries when prefix ended with a delimiter. Added regression tests to verify deep directory traversal.
* Address PR comments: condense test comments
4 weeks ago
e863767ac7
cleanup(iam): final removal of temporary debug logging from STS and S3 API
4 weeks ago
e29a7f1741
cleanup(iam): remove temporary debug logging from STS and S3 API (redo)
4 weeks ago
cf8e383e1e
STS: Fallback to Caller Identity when RoleArn is missing in AssumeRole ( #8345 )
* s3api: make RoleArn optional in AssumeRole
* s3api: address PR feedback for optional RoleArn
* iam: add configurable default role for AssumeRole
* S3 STS: Use caller identity when RoleArn is missing
- Fallback to PrincipalArn/Context in AssumeRole if RoleArn is empty
- Handle User ARNs in prepareSTSCredentials
- Fix PrincipalArn generation for env var credentials
* Test: Add unit test for AssumeRole caller identity fallback
* fix(s3api): propagate admin permissions to assumed role session when using caller identity fallback
* STS: Fix is_admin propagation and optimize IAM policy evaluation for assumed roles
- Restore is_admin propagation via JWT req_ctx
- Optimize IsActionAllowed to skip role lookups for admin sessions
- Ensure session policies are still applied for downscoping
- Remove debug logging
- Fix syntax errors in cleanup
* fix(iam): resolve STS policy bypass for admin sessions
- Fixed IsActionAllowed in iam_manager.go to correctly identify and validate internal STS tokens, ensuring session policies are enforced.
- Refactored VerifyActionPermission in auth_credentials.go to properly handle session tokens and avoid legacy authorization short-circuits.
- Added debug logging for better tracing of policy evaluation and session validation.
4 weeks ago
f49f6c6876
FUSE mount: fix failed git clone ( #8344 )
tests: reset MemoryStore to avoid test pollution; fix port reservation to prevent duplicate ports in mini
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 month ago
7799915e50
Fix IAM identity loss on S3 restart migration ( #8343 )
* Fix IAM reload after legacy config migration
Handle legacy identity.json metadata events by reloading from the credential manager instead of parsing event content, and watch the correct /etc/iam multi-file directories so identity changes are applied.
Add regression tests for legacy deletion and /etc/iam/identities change events.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix auth_credentials_subscribe_test helper to not pollute global memory store
The SaveConfiguration call was affecting other tests. Use local credential manager and ReplaceS3ApiConfiguration instead.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix IAM event watching: subscribe to IAM directories and improve directory matching
- Add /etc/iam and its subdirectories (identities, policies, service_accounts) to directoriesToWatch
- Fix directory matching to avoid false positives from sibling directories
- Use exact match or prefix with trailing slash instead of plain HasPrefix
- Prevents matching hypothetical /etc/iam/identities_backup directory
This ensures IAM config change events are actually delivered to the handler.
* fix tests
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 month ago
c090604143
Add UpdateAccessKey support to IAM API ( #8342 )
* Add UpdateAccessKey support to IAM API
* simplify
1 month ago
f44e25b422
fix(iam): ensure access key status is persisted and defaulted to Active ( #8341 )
* Fix master leader election startup issue
Fixes #error-log-leader-not-selected-yet
* not useful test
* fix(iam): ensure access key status is persisted and defaulted to Active
* make pb
* update tests
* using constants
1 month ago
fbe7dd32c2
Implement full scrubbing for regular volumes ( #8254 )
Implement full scrubbing for regular volumes.
1 month ago
1ebc9dd530
Have local EC volume scrubbing check needle integrity whenever possible. ( #8334 )
If local EC scrubbing hits needles whose chunk location reside entirely
in local shards, we can fully reconstruct them, and check CRCs for
data integrity.
1 month ago
b08bb8237c
Fix master leader election startup issue ( #8340 )
* Fix master leader election startup issue
Fixes #error-log-leader-not-selected-yet
* Fix master leader election startup issue
This change improves server address comparison using the 'Equals' method and handles recursion in topology leader lookup, resolving the 'leader not selected yet' error during master startup.
* Merge user improvements: use MaybeLeader for non-blocking checks
* not useful test
* Address code review: optimize Equals, fix deadlock in IsLeader, safe access in Leader
1 month ago
f1bf60d288
faster
1 month ago
35b6e895cc
build(deps): bump org.apache.avro:avro from 1.11.4 to 1.11.5 in /test/kafka/kafka-client-loadtest/tools ( #8339 )
build(deps): bump org.apache.avro:avro
Bumps org.apache.avro:avro from 1.11.4 to 1.11.5.
---
updated-dependencies:
- dependency-name: org.apache.avro:avro
dependency-version: 1.11.5
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1 month ago
49a64f50f1
Add session policy support to IAM ( #8338 )
* Add session policy support to IAM
- Implement policy evaluation for session tokens in policy_engine.go
- Add session_policy field to session claims for tracking applied policies
- Update STS service to include session policies in token generation
- Add IAM integration tests for session policy validation
- Update IAM manager to support policy attachment to sessions
- Extend S3 API STS endpoint to handle session policy restrictions
* fix: optimize session policy evaluation and add documentation
* sts: add NormalizeSessionPolicy helper for inline session policies
* sts: support inline session policies for AssumeRoleWithWebIdentity and credential-based flows
* s3api: parse and normalize Policy parameter for STS HTTP handlers
* tests: add session policy unit tests and integration tests for inline policy downscoping
* tests: add s3tables STS inline policy integration
* iam: handle user principals and validate tokens
* sts: enforce inline session policy size limit
* tests: harden s3tables STS integration config
* iam: clarify principal policy resolution errors
* tests: improve STS integration endpoint selection
1 month ago
beeb375a88
Add volume server integration test suite and CI workflow ( #8322 )
* docs(volume_server): add integration test development plan
* test(volume_server): add integration harness and profile matrix
* test(volume_server/http): add admin and options integration coverage
* test(volume_server/grpc): add state and status integration coverage
* test(volume_server): auto-build weed binary and harden cluster startup
* test(volume_server/http): add upload read range head delete coverage
* test(volume_server/grpc): expand admin lifecycle and state coverage
* docs(volume_server): update progress tracker for implemented tests
* test(volume_server/http): cover if-none-match and invalid-range branches
* test(volume_server/grpc): add batch delete integration coverage
* docs(volume_server): log latest HTTP and gRPC test coverage
* ci(volume_server): run volume server integration tests in github actions
* test(volume_server/grpc): add needle status configure ping and leave coverage
* docs(volume_server): record additional grpc coverage progress
* test(volume_server/grpc): add vacuum integration coverage
* docs(volume_server): record vacuum test coverage progress
* test(volume_server/grpc): add read and write needle blob error-path coverage
* docs(volume_server): record data rw grpc coverage progress
* test(volume_server/http): add jwt auth integration coverage
* test(volume_server/grpc): add sync copy and stream error-path coverage
* docs(volume_server): record jwt and sync/copy test coverage
* test(volume_server/grpc): add scrub and query integration coverage
* test(volume_server/grpc): add volume tail sender and receiver coverage
* docs(volume_server): record scrub query and tail test progress
* test(volume_server/grpc): add readonly writable and collection lifecycle coverage
* test(volume_server/http): add public-port cors and method parity coverage
* test(volume_server/grpc): add blob meta and read-all success path coverage
* test(volume_server/grpc): expand scrub and query variation coverage
* test(volume_server/grpc): add tiering and remote fetch error-path coverage
* test(volume_server/http): add unchanged write and delete edge-case coverage
* test(volume_server/grpc): add ping unknown and unreachable target coverage
* test(volume_server/grpc): add volume delete only-empty variation coverage
* test(volume_server/http): add jwt fid-mismatch auth coverage
* test(volume_server/grpc): add scrub ec auto-select empty coverage
* test(volume_server/grpc): stabilize ping timestamp assertion
* docs(volume_server): update integration coverage progress log
* test(volume_server/grpc): add tier remote backend and config variation coverage
* docs(volume_server): record tier remote variation progress
* test(volume_server/grpc): add incremental copy and receive-file protocol coverage
* test(volume_server/http): add read path shape and if-modified-since coverage
* test(volume_server/grpc): add copy-file compaction and receive-file success coverage
* test(volume_server/http): add passthrough headers and static asset coverage
* test(volume_server/grpc): add ping filer unreachable coverage
* docs(volume_server): record copy receive and http variant progress
* test(volume_server/grpc): add erasure coding maintenance and missing-path coverage
* docs(volume_server): record initial erasure coding rpc coverage
* test(volume_server/http): add multi-range multipart response coverage
* docs(volume_server): record multi-range http coverage progress
* test(volume_server/grpc): add query empty-stripe no-match coverage
* docs(volume_server): record query no-match stream behavior coverage
* test(volume_server/http): add upload throttling timeout and replicate bypass coverage
* docs(volume_server): record upload throttling coverage progress
* test(volume_server/http): add download throttling timeout coverage
* docs(volume_server): record download throttling coverage progress
* test(volume_server/http): add jwt wrong-cookie fid mismatch coverage
* docs(volume_server): record jwt wrong-cookie mismatch coverage
* test(volume_server/http): add jwt expired-token rejection coverage
* docs(volume_server): record jwt expired-token coverage
* test(volume_server/http): add jwt query and cookie transport coverage
* docs(volume_server): record jwt token transport coverage
* test(volume_server/http): add jwt token-source precedence coverage
* docs(volume_server): record jwt token-source precedence coverage
* test(volume_server/http): add jwt header-over-cookie precedence coverage
* docs(volume_server): record jwt header cookie precedence coverage
* test(volume_server/http): add jwt query-over-cookie precedence coverage
* docs(volume_server): record jwt query cookie precedence coverage
* test(volume_server/grpc): add setstate version mismatch and nil-state coverage
* docs(volume_server): record setstate validation coverage
* test(volume_server/grpc): add readonly persist-true lifecycle coverage
* docs(volume_server): record readonly persist variation coverage
* test(volume_server/http): add options origin cors header coverage
* docs(volume_server): record options origin cors coverage
* test(volume_server/http): add trace unsupported-method parity coverage
* docs(volume_server): record trace method parity coverage
* test(volume_server/grpc): add batch delete cookie-check variation coverage
* docs(volume_server): record batch delete cookie-check coverage
* test(volume_server/grpc): add admin lifecycle missing and maintenance variants
* docs(volume_server): record admin lifecycle edge-case coverage
* test(volume_server/grpc): add mixed batch delete status matrix coverage
* docs(volume_server): record mixed batch delete matrix coverage
* test(volume_server/http): add jwt-profile ui access gating coverage
* docs(volume_server): record jwt ui-gating http coverage
* test(volume_server/http): add propfind unsupported-method parity coverage
* docs(volume_server): record propfind method parity coverage
* test(volume_server/grpc): add volume configure success and rollback-path coverage
* docs(volume_server): record volume configure branch coverage
* test(volume_server/grpc): add volume needle status missing-path coverage
* docs(volume_server): record volume needle status error-path coverage
* test(volume_server/http): add readDeleted query behavior coverage
* docs(volume_server): record readDeleted http behavior coverage
* test(volume_server/http): add delete ts override parity coverage
* docs(volume_server): record delete ts parity coverage
* test(volume_server/grpc): add invalid blob/meta offset coverage
* docs(volume_server): record invalid blob/meta offset coverage
* test(volume_server/grpc): add read-all mixed volume abort coverage
* docs(volume_server): record read-all mixed-volume abort coverage
* test(volume_server/http): assert head response body parity
* docs(volume_server): record head body parity assertion
* test(volume_server/grpc): assert status state and memory payload completeness
* docs(volume_server): record volume server status payload coverage
* test(volume_server/grpc): add batch delete chunk-manifest rejection coverage
* docs(volume_server): record batch delete chunk-manifest coverage
* test(volume_server/grpc): add query cookie-mismatch eof parity coverage
* docs(volume_server): record query cookie-mismatch parity coverage
* test(volume_server/grpc): add ping master success target coverage
* docs(volume_server): record ping master success coverage
* test(volume_server/http): add head if-none-match conditional parity
* docs(volume_server): record head if-none-match parity coverage
* test(volume_server/http): add head if-modified-since parity coverage
* docs(volume_server): record head if-modified-since parity coverage
* test(volume_server/http): add connect unsupported-method parity coverage
* docs(volume_server): record connect method parity coverage
* test(volume_server/http): assert options allow-headers cors parity
* docs(volume_server): record options allow-headers coverage
* test(volume_server/framework): add dual volume cluster integration harness
* test(volume_server/http): add missing-local read mode proxy redirect local coverage
* docs(volume_server): record read mode missing-local matrix coverage
* test(volume_server/http): add download over-limit replica proxy fallback coverage
* docs(volume_server): record download replica fallback coverage
* test(volume_server/http): add missing-local readDeleted proxy redirect parity coverage
* docs(volume_server): record missing-local readDeleted mode coverage
* test(volume_server/framework): add single-volume cluster with filer harness
* test(volume_server/grpc): add ping filer success target coverage
* docs(volume_server): record ping filer success coverage
* test(volume_server/http): add proxied-loop guard download timeout coverage
* docs(volume_server): record proxied-loop download coverage
* test(volume_server/http): add disabled upload and download limit coverage
* docs(volume_server): record disabled throttling path coverage
* test(volume_server/grpc): add idempotent volume server leave coverage
* docs(volume_server): record leave idempotence coverage
* test(volume_server/http): add redirect collection query preservation coverage
* docs(volume_server): record redirect collection query coverage
* test(volume_server/http): assert admin server headers on status and health
* docs(volume_server): record admin server header coverage
* test(volume_server/http): assert healthz request-id echo parity
* docs(volume_server): record healthz request-id parity coverage
* test(volume_server/http): add over-limit invalid-vid download branch coverage
* docs(volume_server): record over-limit invalid-vid branch coverage
* test(volume_server/http): add public-port static asset coverage
* docs(volume_server): record public static endpoint coverage
* test(volume_server/http): add public head method parity coverage
* docs(volume_server): record public head parity coverage
* test(volume_server/http): add throttling wait-then-proceed path coverage
* docs(volume_server): record throttling wait-then-proceed coverage
* test(volume_server/http): add read cookie-mismatch not-found coverage
* docs(volume_server): record read cookie-mismatch coverage
* test(volume_server/http): add throttling timeout-recovery coverage
* docs(volume_server): record throttling timeout-recovery coverage
* test(volume_server/grpc): add ec generate mount info unmount lifecycle coverage
* docs(volume_server): record ec positive lifecycle coverage
* test(volume_server/grpc): add ec shard read and blob delete lifecycle coverage
* docs(volume_server): record ec shard read/blob delete lifecycle coverage
* test(volume_server/grpc): add ec rebuild and to-volume error branch coverage
* docs(volume_server): record ec rebuild and to-volume branch coverage
* test(volume_server/grpc): add ec shards-to-volume success roundtrip coverage
* docs(volume_server): record ec shards-to-volume success coverage
* test(volume_server/grpc): add ec receive and copy-file missing-source coverage
* docs(volume_server): record ec receive and copy-file coverage
* test(volume_server/grpc): add ec last-shard delete cleanup coverage
* docs(volume_server): record ec last-shard delete cleanup coverage
* test(volume_server/grpc): add volume copy success path coverage
* docs(volume_server): record volume copy success coverage
* test(volume_server/grpc): add volume copy overwrite-destination coverage
* docs(volume_server): record volume copy overwrite coverage
* test(volume_server/http): add write error-path variant coverage
* docs(volume_server): record http write error-path coverage
* test(volume_server/http): add conditional header precedence coverage
* docs(volume_server): record conditional header precedence coverage
* test(volume_server/http): add oversized combined range guard coverage
* docs(volume_server): record oversized range guard coverage
* test(volume_server/http): add image resize and crop read coverage
* docs(volume_server): record image transform coverage
* test(volume_server/http): add chunk-manifest expansion and bypass coverage
* docs(volume_server): record chunk-manifest read coverage
* test(volume_server/http): add compressed read encoding matrix coverage
* docs(volume_server): record compressed read matrix coverage
* test(volume_server/grpc): add tail receiver source replication coverage
* docs(volume_server): record tail receiver replication coverage
* test(volume_server/grpc): add tail sender large-needle chunking coverage
* docs(volume_server): record tail sender chunking coverage
* test(volume_server/grpc): add ec-backed volume needle status coverage
* docs(volume_server): record ec-backed needle status coverage
* test(volume_server/grpc): add ec shard copy from peer success coverage
* docs(volume_server): record ec shard copy success coverage
* test(volume_server/http): add chunk-manifest delete child cleanup coverage
* docs(volume_server): record chunk-manifest delete cleanup coverage
* test(volume_server/http): add chunk-manifest delete failure-path coverage
* docs(volume_server): record chunk-manifest delete failure coverage
* test(volume_server/grpc): add ec shard copy source-unavailable coverage
* docs(volume_server): record ec shard copy source-unavailable coverage
* parallel
1 month ago
Chris Lu
Chris Lu
Lisandro Pin
dependabot[bot]