Branch:
feature/mq-kafka-gateway-m1
add-ec-vacuum
add-foundation-db
add_fasthttp_client
add_remote_storage
adding-message-queue-integration-tests
avoid_releasing_temp_file_on_write
changing-to-zap
collect-public-metrics
create-table-snapshot-api-design
data_query_pushdown
dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5
dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0
detect-and-plan-ec-tasks
do-not-retry-if-error-is-NotFound
fasthttp
feature/mq-kafka-gateway-m1
filer1_maintenance_branch
fix-GetObjectLockConfigurationHandler
fix-versioning-listing-only
ftp
gh-pages
improve-fuse-mount
improve-fuse-mount2
logrus
master
message_send
mount2
mq-subscribe
mq2
original_weed_mount
random_access_file
refactor-needle-read-operations
refactor-volume-write
remote_overlay
revert-5134-patch-1
revert-5819-patch-1
revert-6434-bugfix-missing-s3-audit
s3-select
sub
tcp_read
test-reverting-lock-table
test_udp
testing
testing-sdx-generation
tikv
track-mount-e2e
volume_buffered_writes
worker-execute-ec-tasks
0.72
0.72.release
0.73
0.74
0.75
0.76
0.77
0.90
0.91
0.92
0.93
0.94
0.95
0.96
0.97
0.98
0.99
1.00
1.01
1.02
1.03
1.04
1.05
1.06
1.07
1.08
1.09
1.10
1.11
1.12
1.14
1.15
1.16
1.17
1.18
1.19
1.20
1.21
1.22
1.23
1.24
1.25
1.26
1.27
1.28
1.29
1.30
1.31
1.32
1.33
1.34
1.35
1.36
1.37
1.38
1.40
1.41
1.42
1.43
1.44
1.45
1.46
1.47
1.48
1.49
1.50
1.51
1.52
1.53
1.54
1.55
1.56
1.57
1.58
1.59
1.60
1.61
1.61RC
1.62
1.63
1.64
1.65
1.66
1.67
1.68
1.69
1.70
1.71
1.72
1.73
1.74
1.75
1.76
1.77
1.78
1.79
1.80
1.81
1.82
1.83
1.84
1.85
1.86
1.87
1.88
1.90
1.91
1.92
1.93
1.94
1.95
1.96
1.97
1.98
1.99
1;70
2.00
2.01
2.02
2.03
2.04
2.05
2.06
2.07
2.08
2.09
2.10
2.11
2.12
2.13
2.14
2.15
2.16
2.17
2.18
2.19
2.20
2.21
2.22
2.23
2.24
2.25
2.26
2.27
2.28
2.29
2.30
2.31
2.32
2.33
2.34
2.35
2.36
2.37
2.38
2.39
2.40
2.41
2.42
2.43
2.47
2.48
2.49
2.50
2.51
2.52
2.53
2.54
2.55
2.56
2.57
2.58
2.59
2.60
2.61
2.62
2.63
2.64
2.65
2.66
2.67
2.68
2.69
2.70
2.71
2.72
2.73
2.74
2.75
2.76
2.77
2.78
2.79
2.80
2.81
2.82
2.83
2.84
2.85
2.86
2.87
2.88
2.89
2.90
2.91
2.92
2.93
2.94
2.95
2.96
2.97
2.98
2.99
3.00
3.01
3.02
3.03
3.04
3.05
3.06
3.07
3.08
3.09
3.10
3.11
3.12
3.13
3.14
3.15
3.16
3.18
3.19
3.20
3.21
3.22
3.23
3.24
3.25
3.26
3.27
3.28
3.29
3.30
3.31
3.32
3.33
3.34
3.35
3.36
3.37
3.38
3.39
3.40
3.41
3.42
3.43
3.44
3.45
3.46
3.47
3.48
3.50
3.51
3.52
3.53
3.54
3.55
3.56
3.57
3.58
3.59
3.60
3.61
3.62
3.63
3.64
3.65
3.66
3.67
3.68
3.69
3.71
3.72
3.73
3.74
3.75
3.76
3.77
3.78
3.79
3.80
3.81
3.82
3.83
3.84
3.85
3.86
3.87
3.88
3.89
3.90
3.91
3.92
3.93
3.94
3.95
3.96
3.97
dev
helm-3.65.1
v0.69
v0.70beta
v3.33
${ item.name }
${ noResults }
4 Commits (feature/mq-kafka-gateway-m1)
| Author | SHA1 | Message | Date |
|---|---|---|---|
chrislu
|
093b5062e4 |
Add detailed debug logging for consumer group coordination
- JoinGroup: request parsing, state transitions, member add/update, response summary - SyncGroup: pre/post state, leader/non-leader paths, assignment summary - Heartbeat: parse/validate, member/gen checks, response details - LeaveGroup: parse, pre/post state, member removal, leader transitions - OffsetCommit/Fetch: parse errors, request summaries, response counts - Assignment strategies: topic/member/partition assignment tracing - Group cleanup: expired member removal, state transitions, stuck rebalance These logs will help trace timeouts in OffsetManagement by showing exactly where coordination stalls or misaligns with client expectations. |
7 hours ago |
|
chrislu
|
de85e9b90d |
Implement incremental cooperative rebalancing protocol
- Add IncrementalCooperativeAssignmentStrategy with two-phase rebalancing: * Revocation phase: Members give up partitions that need reassignment * Assignment phase: Members receive new partitions after revocation - Implement IncrementalRebalanceState to track rebalance progress: * Phase tracking (None, Revocation, Assignment) * Revocation timeout handling with configurable timeouts * Partition tracking for revoked and pending assignments - Add sophisticated assignment logic: * Respect member topic subscriptions when distributing partitions * Calculate ideal assignments and determine necessary revocations * Support multiple topics with different subscription patterns * Minimize partition movement while ensuring fairness - Add comprehensive test coverage: * Basic assignment scenarios without rebalancing * Rebalance scenarios with revocation and assignment phases * Multiple topic scenarios with mixed subscriptions * Timeout handling and forced completion * State transition verification - Update GetAssignmentStrategy to support 'incremental-cooperative' protocol - Implement monitoring methods: * IsRebalanceInProgress() for status checking * GetRebalanceState() for detailed state inspection * ForceCompleteRebalance() for timeout scenarios This enables advanced rebalancing that reduces 'stop-the-world' effects by allowing consumers to incrementally give up and receive partitions during rebalancing. |
4 days ago |
|
chrislu
|
549d88bc7b |
Phase 2 Part 1: Consumer group robustness improvements
- Implement comprehensive rebalancing tests with multiple scenarios: * Single consumer gets all partitions * Two consumers rebalance when second joins * Consumer leave triggers rebalancing * Multiple consumers join simultaneously - Add cooperative-sticky assignment strategy with: * Sticky behavior to minimize partition movement * Fair distribution respecting target counts * Support for multiple topics and partial subscriptions * Comprehensive test coverage for all scenarios - Implement advanced rebalance timeout handling: * RebalanceTimeoutManager for sophisticated timeout logic * Member eviction based on rebalance and session timeouts * Leader eviction and automatic leader selection * Stuck rebalance detection and forced completion * Detailed rebalance status reporting with member timeout info - Add ProduceMessageToPartition method for partition-specific testing - All new functionality includes comprehensive test coverage Consumer group robustness significantly improved with production-ready timeout handling and assignment strategies. |
4 days ago |
|
chrislu
|
d415911943 |
mq(kafka): Phase 3 Step 1 - Consumer Group Foundation
- Implement comprehensive consumer group coordinator with state management - Add JoinGroup API (key 11) for consumer group membership - Add SyncGroup API (key 14) for partition assignment coordination - Create Range and RoundRobin assignment strategies - Support consumer group lifecycle: Empty -> PreparingRebalance -> CompletingRebalance -> Stable - Add automatic member cleanup and expired session handling - Comprehensive test coverage for consumer groups, assignment strategies - Update ApiVersions to advertise 9 APIs total (was 7) - All existing integration tests pass with new consumer group support This provides the foundation for distributed Kafka consumers with automatic partition rebalancing and group coordination, compatible with standard Kafka clients. |
1 week ago |