PROBLEM:
Consumer throughput still 45.46 msgs/sec vs producer 50.29 msgs/sec (10% gap).
ROOT CAUSE:
Manual session.Commit() every 10 messages creates excessive overhead:
- 1,880 messages consumed → 188 commit operations
- Each commit is SYNCHRONOUS and blocks message processing
- Auto-commit is already enabled (5s interval)
- Double-committing reduces effective throughput
ANALYSIS:
- Test showed consumer lag at 0 at end (not falling behind)
- Only ~1,880 of 12,200 messages consumed during 2-minute window
- Consumers start 2s late, need ~262s to consume all at current rate
- Commit overhead: 188 RPC round trips = significant latency
FIX:
Reduce manual commit frequency from every 10 to every 100 messages:
- Only 18-20 manual commits during entire test
- Auto-commit handles primary offset persistence (5s interval)
- Manual commits serve as backup for edge cases
- Unblocks message processing loop for higher throughput
EXPECTED IMPACT:
- Consumer throughput: 45.46 → ~49+ msgs/sec (match producer!)
- Latency reduction: Fewer synchronous commits
- Test duration: Should consume all messages before test ends
chrislu
5ae0b21b22