Browse Source
Phase 12: ROOT CAUSE FOUND - Duplicates due to Topic Persistence Bug
Duplicate Analysis:
- 8104 duplicates (66.5%), ALL read exactly 2 times
- Suggests single rebalance/restart event
- Duplicates start at offset 0, go to ~800 (50% of data)
Investigation Results:
1. Offset commits ARE working (logging shows commits every 20 msgs)
2. NO rebalance during normal operation (only 10 OFFSET_FETCH at start)
3. Consumer error logs show REPEATED failures:
'Request was for a topic or partition that does not exist'
4. Broker logs show: 'no entry is found in filer store' for topic-2
Root Cause:
Auto-created topics are NOT being reliably persisted to filer!
- Producer auto-creates topic-2
- Topic config NOT saved to filer
- Consumer tries to fetch metadata → broker says 'doesn't exist'
- Consumer group errors → Sarama triggers rebalance
- During rebalance, OffsetFetch returns -1 (no offset found)
- Consumer starts from offset 0 again → DUPLICATES!
The Flow:
1. Consumers start, read 0-800, commit offsets
2. Consumer tries to fetch metadata for topic-2
3. Broker can't find topic config in filer
4. Consumer group crashes/rebalances
5. OffsetFetch during rebalance returns -1
6. Consumers restart from offset 0 → re-read 0-800
7. Then continue from 800-1600 → 66% duplicates
Next Fix:
Ensure topic auto-creation RELIABLY persists config to filer
before returning success to producers.
pull/7329/head
1 changed files with 5 additions and 5 deletions
Loading…
Reference in new issue