Phase 4.5: Root Cause Identified - Broker-Side Bug
Added detailed logging to detect when broker returns 0 messages despite HWM indicating data exists:
- CRITICAL BUG log when broker returns empty but HWM > requestedOffset
- Logs broker metadata (logStart, nextOffset, endOfPartition)
- Per-message logging for debugging
Changes:
- broker_client_fetch.go: Added CRITICAL BUG detection and logging
Test Results:
- 87.9% delivery (2067/2350) - consistent with previous
- Confirmed broker bug: Returns 0 messages for offset 1424 when HWM=1428
Root Cause Discovered:
✅ Gateway fetch logic is CORRECT
✅ HWM calculation is CORRECT
❌ Broker's ReadMessagesAtOffset or disk read function FAILING SILENTLY
Evidence:
Multiple CRITICAL BUG logs show broker can't retrieve data that exists:
- topic-3[0] offset 1424 (HWM=1428)
- topic-2[0] offset 968 (HWM=969)
Answer to 'Why does stream stop?':
1. Broker can't retrieve data from storage for certain offsets
2. Gateway gets empty responses repeatedly
3. Sarama gives up thinking no more data
4. Channel closes cleanly (not a crash)
Next: Investigate broker's ReadMessagesAtOffset and disk read path