Phase 9: Root Cause Identified - Disk Cache Not Updated on Flush
Analysis:
- Consumer stops at offset 600/601 (pattern repeats at multiples of ~600)
- Buffer state shows: startOffset=601, bufferStart=602 (data flushed!)
- Disk read attempts to read offset 601
- Disk cache contains ONLY offsets 0-100 (first flush)
- Subsequent flushes (101-150, 151-200, ..., 551-601) NOT in cache
Flush logs confirm regular flushes:
- offset 51: First flush (0-50)
- offset 101: Second flush (51-100)
- offset 151, 201, 251, ..., 602: Subsequent flushes
- ALL flushes succeed, but cache not updated!
ROOT CAUSE:
The disk cache (diskChunkCache) is only populated on the FIRST
flush. Subsequent flushes write to disk successfully, but the
cache is never updated with the new chunk boundaries.
When a consumer requests offset 601:
1. Buffer has flushed, so bufferStart=602
2. Code correctly tries disk read
3. Cache has chunk 0-100, returns 'data not on disk'
4. Code returns empty, consumer stalls
FIX NEEDED:
Update diskChunkCache after EVERY flush, not just first one.
OR invalidate cache more aggressively to force fresh reads.
Next: Fix diskChunkCache update in flush logic
chrislu
b1856dfc9f