Phase 6: Disk Read Fix Attempt #1
Added cache invalidation when extraction fails due to offset beyond cached chunk:
- extractMessagesFromCache: Returns error when offset beyond cache
- readHistoricalDataFromDisk: Invalidates bad cache and retries
- invalidateCachedDiskChunk: New function to remove stale cache
Problem Discovered:
Cache invalidation works, but re-reading returns SAME incomplete data!
Example:
- Request offset 1764
- Disk read returns 764 messages (1000-1763)
- Cache stores 1000-1763
- Request 1764 again → cache invalid → re-read → SAME 764 messages!
Root Cause:
ReadFromDiskFn (GenLogOnDiskReadFunc) is NOT returning incomplete data
The disk files ACTUALLY only contain up to offset 1763
Messages 1764+ are either:
1. Still in memory (not yet flushed)
2. In a different file not being read
3. Lost during flush
Test Results: 73.3% delivery (worse than before 87.9%)
Cache thrashing causing performance degradation
Next: Fix the actual disk read to handle gaps between flushed data and in-memory data
chrislu