Fixed critical race condition in CompactMap where Set(), Delete(), and
Get() methods had issues with concurrent map access.
Root cause: segmentForKey() can create new map segments, which modifies
the cm.segments map. Calling this under a read lock caused concurrent
map write panics when multiple goroutines accessed the map simultaneously
(e.g., during VolumeNeedleStatus gRPC calls).
Changes:
- Set() method: Changed RLock/RUnlock to Lock/Unlock
- Delete() method: Changed RLock/RUnlock to Lock/Unlock, optimized to
avoid creating empty segments when key doesn't exist
- Get() method: Removed segmentForKey() call to avoid race condition,
now checks segment existence directly and returns early if segment
doesn't exist (optimization: avoids unnecessary segment creation)
This fix resolves the runtime/maps.fatal panic that occurred under
concurrent load.
Tested with race detector: go test -v -race ./weed/storage/needle_map/...
This is a follow-up fix to PR #7332 which partially addressed the issue.
The problem is that size=0 needles are in a gray area:
- IsValid() returns false for size=0 (because size must be > 0)
- IsDeleted() returns false for size=0 (because size must be < 0 or == TombstoneFileSize)
PR #7332 only fixed 2 places, but several other places still had the same bug:
1. needle_map_memory.go:doLoading - line 43 still used oldSize.IsValid()
2. needle_map_memory.go:DoOffsetLoading - used during vacuum and incremental loading
3. needle_map_leveldb.go:generateLevelDbFile - used when generating LevelDB needle maps
4. needle_map_leveldb.go:DoOffsetLoading - used during incremental loading for LevelDB
5. needle_map/compact_map.go:delete - couldn't delete size=0 entries because:
- The condition 'size > 0' failed for size=0
- Even if it passed, negating 0 gives 0 (not marking as deleted)
Changes:
- Changed size.IsValid() to !size.IsDeleted() in doLoading and DoOffsetLoading functions
- Fixed compact_map delete to use TombstoneFileSize for size=0 entries
Fixes#7293
* Rework `needle_map.CompactMap()` to maximize memory efficiency.
* Use a memory-efficient structure for `CompactMap` needle value entries.
This slightly complicates the code, but makes a **massive** difference
in memory efficiency - preliminary results show a ~30% reduction in
heap usage, with no measurable performance impact otherwise.
* Clean up type for `CompactMap` chunk IDs.
* Add a small comment description for `CompactMap()`.
* Add the old version of `CompactMap()` for comparison purposes.
Chris Lu
Lisandro Pin
Chris Lu
binbinshi
j.laycock