New issue 45 by hieu.hcmus@gmail.com: [Compact issue] Offset overflow
http://code.google.com/p/weed-fs/issues/detail?id=45
You are using uint32(Maximum 4Gb) to store needle offset(Maximum 32Gb)
when compacting.
Currently It is ok if the volume size is < 4gb
Change variable "offset" in ScanVolumeFile function to uint64 to fix the
issue.
Checked in for GThomas(tgulacsi78)
I've also met with partial writes (pipe closed on localhost), and also
met with real corruption: after some partial write, the data size in
needle header was read as some huuuuge number, which resulted in memory
panic.
Please consider the attached patch for ensuring full writes (seeks back
to the beginning on needle append error). Hope this helps.
Tested with a small tmpfs, lot of "no space left on device" :)
(sudo umount -lf /tmp/weed; mkdir -p /tmp/weed && sudo mount -o
size=128M,mode=4777 -t tmpfs tmpfs /tmp/weed && bin/weed master
-mdir=/tmp/weed -debug=true & bin/weed volume -dir=/tmp/weed
-debug=true)
GThomas
Walk needed to be added to NeedleMap and CompactMap, to be able to add WalkKeys and WalkValues to volume. This is needed for iterating through all the stored needles in a volume - this was dump's purpose.
Chris Lu
Tamás Gulácsi