Browse Source

fix: properly handle errors in writeToFile to prevent 0-byte EC shards (#7620)

Fixes #7619

The writeToFile function had two critical bugs that could cause data loss
during EC shard evacuation when the destination disk is full:

Bug 1: When os.OpenFile fails (e.g., disk full), the error was silently
ignored and nil was returned. This caused the caller to think the copy
succeeded.

Bug 2: When dst.Write fails (e.g., 'no space left on device'), the error
was completely ignored because the return value was not checked.

When evacuating EC shards to a full volume server (especially on BTRFS):
1. OpenFile may succeed (creates 0-byte file inode)
2. Write fails with 'no space left on device'
3. Errors were ignored, function returned nil
4. Caller thinks copy succeeded and deletes source shard
5. Result: 0-byte shard on destination, data loss!

This fix ensures both errors are properly returned, preventing data loss.
Added unit tests to verify the fix.
pull/7626/head
Chris Lu 4 weeks ago
committed by GitHub
parent
commit
fdb888729b
No known key found for this signature in database GPG Key ID: B5690EEEBB952194
  1. 8
      weed/server/volume_grpc_copy.go

8
weed/server/volume_grpc_copy.go

@ -264,7 +264,7 @@ func writeToFile(client volume_server_pb.VolumeServer_CopyFileClient, fileName s
}
dst, err := os.OpenFile(fileName, flags, 0644)
if err != nil {
return modifiedTsNs, nil
return modifiedTsNs, fmt.Errorf("open file %s: %w", fileName, err)
}
defer dst.Close()
@ -278,9 +278,11 @@ func writeToFile(client volume_server_pb.VolumeServer_CopyFileClient, fileName s
modifiedTsNs = resp.ModifiedTsNs
}
if receiveErr != nil {
return modifiedTsNs, fmt.Errorf("receiving %s: %v", fileName, receiveErr)
return modifiedTsNs, fmt.Errorf("receiving %s: %w", fileName, receiveErr)
}
if _, writeErr := dst.Write(resp.FileContent); writeErr != nil {
return modifiedTsNs, fmt.Errorf("write file %s: %w", fileName, writeErr)
}
dst.Write(resp.FileContent)
progressedBytes += int64(len(resp.FileContent))
if progressFn != nil {
if !progressFn(progressedBytes) {

Loading…
Cancel
Save