seaweedfs

13201 Commits

119 Branches

330 Tags

364 MiB

Tree: a080fbb495

Commit Graph

Author	SHA1	Message	Date
Chris Lu	51ec0d2122	fix(remote_gateway): prevent double-versioning when syncing to versioned central bucket (#8710 ) * fix(remote_gateway): prevent double-versioning when syncing to versioned central bucket When a file is uploaded to a versioned bucket on edge, SeaweedFS stores it internally as {object}.versions/v_{versionId}. The remote_gateway was syncing this internal path directly to the central S3 endpoint. When central's bucket also has versioning enabled, this caused central to apply its own versioning on top, producing corrupt paths like: object.versions/v_{edgeId}.versions/v_{centralId} Fix: rewrite internal .versions/v_{id} paths to the original S3 object key before uploading to the remote. Skip version file delete/update events that are internal bookkeeping. Fixes https://github.com/seaweedfs/seaweedfs/discussions/8481#discussioncomment-16209342 * fix(remote_gateway): propagate delete markers to remote as deletions Delete markers are zero-content version entries (ExtDeleteMarkerKey=true) created by S3 DELETE on a versioned bucket. Previously they were silently dropped by the HasData() filter, so deletions on edge never reached central. Now: detect delete markers before the HasData check, rewrite the .versions path to the original S3 key, and issue client.DeleteFile() on the remote. * fix(remote_gateway): tighten isVersionedPath to avoid false positives Address PR review feedback: - Add isDir parameter to isVersionedPath so it only matches the exact internal shapes: directories whose name ends with .versions (isDir=true), and files with the v_ prefix inside a .versions parent (isDir=false). Previously the function was too broad and could match user-created paths like "my.versions/data.txt". - Update all 4 call sites to pass the entry's IsDirectory field. - Rename TestVersionedDirectoryNotFilteredByHasData to TestVersionsDirectoryFilteredByHasData so the name reflects the actual assertion (directories ARE filtered by HasData). - Expand TestIsVersionedPath with isDir cases and false-positive checks. * fix(remote_gateway): persist sync marker after delete-marker propagation The delete-marker branch was calling client.DeleteFile() and returning without updating the local entry, making event replay re-issue the remote delete. Now call updateLocalEntry after a successful DeleteFile to stamp the delete-marker entry with a RemoteEntry, matching the pattern used by the normal create path. * refactor(remote_gateway): extract syncDeleteMarker and fix root path edge case - Extract syncDeleteMarker() shared helper used by both bucketed and mounted-dir event processors, replacing the duplicated delete + persist local marker logic. - Fix rewriteVersionedSourcePath for root-level objects: when lastSlash is 0 (e.g. "/file.xml.versions"), return "/" as the parent dir instead of an empty string. - The strings.Contains(dir, ".versions/") condition flagged in review was already removed in a prior commit that tightened isVersionedPath. * fix(remote_gateway): skip updateLocalEntry for versioned path rewrites After rewriting a .versions/v_{id} path to the logical S3 key and uploading, the code was calling updateLocalEntry on the original v_* entry, stamping it with a RemoteEntry for the logical key. This is semantically wrong: the logical object has no filer entry in versioned buckets, and the internal v_* entry should not carry a RemoteEntry for a different path. Skip updateLocalEntry when the path was rewritten from a versioned source. Replay safety is preserved because S3 PutObject is idempotent. * fix(remote_gateway): scope versioning checks to /buckets/ namespace isVersionedPath and rewriteVersionedSourcePath could wrongly match paths in non-bucket mounts (e.g. /mnt/remote/file.xml.versions). Add the same /buckets/ prefix guard used by isMultipartUploadDir so the .versions / v_ logic only applies within the bucket namespace.	1 week ago

Author

SHA1

Message

Date

Chris Lu

51ec0d2122

fix(remote_gateway): prevent double-versioning when syncing to versioned central bucket (#8710 )

* fix(remote_gateway): prevent double-versioning when syncing to versioned central bucket

When a file is uploaded to a versioned bucket on edge, SeaweedFS stores
it internally as {object}.versions/v_{versionId}. The remote_gateway was
syncing this internal path directly to the central S3 endpoint. When
central's bucket also has versioning enabled, this caused central to
apply its own versioning on top, producing corrupt paths like:
  object.versions/v_{edgeId}.versions/v_{centralId}

Fix: rewrite internal .versions/v_{id} paths to the original S3 object
key before uploading to the remote. Skip version file delete/update
events that are internal bookkeeping.

Fixes https://github.com/seaweedfs/seaweedfs/discussions/8481#discussioncomment-16209342

* fix(remote_gateway): propagate delete markers to remote as deletions

Delete markers are zero-content version entries (ExtDeleteMarkerKey=true)
created by S3 DELETE on a versioned bucket. Previously they were silently
dropped by the HasData() filter, so deletions on edge never reached
central.

Now: detect delete markers before the HasData check, rewrite the
.versions path to the original S3 key, and issue client.DeleteFile()
on the remote.

* fix(remote_gateway): tighten isVersionedPath to avoid false positives

Address PR review feedback:

- Add isDir parameter to isVersionedPath so it only matches the exact
  internal shapes: directories whose name ends with .versions (isDir=true),
  and files with the v_ prefix inside a .versions parent (isDir=false).
  Previously the function was too broad and could match user-created paths
  like "my.versions/data.txt".

- Update all 4 call sites to pass the entry's IsDirectory field.

- Rename TestVersionedDirectoryNotFilteredByHasData to
  TestVersionsDirectoryFilteredByHasData so the name reflects the
  actual assertion (directories ARE filtered by HasData).

- Expand TestIsVersionedPath with isDir cases and false-positive checks.

* fix(remote_gateway): persist sync marker after delete-marker propagation

The delete-marker branch was calling client.DeleteFile() and returning
without updating the local entry, making event replay re-issue the
remote delete. Now call updateLocalEntry after a successful DeleteFile
to stamp the delete-marker entry with a RemoteEntry, matching the
pattern used by the normal create path.

* refactor(remote_gateway): extract syncDeleteMarker and fix root path edge case

- Extract syncDeleteMarker() shared helper used by both bucketed and
  mounted-dir event processors, replacing the duplicated delete + persist
  local marker logic.

- Fix rewriteVersionedSourcePath for root-level objects: when lastSlash
  is 0 (e.g. "/file.xml.versions"), return "/" as the parent dir instead
  of an empty string.

- The strings.Contains(dir, ".versions/") condition flagged in review was
  already removed in a prior commit that tightened isVersionedPath.

* fix(remote_gateway): skip updateLocalEntry for versioned path rewrites

After rewriting a .versions/v_{id} path to the logical S3 key and
uploading, the code was calling updateLocalEntry on the original v_*
entry, stamping it with a RemoteEntry for the logical key. This is
semantically wrong: the logical object has no filer entry in versioned
buckets, and the internal v_* entry should not carry a RemoteEntry for
a different path.

Skip updateLocalEntry when the path was rewritten from a versioned
source. Replay safety is preserved because S3 PutObject is idempotent.

* fix(remote_gateway): scope versioning checks to /buckets/ namespace

isVersionedPath and rewriteVersionedSourcePath could wrongly match
paths in non-bucket mounts (e.g. /mnt/remote/file.xml.versions).

Add the same /buckets/ prefix guard used by isMultipartUploadDir so
the .versions / v_ logic only applies within the bucket namespace.

1 week ago

1 Commits (a080fbb495278c66d74762d683e61dcde0d778d3)