Browse Source

fix: CompleteMultipartUpload fails for uploads with more than 1000 parts (#7641)

When completing a multipart upload, the code was listing parts with limit=0,
which relies on the server's DirListingLimit default. In 'weed server' mode,
this defaults to 1000, causing uploads with more than 1000 parts to fail
with InvalidPart error.

For a 38GB file with 8MB parts (AWS CLI default), this results in ~4564 parts,
far exceeding the 1000 limit.

Fix: Use explicit limit of MaxS3MultipartParts+1 (10001) to ensure all parts
are listed regardless of server configuration.

Fixes #7638
pull/7642/head
Chris Lu 4 days ago
committed by GitHub
parent
commit
9c266fac29
No known key found for this signature in database GPG Key ID: B5690EEEBB952194
  1. 5
      weed/s3api/filer_multipart.go

5
weed/s3api/filer_multipart.go

@ -187,7 +187,10 @@ func (s3a *S3ApiServer) completeMultipartUpload(r *http.Request, input *s3.Compl
sort.Ints(completedPartNumbers)
uploadDirectory := s3a.genUploadsFolder(*input.Bucket) + "/" + *input.UploadId
entries, _, err := s3a.list(uploadDirectory, "", "", false, 0)
// Use explicit limit to ensure all parts are listed (up to S3's max of 10,000 parts)
// Previously limit=0 relied on server's DirListingLimit default (1000 in weed server mode),
// which caused CompleteMultipartUpload to fail for uploads with more than 1000 parts.
entries, _, err := s3a.list(uploadDirectory, "", "", false, s3_constants.MaxS3MultipartParts+1)
if err != nil {
glog.Errorf("completeMultipartUpload %s %s error: %v, entries:%d", *input.Bucket, *input.UploadId, err, len(entries))
stats.S3HandlerCounter.WithLabelValues(stats.ErrorCompletedNoSuchUpload).Inc()

Loading…
Cancel
Save