Chris Lu
e88392b50f
Fix s3 pagination ( #3436 )
* Revert previous changes
* s3: use cursor to track tree traversal
fix https://github.com/seaweedfs/seaweedfs/issues/3166
* special cases for empty prefix and empty directory
* use constants
* address empty folder
* undo local changes
* fix IsTruncated
* adjust counting directories
* fix cases when prefix is a directory
* s3: handle directory object
works for
aws --endpoint-url http://127.0.0.1:8333/ s3api list-objects-v2 --bucket test --prefix "fakedir"
2 years ago
chrislu
90db4b5a44
minor
2 years ago
chrislu
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
2 years ago
chrislu
69ef6459c3
S3 ListObjectsV2 fix pagination under bucket root
fix https://github.com/chrislusf/seaweedfs/issues/3166
3 years ago
Konstantin Lebedev
3d5f1cbc5e
fix pagination
3 years ago
Konstantin Lebedev
bcbdc4cb37
use const multipart uploads folder
avoid error bucket NotEmpty if multipart uploads folder exist
3 years ago
Konstantin Lebedev
c07820178f
fix s3 tests
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
3 years ago
Konstantin Lebedev
b86628f85d
fix show empty dir
https://github.com/chrislusf/seaweedfs/issues/3086
3 years ago
chrislu
27732ecfa4
move s3 related constants from package http to s3_constants
3 years ago
Konstantin Lebedev
62e5e3822c
list self dir
https://github.com/chrislusf/seaweedfs/issues/3086
3 years ago
Konstantin Lebedev
26f3646961
show fake dirs
https://github.com/chrislusf/seaweedfs/issues/3086
3 years ago
Konstantin Lebedev
54356211c2
list empty folder
3 years ago
chrislu
9f9ef1340c
use streaming mode for long poll grpc calls
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
3 years ago
Konstantin Lebedev
4ec8715f20
audit log
3 years ago
Chris Lu
6e999f1176
s3: fix ListObject if more than 10000 objects
3 years ago
Chris Lu
f17fa400d5
refactoring
3 years ago
Chris Lu
006b4ec0f8
refactoring
3 years ago
Chris Lu
71175461ef
add glog for s3 handlers
3 years ago
Chris Lu
8b382a8209
refactor
4 years ago
Chris Lu
33b87244ef
refactoring
4 years ago
Chris Lu
e87e6ef33c
s3: return 404 if bucket does not exist
fix https://github.com/chrislusf/seaweedfs/issues/2039
4 years ago
wuh-fnst
050a7f9599
s3: count correction for the number of files in the dir
4 years ago
Chris Lu
ccbdb38c89
s3: fix for listing objects if more than 1000 in the folder
many existing tools have max key set to 1000
4 years ago
Chris Lu
389426bbb7
s3: listing may repeat on the edge
4 years ago
Chris Lu
8a7302ce37
s3: KeyCount is zero
fix https://github.com/chrislusf/seaweedfs/issues/1741
4 years ago
Chris Lu
561a2ff0bc
s3: add option for "alllowEmptyFolder"
4 years ago
Chris Lu
8e48a235e2
s3: avoid looping if the directory is empty
fix https://github.com/chrislusf/seaweedfs/issues/1701
4 years ago
Chris Lu
1620de08ae
added more logs
4 years ago
Chris Lu
a09cd28986
add debug messages
4 years ago
Chris Lu
3be3635799
just log error when checking empty dir failed
4 years ago
Chris Lu
0ca9d89589
s3: break loop if error
fix #1701
4 years ago
Chris Lu
132f275d04
s3: log errors instead of stopping when checking isDirectoryAllEmpty has error
4 years ago
Chris Lu
db62090f88
s3: break loop if error
fix https://github.com/chrislusf/seaweedfs/issues/1701
4 years ago
Chris Lu
5c465293e9
correctly determine whether a folder is empty
avoid edge cases that deleting the folder if previous 32 directories are all empty
early terminate if one file is found
4 years ago
Chris Lu
f930c713fc
more efficient recursion
4 years ago
Chris Lu
03637d6f57
s3: move "delete-directory-if-empty" to read time
move "delete-directory-if-empty" to read time instead of entry deletion time
the listing speed for a s3 bucket folder will slow down if it has many sub folders
related to 0d345ac97d
fix https://github.com/chrislusf/seaweedfs/issues/1647
fix https://github.com/chrislusf/seaweedfs/issues/1670
4 years ago
Chris Lu
02a160c3fd
s3: recursively iterate all sub folders
fix https://github.com/chrislusf/seaweedfs/issues/1656
4 years ago
ruitao.liu
22a9ea0512
adjust s3 header file.
4 years ago
ruitao.liu
b917be7955
S3 bucket list, response with uploaded storageclass.
4 years ago
Chris Lu
5b40a2690a
refactoring
4 years ago
Chris Lu
daf0a449f7
properly cancel context for streaming grpc
4 years ago
Chris Lu
eb7929a971
rename filer2 to filer
4 years ago
Chris Lu
99ecf63276
go fmt
4 years ago
Chris Lu
c45ba5d7d4
fix listObjectsV2 response format
fix https://github.com/chrislusf/seaweedfs/issues/1426 issue 1
4 years ago
Chris Lu
c647deace1
file size support set file length
use Attr.FileSize and TotalChunkSize to determine file size
4 years ago
Chris Lu
f86c7d911a
remove println
4 years ago
Chris Lu
9ecc1170a3
istObjects和listObjectsV2不能查询子目录 #1418
fix https://github.com/chrislusf/seaweedfs/issues/1418
4 years ago
Chris Lu
3f4aff5dde
s3: fix delimiter in list response
4 years ago
limd
b41b7ea4d0
fix spark read s3 bug (sc.binaryFiles)
4 years ago
Chris Lu
54cd582653
refactor
4 years ago