81624de27b
Include name/mime in ReadAllNeedles ( #4005 )
2 years ago
26d9b662b1
ADHOC: fix fs.meta.save typo ( #3993 )
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2 years ago
4c85da7844
Include meta in ReadAllNeedles ( #3991 )
This is useful for doing backups on the data so we can accurately store the
last modified time, the compression state, and verify the crc.
Previously we were doing VolumeNeedleStatus and then an HTTP request which
needlessly read from the dat file twice.
2 years ago
70a4c98b00
refactor filer_pb.Entry and filer.Entry to use GetChunks()
for later locking on reading chunks
2 years ago
b1d7963e57
3.35
2 years ago
5b905fb2b7
Lazy loading ( #3958 )
* types packages is imported more than onece
* lazy-loading
* fix bugs
* fix bugs
* fix unit tests
* fix test error
* rename function
* unload ldb after initial startup
* Don't load ldb when starting volume server if ldbtimeout is set.
* remove uncessary unloadldb
* Update weed/command/server.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/command/volume.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2 years ago
d90aa31d5f
filer store: skip disabled location specific filer store initialization
fix https://github.com/seaweedfs/seaweedfs/issues/3971
2 years ago
aae82dca6f
file name escape uri
fix https://github.com/seaweedfs/seaweedfs/issues/3969
2 years ago
39f0246deb
fix 'NoSuchUpload' when upload part ( #3968 )
2 years ago
df0662038c
feat:add a uuid after uploadid ( #3963 )
2 years ago
180853a2c9
Replace dashes with underscores in x-amz-meta headers ( #3965 )
2 years ago
47db75a695
3.34
2 years ago
5431c445cd
fix filer.remote.sync to azure with ContentType ( #3949 )
* fix filer.remote.sync to azure with ContentType
* fix pass X-Amz-Meta to X-Ms-Meta
2 years ago
cec55feeb0
[filer.backup] add retention_duration ( #3941 )
2 years ago
4193dafce1
azure metadata: skip metadata prefixed with "X-"
fix https://github.com/seaweedfs/seaweedfs/issues/3875
2 years ago
abe4a61659
Bug fix: empty key in DeleteMultipleObjects request caused bucket delete ( #3939 )
2 years ago
0999f9b7ff
[volume.fsck] collect ids without cut off time for finding missing data from volumes ( #3934 )
collect all file ids from the file without cut off time for finding missing data from volumes
2 years ago
a322ba042e
[volume.fsck] param volumeId is comma separated the volume id ( #3933 )
volume.fsck param volumeId is comma separated the volume id
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2 years ago
4a48332248
refactor error contains already deleted ( #3932 )
2 years ago
c0deaa4948
[volume.fsck] check needles status from volume server ( #3926 )
check needles status from volume server
2 years ago
0f360862bf
3.33
2 years ago
409c9328de
[master] avoid vacuum if not enough replica copies ( #3924 )
avoid vacuum if not enough replica copies
2 years ago
bf8a9d2db1
[volume.chek.disk] sync of deletions the fix ( #3923 )
* sync of deletions the fix
* avoid return if only partiallyDeletedNeedles
* refactor sync deletions
2 years ago
8b9957d461
add back "/" prefix if it is missing in object
fix https://github.com/seaweedfs/seaweedfs/issues/3737
2 years ago
ea2637734a
refactor filer proto chunk variable from mtime to modified_ts_ns
2 years ago
764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet ( #3922 )
needles older than the cutoff time are not missing yet
https://github.com/seaweedfs/seaweedfs/issues/3919
2 years ago
f9f499b8d5
[fix] add param for ignore error ( #3918 )
2 years ago
1e0d64c048
master: delete partially created replicated volumes during volume growth
fix https://github.com/seaweedfs/seaweedfs/discussions/3792#discussioncomment-3973120
2 years ago
c6f2899375
avoid fix missing of deleted needles ( #3909 )
2 years ago
51d462f204
ADHOC: volume fsck using append at ns ( #3906 )
* ADHOC: volume fsck using append at ns
* nit
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2 years ago
34132b2c9f
fix: isTruncated returns problematic ( #3907 )
2 years ago
874fd197b5
feat: simplify a bit ( #3905 )
2 years ago
25e012d30b
fix: set user metadata key to lowercase ( #3894 )
* fix: set user metadata key to lowercase
* feat: simplify a bit
2 years ago
377870f4a9
keep system log data
2 years ago
97edb40275
Fix errinfo ( #3893 )
* types packages is imported more than onece
* Fix error response when format of --expires is wrong.
It MUST be in RFC 1123 date format.
2 years ago
d5364218b2
adjust help message
2 years ago
d880fc2bb3
fix merge
2 years ago
6253058d9d
ensure monotonic n.AppendAtNs in each place ( #3880 )
https://github.com/seaweedfs/seaweedfs/issues/3852
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2 years ago
452202f35b
ReadNeedleMeta recovery panic occurred ( #3883 )
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
2 years ago
e3ea544d40
skip reading needle if size is invalid
related to https://github.com/seaweedfs/seaweedfs/issues/3882
2 years ago
5294b6dd4d
remove volume server tcp client
2 years ago
f2d9049e6a
fix size variable
2 years ago
184fbb6c50
volume server: remote tier volumes only soft delete in local index
fix https://github.com/seaweedfs/seaweedfs/issues/3889
2 years ago
25471d579a
add md5 header when UploadData to replication in ReplicatedWrite ( #3881 )
2 years ago
758d70bdc0
3.32
2 years ago
de286fe662
shell: volume.move handles volume moved to cloud tier
fix https://github.com/seaweedfs/seaweedfs/issues/3803
2 years ago
52e0a88a15
shell: volume.tier.upload clean up replicated copies
fix https://github.com/seaweedfs/seaweedfs/issues/3804
2 years ago
e20f0dbd2d
avoid data race of TraverseBfs ( #3856 )
* avoid data race of TraverseBfs
* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]
goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
2 years ago
ee38ab8581
avoid data race of LogBuffer isStopping ( #3859 )
2 years ago
7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 ( #3861 )
* fix apply purging and add verifyNeedle
* common readSourceNeedleBlob
* use consts
2 years ago
James Hartig
Eric Yang
chrislu
Guo Lei
LHHDZ
famosss
aronneagu
Konstantin Lebedev
zemul
pavelzhurov
Guo Lei
liubaojiang