5b905fb2b7
Lazy loading ( #3958 )
* types packages is imported more than onece
* lazy-loading
* fix bugs
* fix bugs
* fix unit tests
* fix test error
* rename function
* unload ldb after initial startup
* Don't load ldb when starting volume server if ldbtimeout is set.
* remove uncessary unloadldb
* Update weed/command/server.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/command/volume.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2 years ago
bf8a9d2db1
[volume.chek.disk] sync of deletions the fix ( #3923 )
* sync of deletions the fix
* avoid return if only partiallyDeletedNeedles
* refactor sync deletions
2 years ago
764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet ( #3922 )
needles older than the cutoff time are not missing yet
https://github.com/seaweedfs/seaweedfs/issues/3919
2 years ago
51d462f204
ADHOC: volume fsck using append at ns ( #3906 )
* ADHOC: volume fsck using append at ns
* nit
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2 years ago
d880fc2bb3
fix merge
2 years ago
6253058d9d
ensure monotonic n.AppendAtNs in each place ( #3880 )
https://github.com/seaweedfs/seaweedfs/issues/3852
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2 years ago
452202f35b
ReadNeedleMeta recovery panic occurred ( #3883 )
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
2 years ago
e3ea544d40
skip reading needle if size is invalid
related to https://github.com/seaweedfs/seaweedfs/issues/3882
2 years ago
f2d9049e6a
fix size variable
2 years ago
184fbb6c50
volume server: remote tier volumes only soft delete in local index
fix https://github.com/seaweedfs/seaweedfs/issues/3889
2 years ago
e20f0dbd2d
avoid data race of TraverseBfs ( #3856 )
* avoid data race of TraverseBfs
* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]
goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
2 years ago
2f72103c83
avoid load volume file with BytesOffset mismatch ( #3841 )
* avoid load volume file with BytesOffset mismatch
https://github.com/seaweedfs/seaweedfs/issues/2966
* set BytesOffset if has not VolumeInfoFile
* typos fail => failed
* exit if bytesOffset mismatch
2 years ago
9c8678ded9
ensure monotonic n.AppendAtNs
fix https://github.com/seaweedfs/seaweedfs/issues/3852
2 years ago
1f7e52c63e
vacuum metrics and force sync dst files ( #3832 )
2 years ago
f95c25e113
types packages is imported more than onece ( #3838 )
2 years ago
e55076c46f
cloud tier: add retry when copying data file
fix https://github.com/seaweedfs/seaweedfs/issues/3828
2 years ago
84c401e693
Optimiz leveldb metric ( #3830 )
* optimiz updating mapmetric for leveldb
* import loading leveldb
* add comments
2 years ago
5b28c3f728
revert disabling FSync for non Mac ( #3814 )
2 years ago
b7de4a967e
fix: compact_map get error mismatching cokie ( #3748 )
* fix: compact_map get error
* fix: CompactSection delete lock and move test to compact_map
Co-authored-by: shibinbin <shibinbin@megvii.com>
2 years ago
f8d3ff466d
Allow parallel volume loading from different dirs during startup. ( #3802 )
2 years ago
56c94cc08e
ADHOC: filter deleted files from idx file binary search ( #3763 )
* ADHOC: filter deleted files from idx file binary search
* remove unwanted check
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2 years ago
301b678147
[volume] Add new volumes to HUP(reload) signal ( #3755 )
Add new volumes to HUP(reload) signal
2 years ago
06748f2771
remove unused code
2 years ago
277976bd76
refactor(storage): readability improvements ( #3703 )
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2 years ago
d949a238b8
volume: add "readBufSize" option to customize read optimization ( #3702 )
* simplify a bit
* feat: volume: add "readBufSize" option to customize read optimization
* refactor : redbufSIze -> readBufferSize
* simplify a bit
* simplify a bit
2 years ago
896a85d6e4
volume: add "hasSlowRead" option to customize read optimization
2 years ago
21c0587900
go fmt
2 years ago
0baf3d38c2
adjust logs
2 years ago
ddd6bee970
ADHOC: Volume fsck use a time cutoff param ( #3626 )
* ADHOC: cut off volumn fsck
* more
* fix typo
* add test
* modify name
* fix comment
* fix comments
* nit
* fix typo
* Update weed/shell/command_volume_fsck.go
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2 years ago
41af5a6ee5
simplify a bit ( #3631 )
2 years ago
64e75a286e
simplify a bit
2 years ago
d14d029c73
reduce v.dataFileAccessLock lock scope
2 years ago
406a80da4b
adjust variable names
2 years ago
2bfc8970d2
refactor: move ReadNeedleDataInto into volume_read.go
2 years ago
9b084d4c88
purge tcp implementation
2 years ago
b324a6536c
ADHOC: add read needle meta grpc ( #3581 )
* ADHOC: add read needle meta grpc
* add test
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2 years ago
39340f7e42
cloud tier: s3 consume all read response body
fix https://github.com/seaweedfs/seaweedfs/issues/3584
2 years ago
7c6324b114
adjust log level
2 years ago
8c5759a06d
minor optimization
2 years ago
ade94b0d0a
avoid race conditions access to SuperBlock.Version ( #3539 )
* avoid race conditions access to SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515
* superBlockAccessLock replace to sync.Mutex
2 years ago
3f3a1341d8
make CodeQL happy
2 years ago
e0f4366f4c
more detailed error
related to https://github.com/seaweedfs/seaweedfs/issues/3528
2 years ago
4f7a1f67cd
avoid race conditions for diskLocation.MaxVolumeCount ( #3526 )
2 years ago
c57c79a0ab
optimiz commitig compact ( #3388 )
* optimiz vacuuming volume
* fix bugx
* rename parameters
* fix conflict
* change copyDataBasedOnIndexFile to an instance method
* close needlemap
* optimiz commiting Vacuum volume for leveldb index
* fix bugs
* fix leveldb loading bugs
* refactor
* fix leveldb loading bug
* add leveldb recovery
* add test case for levelDB
* modify test case to cover all the new branches
* use one tmpNm instead of two instances
* refactor
* refactor
* move setWatermark to the end
* add test for watermark and updating leveldb
* fix error logic
* refactor, add test
* check nil before close needlemapeer
add test case
fix metric bug
* add tests, fix bugs
* adjust log level
remove wrong test case
refactor
* avoid duplicate updating metric for leveldb index
2 years ago
762dc219a6
fix deletecount bug ( #3499 )
2 years ago
81e7e6ceb3
the statistical methord for fileCount should stay same ( #3477 )
the statistical methord for fileCount between should stay same
They all equal to the entry count in .idx file.
relate to commit(c7892bc
2 years ago
c7892bc7c4
volume: file counter should be all files
address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228
2 years ago
4d72a1ce6e
s3: remove deleted collection during metrics reporting
fix https://github.com/seaweedfs/seaweedfs/issues/3453
2 years ago
2b580a7566
also migrate jsonpb
2 years ago
eaeb141b09
move proto package
2 years ago
Guo Lei
Konstantin Lebedev
Eric Yang
chrislu
Guo Lei
binbinshi
Viktor Kuzmin
Ryan Russell
famosss