Guo Lei
5b905fb2b7
Lazy loading ( #3958 )
* types packages is imported more than onece
* lazy-loading
* fix bugs
* fix bugs
* fix unit tests
* fix test error
* rename function
* unload ldb after initial startup
* Don't load ldb when starting volume server if ldbtimeout is set.
* remove uncessary unloadldb
* Update weed/command/server.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/command/volume.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2 years ago
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch ( #3841 )
* avoid load volume file with BytesOffset mismatch
https://github.com/seaweedfs/seaweedfs/issues/2966
* set BytesOffset if has not VolumeInfoFile
* typos fail => failed
* exit if bytesOffset mismatch
2 years ago
Guo Lei
c57c79a0ab
optimiz commitig compact ( #3388 )
* optimiz vacuuming volume
* fix bugx
* rename parameters
* fix conflict
* change copyDataBasedOnIndexFile to an instance method
* close needlemap
* optimiz commiting Vacuum volume for leveldb index
* fix bugs
* fix leveldb loading bugs
* refactor
* fix leveldb loading bug
* add leveldb recovery
* add test case for levelDB
* modify test case to cover all the new branches
* use one tmpNm instead of two instances
* refactor
* refactor
* move setWatermark to the end
* add test for watermark and updating leveldb
* fix error logic
* refactor, add test
* check nil before close needlemapeer
add test case
fix metric bug
* add tests, fix bugs
* adjust log level
remove wrong test case
refactor
* avoid duplicate updating metric for leveldb index
2 years ago
chrislu
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
2 years ago
chrislu
59e58c4b23
volume: fix loading old volume format
fix https://github.com/chrislusf/seaweedfs/issues/2487
3 years ago
Chris Lu
c46d16b2f3
skip network error when remote volumes start
4 years ago
Chris Lu
6e43e8ce17
add logs for remote file loading
4 years ago
Chris Lu
015d16f43f
add vif file versions in case loading superblock fails
4 years ago
Chris Lu
885ca34748
volume: fail fast if idx files are missing
fix https://github.com/chrislusf/seaweedfs/issues/1796
4 years ago
bingoohuang
94ea3bd3a5
renaming NeedleMapType to NeedleMapKind
4 years ago
Chris Lu
4f31c1bb94
go fmt
4 years ago
Chris Lu
1bf22c0b5b
go fmt
4 years ago
Chris Lu
d171d9f988
volume: ensure the volume dat and idx files are always closed
fix https://github.com/chrislusf/seaweedfs/issues/1646
4 years ago
Chris Lu
6d30b21b10
volume: add "-dir.idx" option for separate index storage
fix https://github.com/chrislusf/seaweedfs/issues/1265
4 years ago
Chris Lu
744c5594da
volume: automatically trim out unreachable entries
4 years ago
Chris Lu
faa5c2e89a
refactoring
5 years ago
Chris Lu
c8ca234773
refactoring
5 years ago
Chris Lu
df9d538044
rename function
5 years ago
Chris Lu
c3cb6fa1d7
volume: compaction can cause readonly volumes
address https://github.com/chrislusf/seaweedfs/issues/1233
5 years ago
Chris Lu
288baf37fd
saving .vif files correctly
5 years ago
Chris Lu
9134092a8d
endure creating .vif files
5 years ago
Chris Lu
0cc8347a3a
go fmt
5 years ago
Chris Lu
c06f7eb48a
load volume info from .vif file, use superblock as a backup
5 years ago
Chris Lu
f4a74e03d1
refactoring: separating .vif from tier file loading
5 years ago
Chris Lu
2000284435
rename volume tier info to volume info
5 years ago
Chris Lu
efd2f50ede
compaction changed to .idx based deletion
5 years ago
Chris Lu
f42b5bd0f5
rename .sdb to .sdx
5 years ago
Chris Lu
09ca936c78
shell: add ec.decode command
5 years ago
Chris Lu
ab966d7192
refactoring variable name
5 years ago
Chris Lu
f61de28c69
volume: add deletion capability for previously readonly volumes
5 years ago
Chris Lu
39810a9951
tiered volume is read only
5 years ago
Chris Lu
58f88e530c
volume: use sorted index map for readonly volumes
5 years ago
Chris Lu
ec8de250e2
tiered storage: can copy to s3, read from s3
master not aware tiered volume yet, file assigning is not working yet
5 years ago
Chris Lu
85f8649320
refactor memory mapped file into backend storage
5 years ago
Chris Lu
4b5ba4927f
adjust NewDiskFile() paramaters
5 years ago
Chris Lu
19b6a16003
changed from os.file to backend.DataStorageBackend
5 years ago
Chris Lu
faec9076a4
adjust parameter names
5 years ago
Chris Lu
09874f0d16
volume: return error if superblock is not initialized
fix https://github.com/chrislusf/seaweedfs/issues/1079
5 years ago
j.laycock
6fc6322c90
Change joeslay paths to chrislusf paths
5 years ago
j.laycock
5885ab67b3
rename volume property MemoryMap to MemoryMapMaxSizeMB
5 years ago
j.laycock
1f01eb78e8
Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct
5 years ago
Tom Maxwell
d637d86d22
Changes to try and pass the URL parameters through - in memory flag not working still
5 years ago
j.laycock
595a1beff0
Swap imports to use joeslay
5 years ago
xushuxun
9af9f36b89
volume: read dat files' last modified time no matter dat files are catWrite or readonly
5 years ago
Chris Lu
115558e5f5
adjust counters
6 years ago
Chris Lu
289fd7eb39
count number of volumes and ec shards
6 years ago
Chris Lu
76cf57a0e1
volume: optimzation for leveldb
6 years ago
Chris Lu
0be2d51c96
read volume lastAppendAtNs when loading a volume
6 years ago
Chris Lu
ac2727853f
fix needle map entry size
6 years ago
Chris Lu
e5506152c0
refactoring
6 years ago