Lars Lehtonen
f24c7e803f
weed/storage/erasure_coding: Close() after error handling
1 year ago
Lars Lehtonen
d762b12e11
weed/storage/erasure_coding: fix dropped test error
1 year ago
Lars Lehtonen
a879e1bbb5
weed/storage/erasure_coding: remove unused err from encodeDatFile() signature
1 year ago
Chris Lu
9215ba24be
Revert "Add disk type to prometheus metrics" ( #4777 )
Revert "Add disk type to prometheus metrics (#4736 )"
This reverts commit 9956d93a40
.
1 year ago
Dmitry Mishin
9956d93a40
Add disk type to prometheus metrics ( #4736 )
* Add disk type to prometheus metrics
* Del metrics
* Disk type as readable string
---------
Co-authored-by: Dima Mishin <dimm@dimm.dev>
1 year ago
Nikita Mochalov
4b1ba7f5b2
Save disk space metrics immediately ( #4740 )
1 year ago
Nikita Mochalov
e6a49dc533
Fix resource leaks ( #4737 )
* Fix division by zero
* Fix file handle leak
* Fix file handle leak
* Fix file handle leak
* Fix goroutine leak
1 year ago
LHHDZ
3730772e05
refresh max volume count after adjust ( #4662 )
Signed-off-by: changlin.shi <changlin.shi@ly.com>
1 year ago
chrislu
c278bac263
avoid nil needle map
fix https://github.com/seaweedfs/seaweedfs/issues/4640
1 year ago
chrislu
14546f0ad8
minor
1 year ago
wusong
61553beba2
Fix DataBackend nil pointer ( #4641 )
1 year ago
chrislu
2aa59ab37c
fix copying level db files during commi
fix https://github.com/seaweedfs/seaweedfs/issues/4635
1 year ago
Damiano Albani
0d9ba8c612
Support templating name of files stored via Rclone backend ( #4606 )
2 years ago
Bai Jie
3b88ab42aa
remove duplicate `fileCount` query ( #4588 )
2 years ago
Bai Jie
44b9d72ef0
doIsEmpty() return error if v.DataBackend is nil ( #4587 )
2 years ago
柏杰
0b0fb9b9e4
avoid data race read volume.IsEmpty ( #4574 )
* avoid data race read volume.IsEmpty
- avoid phantom read isEmpty for onlyEmpty
- use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope
* add Destroy(onlyEmpty: true) test
* add Destroy(onlyEmpty: false) test
* remove unused `IsEmpty()`
* change literal `8` to `SuperBlockSize`
2 years ago
Konstantin Lebedev
1e22d5caf2
fix get file stats for IsEmpty ( #4576 )
2 years ago
Konstantin Lebedev
4527ead295
fix from comment delete volume is empty ( #4573 )
* fix from coments
https://github.com/seaweedfs/seaweedfs/pull/4561
* fix tests
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2 years ago
Konstantin Lebedev
25535e9c36
Delete volume is empty ( #4561 )
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559
* fix IsEmpty
* fix test
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2 years ago
wusong
26f15d0079
Fix no more writable volumes by delay judgment ( #4548 )
* fix nomore writables volumes while disk free space is sufficient by time delay
* reset
---------
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2 years ago
chrislu
e23f3d6eca
set df.File to nil after it is closed
possibly fix https://github.com/seaweedfs/seaweedfs/issues/4530
2 years ago
chrislu
ff7b6d779e
avoid overwriting variables
fix https://github.com/seaweedfs/seaweedfs/issues/4365
2 years ago
Damiano Albani
9f55c7c90e
Rclone storage backend ( #4402 )
* Add Rclone storage backend
* Support templating the name of files stored via Rclone
* Enable Rclone accounting
* Remove redundant type conversion
* Provide progress information for Rclone download/upload operations
* Log error when Rclone can't instantiate filesystem
* Remove filename templating functionality for Rclone storage
To (maybe) be later reintroduced as a generic functionality for all
storage backends.
* Remove S3 specific check
* Move Rclone config initialisation to init() method
2 years ago
chrislu
0a22eea55d
collect ec shard from multiple locations
fix https://github.com/seaweedfs/seaweedfs/issues/4365
2 years ago
chrislu
f5854d13df
fix test
2 years ago
chrislu
0454bb2d88
fix test
when size is zero, the "LastModified" is not written
2 years ago
chrislu
c589e9837c
fix tests
2 years ago
zemul
bd6f2716d3
fix_read_tomb_needle_meta ( #4263 )
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2 years ago
James Hartig
81624de27b
Include name/mime in ReadAllNeedles ( #4005 )
2 years ago
James Hartig
4c85da7844
Include meta in ReadAllNeedles ( #3991 )
This is useful for doing backups on the data so we can accurately store the
last modified time, the compression state, and verify the crc.
Previously we were doing VolumeNeedleStatus and then an HTTP request which
needlessly read from the dat file twice.
2 years ago
Guo Lei
5b905fb2b7
Lazy loading ( #3958 )
* types packages is imported more than onece
* lazy-loading
* fix bugs
* fix bugs
* fix unit tests
* fix test error
* rename function
* unload ldb after initial startup
* Don't load ldb when starting volume server if ldbtimeout is set.
* remove uncessary unloadldb
* Update weed/command/server.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/command/volume.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2 years ago
Konstantin Lebedev
bf8a9d2db1
[volume.chek.disk] sync of deletions the fix ( #3923 )
* sync of deletions the fix
* avoid return if only partiallyDeletedNeedles
* refactor sync deletions
2 years ago
Konstantin Lebedev
764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet ( #3922 )
needles older than the cutoff time are not missing yet
https://github.com/seaweedfs/seaweedfs/issues/3919
2 years ago
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns ( #3906 )
* ADHOC: volume fsck using append at ns
* nit
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2 years ago
chrislu
d880fc2bb3
fix merge
2 years ago
Konstantin Lebedev
6253058d9d
ensure monotonic n.AppendAtNs in each place ( #3880 )
https://github.com/seaweedfs/seaweedfs/issues/3852
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2 years ago
Konstantin Lebedev
452202f35b
ReadNeedleMeta recovery panic occurred ( #3883 )
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
2 years ago
chrislu
e3ea544d40
skip reading needle if size is invalid
related to https://github.com/seaweedfs/seaweedfs/issues/3882
2 years ago
chrislu
f2d9049e6a
fix size variable
2 years ago
chrislu
184fbb6c50
volume server: remote tier volumes only soft delete in local index
fix https://github.com/seaweedfs/seaweedfs/issues/3889
2 years ago
Konstantin Lebedev
e20f0dbd2d
avoid data race of TraverseBfs ( #3856 )
* avoid data race of TraverseBfs
* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]
goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
2 years ago
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch ( #3841 )
* avoid load volume file with BytesOffset mismatch
https://github.com/seaweedfs/seaweedfs/issues/2966
* set BytesOffset if has not VolumeInfoFile
* typos fail => failed
* exit if bytesOffset mismatch
2 years ago
chrislu
9c8678ded9
ensure monotonic n.AppendAtNs
fix https://github.com/seaweedfs/seaweedfs/issues/3852
2 years ago
Konstantin Lebedev
1f7e52c63e
vacuum metrics and force sync dst files ( #3832 )
2 years ago
Guo Lei
f95c25e113
types packages is imported more than onece ( #3838 )
2 years ago
chrislu
e55076c46f
cloud tier: add retry when copying data file
fix https://github.com/seaweedfs/seaweedfs/issues/3828
2 years ago
Guo Lei
84c401e693
Optimiz leveldb metric ( #3830 )
* optimiz updating mapmetric for leveldb
* import loading leveldb
* add comments
2 years ago
Konstantin Lebedev
5b28c3f728
revert disabling FSync for non Mac ( #3814 )
2 years ago
binbinshi
b7de4a967e
fix: compact_map get error mismatching cokie ( #3748 )
* fix: compact_map get error
* fix: CompactSection delete lock and move test to compact_map
Co-authored-by: shibinbin <shibinbin@megvii.com>
2 years ago
Viktor Kuzmin
f8d3ff466d
Allow parallel volume loading from different dirs during startup. ( #3802 )
2 years ago