chrislu
896a85d6e4
volume: add "hasSlowRead" option to customize read optimization
2 years ago
chrislu
21c0587900
go fmt
2 years ago
chrislu
0baf3d38c2
adjust logs
2 years ago
Eric Yang
ddd6bee970
ADHOC: Volume fsck use a time cutoff param ( #3626 )
* ADHOC: cut off volumn fsck
* more
* fix typo
* add test
* modify name
* fix comment
* fix comments
* nit
* fix typo
* Update weed/shell/command_volume_fsck.go
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2 years ago
famosss
41af5a6ee5
simplify a bit ( #3631 )
2 years ago
chrislu
64e75a286e
simplify a bit
2 years ago
chrislu
d14d029c73
reduce v.dataFileAccessLock lock scope
2 years ago
chrislu
406a80da4b
adjust variable names
2 years ago
chrislu
2bfc8970d2
refactor: move ReadNeedleDataInto into volume_read.go
2 years ago
chrislu
9b084d4c88
purge tcp implementation
2 years ago
Eric Yang
b324a6536c
ADHOC: add read needle meta grpc ( #3581 )
* ADHOC: add read needle meta grpc
* add test
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2 years ago
chrislu
39340f7e42
cloud tier: s3 consume all read response body
fix https://github.com/seaweedfs/seaweedfs/issues/3584
2 years ago
chrislu
7c6324b114
adjust log level
2 years ago
chrislu
8c5759a06d
minor optimization
2 years ago
Konstantin Lebedev
ade94b0d0a
avoid race conditions access to SuperBlock.Version ( #3539 )
* avoid race conditions access to SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515
* superBlockAccessLock replace to sync.Mutex
2 years ago
chrislu
3f3a1341d8
make CodeQL happy
2 years ago
chrislu
e0f4366f4c
more detailed error
related to https://github.com/seaweedfs/seaweedfs/issues/3528
2 years ago
Konstantin Lebedev
4f7a1f67cd
avoid race conditions for diskLocation.MaxVolumeCount ( #3526 )
2 years ago
Guo Lei
c57c79a0ab
optimiz commitig compact ( #3388 )
* optimiz vacuuming volume
* fix bugx
* rename parameters
* fix conflict
* change copyDataBasedOnIndexFile to an instance method
* close needlemap
* optimiz commiting Vacuum volume for leveldb index
* fix bugs
* fix leveldb loading bugs
* refactor
* fix leveldb loading bug
* add leveldb recovery
* add test case for levelDB
* modify test case to cover all the new branches
* use one tmpNm instead of two instances
* refactor
* refactor
* move setWatermark to the end
* add test for watermark and updating leveldb
* fix error logic
* refactor, add test
* check nil before close needlemapeer
add test case
fix metric bug
* add tests, fix bugs
* adjust log level
remove wrong test case
refactor
* avoid duplicate updating metric for leveldb index
2 years ago
Guo Lei
762dc219a6
fix deletecount bug ( #3499 )
2 years ago
Guo Lei
81e7e6ceb3
the statistical methord for fileCount should stay same ( #3477 )
the statistical methord for fileCount between should stay same
They all equal to the entry count in .idx file.
relate to commit(c7892bc
)
2 years ago
chrislu
c7892bc7c4
volume: file counter should be all files
address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228
2 years ago
chrislu
4d72a1ce6e
s3: remove deleted collection during metrics reporting
fix https://github.com/seaweedfs/seaweedfs/issues/3453
2 years ago
chrislu
2b580a7566
also migrate jsonpb
2 years ago
chrislu
eaeb141b09
move proto package
2 years ago
chrislu
93261f5199
Revert "Refactor for Sync method ( #3426 )"
This reverts commit 670cb759f8
.
with the pr
weed/storage () - (master) > go test -count=1 ./...
ok github.com/seaweedfs/seaweedfs/weed/storage 18.486s
? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files]
ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.025s
? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files]
ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.864s
? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files]
ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.110s
ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.414s
ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.203s
? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files]
? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files]
weed/storage () - (master) >
weed/storage () - (master) >
without the pr
weed/storage () - (master) >
weed/storage () - (master) > go test -count=1 ./...
ok github.com/seaweedfs/seaweedfs/weed/storage 1.617s
? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files]
ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.026s
? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files]
ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.906s
? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files]
ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.202s
ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.533s
ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.280s
? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files]
? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files]
2 years ago
Rain Li
670cb759f8
Refactor for Sync method ( #3426 )
2 years ago
Abirdcfly
b0633716b7
delete minor unreachable code ( #3423 )
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2 years ago
chrislu
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
2 years ago
Konstantin Lebedev
2f0dda384d
vacuum show LA
2 years ago
Evgeny Kuzhelev
47c72e6f35
remove all (currently existing) collection volume metrics
2 years ago
Evgeny Kuzhelev
709c83716c
delete disk_size metrics when collection deleted
2 years ago
guosj
cc7a4b0a6e
correct comment
2 years ago
guol-fnst
ac694f0c8f
rename parameter and reuse functions
rename milestone to watermark
2 years ago
guol-fnst
91285bb51d
remove uncessary code
2 years ago
guol-fnst
b9256e0b34
optimiz
2 years ago
guol-fnst
bec9f79659
use snapshot
2 years ago
guol-fnst
300b383cdf
use 10 or numCPU workers if env is not found
2 years ago
guol-fnst
308a48c0c2
optimiz concurrency
user can customize number of workers via env "GOMAXPROCS"
2 years ago
guol-fnst
26313060a3
speeding up loading volumes
2 years ago
chrislu
06a8b174b5
also remove Sync() for idx file
3 years ago
chrislu
52580743b9
stuck with file.Sync()
starting to get weed/storage () - (master) > go test ./...
I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648
I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959
I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000
I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory
I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming...
panic: test timed out after 10m0s
goroutine 61 [running]:
testing.(*M).startAlarm.func1()
/usr/local/go/src/testing/testing.go:2029 +0x8e
created by time.goFunc
/usr/local/go/src/time/sleep.go:176 +0x32
goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1487 +0x37a
testing.runTests.func1(0xc0000cd520?)
/usr/local/go/src/testing/testing.go:1839 +0x6e
testing.tRunner(0xc0000cd520, 0xc00039fcd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?})
/usr/local/go/src/testing/testing.go:1837 +0x457
testing.(*M).Run(0xc0000dadc0)
/usr/local/go/src/testing/testing.go:1719 +0x5d9
main.main()
_testmain.go:57 +0x1aa
goroutine 21 [chan receive]:
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a
created by github.com/chrislusf/seaweedfs/weed/glog.init.0
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf
goroutine 55 [syscall]:
syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?)
/usr/local/go/src/runtime/sys_darwin.go:22 +0x4e
syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?)
/usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30
internal/poll.(*FD).Fsync.func1(...)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:18
internal/poll.ignoringEINTR(...)
/usr/local/go/src/internal/poll/fd_posix.go:74
internal/poll.(*FD).Fsync(0xc00038f1e0?)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc
os.(*File).Sync(0xc00012a030)
/usr/local/go/src/os/file_posix.go:168 +0x4e
github.com/chrislusf/seaweedfs/weed/storage/backend.(*DiskFile).Sync(...)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52})
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc0000bb440)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8
github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305
testing.tRunner(0xc00025a000, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1486 +0x35f
goroutine 56 [chan receive, 10 minutes]:
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker.func1()
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a
created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56
FAIL github.com/chrislusf/seaweedfs/weed/storage 600.194s
? github.com/chrislusf/seaweedfs/weed/storage/backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map (cached)
? github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/erasure_coding (cached)
? github.com/chrislusf/seaweedfs/weed/storage/idx [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/needle (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/needle_map (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/super_block (cached)
? github.com/chrislusf/seaweedfs/weed/storage/types [no test files]
? github.com/chrislusf/seaweedfs/weed/storage/volume_info [no test files]
FAIL
3 years ago
garenchan
8aa19577f4
fix 3238: handle errors for GenerateDirUuid method
3 years ago
chrislu
509a9047db
test compact map with snowflake sequencer
3 years ago
chrislu
625fd16a2e
reduce upfront memory usage for low density volume
3 years ago
chrislu
36d2756978
volume: avoid writing too much data for range requests in large files
related to https://github.com/chrislusf/seaweedfs/issues/3178
3 years ago
chrislu
9f8b72a54d
Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID"
This reverts commit 37da689319
, reversing
changes made to 00d53c34c4
.
3 years ago
shichanglin5
f5b0c04b14
perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal
Under normal circumstances, there will be no problems, but when the
master is debugged in the local environment, the volume client cannot
communicate with the master normally, so the sendHeartBeat logic is
restarted, and a new connection is created to report the heartbeat. If
the master has not cleared the uuid of the volume at this time, then The
master will respond to volume duplicateUUIDS, and the volume service
will exit, but in fact the uuid of the volume is not duplicated
3 years ago
chrislu
11f1e7996b
fix test
3 years ago
chrislu
056c480eb0
volume: checksum remove the hashing step in Value()
3 years ago