seaweedfs

Commit Graph

Author	SHA1	Message	Date
chrislu	06748f2771	remove unused code	3 years ago
Ryan Russell	277976bd76	refactor(storage): readability improvements (#3703 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	3 years ago
famosss	d949a238b8	volume: add "readBufSize" option to customize read optimization (#3702 ) * simplify a bit * feat: volume: add "readBufSize" option to customize read optimization * refactor : redbufSIze -> readBufferSize * simplify a bit * simplify a bit	3 years ago
chrislu	896a85d6e4	volume: add "hasSlowRead" option to customize read optimization	3 years ago
chrislu	21c0587900	go fmt	3 years ago
chrislu	0baf3d38c2	adjust logs	3 years ago
Eric Yang	ddd6bee970	ADHOC: Volume fsck use a time cutoff param (#3626 ) * ADHOC: cut off volumn fsck * more * fix typo * add test * modify name * fix comment * fix comments * nit * fix typo * Update weed/shell/command_volume_fsck.go Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	3 years ago
famosss	41af5a6ee5	simplify a bit (#3631 )	3 years ago
chrislu	64e75a286e	simplify a bit	3 years ago
chrislu	d14d029c73	reduce v.dataFileAccessLock lock scope	3 years ago
chrislu	406a80da4b	adjust variable names	3 years ago
chrislu	2bfc8970d2	refactor: move ReadNeedleDataInto into volume_read.go	3 years ago
chrislu	9b084d4c88	purge tcp implementation	3 years ago
Eric Yang	b324a6536c	ADHOC: add read needle meta grpc (#3581 ) * ADHOC: add read needle meta grpc * add test * nit Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	3 years ago
chrislu	39340f7e42	cloud tier: s3 consume all read response body fix https://github.com/seaweedfs/seaweedfs/issues/3584	3 years ago
chrislu	7c6324b114	adjust log level	3 years ago
chrislu	8c5759a06d	minor optimization	3 years ago
Konstantin Lebedev	ade94b0d0a	avoid race conditions access to SuperBlock.Version (#3539 ) * avoid race conditions access to SuperBlock.Version https://github.com/seaweedfs/seaweedfs/issues/3515 * superBlockAccessLock replace to sync.Mutex	3 years ago
chrislu	3f3a1341d8	make CodeQL happy	3 years ago
chrislu	e0f4366f4c	more detailed error related to https://github.com/seaweedfs/seaweedfs/issues/3528	3 years ago
Konstantin Lebedev	4f7a1f67cd	avoid race conditions for diskLocation.MaxVolumeCount (#3526 )	3 years ago
Guo Lei	c57c79a0ab	optimiz commitig compact (#3388 ) * optimiz vacuuming volume * fix bugx * rename parameters * fix conflict * change copyDataBasedOnIndexFile to an instance method * close needlemap * optimiz commiting Vacuum volume for leveldb index * fix bugs * fix leveldb loading bugs * refactor * fix leveldb loading bug * add leveldb recovery * add test case for levelDB * modify test case to cover all the new branches * use one tmpNm instead of two instances * refactor * refactor * move setWatermark to the end * add test for watermark and updating leveldb * fix error logic * refactor, add test * check nil before close needlemapeer add test case fix metric bug * add tests, fix bugs * adjust log level remove wrong test case refactor * avoid duplicate updating metric for leveldb index	3 years ago
Guo Lei	762dc219a6	fix deletecount bug (#3499 )	3 years ago
Guo Lei	81e7e6ceb3	the statistical methord for fileCount should stay same (#3477 ) the statistical methord for fileCount between should stay same They all equal to the entry count in .idx file. relate to commit(`c7892bc`)	3 years ago
chrislu	c7892bc7c4	volume: file counter should be all files address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228	3 years ago
chrislu	4d72a1ce6e	s3: remove deleted collection during metrics reporting fix https://github.com/seaweedfs/seaweedfs/issues/3453	3 years ago
chrislu	2b580a7566	also migrate jsonpb	3 years ago
chrislu	eaeb141b09	move proto package	3 years ago
chrislu	93261f5199	Revert "Refactor for Sync method (#3426 )" This reverts commit `670cb759f8`. with the pr weed/storage () - (master) > go test -count=1 ./... ok github.com/seaweedfs/seaweedfs/weed/storage 18.486s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.025s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.864s ? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.110s ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.414s ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.203s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files] weed/storage () - (master) > weed/storage () - (master) > without the pr weed/storage () - (master) > weed/storage () - (master) > go test -count=1 ./... ok github.com/seaweedfs/seaweedfs/weed/storage 1.617s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.026s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.906s ? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.202s ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.533s ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.280s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files]	3 years ago
Rain Li	670cb759f8	Refactor for Sync method (#3426 )	3 years ago
Abirdcfly	b0633716b7	delete minor unreachable code (#3423 ) Signed-off-by: Abirdcfly <fp544037857@gmail.com>	3 years ago
chrislu	26dbc6c905	move to https://github.com/seaweedfs/seaweedfs	3 years ago
Konstantin Lebedev	2f0dda384d	vacuum show LA	3 years ago
Evgeny Kuzhelev	47c72e6f35	remove all (currently existing) collection volume metrics	3 years ago
Evgeny Kuzhelev	709c83716c	delete disk_size metrics when collection deleted	3 years ago
guosj	cc7a4b0a6e	correct comment	3 years ago
guol-fnst	ac694f0c8f	rename parameter and reuse functions rename milestone to watermark	3 years ago
guol-fnst	91285bb51d	remove uncessary code	3 years ago
guol-fnst	b9256e0b34	optimiz	3 years ago
guol-fnst	bec9f79659	use snapshot	3 years ago
guol-fnst	300b383cdf	use 10 or numCPU workers if env is not found	3 years ago
guol-fnst	308a48c0c2	optimiz concurrency user can customize number of workers via env "GOMAXPROCS"	3 years ago
guol-fnst	26313060a3	speeding up loading volumes	3 years ago
chrislu	06a8b174b5	also remove Sync() for idx file	3 years ago
chrislu	52580743b9	stuck with file.Sync() starting to get weed/storage () - (master) > go test ./... I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000 I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648 I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000 I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959 I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000 I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming... panic: test timed out after 10m0s goroutine 61 [running]: testing.(M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2029 +0x8e created by time.goFunc /usr/local/go/src/time/sleep.go:176 +0x32 goroutine 1 [chan receive, 10 minutes]: testing.(T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8) /usr/local/go/src/testing/testing.go:1487 +0x37a testing.runTests.func1(0xc0000cd520?) /usr/local/go/src/testing/testing.go:1839 +0x6e testing.tRunner(0xc0000cd520, 0xc00039fcd8) /usr/local/go/src/testing/testing.go:1439 +0x102 testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?}) /usr/local/go/src/testing/testing.go:1837 +0x457 testing.(M).Run(0xc0000dadc0) /usr/local/go/src/testing/testing.go:1719 +0x5d9 main.main() _testmain.go:57 +0x1aa goroutine 21 [chan receive]: github.com/chrislusf/seaweedfs/weed/glog.(loggingT).flushDaemon(0x0?) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a created by github.com/chrislusf/seaweedfs/weed/glog.init.0 /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf goroutine 55 [syscall]: syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?) /usr/local/go/src/runtime/sys_darwin.go:22 +0x4e syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?) /usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30 internal/poll.(FD).Fsync.func1(...) /usr/local/go/src/internal/poll/fd_fsync_darwin.go:18 internal/poll.ignoringEINTR(...) /usr/local/go/src/internal/poll/fd_posix.go:74 internal/poll.(FD).Fsync(0xc00038f1e0?) /usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc os.(File).Sync(0xc00012a030) /usr/local/go/src/os/file_posix.go:168 +0x4e github.com/chrislusf/seaweedfs/weed/storage/backend.(DiskFile).Sync(...) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84 github.com/chrislusf/seaweedfs/weed/storage.(Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52}) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa github.com/chrislusf/seaweedfs/weed/storage.(Volume).CommitCompact(0xc0000bb440) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8 github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305 testing.tRunner(0xc00025a000, 0x1c67bd8) /usr/local/go/src/testing/testing.go:1439 +0x102 created by testing.(T).Run /usr/local/go/src/testing/testing.go:1486 +0x35f goroutine 56 [chan receive, 10 minutes]: github.com/chrislusf/seaweedfs/weed/storage.(Volume).startWorker.func1() /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56 FAIL github.com/chrislusf/seaweedfs/weed/storage 600.194s ? github.com/chrislusf/seaweedfs/weed/storage/backend [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map (cached) ? github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/erasure_coding (cached) ? github.com/chrislusf/seaweedfs/weed/storage/idx [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/needle (cached) ok github.com/chrislusf/seaweedfs/weed/storage/needle_map (cached) ok github.com/chrislusf/seaweedfs/weed/storage/super_block (cached) ? github.com/chrislusf/seaweedfs/weed/storage/types [no test files] ? github.com/chrislusf/seaweedfs/weed/storage/volume_info [no test files] FAIL	3 years ago
garenchan	8aa19577f4	fix 3238: handle errors for GenerateDirUuid method	3 years ago
chrislu	509a9047db	test compact map with snowflake sequencer	3 years ago
chrislu	625fd16a2e	reduce upfront memory usage for low density volume	3 years ago
chrislu	36d2756978	volume: avoid writing too much data for range requests in large files related to https://github.com/chrislusf/seaweedfs/issues/3178	3 years ago
chrislu	9f8b72a54d	Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID" This reverts commit `37da689319`, reversing changes made to `00d53c34c4`.	3 years ago

1 2 3 4 5 ...

761 Commits (b2a171632bde65c6ff3e668245b2461ce6b44ced)