seaweedfs

Commit Graph

Author	SHA1	Message	Date
Eric Yang	ddd6bee970	ADHOC: Volume fsck use a time cutoff param (#3626 ) * ADHOC: cut off volumn fsck * more * fix typo * add test * modify name * fix comment * fix comments * nit * fix typo * Update weed/shell/command_volume_fsck.go Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2 years ago
famosss	41af5a6ee5	simplify a bit (#3631 )	2 years ago
chrislu	64e75a286e	simplify a bit	2 years ago
chrislu	d14d029c73	reduce v.dataFileAccessLock lock scope	2 years ago
chrislu	406a80da4b	adjust variable names	2 years ago
chrislu	2bfc8970d2	refactor: move ReadNeedleDataInto into volume_read.go	2 years ago
chrislu	9b084d4c88	purge tcp implementation	2 years ago
Eric Yang	b324a6536c	ADHOC: add read needle meta grpc (#3581 ) * ADHOC: add read needle meta grpc * add test * nit Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	2 years ago
chrislu	39340f7e42	cloud tier: s3 consume all read response body fix https://github.com/seaweedfs/seaweedfs/issues/3584	2 years ago
chrislu	7c6324b114	adjust log level	2 years ago
chrislu	8c5759a06d	minor optimization	2 years ago
Konstantin Lebedev	ade94b0d0a	avoid race conditions access to SuperBlock.Version (#3539 ) * avoid race conditions access to SuperBlock.Version https://github.com/seaweedfs/seaweedfs/issues/3515 * superBlockAccessLock replace to sync.Mutex	2 years ago
chrislu	3f3a1341d8	make CodeQL happy	2 years ago
chrislu	e0f4366f4c	more detailed error related to https://github.com/seaweedfs/seaweedfs/issues/3528	2 years ago
Konstantin Lebedev	4f7a1f67cd	avoid race conditions for diskLocation.MaxVolumeCount (#3526 )	2 years ago
Guo Lei	c57c79a0ab	optimiz commitig compact (#3388 ) * optimiz vacuuming volume * fix bugx * rename parameters * fix conflict * change copyDataBasedOnIndexFile to an instance method * close needlemap * optimiz commiting Vacuum volume for leveldb index * fix bugs * fix leveldb loading bugs * refactor * fix leveldb loading bug * add leveldb recovery * add test case for levelDB * modify test case to cover all the new branches * use one tmpNm instead of two instances * refactor * refactor * move setWatermark to the end * add test for watermark and updating leveldb * fix error logic * refactor, add test * check nil before close needlemapeer add test case fix metric bug * add tests, fix bugs * adjust log level remove wrong test case refactor * avoid duplicate updating metric for leveldb index	2 years ago
Guo Lei	762dc219a6	fix deletecount bug (#3499 )	2 years ago
Guo Lei	81e7e6ceb3	the statistical methord for fileCount should stay same (#3477 ) the statistical methord for fileCount between should stay same They all equal to the entry count in .idx file. relate to commit(`c7892bc`)	2 years ago
chrislu	c7892bc7c4	volume: file counter should be all files address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228	2 years ago
chrislu	4d72a1ce6e	s3: remove deleted collection during metrics reporting fix https://github.com/seaweedfs/seaweedfs/issues/3453	2 years ago
chrislu	2b580a7566	also migrate jsonpb	2 years ago
chrislu	eaeb141b09	move proto package	2 years ago
chrislu	93261f5199	Revert "Refactor for Sync method (#3426 )" This reverts commit `670cb759f8`. with the pr weed/storage () - (master) > go test -count=1 ./... ok github.com/seaweedfs/seaweedfs/weed/storage 18.486s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.025s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.864s ? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.110s ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.414s ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.203s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files] weed/storage () - (master) > weed/storage () - (master) > without the pr weed/storage () - (master) > weed/storage () - (master) > go test -count=1 ./... ok github.com/seaweedfs/seaweedfs/weed/storage 1.617s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.026s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.906s ? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.202s ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.533s ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.280s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files]	2 years ago
Rain Li	670cb759f8	Refactor for Sync method (#3426 )	2 years ago
Abirdcfly	b0633716b7	delete minor unreachable code (#3423 ) Signed-off-by: Abirdcfly <fp544037857@gmail.com>	2 years ago
chrislu	26dbc6c905	move to https://github.com/seaweedfs/seaweedfs	2 years ago
Konstantin Lebedev	2f0dda384d	vacuum show LA	2 years ago
Evgeny Kuzhelev	47c72e6f35	remove all (currently existing) collection volume metrics	2 years ago
Evgeny Kuzhelev	709c83716c	delete disk_size metrics when collection deleted	2 years ago
guosj	cc7a4b0a6e	correct comment	2 years ago
guol-fnst	ac694f0c8f	rename parameter and reuse functions rename milestone to watermark	2 years ago
guol-fnst	91285bb51d	remove uncessary code	2 years ago
guol-fnst	b9256e0b34	optimiz	2 years ago
guol-fnst	bec9f79659	use snapshot	2 years ago
guol-fnst	300b383cdf	use 10 or numCPU workers if env is not found	2 years ago
guol-fnst	308a48c0c2	optimiz concurrency user can customize number of workers via env "GOMAXPROCS"	2 years ago
guol-fnst	26313060a3	speeding up loading volumes	2 years ago
chrislu	06a8b174b5	also remove Sync() for idx file	3 years ago
chrislu	52580743b9	stuck with file.Sync() starting to get weed/storage () - (master) > go test ./... I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000 I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648 I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000 I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959 I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000 I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming... panic: test timed out after 10m0s goroutine 61 [running]: testing.(M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2029 +0x8e created by time.goFunc /usr/local/go/src/time/sleep.go:176 +0x32 goroutine 1 [chan receive, 10 minutes]: testing.(T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8) /usr/local/go/src/testing/testing.go:1487 +0x37a testing.runTests.func1(0xc0000cd520?) /usr/local/go/src/testing/testing.go:1839 +0x6e testing.tRunner(0xc0000cd520, 0xc00039fcd8) /usr/local/go/src/testing/testing.go:1439 +0x102 testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?}) /usr/local/go/src/testing/testing.go:1837 +0x457 testing.(M).Run(0xc0000dadc0) /usr/local/go/src/testing/testing.go:1719 +0x5d9 main.main() _testmain.go:57 +0x1aa goroutine 21 [chan receive]: github.com/chrislusf/seaweedfs/weed/glog.(loggingT).flushDaemon(0x0?) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a created by github.com/chrislusf/seaweedfs/weed/glog.init.0 /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf goroutine 55 [syscall]: syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?) /usr/local/go/src/runtime/sys_darwin.go:22 +0x4e syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?) /usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30 internal/poll.(FD).Fsync.func1(...) /usr/local/go/src/internal/poll/fd_fsync_darwin.go:18 internal/poll.ignoringEINTR(...) /usr/local/go/src/internal/poll/fd_posix.go:74 internal/poll.(FD).Fsync(0xc00038f1e0?) /usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc os.(File).Sync(0xc00012a030) /usr/local/go/src/os/file_posix.go:168 +0x4e github.com/chrislusf/seaweedfs/weed/storage/backend.(DiskFile).Sync(...) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84 github.com/chrislusf/seaweedfs/weed/storage.(Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52}) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa github.com/chrislusf/seaweedfs/weed/storage.(Volume).CommitCompact(0xc0000bb440) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8 github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305 testing.tRunner(0xc00025a000, 0x1c67bd8) /usr/local/go/src/testing/testing.go:1439 +0x102 created by testing.(T).Run /usr/local/go/src/testing/testing.go:1486 +0x35f goroutine 56 [chan receive, 10 minutes]: github.com/chrislusf/seaweedfs/weed/storage.(Volume).startWorker.func1() /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56 FAIL github.com/chrislusf/seaweedfs/weed/storage 600.194s ? github.com/chrislusf/seaweedfs/weed/storage/backend [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map (cached) ? github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/erasure_coding (cached) ? github.com/chrislusf/seaweedfs/weed/storage/idx [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/needle (cached) ok github.com/chrislusf/seaweedfs/weed/storage/needle_map (cached) ok github.com/chrislusf/seaweedfs/weed/storage/super_block (cached) ? github.com/chrislusf/seaweedfs/weed/storage/types [no test files] ? github.com/chrislusf/seaweedfs/weed/storage/volume_info [no test files] FAIL	3 years ago
garenchan	8aa19577f4	fix 3238: handle errors for GenerateDirUuid method	3 years ago
chrislu	509a9047db	test compact map with snowflake sequencer	3 years ago
chrislu	625fd16a2e	reduce upfront memory usage for low density volume	3 years ago
chrislu	36d2756978	volume: avoid writing too much data for range requests in large files related to https://github.com/chrislusf/seaweedfs/issues/3178	3 years ago
chrislu	9f8b72a54d	Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID" This reverts commit `37da689319`, reversing changes made to `00d53c34c4`.	3 years ago
shichanglin5	f5b0c04b14	perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal Under normal circumstances, there will be no problems, but when the master is debugged in the local environment, the volume client cannot communicate with the master normally, so the sendHeartBeat logic is restarted, and a new connection is created to report the heartbeat. If the master has not cleared the uuid of the volume at this time, then The master will respond to volume duplicateUUIDS, and the volume service will exit, but in fact the uuid of the volume is not duplicated	3 years ago
chrislu	11f1e7996b	fix test	3 years ago
chrislu	056c480eb0	volume: checksum remove the hashing step in Value()	3 years ago
chrislu	ecef844dfc	stream read large files	3 years ago
chrislu	685643d506	needle read into a writer	3 years ago
chrislu	663bc5dc23	add needle reading in chunks	3 years ago

1 2 3 4 5 ...

853 Commits (e069009f09b6ff73cf3c8359a10a73372c530c59)