300b383cdf
use 10 or numCPU workers if env is not found
3 years ago
308a48c0c2
optimiz concurrency
user can customize number of workers via env "GOMAXPROCS"
3 years ago
26313060a3
speeding up loading volumes
3 years ago
06a8b174b5
also remove Sync() for idx file
3 years ago
52580743b9
stuck with file.Sync()
starting to get weed/storage () - (master) > go test ./...
I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648
I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959
I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000
I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory
I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming...
panic: test timed out after 10m0s
goroutine 61 [running]:
testing.(*M).startAlarm.func1()
/usr/local/go/src/testing/testing.go:2029 +0x8e
created by time.goFunc
/usr/local/go/src/time/sleep.go:176 +0x32
goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1487 +0x37a
testing.runTests.func1(0xc0000cd520?)
/usr/local/go/src/testing/testing.go:1839 +0x6e
testing.tRunner(0xc0000cd520, 0xc00039fcd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?})
/usr/local/go/src/testing/testing.go:1837 +0x457
testing.(*M).Run(0xc0000dadc0)
/usr/local/go/src/testing/testing.go:1719 +0x5d9
main.main()
_testmain.go:57 +0x1aa
goroutine 21 [chan receive]:
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a
created by github.com/chrislusf/seaweedfs/weed/glog.init.0
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf
goroutine 55 [syscall]:
syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?)
/usr/local/go/src/runtime/sys_darwin.go:22 +0x4e
syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?)
/usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30
internal/poll.(*FD).Fsync.func1(...)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:18
internal/poll.ignoringEINTR(...)
/usr/local/go/src/internal/poll/fd_posix.go:74
internal/poll.(*FD).Fsync(0xc00038f1e0?)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc
os.(*File).Sync(0xc00012a030)
/usr/local/go/src/os/file_posix.go:168 +0x4e
github.com/chrislusf/seaweedfs/weed/storage/backend.(*DiskFile).Sync(...)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52})
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc0000bb440)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8
github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305
testing.tRunner(0xc00025a000, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1486 +0x35f
goroutine 56 [chan receive, 10 minutes]:
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker.func1()
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a
created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56
FAIL github.com/chrislusf/seaweedfs/weed/storage 600.194s
? github.com/chrislusf/seaweedfs/weed/storage/backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map (cached)
? github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/erasure_coding (cached)
? github.com/chrislusf/seaweedfs/weed/storage/idx [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/needle (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/needle_map (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/super_block (cached)
? github.com/chrislusf/seaweedfs/weed/storage/types [no test files]
? github.com/chrislusf/seaweedfs/weed/storage/volume_info [no test files]
FAIL
3 years ago
8aa19577f4
fix 3238: handle errors for GenerateDirUuid method
3 years ago
509a9047db
test compact map with snowflake sequencer
3 years ago
625fd16a2e
reduce upfront memory usage for low density volume
3 years ago
36d2756978
volume: avoid writing too much data for range requests in large files
related to https://github.com/chrislusf/seaweedfs/issues/3178
3 years ago
9f8b72a54d
Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID"
This reverts commit 37da68931900d53c34c4
3 years ago
f5b0c04b14
perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal
Under normal circumstances, there will be no problems, but when the
master is debugged in the local environment, the volume client cannot
communicate with the master normally, so the sendHeartBeat logic is
restarted, and a new connection is created to report the heartbeat. If
the master has not cleared the uuid of the volume at this time, then The
master will respond to volume duplicateUUIDS, and the volume service
will exit, but in fact the uuid of the volume is not duplicated
3 years ago
11f1e7996b
fix test
3 years ago
056c480eb0
volume: checksum remove the hashing step in Value()
3 years ago
ecef844dfc
stream read large files
3 years ago
685643d506
needle read into a writer
3 years ago
663bc5dc23
add needle reading in chunks
3 years ago
18b29f709c
refactoring, split file
3 years ago
81d6159290
volume: report error if a volume has nil data backend
fix https://github.com/chrislusf/seaweedfs/issues/3105
3 years ago
b12944f9c6
fix naming convention
notify volume server of duplicate directoris
improve searching efficiency
4 years ago
8fab39e775
rename UUID file
fix typo
move locationUUID into DiskLocation
4 years ago
de6aa9cce8
avoid duplicated volume directory
4 years ago
3e20336bde
fix atomic add int64
fix https://github.com/chrislusf/seaweedfs/issues/3038#issuecomment-1123269831
4 years ago
76148ce0f7
use storage_class for backend configuration
fix https://github.com/chrislusf/seaweedfs/issues/3008
4 years ago
1aae7a3f1b
volume.tier.upload progress starts negative #2992
fix https://github.com/chrislusf/seaweedfs/issues/2992
4 years ago
192983b464
s3 backend support customizing storage class
4 years ago
70e5a1b632
volume close should wait for committing compaction
4 years ago
37ab8909b0
use two flags: v.isCompacting and v.isCommitCompacting
4 years ago
94f824e1ce
volume: sync to disk before copying volume files
address https://github.com/chrislusf/seaweedfs/issues/2976
4 years ago
7315d1d039
waite volume being closed during compression idx
4 years ago
8e9ad7db5a
fix bug deleting volume or unmount volume if a volume server has multiple directories
4 years ago
1e35b4929f
shell vacuum volume by collection and volume id
4 years ago
9438738693
avoid invalid memory address or nil pointer dereference
4 years ago
3551ca2fcf
enhancement: replace sort.Slice with slices.SortFunc to reduce reflection
4 years ago
d51a724101
fix: encode small chunk return error maybe have some bug.
4 years ago
f53cff045f
checks disk file exist
4 years ago
3e7aa1caf5
fix return value in storage/volume_vacuum.go:444
4 years ago
c29bc9a367
fix error handling
4 years ago
d68c27f82d
fix another return value bug
4 years ago
8f9aa0cddd
fix bugs in return value
4 years ago
a129bda7d9
sync data first before stopping
4 years ago
b92df1654c
test: use `T.TempDir` to create temporary test directory
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
4 years ago
ef541972f8
updated needle with fsync
4 years ago
fb97e234c9
skips compact if store is stopping
4 years ago
433fde4b18
move error to a separate file
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
4 years ago
76e297d64f
sync call to write file, avoid vif loading error
fix https://github.com/chrislusf/seaweedfs/issues/2633
4 years ago
3f4e17aa24
error metrics for filer and store
4 years ago
9f9ef1340c
use streaming mode for long poll grpc calls
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
4 years ago
50ddd8c8e2
remove debug messages
fix https://github.com/chrislusf/seaweedfs/issues/2514
4 years ago
488afa5002
volume: load volume can optionally be skipped, if ec volume exists
fix https://github.com/chrislusf/seaweedfs/issues/2489
4 years ago
59e58c4b23
volume: fix loading old volume format
fix https://github.com/chrislusf/seaweedfs/issues/2487
4 years ago
guol-fnst
chrislu
garenchan
shichanglin5
guol-fnst
Konstantin Lebedev
justin
guosj
Eng Zer Jun
chrislu