952afd810c
Fix dead lock ( #5815 )
* reduce locks to avoid dead lock
Flush->FlushData->uplloadPipeline.FluahAll
uploaderCount>0
goroutine 1 [sync.Cond.Wait, 71 minutes]:
sync.runtime_notifyListWait(0xc0007ae4d0, 0x0)
/usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc001a59290?)
/usr/local/go/src/sync/cond.go:70 +0x85
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).waitForCurrentWritersToComplete(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline_lock.go:58 +0x32
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).FlushAll(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline.go:151 +0x25
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).FlushData(0xc00087e840)
/github/workspace/weed/mount/dirty_pages_chunked.go:54 +0x29
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).FlushData(...)
/github/workspace/weed/mount/page_writer.go:50
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).doFlush(0xc0006ad600, 0xc00030d380, 0x0, 0x0)
/github/workspace/weed/mount/weedfs_file_sync.go:101 +0x169
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).Flush(0xc0006ad600, 0xc001a594a8?, 0xc0004c1ca0)
/github/workspace/weed/mount/weedfs_file_sync.go:59 +0x48
github.com/hanwen/go-fuse/v2/fuse.doFlush(0xc0000da870?, 0xc0004c1b08)
SaveContent -> MemChunk.RLock ->
ChunkedDirtyPages.saveChunkedFileIntervalToStorage
pages.fh.AddChunks([]*filer_pb.FileChunk{chunk})
fh.entryLock.Lock()
sync.(*RWMutex).Lock(0x0?)
/usr/local/go/src/sync/rwmutex.go:146 +0x31
github.com/seaweedfs/seaweedfs/weed/mount.(*FileHandle).AddChunks(0xc00030d380, {0xc00028bdc8, 0x1, 0x1})
/github/workspace/weed/mount/filehandle.go:93 +0x45
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).saveChunkedFileIntervalToStorage(0xc00087e840, {0x2be7ac0, 0xc00018d9e0}, 0x0, 0x121, 0x17e3c624565ace45, 0x1?)
/github/workspace/weed/mount/dirty_pages_chunked.go:80 +0x2d4
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).SaveContent(0xc0008d9130, 0xc0008093e0)
/github/workspace/weed/mount/page_writer/page_chunk_mem.go:115 +0x112
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).moveToSealed.func1()
/github/workspace/weed/mount/page_writer/upload_pipeline.go:187 +0x55
github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute.func1()
/github/workspace/weed/util/limited_executor.go:38 +0x62
created by github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute in goroutine 1
/github/workspace/weed/util/limited_executor.go:33 +0x97
On metadata update
fh.entryLock.Lock()
fh.dirtyPages.Destroy()
up.chunksLock.Lock => each sealed chunk.FreeReference => MemChunk.Lock
goroutine 134 [sync.RWMutex.Lock, 71 minutes]:
sync.runtime_SemacquireRWMutex(0xc0007c3558?, 0xea?, 0x3fb0800?)
/usr/local/go/src/runtime/sema.go:87 +0x25
sync.(*RWMutex).Lock(0xc0007c35a8?)
/usr/local/go/src/sync/rwmutex.go:151 +0x6a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).FreeResource(0xc0008d9130)
/github/workspace/weed/mount/page_writer/page_chunk_mem.go:38 +0x2a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*SealedChunk).FreeReference(0xc00071cdb0, {0xc0006ba1a0, 0x20})
/github/workspace/weed/mount/page_writer/upload_pipeline.go:38 +0xb7
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).Shutdown(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline.go:220 +0x185
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).Destroy(0xc0008cea40?)
/github/workspace/weed/mount/dirty_pages_chunked.go:87 +0x17
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).Destroy(...)
/github/workspace/weed/mount/page_writer.go:78
github.com/seaweedfs/seaweedfs/weed/mount.NewSeaweedFileSystem.func3({0xc00069a6c0, 0x30}, 0x6?)
/github/workspace/weed/mount/weedfs.go:119 +0x17a
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.NewMetaCache.func1({0xc00069a6c0?, 0xc00069a480?}, 0x4015b40?)
/github/workspace/weed/mount/meta_cache/meta_cache.go:37 +0x1c
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func1(0xc000661810)
/github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:43 +0x570
* use locked entry everywhere
* modifiable remote entry
* skip locking after getting lock from fhLockTable
9 months ago
ed7e721efe
3.71
9 months ago
9265be43c0
avoid nil
9 months ago
4020149690
3.70
9 months ago
9dd008f8f1
add version to filer
9 months ago
9dcc576499
Revert "add collection for buckets"
This reverts commit 96af571219
9 months ago
aed2a18223
get, set and add path conf
9 months ago
96af571219
add collection for buckets
9 months ago
a8fa78b892
refactoring
9 months ago
7a75928e7d
Azure sink: avoid overwriting existing files
9 months ago
86d92a42b4
Added tls for http clients ( #5766 )
* Added global http client
* Added Do func for global http client
* Changed the code to use the global http client
* Fix http client in volume uploader
* Fixed pkg name
* Fixed http util funcs
* Fixed http client for bench_filer_upload
* Fixed http client for stress_filer_upload
* Fixed http client for filer_server_handlers_proxy
* Fixed http client for command_fs_merge_volumes
* Fixed http client for command_fs_merge_volumes and command_volume_fsck
* Fixed http client for s3api_server
* Added init global client for main funcs
* Rename global_client to client
* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold
* Reduce the visibility of some functions in the util/http/client pkg
* Added the loadSecurityConfig function
* Use util.LoadSecurityConfiguration() in NewHttpClient func
9 months ago
c6dec11ea5
[filer.sync] skip overwriting existing fresh entry
10 months ago
19d14b4c71
minor
10 months ago
b796c21fa9
Added loadSecurityConfigOnce ( #5792 )
10 months ago
67edf1d014
[master] Do Automatic Volume Grow in background ( #5781 )
* Do Automatic Volume Grow in backgound
* pass lastGrowCount to master
* fix build
* fix count to uint64
10 months ago
ce61a66b65
Fix mmap write fail ( #5791 )
* [mount] fix GetAttr blocks count
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
* [mount] fix mmap read
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
---------
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
10 months ago
794a5aa67d
Add warning with hints to why a EC shard cant be moved ( #5790 )
10 months ago
a53e406c99
[master] refactor HasGrowRequest to atomic bool ( #5782 )
refactor HasGrowRequest to atomit bool
10 months ago
33964fa292
metrics stats of volume layout depends on the data center ( #5775 )
stats volume layout depends on the data center
10 months ago
04f4b10884
fix: avoid timeout if datacenter does not exist in topology ( #5772 )
* fix: avoid timeout if datacenter does not exist in topology
* fix: error msg
* fix: rm dublicate check
* fix: compare
* revert minor change
10 months ago
3a82f5ffad
ensure metadata follow a specific folder
fix https://github.com/seaweedfs/seaweedfs/issues/5774
10 months ago
ec9e7493b3
-metricsIp cmd flag ( #5773 )
* Added/Updated:
- Added metrics ip options for all servers;
- Fixed a bug with the selection of the binIp or ip parameter for the metrics handler;
* Fixed cmd flags
10 months ago
cb5dae0c9c
Fix "no more writable volumes" error when volume grows and master leader changed at the same time ( #5771 )
10 months ago
e8537d7172
Different disk labels should not use the same DiskUsages instance while master received volume heatbeat ( #5770 )
10 months ago
4e7d8eb3f1
small refactoring
10 months ago
de60f383de
Added tls for http handlers ( #5764 )
* Added https handler for filer
* Added example for security.toml
10 months ago
2addcd1623
fix test
10 months ago
590397229e
fix heap init
10 months ago
06e3dd306f
clean up
10 months ago
c59517e284
upgrade to github.com/redis/go-redis/v9
fix https://github.com/seaweedfs/seaweedfs/issues/4532
10 months ago
33b11f92bc
Squashed commit of the following:
commit 465b47b8f7
10 months ago
9c440d472c
[s3] revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function ( #5760 )
* revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function
* fix response error
* fix test_lifecycle_get
* Revert "fix test_lifecycle_get"
This reverts commit 8f991bdcf9
10 months ago
182f6d00a9
always copy ecj files
10 months ago
bd54669d58
Detect underflow when calculating unused space ( #5758 )
* Detect underflow when calculating unused space
* Detect underflow when calculating unused space
10 months ago
b8121e4b24
simplify
10 months ago
f618121e63
Fix S3 deletion in deep folders, and names with empty spaces
fix https://github.com/seaweedfs/seaweedfs/issues/5748
10 months ago
f3d708d3b6
add a test case
10 months ago
d29fccc521
skip checking if it is the root directory
10 months ago
114ac39917
add a simple test
10 months ago
f77eee667d
add s3test for sql ( #5718 )
* add s3test for sql
* fix test test_bucket_listv2_delimiter_basic for s3
* fix action s3tests
* regen s3 api xsd
* rm minor s3 test test_bucket_listv2_fetchowner_defaultempty
* add docs
* without xmlns
10 months ago
7c06306857
fix breadcrumb
10 months ago
7ee1f520a4
volume.tier.upload: Fix deleting replicated volumes ( #5743 )
Co-authored-by: Rhys M <rhysm@protonmail.com>
10 months ago
8943a572d8
log error message
10 months ago
62c29b2398
Revert "directory can paginate"
This reverts commit 69003d0e90
10 months ago
761a1555de
fix ordering
10 months ago
69003d0e90
directory can paginate
10 months ago
5ffacbb6ea
refactor all methods strings to const ( #5726 )
10 months ago
0fdf7eca48
avoid possible nil case during other error cases
10 months ago
ac6fd36c06
using fixed bootstrap from peer filer
10 months ago
fbe1316c92
3.69
10 months ago
Chris Lu
chrislu
vadimartynov
Konstantin Lebedev
wusong
Dan
小羽
rhysm