d15966ae8e
Check `ShouldGrowVolumes` before returning error in assign. ( #5819 )
1 year ago
952afd810c
Fix dead lock ( #5815 )
* reduce locks to avoid dead lock
Flush->FlushData->uplloadPipeline.FluahAll
uploaderCount>0
goroutine 1 [sync.Cond.Wait, 71 minutes]:
sync.runtime_notifyListWait(0xc0007ae4d0, 0x0)
/usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc001a59290?)
/usr/local/go/src/sync/cond.go:70 +0x85
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).waitForCurrentWritersToComplete(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline_lock.go:58 +0x32
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).FlushAll(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline.go:151 +0x25
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).FlushData(0xc00087e840)
/github/workspace/weed/mount/dirty_pages_chunked.go:54 +0x29
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).FlushData(...)
/github/workspace/weed/mount/page_writer.go:50
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).doFlush(0xc0006ad600, 0xc00030d380, 0x0, 0x0)
/github/workspace/weed/mount/weedfs_file_sync.go:101 +0x169
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).Flush(0xc0006ad600, 0xc001a594a8?, 0xc0004c1ca0)
/github/workspace/weed/mount/weedfs_file_sync.go:59 +0x48
github.com/hanwen/go-fuse/v2/fuse.doFlush(0xc0000da870?, 0xc0004c1b08)
SaveContent -> MemChunk.RLock ->
ChunkedDirtyPages.saveChunkedFileIntervalToStorage
pages.fh.AddChunks([]*filer_pb.FileChunk{chunk})
fh.entryLock.Lock()
sync.(*RWMutex).Lock(0x0?)
/usr/local/go/src/sync/rwmutex.go:146 +0x31
github.com/seaweedfs/seaweedfs/weed/mount.(*FileHandle).AddChunks(0xc00030d380, {0xc00028bdc8, 0x1, 0x1})
/github/workspace/weed/mount/filehandle.go:93 +0x45
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).saveChunkedFileIntervalToStorage(0xc00087e840, {0x2be7ac0, 0xc00018d9e0}, 0x0, 0x121, 0x17e3c624565ace45, 0x1?)
/github/workspace/weed/mount/dirty_pages_chunked.go:80 +0x2d4
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).SaveContent(0xc0008d9130, 0xc0008093e0)
/github/workspace/weed/mount/page_writer/page_chunk_mem.go:115 +0x112
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).moveToSealed.func1()
/github/workspace/weed/mount/page_writer/upload_pipeline.go:187 +0x55
github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute.func1()
/github/workspace/weed/util/limited_executor.go:38 +0x62
created by github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute in goroutine 1
/github/workspace/weed/util/limited_executor.go:33 +0x97
On metadata update
fh.entryLock.Lock()
fh.dirtyPages.Destroy()
up.chunksLock.Lock => each sealed chunk.FreeReference => MemChunk.Lock
goroutine 134 [sync.RWMutex.Lock, 71 minutes]:
sync.runtime_SemacquireRWMutex(0xc0007c3558?, 0xea?, 0x3fb0800?)
/usr/local/go/src/runtime/sema.go:87 +0x25
sync.(*RWMutex).Lock(0xc0007c35a8?)
/usr/local/go/src/sync/rwmutex.go:151 +0x6a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).FreeResource(0xc0008d9130)
/github/workspace/weed/mount/page_writer/page_chunk_mem.go:38 +0x2a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*SealedChunk).FreeReference(0xc00071cdb0, {0xc0006ba1a0, 0x20})
/github/workspace/weed/mount/page_writer/upload_pipeline.go:38 +0xb7
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).Shutdown(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline.go:220 +0x185
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).Destroy(0xc0008cea40?)
/github/workspace/weed/mount/dirty_pages_chunked.go:87 +0x17
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).Destroy(...)
/github/workspace/weed/mount/page_writer.go:78
github.com/seaweedfs/seaweedfs/weed/mount.NewSeaweedFileSystem.func3({0xc00069a6c0, 0x30}, 0x6?)
/github/workspace/weed/mount/weedfs.go:119 +0x17a
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.NewMetaCache.func1({0xc00069a6c0?, 0xc00069a480?}, 0x4015b40?)
/github/workspace/weed/mount/meta_cache/meta_cache.go:37 +0x1c
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func1(0xc000661810)
/github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:43 +0x570
* use locked entry everywhere
* modifiable remote entry
* skip locking after getting lock from fhLockTable
1 year ago
ed7e721efe
3.71
1 year ago
9265be43c0
avoid nil
1 year ago
4020149690
3.70
1 year ago
9dd008f8f1
add version to filer
1 year ago
9dcc576499
Revert "add collection for buckets"
This reverts commit 96af571219
1 year ago
aed2a18223
get, set and add path conf
1 year ago
96af571219
add collection for buckets
1 year ago
a8fa78b892
refactoring
1 year ago
7a75928e7d
Azure sink: avoid overwriting existing files
1 year ago
86d92a42b4
Added tls for http clients ( #5766 )
* Added global http client
* Added Do func for global http client
* Changed the code to use the global http client
* Fix http client in volume uploader
* Fixed pkg name
* Fixed http util funcs
* Fixed http client for bench_filer_upload
* Fixed http client for stress_filer_upload
* Fixed http client for filer_server_handlers_proxy
* Fixed http client for command_fs_merge_volumes
* Fixed http client for command_fs_merge_volumes and command_volume_fsck
* Fixed http client for s3api_server
* Added init global client for main funcs
* Rename global_client to client
* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold
* Reduce the visibility of some functions in the util/http/client pkg
* Added the loadSecurityConfig function
* Use util.LoadSecurityConfiguration() in NewHttpClient func
1 year ago
c6dec11ea5
[filer.sync] skip overwriting existing fresh entry
1 year ago
19d14b4c71
minor
1 year ago
b796c21fa9
Added loadSecurityConfigOnce ( #5792 )
1 year ago
67edf1d014
[master] Do Automatic Volume Grow in background ( #5781 )
* Do Automatic Volume Grow in backgound
* pass lastGrowCount to master
* fix build
* fix count to uint64
1 year ago
ce61a66b65
Fix mmap write fail ( #5791 )
* [mount] fix GetAttr blocks count
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
* [mount] fix mmap read
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
---------
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
1 year ago
794a5aa67d
Add warning with hints to why a EC shard cant be moved ( #5790 )
1 year ago
a53e406c99
[master] refactor HasGrowRequest to atomic bool ( #5782 )
refactor HasGrowRequest to atomit bool
1 year ago
33964fa292
metrics stats of volume layout depends on the data center ( #5775 )
stats volume layout depends on the data center
1 year ago
04f4b10884
fix: avoid timeout if datacenter does not exist in topology ( #5772 )
* fix: avoid timeout if datacenter does not exist in topology
* fix: error msg
* fix: rm dublicate check
* fix: compare
* revert minor change
1 year ago
3a82f5ffad
ensure metadata follow a specific folder
fix https://github.com/seaweedfs/seaweedfs/issues/5774
1 year ago
ec9e7493b3
-metricsIp cmd flag ( #5773 )
* Added/Updated:
- Added metrics ip options for all servers;
- Fixed a bug with the selection of the binIp or ip parameter for the metrics handler;
* Fixed cmd flags
1 year ago
cb5dae0c9c
Fix "no more writable volumes" error when volume grows and master leader changed at the same time ( #5771 )
1 year ago
e8537d7172
Different disk labels should not use the same DiskUsages instance while master received volume heatbeat ( #5770 )
1 year ago
4e7d8eb3f1
small refactoring
1 year ago
de60f383de
Added tls for http handlers ( #5764 )
* Added https handler for filer
* Added example for security.toml
1 year ago
2addcd1623
fix test
1 year ago
590397229e
fix heap init
1 year ago
06e3dd306f
clean up
1 year ago
c59517e284
upgrade to github.com/redis/go-redis/v9
fix https://github.com/seaweedfs/seaweedfs/issues/4532
1 year ago
33b11f92bc
Squashed commit of the following:
commit 465b47b8f7
1 year ago
9c440d472c
[s3] revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function ( #5760 )
* revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function
* fix response error
* fix test_lifecycle_get
* Revert "fix test_lifecycle_get"
This reverts commit 8f991bdcf9
1 year ago
182f6d00a9
always copy ecj files
1 year ago
bd54669d58
Detect underflow when calculating unused space ( #5758 )
* Detect underflow when calculating unused space
* Detect underflow when calculating unused space
1 year ago
b8121e4b24
simplify
1 year ago
f618121e63
Fix S3 deletion in deep folders, and names with empty spaces
fix https://github.com/seaweedfs/seaweedfs/issues/5748
1 year ago
f3d708d3b6
add a test case
1 year ago
d29fccc521
skip checking if it is the root directory
1 year ago
114ac39917
add a simple test
1 year ago
f77eee667d
add s3test for sql ( #5718 )
* add s3test for sql
* fix test test_bucket_listv2_delimiter_basic for s3
* fix action s3tests
* regen s3 api xsd
* rm minor s3 test test_bucket_listv2_fetchowner_defaultempty
* add docs
* without xmlns
1 year ago
7c06306857
fix breadcrumb
1 year ago
7ee1f520a4
volume.tier.upload: Fix deleting replicated volumes ( #5743 )
Co-authored-by: Rhys M <rhysm@protonmail.com>
1 year ago
8943a572d8
log error message
1 year ago
62c29b2398
Revert "directory can paginate"
This reverts commit 69003d0e90
1 year ago
761a1555de
fix ordering
1 year ago
69003d0e90
directory can paginate
1 year ago
5ffacbb6ea
refactor all methods strings to const ( #5726 )
1 year ago
0fdf7eca48
avoid possible nil case during other error cases
1 year ago
ac6fd36c06
using fixed bootstrap from peer filer
1 year ago
Ruoxi
Chris Lu
chrislu
vadimartynov
Konstantin Lebedev
wusong
Dan
小羽
rhysm