0581ce6096
fix delete chunk failed if volumeSever specified grpc.port ( #5820 )
Co-authored-by: Yang Wang <yangwang@weride.ai>
9 months ago
ec9455dab7
same change as #5819
follow up with https://github.com/seaweedfs/seaweedfs/pull/5819
9 months ago
d15966ae8e
Check `ShouldGrowVolumes` before returning error in assign. ( #5819 )
9 months ago
952afd810c
Fix dead lock ( #5815 )
* reduce locks to avoid dead lock
Flush->FlushData->uplloadPipeline.FluahAll
uploaderCount>0
goroutine 1 [sync.Cond.Wait, 71 minutes]:
sync.runtime_notifyListWait(0xc0007ae4d0, 0x0)
/usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc001a59290?)
/usr/local/go/src/sync/cond.go:70 +0x85
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).waitForCurrentWritersToComplete(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline_lock.go:58 +0x32
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).FlushAll(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline.go:151 +0x25
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).FlushData(0xc00087e840)
/github/workspace/weed/mount/dirty_pages_chunked.go:54 +0x29
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).FlushData(...)
/github/workspace/weed/mount/page_writer.go:50
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).doFlush(0xc0006ad600, 0xc00030d380, 0x0, 0x0)
/github/workspace/weed/mount/weedfs_file_sync.go:101 +0x169
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).Flush(0xc0006ad600, 0xc001a594a8?, 0xc0004c1ca0)
/github/workspace/weed/mount/weedfs_file_sync.go:59 +0x48
github.com/hanwen/go-fuse/v2/fuse.doFlush(0xc0000da870?, 0xc0004c1b08)
SaveContent -> MemChunk.RLock ->
ChunkedDirtyPages.saveChunkedFileIntervalToStorage
pages.fh.AddChunks([]*filer_pb.FileChunk{chunk})
fh.entryLock.Lock()
sync.(*RWMutex).Lock(0x0?)
/usr/local/go/src/sync/rwmutex.go:146 +0x31
github.com/seaweedfs/seaweedfs/weed/mount.(*FileHandle).AddChunks(0xc00030d380, {0xc00028bdc8, 0x1, 0x1})
/github/workspace/weed/mount/filehandle.go:93 +0x45
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).saveChunkedFileIntervalToStorage(0xc00087e840, {0x2be7ac0, 0xc00018d9e0}, 0x0, 0x121, 0x17e3c624565ace45, 0x1?)
/github/workspace/weed/mount/dirty_pages_chunked.go:80 +0x2d4
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).SaveContent(0xc0008d9130, 0xc0008093e0)
/github/workspace/weed/mount/page_writer/page_chunk_mem.go:115 +0x112
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).moveToSealed.func1()
/github/workspace/weed/mount/page_writer/upload_pipeline.go:187 +0x55
github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute.func1()
/github/workspace/weed/util/limited_executor.go:38 +0x62
created by github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute in goroutine 1
/github/workspace/weed/util/limited_executor.go:33 +0x97
On metadata update
fh.entryLock.Lock()
fh.dirtyPages.Destroy()
up.chunksLock.Lock => each sealed chunk.FreeReference => MemChunk.Lock
goroutine 134 [sync.RWMutex.Lock, 71 minutes]:
sync.runtime_SemacquireRWMutex(0xc0007c3558?, 0xea?, 0x3fb0800?)
/usr/local/go/src/runtime/sema.go:87 +0x25
sync.(*RWMutex).Lock(0xc0007c35a8?)
/usr/local/go/src/sync/rwmutex.go:151 +0x6a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).FreeResource(0xc0008d9130)
/github/workspace/weed/mount/page_writer/page_chunk_mem.go:38 +0x2a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*SealedChunk).FreeReference(0xc00071cdb0, {0xc0006ba1a0, 0x20})
/github/workspace/weed/mount/page_writer/upload_pipeline.go:38 +0xb7
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).Shutdown(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline.go:220 +0x185
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).Destroy(0xc0008cea40?)
/github/workspace/weed/mount/dirty_pages_chunked.go:87 +0x17
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).Destroy(...)
/github/workspace/weed/mount/page_writer.go:78
github.com/seaweedfs/seaweedfs/weed/mount.NewSeaweedFileSystem.func3({0xc00069a6c0, 0x30}, 0x6?)
/github/workspace/weed/mount/weedfs.go:119 +0x17a
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.NewMetaCache.func1({0xc00069a6c0?, 0xc00069a480?}, 0x4015b40?)
/github/workspace/weed/mount/meta_cache/meta_cache.go:37 +0x1c
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func1(0xc000661810)
/github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:43 +0x570
* use locked entry everywhere
* modifiable remote entry
* skip locking after getting lock from fhLockTable
9 months ago
8b402fb4b9
chore(deps): bump google.golang.org/api from 0.187.0 to 0.189.0 ( #5808 )
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client ) from 0.187.0 to 0.189.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases )
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md )
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.187.0...v0.189.0 )
---
updated-dependencies:
- dependency-name: google.golang.org/api
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
9 months ago
a7b48b971e
chore(deps): bump github.com/Jille/raft-grpc-transport from 1.5.0 to 1.6.1 ( #5807 )
chore(deps): bump github.com/Jille/raft-grpc-transport
Bumps [github.com/Jille/raft-grpc-transport](https://github.com/Jille/raft-grpc-transport ) from 1.5.0 to 1.6.1.
- [Release notes](https://github.com/Jille/raft-grpc-transport/releases )
- [Commits](https://github.com/Jille/raft-grpc-transport/compare/v1.5.0...v1.6.1 )
---
updated-dependencies:
- dependency-name: github.com/Jille/raft-grpc-transport
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
9 months ago
5e554582f5
chore(deps): bump go.etcd.io/etcd/client/pkg/v3 from 3.5.14 to 3.5.15 ( #5806 )
Bumps [go.etcd.io/etcd/client/pkg/v3](https://github.com/etcd-io/etcd ) from 3.5.14 to 3.5.15.
- [Release notes](https://github.com/etcd-io/etcd/releases )
- [Commits](https://github.com/etcd-io/etcd/compare/v3.5.14...v3.5.15 )
---
updated-dependencies:
- dependency-name: go.etcd.io/etcd/client/pkg/v3
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
9 months ago
e7cbed63a6
chore(deps): bump go.mongodb.org/mongo-driver from 1.15.0 to 1.16.0 ( #5805 )
Bumps [go.mongodb.org/mongo-driver](https://github.com/mongodb/mongo-go-driver ) from 1.15.0 to 1.16.0.
- [Release notes](https://github.com/mongodb/mongo-go-driver/releases )
- [Commits](https://github.com/mongodb/mongo-go-driver/compare/v1.15.0...v1.16.0 )
---
updated-dependencies:
- dependency-name: go.mongodb.org/mongo-driver
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
9 months ago
c2a6055c04
chore(deps): bump github.com/klauspost/reedsolomon from 1.12.1 to 1.12.3 ( #5804 )
Bumps [github.com/klauspost/reedsolomon](https://github.com/klauspost/reedsolomon ) from 1.12.1 to 1.12.3.
- [Release notes](https://github.com/klauspost/reedsolomon/releases )
- [Commits](https://github.com/klauspost/reedsolomon/compare/v1.12.1...v1.12.3 )
---
updated-dependencies:
- dependency-name: github.com/klauspost/reedsolomon
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
9 months ago
1c32ac133b
chore(deps): bump docker/setup-buildx-action from 3.4.0 to 3.5.0 ( #5803 )
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action ) from 3.4.0 to 3.5.0.
- [Release notes](https://github.com/docker/setup-buildx-action/releases )
- [Commits](4fd812986e
9 months ago
5e2471d790
chore(deps): bump docker/login-action from 3.2.0 to 3.3.0 ( #5802 )
Bumps [docker/login-action](https://github.com/docker/login-action ) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/docker/login-action/releases )
- [Commits](0d4c9c5ea7
9 months ago
171e824444
chore(deps): bump docker/setup-qemu-action from 3.1.0 to 3.2.0 ( #5801 )
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action ) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/docker/setup-qemu-action/releases )
- [Commits](5927c834f5
9 months ago
509ac40936
chore(deps): bump docker/build-push-action from 6.4.0 to 6.5.0 ( #5800 )
Bumps [docker/build-push-action](https://github.com/docker/build-push-action ) from 6.4.0 to 6.5.0.
- [Release notes](https://github.com/docker/build-push-action/releases )
- [Commits](a254f8ca60
9 months ago
ed7e721efe
3.71
9 months ago
9265be43c0
avoid nil
9 months ago
4020149690
3.70
9 months ago
9dd008f8f1
add version to filer
9 months ago
9dcc576499
Revert "add collection for buckets"
This reverts commit 96af571219
10 months ago
aed2a18223
get, set and add path conf
10 months ago
96af571219
add collection for buckets
10 months ago
a8fa78b892
refactoring
10 months ago
7a75928e7d
Azure sink: avoid overwriting existing files
10 months ago
86d92a42b4
Added tls for http clients ( #5766 )
* Added global http client
* Added Do func for global http client
* Changed the code to use the global http client
* Fix http client in volume uploader
* Fixed pkg name
* Fixed http util funcs
* Fixed http client for bench_filer_upload
* Fixed http client for stress_filer_upload
* Fixed http client for filer_server_handlers_proxy
* Fixed http client for command_fs_merge_volumes
* Fixed http client for command_fs_merge_volumes and command_volume_fsck
* Fixed http client for s3api_server
* Added init global client for main funcs
* Rename global_client to client
* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold
* Reduce the visibility of some functions in the util/http/client pkg
* Added the loadSecurityConfig function
* Use util.LoadSecurityConfiguration() in NewHttpClient func
10 months ago
c6dec11ea5
[filer.sync] skip overwriting existing fresh entry
10 months ago
19d14b4c71
minor
10 months ago
b796c21fa9
Added loadSecurityConfigOnce ( #5792 )
10 months ago
67edf1d014
[master] Do Automatic Volume Grow in background ( #5781 )
* Do Automatic Volume Grow in backgound
* pass lastGrowCount to master
* fix build
* fix count to uint64
10 months ago
ce61a66b65
Fix mmap write fail ( #5791 )
* [mount] fix GetAttr blocks count
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
* [mount] fix mmap read
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
---------
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
10 months ago
794a5aa67d
Add warning with hints to why a EC shard cant be moved ( #5790 )
10 months ago
a53e406c99
[master] refactor HasGrowRequest to atomic bool ( #5782 )
refactor HasGrowRequest to atomit bool
10 months ago
feb911a0b4
chore(deps): bump docker/build-push-action from 6.3.0 to 6.4.0 ( #5784 )
Bumps [docker/build-push-action](https://github.com/docker/build-push-action ) from 6.3.0 to 6.4.0.
- [Release notes](https://github.com/docker/build-push-action/releases )
- [Commits](1a162644f9
10 months ago
3083cd537f
chore(deps): bump modernc.org/sqlite from 1.30.1 to 1.30.2 ( #5785 )
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite ) from 1.30.1 to 1.30.2.
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.30.1...v1.30.2 )
---
updated-dependencies:
- dependency-name: modernc.org/sqlite
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
10 months ago
45cde7b5ea
chore(deps): bump cloud.google.com/go/pubsub from 1.39.0 to 1.40.0 ( #5786 )
Bumps [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go ) from 1.39.0 to 1.40.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases )
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md )
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.39.0...pubsub/v1.40.0 )
---
updated-dependencies:
- dependency-name: cloud.google.com/go/pubsub
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
10 months ago
4fd1528534
chore(deps): bump github.com/parquet-go/parquet-go from 0.22.0 to 0.23.0 ( #5787 )
Bumps [github.com/parquet-go/parquet-go](https://github.com/parquet-go/parquet-go ) from 0.22.0 to 0.23.0.
- [Release notes](https://github.com/parquet-go/parquet-go/releases )
- [Changelog](https://github.com/parquet-go/parquet-go/blob/main/CHANGELOG.md )
- [Commits](https://github.com/parquet-go/parquet-go/compare/v0.22.0...v0.23.0 )
---
updated-dependencies:
- dependency-name: github.com/parquet-go/parquet-go
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
10 months ago
6dbfac384a
chore(deps): bump github.com/aws/aws-sdk-go-v2 from 1.30.1 to 1.30.3 ( #5788 )
Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2 ) from 1.30.1 to 1.30.3.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.30.1...v1.30.3 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
10 months ago
7cc5076896
chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.58.0 to 1.58.2 ( #5789 )
chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3
Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2 ) from 1.58.0 to 1.58.2.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.58.0...service/s3/v1.58.2 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
10 months ago
33964fa292
metrics stats of volume layout depends on the data center ( #5775 )
stats volume layout depends on the data center
10 months ago
04f4b10884
fix: avoid timeout if datacenter does not exist in topology ( #5772 )
* fix: avoid timeout if datacenter does not exist in topology
* fix: error msg
* fix: rm dublicate check
* fix: compare
* revert minor change
10 months ago
3a82f5ffad
ensure metadata follow a specific folder
fix https://github.com/seaweedfs/seaweedfs/issues/5774
10 months ago
ec9e7493b3
-metricsIp cmd flag ( #5773 )
* Added/Updated:
- Added metrics ip options for all servers;
- Fixed a bug with the selection of the binIp or ip parameter for the metrics handler;
* Fixed cmd flags
10 months ago
cb5dae0c9c
Fix "no more writable volumes" error when volume grows and master leader changed at the same time ( #5771 )
10 months ago
e8537d7172
Different disk labels should not use the same DiskUsages instance while master received volume heatbeat ( #5770 )
10 months ago
4e7d8eb3f1
small refactoring
10 months ago
b62f7c5122
helm: restart filer when the s3-configuration changes ( #5768 )
Restarts the filer when the s3-config inside the secret (existing or generated) changes
10 months ago
de60f383de
Added tls for http handlers ( #5764 )
* Added https handler for filer
* Added example for security.toml
10 months ago
2addcd1623
fix test
10 months ago
590397229e
fix heap init
10 months ago
06e3dd306f
clean up
10 months ago
c59517e284
upgrade to github.com/redis/go-redis/v9
fix https://github.com/seaweedfs/seaweedfs/issues/4532
10 months ago
33b11f92bc
Squashed commit of the following:
commit 465b47b8f7
10 months ago
wyang
chrislu
Ruoxi
Chris Lu
dependabot[bot]
vadimartynov
Konstantin Lebedev
wusong
Dan
小羽
Gregor Tudan