Improve safety for weed shells `ec.encode`.
The current process for `ec.encode` is:
1. EC shards for a volume are generated and added to a single server
2. The original volume is deleted
3. EC shards get re-balanced across the entire topology
It is then possible to lose data between #2 and #3, if the underlying volume storage/server/rack/DC
happens to fail, for whatever reason. As a fix, this MR reworks `ec.encode` so:
* Newly created EC shards are spread across all locations for the source volume.
* Source volumes are deleted only after EC shards are converted and balanced.
NeedleHeaderSize happen to have the same size as NeedleMapEntrySize, except when running the 5 bytes offset variant of Seaweedfs, because it does not contain OffsetSize. This causes ECX corruption on deletes, due to the drifting offset computation (offset is always computed on a basis of 16 bytes per record instead of 17 bytes)
Signed-off-by: Quentin Devos <4972091+Okhoshi@users.noreply.github.com>
Improve parallelization for `ec.encode`.
Instead of processing one volume at at time, perform all EC conversion
steps (mark readonly -> generate EC shards -> delete volume -> remount) in
parallel for all of them.
This should substantially improve performance when EC encoding
entire collections.
* feat(redis): add mTLS support for Redis connection initialization
- Enhanced the Redis2Store initialization to support mutual TLS (mTLS) by adding configuration options for CA certificate, client certificate, and client key paths.
- Updated the Redis client setup to use TLS configuration when mTLS is enabled, ensuring secure connections to the Redis server.
* feat(redis): extend Redis3Store initialization to support mTLS
- Added configuration options for enabling mutual TLS (mTLS) in Redis3Store.
- Implemented logic to load client certificates and CA certificates for secure Redis connections.
- Updated the Redis client setup to utilize TLS configuration when mTLS is enabled.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
This prevent crash filler with nil pointer dereference as s3 expect this
parameters.
New two parameters are added to filer command - copy of s3 parameters:
- s3.cacert.file - path to the TLS CA certificate file
- s3.tlsVerifyClientCert - whether to verify the client's certificate
* rename
* set agent address
* refactor
* add agent sub
* pub messages
* grpc new client
* can publish records via agent
* send init message with session id
* fmt
* check cancelled request while waiting
* use sessionId
* handle possible nil stream
* subscriber process messages
* separate debug port
* use atomic int64
* less logs
* minor
* skip io.EOF
* rename
* remove unused
* use saved offsets
* do not reuse session, since always session id is new after restart
remove last active ts from SessionEntry
* simplify printing
* purge unused
* just proxy the subscription, skipping the session step
* adjust offset types
* subscribe offset type and possible value
* start after the known tsns
* avoid wrongly set startPosition
* move
* remove
* refactor
* typo
* fix
* fix changed path
SmoothDenis
Kuzmin Anton
Lisandro Pin
Quentin D.
Bruce Zou
Mohamed Sekour
ARibster
chrislu
Jade Devin Cabatlao
Taehyung Lim
NyaMisty
Konstantin Lebedev
zuzuviewer
bwlfhu
dependabot[bot]
MaratKarimov
Chris Lu
alingse
Karol Będkowski
Tom Crasset
Aleksey Kosov
jang1lee
Weihao Jiang
hoverseu
NinaLua
lizhengui007
tiaoxizhan