* fix filer range read
Only return true if we're reading the ENTIRE chunk from the beginning.
// This prevents bandwidth amplification when range requests happen to align
// with chunk boundaries but don't actually want the full chunk.
* Update weed/filer/filechunks.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* refactoring
* add ec shard size
* address comments
* passing task id
There seems to be a disconnect between the pending tasks created in ActiveTopology and the TaskDetectionResult returned by this function. A taskID is generated locally and used to create pending tasks via AddPendingECShardTask, but this taskID is not stored in the TaskDetectionResult or passed along in any way.
This makes it impossible for the worker that eventually executes the task to know which pending task in ActiveTopology it corresponds to. Without the correct taskID, the worker cannot call AssignTask or CompleteTask on the master, breaking the entire task lifecycle and capacity management feature.
A potential solution is to add a TaskID field to TaskDetectionResult and worker_pb.TaskParams, ensuring the ID is propagated from detection to execution.
* 1 source multiple destinations
* task supports multi source and destination
* ec needs to clean up previous shards
* use erasure coding constants
* getPlanningCapacityUnsafe getEffectiveAvailableCapacityUnsafe should return StorageSlotChange for calculation
* use CanAccommodate to calculate
* remove dead code
* address comments
* fix Mutex Copying in Protobuf Structs
* use constants
* fix estimatedSize
The calculation for estimatedSize only considers source.EstimatedSize and dest.StorageChange, but omits dest.EstimatedSize. The TaskDestination struct has an EstimatedSize field, which seems to be ignored here. This could lead to an incorrect estimation of the total size of data involved in tasks on a disk. The loop should probably also include estimatedSize += dest.EstimatedSize.
* at.assignTaskToDisk(task)
* refactoring
* Update weed/admin/topology/internal.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fail fast
* fix compilation
* Update weed/worker/tasks/erasure_coding/detection.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* indexes for volume and shard locations
* dedup with ToVolumeSlots
* return an additional boolean to indicate success, or an error
* Update abstract_sql_store.go
* fix
* Update weed/worker/tasks/erasure_coding/detection.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update weed/admin/topology/task_management.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* faster findVolumeDisk
* Update weed/worker/tasks/erasure_coding/detection.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/admin/topology/storage_slot_test.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* refactor
* simplify
* remove unused GetDiskStorageImpact function
* refactor
* add comments
* Update weed/admin/topology/storage_impact.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update weed/admin/topology/storage_slot_test.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update storage_impact.go
* AddPendingTask
The unified AddPendingTask function now serves as the single entry point for all task creation, successfully consolidating the previously separate functions while maintaining full functionality and improving code organization.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix(ydb): table creation with params
* fix(ydb): migrate to new query client & prevent creation table-bucket on get request
* fix(ydb): use new query client with kv req
* fix(ydb): use directory in every query
* fix(ydb): del unused import
* fix(ydb): tests & default const usage
* feat(redis): add mTLS support for Redis connection initialization
- Enhanced the Redis2Store initialization to support mutual TLS (mTLS) by adding configuration options for CA certificate, client certificate, and client key paths.
- Updated the Redis client setup to use TLS configuration when mTLS is enabled, ensuring secure connections to the Redis server.
* feat(redis): extend Redis3Store initialization to support mTLS
- Added configuration options for enabling mutual TLS (mTLS) in Redis3Store.
- Implemented logic to load client certificates and CA certificates for secure Redis connections.
- Updated the Redis client setup to utilize TLS configuration when mTLS is enabled.
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
* set larger buf size for LogBuffer
* jump to next day when no more entry found
* Update weed/filer/filer_notify_read.go
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* feature: we can check if a fileId is already in the cache
We using this to protect cache from adding the same needle to
the cache over and over.
* fuse mount: Do not start dowloader if needle is already in the cache
* added maxFilePartSizeInCache property to ChunkCache
If file very large only first maxFilePartSizeInCache bytes
are going to be put to the cache (subject to the needle size
constrains).
* feature: for large files put in cache no more than prescribed number of bytes
Before this patch only the first needle of a large file was intended for
caching. This patch uses maximum prescribed amount of bytes to be put in
cache. This allows to bypass default 2MB maximum for a file part stored
in the cache.
* added dummy mock methods to satisfy interfaces of ChunkCache
Chris Lu
Maxim Kostyukov
chrislu
Aleksey Kosov
SmoothDenis
Bruce Zou
FQHSLycopene
bwlfhu
Kuzmin Anton
Jade Devin Cabatlao
MaratKarimov
Numblgw
Guang Jiong Lou
Bruce
Konstantin Lebedev
Eugeniy E. Mikhailov