04b141749f
explicit generation value
5 months ago
a31fc1ac01
Bypass compaction revision check to handle volumes compacted after task creation
6 months ago
05a0cc156b
Self-Contained Design
To prove the system is truly self-contained:
To add a new task:
Create a task package (e.g., worker/tasks/compression/)
Import it: _ "github.com/.../worker/tasks/compression"
That's it! No other changes needed.
To remove a task:
Delete the task package directory
Remove the import line
That's it! No other changes needed.
6 months ago
c7f291e4e4
simplify logging
6 months ago
25bbf4c3d4
Admin UI: Fetch task logs ( #7114 )
* show task details
* loading tasks
* task UI works
* generic rendering
* rendering the export link
* removing placementConflicts from task parameters
* remove TaskSourceLocation
* remove "Server ID" column
* rendering balance task source
* sources and targets
* fix ec task generation
* move info
* render timeline
* simplified worker id
* simplify
* read task logs from worker
* isValidTaskID
* address comments
* Update weed/worker/tasks/balance/execution.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/worker/tasks/erasure_coding/ec_task.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/worker/tasks/task_log_handler.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix shard ids
* plan distributing shard id
* rendering planned shards in task details
* remove Conflicts
* worker logs correctly
* pass in dc and rack
* task logging
* Update weed/admin/maintenance/maintenance_queue.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* display log details
* logs have fields now
* sort field keys
* fix link
* fix collection filtering
* avoid hard coded ec shard counts
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
6 months ago
9d013ea9b8
Admin UI: include ec shard sizes into volume server info ( #7071 )
* show ec shards on dashboard, show max in its own column
* master collect shard size info
* master send shard size via VolumeList
* change to more efficient shard sizes slice
* include ec shard sizes into volume server info
* Eliminated Redundant gRPC Calls
* much more efficient
* Efficient Counting: bits.OnesCount32() uses CPU-optimized instructions to count set bits in O(1)
* avoid extra volume list call
* simplify
* preserve existing shard sizes
* avoid hard coded value
* Update weed/storage/erasure_coding/ec_volume_info.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/admin/dash/volume_management.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update ec_volume_info.go
* address comments
* avoid duplicated functions
* Update weed/admin/dash/volume_management.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* simplify
* refactoring
* fix compilation
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
6 months ago
0975968e71
admin: Refactor task destination planning ( #7063 )
* refactor planning into task detection
* refactoring worker tasks
* refactor
* compiles, but only balance task is registered
* compiles, but has nil exception
* avoid nil logger
* add back ec task
* setting ec log directory
* implement balance and vacuum tasks
* EC tasks will no longer fail with "file not found" errors
* Use ReceiveFile API to send locally generated shards
* distributing shard files and ecx,ecj,vif files
* generate .ecx files correctly
* do not mount all possible EC shards (0-13) on every destination
* use constants
* delete all replicas
* rename files
* pass in volume size to tasks
6 months ago
chrislu
Chris Lu