Tree:
ba318bdac3
add-admin-and-worker-to-helm-charts
add-ec-vacuum
add_fasthttp_client
add_remote_storage
adding-message-queue-integration-tests
also-delete-parent-directory-if-empty
avoid_releasing_temp_file_on_write
changing-to-zap
collect-public-metrics
copilot/fix-helm-chart-installation
copilot/fix-s3-object-tagging-issue
create-table-snapshot-api-design
data_query_pushdown
dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5
dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0
detect-and-plan-ec-tasks
do-not-retry-if-error-is-NotFound
ec-disk-type-support
enhance-erasure-coding
fasthttp
feature/tus-protocol
filer1_maintenance_branch
fix-GetObjectLockConfigurationHandler
fix-admin-user-creation-7624
fix-mount-http-parallelism
fix-s3-object-tagging-issue-7589
fix-versioning-listing-only
ftp
gh-pages
improve-fuse-mount
improve-fuse-mount2
logrus
master
message_send
mount2
mq-subscribe
mq2
original_weed_mount
pr-7412
random_access_file
refactor-needle-read-operations
refactor-volume-write
remote_overlay
revert-5134-patch-1
revert-5819-patch-1
revert-6434-bugfix-missing-s3-audit
s3-select
sub
tcp_read
test-reverting-lock-table
test_udp
testing
testing-sdx-generation
tikv
track-mount-e2e
upgrade-versions-to-4.00
volume_buffered_writes
worker-execute-ec-tasks
0.72
0.72.release
0.73
0.74
0.75
0.76
0.77
0.90
0.91
0.92
0.93
0.94
0.95
0.96
0.97
0.98
0.99
1.00
1.01
1.02
1.03
1.04
1.05
1.06
1.07
1.08
1.09
1.10
1.11
1.12
1.14
1.15
1.16
1.17
1.18
1.19
1.20
1.21
1.22
1.23
1.24
1.25
1.26
1.27
1.28
1.29
1.30
1.31
1.32
1.33
1.34
1.35
1.36
1.37
1.38
1.40
1.41
1.42
1.43
1.44
1.45
1.46
1.47
1.48
1.49
1.50
1.51
1.52
1.53
1.54
1.55
1.56
1.57
1.58
1.59
1.60
1.61
1.61RC
1.62
1.63
1.64
1.65
1.66
1.67
1.68
1.69
1.70
1.71
1.72
1.73
1.74
1.75
1.76
1.77
1.78
1.79
1.80
1.81
1.82
1.83
1.84
1.85
1.86
1.87
1.88
1.90
1.91
1.92
1.93
1.94
1.95
1.96
1.97
1.98
1.99
1;70
2.00
2.01
2.02
2.03
2.04
2.05
2.06
2.07
2.08
2.09
2.10
2.11
2.12
2.13
2.14
2.15
2.16
2.17
2.18
2.19
2.20
2.21
2.22
2.23
2.24
2.25
2.26
2.27
2.28
2.29
2.30
2.31
2.32
2.33
2.34
2.35
2.36
2.37
2.38
2.39
2.40
2.41
2.42
2.43
2.47
2.48
2.49
2.50
2.51
2.52
2.53
2.54
2.55
2.56
2.57
2.58
2.59
2.60
2.61
2.62
2.63
2.64
2.65
2.66
2.67
2.68
2.69
2.70
2.71
2.72
2.73
2.74
2.75
2.76
2.77
2.78
2.79
2.80
2.81
2.82
2.83
2.84
2.85
2.86
2.87
2.88
2.89
2.90
2.91
2.92
2.93
2.94
2.95
2.96
2.97
2.98
2.99
3.00
3.01
3.02
3.03
3.04
3.05
3.06
3.07
3.08
3.09
3.10
3.11
3.12
3.13
3.14
3.15
3.16
3.18
3.19
3.20
3.21
3.22
3.23
3.24
3.25
3.26
3.27
3.28
3.29
3.30
3.31
3.32
3.33
3.34
3.35
3.36
3.37
3.38
3.39
3.40
3.41
3.42
3.43
3.44
3.45
3.46
3.47
3.48
3.50
3.51
3.52
3.53
3.54
3.55
3.56
3.57
3.58
3.59
3.60
3.61
3.62
3.63
3.64
3.65
3.66
3.67
3.68
3.69
3.71
3.72
3.73
3.74
3.75
3.76
3.77
3.78
3.79
3.80
3.81
3.82
3.83
3.84
3.85
3.86
3.87
3.88
3.89
3.90
3.91
3.92
3.93
3.94
3.95
3.96
3.97
3.98
3.99
4.00
4.01
dev
helm-3.65.1
v0.69
v0.70beta
v3.33
${ noResults }
1 Commits (ba318bdac37427f84cd937887710f717d0b6124b)
| Author | SHA1 | Message | Date |
|---|---|---|---|
chrislu
|
ba318bdac3 |
Reorganize ML optimization into dedicated package
- Move ML components to weed/mount/ml package for better organization - Create main MLOptimization interface with configuration - Separate prefetch, access pattern detection, and ML reader cache components - Add comprehensive configuration and metrics interface - Maintain backward compatibility with existing mount package - Package structure: * weed/mount/ml/prefetch.go - Prefetch manager * weed/mount/ml/access_pattern.go - Pattern detection * weed/mount/ml/ml_reader_cache.go - ML-aware reader cache * weed/mount/ml/ml.go - Main interface and configuration Test status: 17/22 tests passing, core functionality solid Package compiles cleanly with proper import structure |
3 months ago |
|
chrislu
|
e76f632907 |
Phase 1: Add smart prefetching foundation for ML workloads
- Implement PrefetchManager with configurable worker pool and deduplication - Add AccessPatternDetector for sequential, strided, and ML-specific patterns - Create MLReaderCache with ML-aware prefetching capabilities - Add comprehensive unit tests for prefetch manager - Include foundation for detecting training datasets, model loading, and epoch patterns - Support configurable prefetch parameters optimized for ML workloads Features: - Concurrent prefetch workers (8 by default) - Pattern detection for sequential, model, epoch, and strided access - ML-specific heuristics for large file and dataset access - Comprehensive metrics and monitoring - Graceful shutdown and cleanup Tests: - PrefetchManager: All tests passing (9/9) - AccessPatternDetector: Core functionality implemented - MLReaderCache: Basic functionality and integration tests |
3 months ago |