Tree:
814e0bb233
add-admin-and-worker-to-helm-charts
add-ec-vacuum
add_fasthttp_client
add_remote_storage
adding-message-queue-integration-tests
also-delete-parent-directory-if-empty
avoid_releasing_temp_file_on_write
changing-to-zap
collect-public-metrics
copilot/fix-helm-chart-installation
copilot/fix-s3-object-tagging-issue
create-table-snapshot-api-design
data_query_pushdown
dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5
dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0
detect-and-plan-ec-tasks
do-not-retry-if-error-is-NotFound
ec-disk-type-support
enhance-erasure-coding
fasthttp
feature/tus-protocol
filer1_maintenance_branch
fix-GetObjectLockConfigurationHandler
fix-mount-http-parallelism
fix-s3-object-tagging-issue-7589
fix-versioning-listing-only
ftp
gh-pages
improve-fuse-mount
improve-fuse-mount2
logrus
master
message_send
mount2
mq-subscribe
mq2
original_weed_mount
pr-7412
random_access_file
refactor-needle-read-operations
refactor-volume-write
remote_overlay
revert-5134-patch-1
revert-5819-patch-1
revert-6434-bugfix-missing-s3-audit
s3-select
sub
tcp_read
test-reverting-lock-table
test_udp
testing
testing-sdx-generation
tikv
track-mount-e2e
upgrade-versions-to-4.00
volume_buffered_writes
worker-execute-ec-tasks
0.72
0.72.release
0.73
0.74
0.75
0.76
0.77
0.90
0.91
0.92
0.93
0.94
0.95
0.96
0.97
0.98
0.99
1.00
1.01
1.02
1.03
1.04
1.05
1.06
1.07
1.08
1.09
1.10
1.11
1.12
1.14
1.15
1.16
1.17
1.18
1.19
1.20
1.21
1.22
1.23
1.24
1.25
1.26
1.27
1.28
1.29
1.30
1.31
1.32
1.33
1.34
1.35
1.36
1.37
1.38
1.40
1.41
1.42
1.43
1.44
1.45
1.46
1.47
1.48
1.49
1.50
1.51
1.52
1.53
1.54
1.55
1.56
1.57
1.58
1.59
1.60
1.61
1.61RC
1.62
1.63
1.64
1.65
1.66
1.67
1.68
1.69
1.70
1.71
1.72
1.73
1.74
1.75
1.76
1.77
1.78
1.79
1.80
1.81
1.82
1.83
1.84
1.85
1.86
1.87
1.88
1.90
1.91
1.92
1.93
1.94
1.95
1.96
1.97
1.98
1.99
1;70
2.00
2.01
2.02
2.03
2.04
2.05
2.06
2.07
2.08
2.09
2.10
2.11
2.12
2.13
2.14
2.15
2.16
2.17
2.18
2.19
2.20
2.21
2.22
2.23
2.24
2.25
2.26
2.27
2.28
2.29
2.30
2.31
2.32
2.33
2.34
2.35
2.36
2.37
2.38
2.39
2.40
2.41
2.42
2.43
2.47
2.48
2.49
2.50
2.51
2.52
2.53
2.54
2.55
2.56
2.57
2.58
2.59
2.60
2.61
2.62
2.63
2.64
2.65
2.66
2.67
2.68
2.69
2.70
2.71
2.72
2.73
2.74
2.75
2.76
2.77
2.78
2.79
2.80
2.81
2.82
2.83
2.84
2.85
2.86
2.87
2.88
2.89
2.90
2.91
2.92
2.93
2.94
2.95
2.96
2.97
2.98
2.99
3.00
3.01
3.02
3.03
3.04
3.05
3.06
3.07
3.08
3.09
3.10
3.11
3.12
3.13
3.14
3.15
3.16
3.18
3.19
3.20
3.21
3.22
3.23
3.24
3.25
3.26
3.27
3.28
3.29
3.30
3.31
3.32
3.33
3.34
3.35
3.36
3.37
3.38
3.39
3.40
3.41
3.42
3.43
3.44
3.45
3.46
3.47
3.48
3.50
3.51
3.52
3.53
3.54
3.55
3.56
3.57
3.58
3.59
3.60
3.61
3.62
3.63
3.64
3.65
3.66
3.67
3.68
3.69
3.71
3.72
3.73
3.74
3.75
3.76
3.77
3.78
3.79
3.80
3.81
3.82
3.83
3.84
3.85
3.86
3.87
3.88
3.89
3.90
3.91
3.92
3.93
3.94
3.95
3.96
3.97
3.98
3.99
4.00
4.01
dev
helm-3.65.1
v0.69
v0.70beta
v3.33
${ noResults }
2 Commits (814e0bb233117ac8e4a101b68418c2109000d994)
| Author | SHA1 | Message | Date |
|---|---|---|---|
chrislu
|
29edb780d9 |
Phase 3: Advanced ML pattern detection and training optimization
- Add DatasetPatternDetector with ML-specific dataset access pattern analysis * Sequential, shuffle, batch, multi-epoch, distributed, and validation patterns * Epoch boundary detection and dataset traversal analysis * Adaptive prefetch recommendations based on detected patterns * Comprehensive throughput and performance metrics - Implement TrainingOptimizer for ML workload lifecycle management * Training phase detection (initialization, training, validation, checkpointing) * Model file access optimization with checkpoint frequency tracking * Training workload registration and multi-workload support * Adaptive optimization levels based on training phase and performance - Create BatchOptimizer for intelligent batch access pattern optimization * Linear, strided, shuffled, hierarchical, multi-GPU, and pipelined batch patterns * Batch sequence detection with predictive next-batch recommendations * Configurable prefetch strategies per batch pattern type * Performance-aware optimization with hit rate tracking - Enhance MLOptimization core integration * Unified interface integrating all Phase 1, 2, and 3 components * Coordinated shutdown and lifecycle management * Comprehensive metrics aggregation across all ML optimization layers - Add Phase 3 comprehensive test coverage * Dataset pattern detection validation * Training optimizer workload management testing * Batch optimization pattern recognition testing * End-to-end ML optimization integration testing Architecture Highlights: - Clean separation of concerns with specialized detectors for different ML patterns - Adaptive optimization that responds to detected training phases and patterns - Scalable design supporting multiple concurrent training workloads - Rich metrics and monitoring for all ML optimization components - Production-ready with proper cleanup, timeouts, and resource management Test Results: Core Phase 3 functionality verified and passing Integration: Seamlessly builds upon Phase 1 prefetching and Phase 2 caching foundations |
3 months ago |
|
chrislu
|
ba318bdac3 |
Reorganize ML optimization into dedicated package
- Move ML components to weed/mount/ml package for better organization - Create main MLOptimization interface with configuration - Separate prefetch, access pattern detection, and ML reader cache components - Add comprehensive configuration and metrics interface - Maintain backward compatibility with existing mount package - Package structure: * weed/mount/ml/prefetch.go - Prefetch manager * weed/mount/ml/access_pattern.go - Pattern detection * weed/mount/ml/ml_reader_cache.go - ML-aware reader cache * weed/mount/ml/ml.go - Main interface and configuration Test status: 17/22 tests passing, core functionality solid Package compiles cleanly with proper import structure |
3 months ago |
|
chrislu
|
e76f632907 |
Phase 1: Add smart prefetching foundation for ML workloads
- Implement PrefetchManager with configurable worker pool and deduplication - Add AccessPatternDetector for sequential, strided, and ML-specific patterns - Create MLReaderCache with ML-aware prefetching capabilities - Add comprehensive unit tests for prefetch manager - Include foundation for detecting training datasets, model loading, and epoch patterns - Support configurable prefetch parameters optimized for ML workloads Features: - Concurrent prefetch workers (8 by default) - Pattern detection for sequential, model, epoch, and strided access - ML-specific heuristics for large file and dataset access - Comprehensive metrics and monitoring - Graceful shutdown and cleanup Tests: - PrefetchManager: All tests passing (9/9) - AccessPatternDetector: Core functionality implemented - MLReaderCache: Basic functionality and integration tests |
3 months ago |