seaweedfs

Commit Graph

Author	SHA1	Message	Date
chrislu	f02c4f816b	Production Integration: ML-aware FUSE mount optimizations OPTION A COMPLETE: Full production integration of ML optimization system ## Major Integration Components: ### 1. Command Line Interface - Add ML optimization flags to 'weed mount' command: * -ml.enabled: Enable/disable ML optimizations * -ml.prefetchWorkers: Configure concurrent prefetch workers (default: 8) * -ml.confidenceThreshold: Set ML confidence threshold (default: 0.6) * -ml.maxPrefetchAhead: Max chunks to prefetch ahead (default: 8) * -ml.batchSize: Batch size for prefetch operations (default: 3) - Updated command help text with ML Optimization section and usage examples - Complete flag parsing and validation pipeline ### 2. Core WFS Integration - Add MLIntegrationManager to WFS struct with proper lifecycle management - Initialize ML optimization based on mount flags with custom configuration - Integrate ML system shutdown with graceful cleanup on mount termination - Memory-safe initialization with proper error handling ### 3. FUSE Operation Hooks - File Open (wfs.Open): Apply ML-specific optimizations (FOPEN_KEEP_CACHE, direct I/O) - File Read (wfs.Read): Record access patterns for ML prefetch decision making - File Close (wfs.Release): Update ML file tracking and cleanup resources - Get Attributes (wfs.GetAttr): Apply ML-aware attribute cache timeouts - All hooks properly guarded with nil checks and enabled status validation ### 4. Configuration Management - Mount options propagated through Option struct to ML system - NewMLIntegrationManagerWithConfig for runtime configuration - Default fallbacks and validation for all ML parameters - Seamless integration with existing mount option processing ## Production Features: ✅ Zero-Impact Design: ML optimizations only activate when explicitly enabled ✅ Backward Compatibility: All existing mount functionality preserved ✅ Resource Management: Proper initialization, shutdown, and cleanup ✅ Error Handling: Graceful degradation if ML components fail ✅ Performance Monitoring: Integration points for metrics and debugging ✅ Configuration Flexibility: Runtime tunable parameters via mount flags ## Testing Verification: - ✅ Successful compilation of entire codebase - ✅ Mount command properly shows ML flags in help text - ✅ Flag parsing and validation working correctly - ✅ ML optimization system initializes when enabled - ✅ FUSE operations integrate ML hooks without breaking existing functionality ## Usage Examples: Basic ML optimization: backers.md bin build cmd CODE_OF_CONDUCT.md DESIGN.md docker examples filerldb2 go.mod go.sum k8s LICENSE Makefile ML_OPTIMIZATION_PLAN.md note other random README.md s3tests_boto3 scripts seaweedfs-rdma-sidecar snap SSE-C_IMPLEMENTATION.md telemetry test test-volume-data unmaintained util venv weed chrislu console Aug 27 13:07 chrislu ttys004 Aug 27 13:11 chrislu ttys012 Aug 28 14:00 Filesystem 512-blocks Used Available Capacity iused ifree %iused Mounted on /dev/disk3s1s1 1942700360 22000776 332038696 7% 425955 1660193480 0% / devfs 494 494 0 100% 856 0 100% /dev /dev/disk3s6 1942700360 6291632 332038696 2% 3 1660193480 0% /System/Volumes/VM /dev/disk3s2 1942700360 13899920 332038696 5% 1270 1660193480 0% /System/Volumes/Preboot /dev/disk3s4 1942700360 4440 332038696 1% 54 1660193480 0% /System/Volumes/Update /dev/disk1s2 1024000 12328 983744 2% 1 4918720 0% /System/Volumes/xarts /dev/disk1s1 1024000 11064 983744 2% 32 4918720 0% /System/Volumes/iSCPreboot /dev/disk1s3 1024000 7144 983744 1% 92 4918720 0% /System/Volumes/Hardware /dev/disk3s5 1942700360 1566013608 332038696 83% 11900819 1660193480 1% /System/Volumes/Data map auto_home 0 0 0 100% 0 0 - /System/Volumes/Data/home Filesystem 512-blocks Used Available Capacity iused ifree %iused Mounted on /dev/disk3s1s1 1942700360 22000776 332038696 7% 425955 1660193480 0% / devfs 494 494 0 100% 856 0 100% /dev /dev/disk3s6 1942700360 6291632 332038696 2% 3 1660193480 0% /System/Volumes/VM /dev/disk3s2 1942700360 13899920 332038696 5% 1270 1660193480 0% /System/Volumes/Preboot /dev/disk3s4 1942700360 4440 332038696 1% 54 1660193480 0% /System/Volumes/Update /dev/disk1s2 1024000 12328 983744 2% 1 4918720 0% /System/Volumes/xarts /dev/disk1s1 1024000 11064 983744 2% 32 4918720 0% /System/Volumes/iSCPreboot /dev/disk1s3 1024000 7144 983744 1% 92 4918720 0% /System/Volumes/Hardware /dev/disk3s5 1942700360 1566013608 332038696 83% 11900819 1660193480 1% /System/Volumes/Data map auto_home 0 0 0 100% 0 0 - /System/Volumes/Data/home /Users/chrislu/go/src/github.com/seaweedfs/seaweedfs HQ-KT6TWPKFQD /Users/chrislu/go/src/github.com/seaweedfs/seaweedfs Custom ML configuration: backers.md bin build cmd CODE_OF_CONDUCT.md DESIGN.md docker examples filerldb2 go.mod go.sum k8s LICENSE Makefile ML_OPTIMIZATION_PLAN.md note other random README.md s3tests_boto3 scripts seaweedfs-rdma-sidecar snap SSE-C_IMPLEMENTATION.md telemetry test test-volume-data unmaintained util venv weed /Users/chrislu/go/src/github.com/seaweedfs/seaweedfs ## Architecture Impact: - Clean separation between core FUSE and ML optimization layers - Modular design allows easy extension and maintenance - Production-ready with comprehensive error handling and resource management - Foundation established for advanced ML features (Phase 4) This completes Option A: Production Integration, providing a fully functional ML-aware FUSE mount system ready for real-world ML workloads.	3 months ago
Chris Lu	4af182f880	Context cancellation during reading range reading large files (#7093 ) * context cancellation during reading range reading large files * address comments * cancellation for fuse read * fix cancellation * pass in context for each function to avoid racing condition * Update reader_at_test.go * remove dead code * Update weed/filer/reader_at.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/filer/filechunk_group.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/filer/filechunk_group.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * address comments * Update weed/mount/weedfs_file_read.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/mount/weedfs_file_lseek.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/mount/weedfs_file_read.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/filer/reader_at.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/mount/weedfs_file_lseek.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * test cancellation --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	4 months ago
chrislu	31fc165715	mount switch to ordered lock requests	2 years ago
wusong	c44a6552d2	Fix buff use (#4628 )	2 years ago
zemul	0122e022ea	Mount concurrent read (#4400 ) * fix:mount deadlock * feat: concurrent read * fix * Remove useless code * fix --------- Co-authored-by: zemul <zhouzemiao@ihuman.com>	3 years ago
Chris Lu	d4566d4aaa	more solid weed mount (#4089 ) * compare chunks by timestamp * fix slab clearing error * fix test compilation * move oldest chunk to sealed, instead of by fullness * lock on fh.entryViewCache * remove verbose logs * revert slat clearing * less logs * less logs * track write and read by timestamp * remove useless logic * add entry lock on file handle release * use mem chunk only, swap file chunk has problems * comment out code that maybe used later * add debug mode to compare data read and write * more efficient readResolvedChunks with linked list * small optimization * fix test compilation * minor fix on writer * add SeparateGarbageChunks * group chunks into sections * turn off debug mode * fix tests * fix tests * tmp enable swap file chunk * Revert "tmp enable swap file chunk" This reverts commit `985137ec47`. * simple refactoring * simple refactoring * do not re-use swap file chunk. Sealed chunks should not be re-used. * comment out debugging facilities * either mem chunk or swap file chunk is fine now * remove orderedMutex as semaphore.Weighted not found impactful optimize size calculation for changing large files * optimize performance to avoid going through the long list of chunks * still problems with swap file chunk * rename * tiny optimization * swap file chunk save only successfully read data * fix * enable both mem and swap file chunk * resolve chunks with range * rename * fix chunk interval list * also change file handle chunk group when adding chunks * pick in-active chunk with time-decayed counter * fix compilation * avoid nil with empty fh.entry * refactoring * rename * rename * refactor visible intervals to list.List refactor chunkViews to list.List add IntervalList for generic interval list * change visible interval to use IntervalList in generics * cahnge chunkViews to IntervalList[ChunkView] * use NewFileChunkSection to create * rename variables * refactor * fix renaming leftover * renaming * renaming * add insert interval * interval list adds lock * incrementally add chunks to readers Fixes: 1. set start and stop offset for the value object 2. clone the value object 3. use pointer instead of copy-by-value when passing to interval.Value 4. use insert interval since adding chunk could be out of order * fix tests compilation * fix tests compilation	3 years ago
chrislu	22064c3425	mount: ensure ordered file handle lock and unlock	3 years ago
Patrick Schmidt	a73e177ecf	Add an End-to-End workflow for FUSE mount (#3562 ) * Add an e2e workflow to test FUSE mount * Fix deadlocks during concurrent r/w	3 years ago
Patrick Schmidt	f875031f06	Reuse readDataByFileHandle in Read call (#3482 )	3 years ago
Patrick Schmidt	3a75d7f7aa	Implement copy_file_range syscall in FUSE (#3475 ) See the man page of copy_file_range: https://man7.org/linux/man-pages/man2/copy_file_range.2.html	3 years ago
chrislu	26dbc6c905	move to https://github.com/seaweedfs/seaweedfs	3 years ago
chrislu	2b955c1713	support read	4 years ago
chrislu	bb9919b07a	add open release, refactor	4 years ago

13 Commits (f02c4f816bd67b3a98cb239c5c18b8445da0f9fc)