seaweedfs

Commit Graph

Author	SHA1	Message	Date
chrislu	f02c4f816b	Production Integration: ML-aware FUSE mount optimizations OPTION A COMPLETE: Full production integration of ML optimization system ## Major Integration Components: ### 1. Command Line Interface - Add ML optimization flags to 'weed mount' command: * -ml.enabled: Enable/disable ML optimizations * -ml.prefetchWorkers: Configure concurrent prefetch workers (default: 8) * -ml.confidenceThreshold: Set ML confidence threshold (default: 0.6) * -ml.maxPrefetchAhead: Max chunks to prefetch ahead (default: 8) * -ml.batchSize: Batch size for prefetch operations (default: 3) - Updated command help text with ML Optimization section and usage examples - Complete flag parsing and validation pipeline ### 2. Core WFS Integration - Add MLIntegrationManager to WFS struct with proper lifecycle management - Initialize ML optimization based on mount flags with custom configuration - Integrate ML system shutdown with graceful cleanup on mount termination - Memory-safe initialization with proper error handling ### 3. FUSE Operation Hooks - File Open (wfs.Open): Apply ML-specific optimizations (FOPEN_KEEP_CACHE, direct I/O) - File Read (wfs.Read): Record access patterns for ML prefetch decision making - File Close (wfs.Release): Update ML file tracking and cleanup resources - Get Attributes (wfs.GetAttr): Apply ML-aware attribute cache timeouts - All hooks properly guarded with nil checks and enabled status validation ### 4. Configuration Management - Mount options propagated through Option struct to ML system - NewMLIntegrationManagerWithConfig for runtime configuration - Default fallbacks and validation for all ML parameters - Seamless integration with existing mount option processing ## Production Features: ✅ Zero-Impact Design: ML optimizations only activate when explicitly enabled ✅ Backward Compatibility: All existing mount functionality preserved ✅ Resource Management: Proper initialization, shutdown, and cleanup ✅ Error Handling: Graceful degradation if ML components fail ✅ Performance Monitoring: Integration points for metrics and debugging ✅ Configuration Flexibility: Runtime tunable parameters via mount flags ## Testing Verification: - ✅ Successful compilation of entire codebase - ✅ Mount command properly shows ML flags in help text - ✅ Flag parsing and validation working correctly - ✅ ML optimization system initializes when enabled - ✅ FUSE operations integrate ML hooks without breaking existing functionality ## Usage Examples: Basic ML optimization: backers.md bin build cmd CODE_OF_CONDUCT.md DESIGN.md docker examples filerldb2 go.mod go.sum k8s LICENSE Makefile ML_OPTIMIZATION_PLAN.md note other random README.md s3tests_boto3 scripts seaweedfs-rdma-sidecar snap SSE-C_IMPLEMENTATION.md telemetry test test-volume-data unmaintained util venv weed chrislu console Aug 27 13:07 chrislu ttys004 Aug 27 13:11 chrislu ttys012 Aug 28 14:00 Filesystem 512-blocks Used Available Capacity iused ifree %iused Mounted on /dev/disk3s1s1 1942700360 22000776 332038696 7% 425955 1660193480 0% / devfs 494 494 0 100% 856 0 100% /dev /dev/disk3s6 1942700360 6291632 332038696 2% 3 1660193480 0% /System/Volumes/VM /dev/disk3s2 1942700360 13899920 332038696 5% 1270 1660193480 0% /System/Volumes/Preboot /dev/disk3s4 1942700360 4440 332038696 1% 54 1660193480 0% /System/Volumes/Update /dev/disk1s2 1024000 12328 983744 2% 1 4918720 0% /System/Volumes/xarts /dev/disk1s1 1024000 11064 983744 2% 32 4918720 0% /System/Volumes/iSCPreboot /dev/disk1s3 1024000 7144 983744 1% 92 4918720 0% /System/Volumes/Hardware /dev/disk3s5 1942700360 1566013608 332038696 83% 11900819 1660193480 1% /System/Volumes/Data map auto_home 0 0 0 100% 0 0 - /System/Volumes/Data/home Filesystem 512-blocks Used Available Capacity iused ifree %iused Mounted on /dev/disk3s1s1 1942700360 22000776 332038696 7% 425955 1660193480 0% / devfs 494 494 0 100% 856 0 100% /dev /dev/disk3s6 1942700360 6291632 332038696 2% 3 1660193480 0% /System/Volumes/VM /dev/disk3s2 1942700360 13899920 332038696 5% 1270 1660193480 0% /System/Volumes/Preboot /dev/disk3s4 1942700360 4440 332038696 1% 54 1660193480 0% /System/Volumes/Update /dev/disk1s2 1024000 12328 983744 2% 1 4918720 0% /System/Volumes/xarts /dev/disk1s1 1024000 11064 983744 2% 32 4918720 0% /System/Volumes/iSCPreboot /dev/disk1s3 1024000 7144 983744 1% 92 4918720 0% /System/Volumes/Hardware /dev/disk3s5 1942700360 1566013608 332038696 83% 11900819 1660193480 1% /System/Volumes/Data map auto_home 0 0 0 100% 0 0 - /System/Volumes/Data/home /Users/chrislu/go/src/github.com/seaweedfs/seaweedfs HQ-KT6TWPKFQD /Users/chrislu/go/src/github.com/seaweedfs/seaweedfs Custom ML configuration: backers.md bin build cmd CODE_OF_CONDUCT.md DESIGN.md docker examples filerldb2 go.mod go.sum k8s LICENSE Makefile ML_OPTIMIZATION_PLAN.md note other random README.md s3tests_boto3 scripts seaweedfs-rdma-sidecar snap SSE-C_IMPLEMENTATION.md telemetry test test-volume-data unmaintained util venv weed /Users/chrislu/go/src/github.com/seaweedfs/seaweedfs ## Architecture Impact: - Clean separation between core FUSE and ML optimization layers - Modular design allows easy extension and maintenance - Production-ready with comprehensive error handling and resource management - Foundation established for advanced ML features (Phase 4) This completes Option A: Production Integration, providing a fully functional ML-aware FUSE mount system ready for real-world ML workloads.	3 months ago
Guang Jiong Lou	3b1ac77e1f	worm grace period and retention time support (#6404 ) Signed-off-by: lou <alex1988@outlook.com>	11 months ago
chrislu	62c4980754	add debug info	1 year ago
chrislu	a9c9e1bcb3	refactor	1 year ago
wusong	a5582cc921	[mount] fix GetAttr blocks count (#5409 )	2 years ago
chrislu	28fe578944	avoid possible nil entry	3 years ago
Chris Lu	d4566d4aaa	more solid weed mount (#4089 ) * compare chunks by timestamp * fix slab clearing error * fix test compilation * move oldest chunk to sealed, instead of by fullness * lock on fh.entryViewCache * remove verbose logs * revert slat clearing * less logs * less logs * track write and read by timestamp * remove useless logic * add entry lock on file handle release * use mem chunk only, swap file chunk has problems * comment out code that maybe used later * add debug mode to compare data read and write * more efficient readResolvedChunks with linked list * small optimization * fix test compilation * minor fix on writer * add SeparateGarbageChunks * group chunks into sections * turn off debug mode * fix tests * fix tests * tmp enable swap file chunk * Revert "tmp enable swap file chunk" This reverts commit `985137ec47`. * simple refactoring * simple refactoring * do not re-use swap file chunk. Sealed chunks should not be re-used. * comment out debugging facilities * either mem chunk or swap file chunk is fine now * remove orderedMutex as semaphore.Weighted not found impactful optimize size calculation for changing large files * optimize performance to avoid going through the long list of chunks * still problems with swap file chunk * rename * tiny optimization * swap file chunk save only successfully read data * fix * enable both mem and swap file chunk * resolve chunks with range * rename * fix chunk interval list * also change file handle chunk group when adding chunks * pick in-active chunk with time-decayed counter * fix compilation * avoid nil with empty fh.entry * refactoring * rename * rename * refactor visible intervals to list.List refactor chunkViews to list.List add IntervalList for generic interval list * change visible interval to use IntervalList in generics * cahnge chunkViews to IntervalList[ChunkView] * use NewFileChunkSection to create * rename variables * refactor * fix renaming leftover * renaming * renaming * add insert interval * interval list adds lock * incrementally add chunks to readers Fixes: 1. set start and stop offset for the value object 2. clone the value object 3. use pointer instead of copy-by-value when passing to interval.Value 4. use insert interval since adding chunk could be out of order * fix tests compilation * fix tests compilation	3 years ago
Chris Lu	dac9c28d05	Revert "refactor: moved to locked entry" (#4035 ) * Revert "refactor: moved to locked entry" This reverts commit `94bc9afd9d`. * only add LockedEntry, no changes to entryLock * fix compilation	3 years ago
chrislu	94bc9afd9d	refactor: moved to locked entry	3 years ago
chrislu	70a4c98b00	refactor filer_pb.Entry and filer.Entry to use GetChunks() for later locking on reading chunks	3 years ago
chrislu	8a880a139d	mount: fix truncate operation fix https://github.com/seaweedfs/seaweedfs/issues/2609	3 years ago
chrislu	bd13a7968f	remove unused symlink resolving	3 years ago
chrislu	26dbc6c905	move to https://github.com/seaweedfs/seaweedfs	3 years ago
chrislu	f401b996eb	mount: fix symlink size reporting	3 years ago
chrislu	b98c0b4bf4	disable symlink translation for now	3 years ago
chrislu	2dc5d4adac	disable symlink translation for now	3 years ago
chrislu	a5c94acd4f	mount: check possible nil entry case	3 years ago
chrislu	d65bb2c6df	mount: file handle locks entry better related to https://github.com/chrislusf/seaweedfs/issues/2952	4 years ago
chrislu	b597baf488	mount: set file write time correct to work well with rsync fix https://github.com/chrislusf/seaweedfs/issues/2867	4 years ago
chrislu	9d44b89c21	Revert "avoid possible nil attributes" This reverts commit `0a6703c7f7`.	4 years ago
chrislu	0a6703c7f7	avoid possible nil attributes	4 years ago
chrislu	b40d252761	mount: chmod for root	4 years ago
chrislu	f3442e36e6	mount: quota adjust error type to be syscall.ENOSPC	4 years ago
chrislu	b7c992f410	add flag to enforce quota	4 years ago
chrislu	ceaf993a27	mount2: add rdev	4 years ago
chrislu	e31ec04f4f	mount2: POSIX deleted opened file nlink should be 0	4 years ago
chrislu	91f0481f4e	mount2: SetAttr set mode correctly	4 years ago
chrislu	e8ce30fdc5	mount2: adjust file mode	4 years ago
chrislu	4244ef8b72	fix building on linux	4 years ago
chrislu	fe57a2e770	file set attribute	4 years ago
chrislu	6a42cb6b0b	supports mknod, unlink	4 years ago
chrislu	e85ca10a1a	add mkdir	4 years ago
chrislu	21046c6a28	split files	4 years ago
chrislu	a4c9223b9d	support xattr	4 years ago
chrislu	c81833a192	add directory setAttr	4 years ago
chrislu	5c48c23235	remove println	4 years ago
chrislu	4c75fd5f9c	sync format	4 years ago
chrislu	b0a5193e32	working	4 years ago
chrislu	5a0a709016	it runs, but directory listing output is not showing up	4 years ago
chrislu	72faae91e1	implement read directory and read directory plus	4 years ago
chrislu	a10c28ba82	simplify	4 years ago
chrislu	f4d88862c4	can attr root directory	4 years ago

42 Commits (814e0bb233117ac8e4a101b68418c2109000d994)