You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Tree: 31cb28d9d3

add-ec-vacuum

add_fasthttp_client

add_remote_storage

adding-message-queue-integration-tests

adjust-fsck-cutoff-default

also-delete-parent-directory-if-empty

avoid_releasing_temp_file_on_write

changing-to-zap

collect-public-metrics

copilot/fix-helm-chart-installation

copilot/fix-s3-object-tagging-issue

copilot/make-renew-interval-configurable

copilot/make-renew-interval-configurable-again

copilot/sub-pr-7677

create-table-snapshot-api-design

data_query_pushdown

dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5

dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0

detect-and-plan-ec-tasks

do-not-retry-if-error-is-NotFound

ec-disk-type-support

enhance-erasure-coding

fasthttp

feature/mini-port-detection

filer1_maintenance_branch

fix-GetObjectLockConfigurationHandler

fix-mount-http-parallelism

fix-mount-read-throughput-7504

fix-s3-object-tagging-issue-7589

fix-versioning-listing-only

fix/issue-7864-s3-delete-authorization

ftp

gh-pages

improve-fuse-mount

improve-fuse-mount2

logrus

master

message_send

mount2

mq-subscribe

mq2

nfs-cookie-prefix-list-fixes

optimize-delete-lookups

original_weed_mount

pr-7412

raft-dual-write

random_access_file

refactor-needle-read-operations

refactor-volume-write

remote_overlay

remove-implicit-directory-handling

revert-5134-patch-1

revert-5819-patch-1

revert-6434-bugfix-missing-s3-audit

s3-remote-cache-singleflight

s3-select

sub

tcp_read

test-reverting-lock-table

test_udp

testing

testing-sdx-generation

tikv

track-mount-e2e

upgrade-versions-to-4.00

volume_buffered_writes

worker-execute-ec-tasks

0.72

0.72.release

0.73

0.74

0.75

0.76

0.77

0.90

0.91

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1.00

1.01

1.02

1.03

1.04

1.05

1.06

1.07

1.08

1.09

1.10

1.11

1.12

1.14

1.15

1.16

1.17

1.18

1.19

1.20

1.21

1.22

1.23

1.24

1.25

1.26

1.27

1.28

1.29

1.30

1.31

1.32

1.33

1.34

1.35

1.36

1.37

1.38

1.40

1.41

1.42

1.43

1.44

1.45

1.46

1.47

1.48

1.49

1.50

1.51

1.52

1.53

1.54

1.55

1.56

1.57

1.58

1.59

1.60

1.61

1.61RC

1.62

1.63

1.64

1.65

1.66

1.67

1.68

1.69

1.70

1.71

1.72

1.73

1.74

1.75

1.76

1.77

1.78

1.79

1.80

1.81

1.82

1.83

1.84

1.85

1.86

1.87

1.88

1.90

1.91

1.92

1.93

1.94

1.95

1.96

1.97

1.98

1.99

1;70

2.00

2.01

2.02

2.03

2.04

2.05

2.06

2.07

2.08

2.09

2.10

2.11

2.12

2.13

2.14

2.15

2.16

2.17

2.18

2.19

2.20

2.21

2.22

2.23

2.24

2.25

2.26

2.27

2.28

2.29

2.30

2.31

2.32

2.33

2.34

2.35

2.36

2.37

2.38

2.39

2.40

2.41

2.42

2.43

2.47

2.48

2.49

2.50

2.51

2.52

2.53

2.54

2.55

2.56

2.57

2.58

2.59

2.60

2.61

2.62

2.63

2.64

2.65

2.66

2.67

2.68

2.69

2.70

2.71

2.72

2.73

2.74

2.75

2.76

2.77

2.78

2.79

2.80

2.81

2.82

2.83

2.84

2.85

2.86

2.87

2.88

2.89

2.90

2.91

2.92

2.93

2.94

2.95

2.96

2.97

2.98

2.99

3.00

3.01

3.02

3.03

3.04

3.05

3.06

3.07

3.08

3.09

3.10

3.11

3.12

3.13

3.14

3.15

3.16

3.18

3.19

3.20

3.21

3.22

3.23

3.24

3.25

3.26

3.27

3.28

3.29

3.30

3.31

3.32

3.33

3.34

3.35

3.36

3.37

3.38

3.39

3.40

3.41

3.42

3.43

3.44

3.45

3.46

3.47

3.48

3.50

3.51

3.52

3.53

3.54

3.55

3.56

3.57

3.58

3.59

3.60

3.61

3.62

3.63

3.64

3.65

3.66

3.67

3.68

3.69

3.71

3.72

3.73

3.74

3.75

3.76

3.77

3.78

3.79

3.80

3.81

3.82

3.83

3.84

3.85

3.86

3.87

3.88

3.89

3.90

3.91

3.92

3.93

3.94

3.95

3.96

3.97

3.98

3.99

4.00

4.01

4.02

4.03

4.04

dev

helm-3.65.1

v0.69

v0.70beta

v3.33

Branches Tags

${ item.name }

Create tag ${ searchTerm }

Create branch ${ searchTerm }

from '31cb28d9d3'

${ noResults }

seaweedfs/test/java/spark

History

Chris Lu 31cb28d9d3 feat: auto-configure optimal volume size limit based on available disk space (#7833) * feat: auto-configure optimal volume size limit based on available disk space - Add calculateOptimalVolumeSizeMB() function with OS-independent disk detection - Reuses existing stats.NewDiskStatus() which works across Linux, macOS, Windows, BSD, Solaris - Algorithm: available disk / 100, rounded up to nearest power of 2 (64MB, 128MB, 256MB, 512MB, 1024MB) - Volume size capped to maximum of 1GB (1024MB) for better stability - Minimum volume size is 64MB - Uses efficient bits.Len() for power-of-2 rounding instead of floating-point operations - Only auto-calculates volume size if user didn't specify a custom value via -master.volumeSizeLimitMB - Respects user-specified values without override - Master logs whether value was auto-calculated or user-specified - Welcome message displays the configured volume size with correct format string ordering - Removed unused autoVolumeSizeMB variable (logging handles source tracking) Fixes: #0 * Refactor: Consolidate volume size constants and use robust flag detection for mini mode This commit addresses all code review feedback on the auto-optimal volume size feature: 1. **Consolidate hardcoded defaults into package-level constants** - Moved minVolumeSizeMB=64 and maxVolumeSizeMB=1024 from local function-scope constants to package-level constants for consistency and maintainability - All three volume size constants (min, default, max) now defined in one place 2. **Implement robust flag detection using flag.Visit()** - Added isFlagPassed() helper function using flag.Visit() to check if a CLI flag was explicitly passed on the command line - Replaces the previous implementation that checked if current value equals default (which could incorrectly assume user intent if default was specified) - Now correctly detects user override regardless of the actual value 3. **Restructure power-of-2 rounding logic for clarity** - Changed from 'only round if above min threshold' to 'always round to power-of-2 first, then apply min/max constraints' - More robust: works correctly even if min/max constants are adjusted in future - Clearer intent: all non-zero values go through consistent rounding logic 4. **Fix import ordering** - Added 'flag' import (aliased to fla9 package) to support isFlagPassed() - Added 'math/bits' import to support power-of-2 rounding Benefits: - Better code organization with all volume size limits in package constants - Correct user override detection that doesn't rely on value equality checks - More maintainable rounding logic that's easier to understand and modify - Consistent with SeaweedFS conventions (uses fla9 package like other commands) * fix: Address code review feedback for volume size calculation This commit resolves three code review comments for better code quality and robustness: 1. **Handle comma-separated directories in -dir flag** - The -dir flag accepts comma-separated list of directories, but the volume size calculation was passing the entire string to util.ResolvePath() - Now splits on comma and uses the first directory for disk space calculation - Added explanatory comment about the multi-directory support - Ensures the optimal size calculation works correctly in all scenarios 2. **Change disk detection failure from verbose log to warning** - When disk status cannot be determined, the warning is now logged via glog.Warningf() instead of glog.V(1).Infof() - Makes the event visible in default logs without requiring verbose mode - Better alerting for operators about fallback to default values 3. **Avoid recalculating availableMB/100 and define bytesPerMB constant** - Added bytesPerMB = 1024*1024 constant for clarity and reusability - Replaced hardcoded (1024 * 1024) with bytesPerMB constant - Store availableMB/100 in initialOptimalMB variable to avoid recalculation - Log message now references initialOptimalMB instead of recalculating - Improves maintainability and reduces redundant computation All three changes maintain the same logic while improving code quality and robustness as requested by the reviewer. * fix: Address rounding logic, logging clarity, and disk capacity measurement issues This commit resolves three additional code review comments to improve robustness and clarity of the volume size calculation: 1. **Fix power-of-2 rounding logic for edge cases** - The previous condition 'if optimalMB > 0' created a bug: when optimalMB=1, bits.Len(0)=0, resulting in 1<<0=1, which is below minimum (64MB) - Changed to explicitly handle zero case first: 'if optimalMB == 0' - Separate zero-handling from power-of-2 rounding ensures correct behavior: * optimalMB=0 → set to minVolumeSizeMB (64) * optimalMB>=1 → apply power-of-2 rounding - Then apply min/max constraints unconditionally - More explicit and easier to reason about correctness 2. **Use total disk capacity instead of free space for stable configuration** - Changed from diskStatus.Free (available space) to diskStatus.All (total capacity) - Free space varies based on current disk usage at startup time - This caused inconsistent volume sizes: same disk could get different sizes depending on how full it is when the service starts - Using total capacity ensures predictable, stable configuration across restarts - Better aligns with the intended behavior of sizing based on disk capacity - Added explanatory comments about why total capacity is more appropriate 3. **Improve log message clarity and accuracy** - Updated message to clearly show: * 'total disk capacity' instead of vague 'available disk' * 'capacity/100 before rounding' to match actual calculation * 'clamped to [min,max]' instead of 'capped to max' to show both bounds * Includes min and max values in log for context - More accurate and helpful for operators troubleshooting volume sizing These changes ensure the volume size calculation is both correct and predictable. * feat: Save mini configuration to file for persistence and documentation This commit adds persistent configuration storage for the 'weed mini' command, saving all non-default parameters to a JSON configuration file for: 1. **Configuration Documentation** - All parameters actually passed on the command line are saved - Provides a clear record of the running configuration - Useful for auditing and understanding how the system is configured 2. **Persistence of Auto-Calculated Values** - The auto-calculated optimal volume size (master.volumeSizeLimitMB) is saved with a note indicating it was auto-calculated - On restart, if the auto-calculated value exists, it won't be recalculated - Users can delete the auto-calculated entry to force recalculation on next startup - Provides stable, predictable configuration across restarts 3. **Configuration File Location** - Saved to: <data-folder>/.seaweedfs/mini.config.json - Uses the first directory from comma-separated -dir list - Directory is created automatically if it doesn't exist - JSON format for easy parsing and manual editing 4. **Implementation Details** - Uses flag.Visit() to collect only explicitly passed flags - Distinguishes between user-specified and auto-calculated values - Includes helpful notes in the JSON file - Graceful handling of save errors (logs warnings, doesn't fail startup) The configuration file includes all parameters such as: - IP and port settings (master, filer, volume, admin) - Data directories and metadata folders - Replication and collection settings - S3 and IAM configurations - Performance tuning parameters (concurrency limits, timeouts, etc.) - Auto-calculated volume size (if applicable) Example mini.config.json output: { "debug": "true", "dir": "/data/seaweedfs", "master.port": "9333", "filer.port": "8888", "volume.port": "9340", "master.volumeSizeLimitMB.auto": "256", "_note_auto_calculated": "This value was auto-calculated. Remove it to recalculate on next startup." } This allows operators to: - Review what configuration was active - Replicate the configuration on other systems - Understand the startup behavior - Control when auto-calculation occurs * refactor: Change configuration file format to match command-line options format Update the saved configuration format from JSON to shell-compatible options format that matches how options are expected to be passed on the command line. Configuration file: .seaweedfs/mini.options Format: Each line contains a command-line option in the format -name=value Benefits: - Format is compatible with shell scripts and can be sourced - Can be easily converted to command-line options - Human-readable and editable - Values with spaces are properly quoted - Includes helpful comments explaining auto-calculated values - Directly usable with weed mini command The file can be used in multiple ways: 1. Extract options: cat .seaweedfs/mini.options | grep -v '^#' | tr '\n' ' ' 2. Inline in command: weed mini \$(cat .seaweedfs/mini.options | grep -v '^#') 3. Manual review: cat .seaweedfs/mini.options * refactor: Save mini.options directly to -dir folder * docs: Update PR description with accurate algorithm and examples Update the function documentation comments to accurately reflect the implemented algorithm and provide real-world examples with actual calculated outputs. Changes: - Clarify that algorithm uses total disk capacity (not free space) - Document exact calculation: capacity/100, round to power of 2, clamp to [64,1024] - Add realistic examples showing input disk sizes and resulting volume sizes: * 10GB disk → 64MB (minimum) * 100GB disk → 64MB (minimum) * 1TB disk → 64MB (minimum) * 6.4TB disk → 64MB * 12.8TB disk → 128MB * 100TB disk → 1024MB (maximum) * 1PB disk → 1024MB (maximum) - Include note that values are rounded to next power of 2 and capped at 1GB This helps users understand the volume size calculation and predict what size will be set for their specific disk configurations. * feat: integrate configuration file loading into mini startup - Load mini.options file at startup if it exists - Apply loaded configuration options before normal initialization - CLI flags override file-based configuration - Exclude 'dir' option from being saved (environment-specific) - Configuration file format: option=value without leading dashes - Auto-calculated volume size persists with recalculation marker		3 days ago
..
src	HDFS: Java client replication configuration (#7526)	4 weeks ago
.gitignore	HDFS: Java client replication configuration (#7526)	4 weeks ago
Makefile	HDFS: Java client replication configuration (#7526)	4 weeks ago
docker-compose.yml	fix: add missing backslash for volume extraArgs in helm chart (#7676)	2 weeks ago
pom.xml	chore(deps): bump io.netty:netty-codec-http from 4.1.125.Final to 4.1.129.Final in /test/java/spark (#7773)	1 week ago
quick-start.sh	HDFS: Java client replication configuration (#7526)	4 weeks ago
run-tests.sh	HDFS: Java client replication configuration (#7526)	4 weeks ago