Issue: Files written successfully but truncated when read back
Error: 'EOFException: Reached the end of stream. Still have: 78 bytes left'
Root cause: Potential race condition between write completion and read
- File metadata updated before all chunks fully flushed
- Spark immediately reads after write without ensuring sync
- Parquet reader gets incomplete file
Solutions applied:
1. Disable filesystem cache to avoid stale file handles
- spark.hadoop.fs.seaweedfs.impl.disable.cache=true
2. Enable explicit flush/sync on write (if supported by client)
- spark.hadoop.fs.seaweed.write.flush.sync=true
3. Add SPARK_SUBMIT_OPTS for cache disabling
These settings ensure:
- Files are fully flushed before close() returns
- No cached file handles with stale metadata
- Fresh reads always get current file state
Note: If issue persists, may need to add explicit delay between
write and read, or investigate seaweedfs-hadoop3-client flush behavior.
Troubleshooting 'seaweedfs-volume: Temporary failure in name resolution':
docker-compose.yml changes:
- Add MAVEN_OPTS to disable Java DNS caching (ttl=0)
Java caches DNS lookups which can cause stale results
- Add ping tests before mvn test to verify DNS resolution
Tests: ping -c 1 seaweedfs-volume && ping -c 1 seaweedfs-filer
- This will show if DNS works before tests run
workflow changes:
- List Docker networks before running tests
- Shows network configuration for debugging
- Helps verify spark-tests joins correct network
If ping succeeds but tests fail, it's a Java/Maven DNS issue.
If ping fails, it's a Docker networking configuration issue.
Note: Previous test failures may be from old code before Docker networking fix.
Better approach than mixing host and container networks.
Changes to docker-compose.yml:
- Remove 'network_mode: host' from spark-tests container
- Add spark-tests to seaweedfs-spark bridge network
- Update SEAWEEDFS_FILER_HOST from 'localhost' to 'seaweedfs-filer'
- Add depends_on to ensure services are healthy before tests
- Update volume publicUrl from 'localhost:8080' to 'seaweedfs-volume:8080'
Changes to workflow:
- Remove separate build and test steps
- Run tests via 'docker compose up spark-tests'
- Use --abort-on-container-exit and --exit-code-from for proper exit codes
- Simpler: one step instead of two
Benefits:
✓ All components use Docker DNS (seaweedfs-master, seaweedfs-volume, seaweedfs-filer)
✓ No host/container network split or DNS resolution issues
✓ Consistent with how other SeaweedFS integration tests work
✓ Tests are fully containerized and reproducible
✓ Volume server accessible via seaweedfs-volume:8080 for all clients
✓ Automatic volume creation works (master can reach volume via gRPC)
✓ Data writes work (Spark can reach volume via Docker network)
This matches the architecture of other integration tests and is cleaner.
The previous fix enabled master-to-volume communication but broke client writes.
Problem:
- Volume server uses -ip=seaweedfs-volume (Docker hostname)
- Master can reach it ✓
- Spark tests run on HOST (not in Docker container)
- Host can't resolve 'seaweedfs-volume' → UnknownHostException ✗
Solution:
- Keep -ip=seaweedfs-volume for master gRPC communication
- Change -publicUrl to 'localhost:8080' for host-based clients
- Change -max=0 to -max=100 (matches other integration tests)
Why -max=100:
- Pre-allocates volume capacity at startup
- Volumes ready immediately for writes
- Consistent with other test configurations
- More reliable than on-demand (-max=0)
This configuration allows:
- Master → Volume: seaweedfs-volume:18080 (Docker network)
- Clients → Volume: localhost:8080 (host network via port mapping)
Root cause identified:
- Volume server was using -ip=127.0.0.1
- Master couldn't reach volume server at 127.0.0.1 from its container
- When Spark requested assignment, master tried to create volume via gRPC
- Master's gRPC call to 127.0.0.1:18080 failed (reached itself, not volume server)
- Result: 'No writable volumes' error
Solution:
- Change volume server to use -ip=seaweedfs-volume (container hostname)
- Master can now reach volume server at seaweedfs-volume:18080
- Automatic volume creation works as designed
- Kept -publicUrl=127.0.0.1:8080 for external clients (host network)
Workflow changes:
- Remove forced volume creation (curl POST to /vol/grow)
- Volumes will be created automatically on first write request
- Keep diagnostic output for troubleshooting
- Simplified startup verification
This matches how other SeaweedFS tests work with Docker networking.
- Add -dir=/data flag to volume server command
- Mount Docker volume seaweedfs-volume-data to /data
- Ensures volume server has persistent storage for volume files
- Fixes issue where volume server couldn't create writable volumes
- Volume data persists across container restarts during tests
- Add -max=0 flag to volume server command
- Allows volume server to create unlimited 50MB volumes
- Fixes 'No writable volumes' error during Spark tests
- Volume server will create new volumes as needed for writes
- Consistent with other integration test configurations
- Change volume -ip from seaweedfs-volume to 127.0.0.1
- Change -publicUrl from localhost:8080 to 127.0.0.1:8080
- Volume server now registers with master using 127.0.0.1
- Filer will return 127.0.0.1:8080 URL that's resolvable from host
- Fixes UnknownHostException for seaweedfs-volume hostname
- Add -publicUrl=localhost:8080 to volume server command
- Ensures filer returns localhost URL instead of Docker service name
- Fixes UnknownHostException when tests run on host network
- Volume server is accessible via localhost from CI runner
- Add -volumeSizeLimitMB=50 to master (consistent with other integration tests)
- Add -defaultReplication=000 to master for explicit single-copy storage
- Add explicit -port and -port.grpc flags to all services
- Add -preStopSeconds=1 to volume for faster shutdown
- Add healthchecks to master and volume services
- Use service_healthy conditions for proper startup ordering
- Improve healthcheck intervals and timeouts for faster startup
- Use -ip flag instead of -ip.bind for service identity
- Ensures master runs in standalone single-node mode
- Prevents master from trying to form a cluster
- Required for proper initialization in test environment
- Add step to restore Maven artifacts from download to ~/.m2/repository
- Restructure artifact upload to use consistent directory layout
- Remove obsolete 'version' field from docker-compose.yml to eliminate warnings
- Ensures SeaweedFS Java dependencies are available during test execution
chrislu