chrislu
7 days ago
14 changed files with 1942 additions and 0 deletions
-
31test/java/spark/.gitignore
-
273test/java/spark/CI_SETUP.md
-
76test/java/spark/Makefile
-
361test/java/spark/README.md
-
72test/java/spark/docker-compose.yml
-
159test/java/spark/pom.xml
-
147test/java/spark/quick-start.sh
-
44test/java/spark/run-tests.sh
-
143test/java/spark/src/main/java/seaweed/spark/SparkSeaweedFSExample.java
-
216test/java/spark/src/test/java/seaweed/spark/SparkReadWriteTest.java
-
236test/java/spark/src/test/java/seaweed/spark/SparkSQLTest.java
-
126test/java/spark/src/test/java/seaweed/spark/SparkTestBase.java
-
18test/java/spark/src/test/resources/log4j.properties
-
40test/java/spark/test-one.sh
@ -0,0 +1,31 @@ |
|||
# Maven |
|||
target/ |
|||
pom.xml.tag |
|||
pom.xml.releaseBackup |
|||
pom.xml.versionsBackup |
|||
pom.xml.next |
|||
release.properties |
|||
dependency-reduced-pom.xml |
|||
buildNumber.properties |
|||
.mvn/timing.properties |
|||
|
|||
# IDE |
|||
.idea/ |
|||
*.iml |
|||
.vscode/ |
|||
.classpath |
|||
.project |
|||
.settings/ |
|||
|
|||
# Spark |
|||
spark-warehouse/ |
|||
metastore_db/ |
|||
derby.log |
|||
|
|||
# Logs |
|||
*.log |
|||
|
|||
# OS |
|||
.DS_Store |
|||
Thumbs.db |
|||
|
|||
@ -0,0 +1,273 @@ |
|||
# GitHub Actions CI/CD Setup |
|||
|
|||
## Overview |
|||
|
|||
The Spark integration tests are now configured to run automatically via GitHub Actions. |
|||
|
|||
## Workflow File |
|||
|
|||
**Location**: `.github/workflows/spark-integration-tests.yml` |
|||
|
|||
## Triggers |
|||
|
|||
The workflow runs automatically on: |
|||
|
|||
1. **Push to master/main** - When code is pushed to main branches |
|||
2. **Pull Requests** - When PRs target master/main |
|||
3. **Manual Trigger** - Via workflow_dispatch in GitHub UI |
|||
|
|||
The workflow only runs when changes are detected in: |
|||
- `test/java/spark/**` |
|||
- `other/java/hdfs2/**` |
|||
- `other/java/hdfs3/**` |
|||
- `other/java/client/**` |
|||
- The workflow file itself |
|||
|
|||
## Jobs |
|||
|
|||
### Job 1: spark-tests (Required) |
|||
**Duration**: ~5-10 minutes |
|||
|
|||
Steps: |
|||
1. ✓ Checkout code |
|||
2. ✓ Setup JDK 11 |
|||
3. ✓ Start SeaweedFS (master, volume, filer) |
|||
4. ✓ Build project |
|||
5. ✓ Run all integration tests (10 tests) |
|||
6. ✓ Upload test results |
|||
7. ✓ Publish test report |
|||
8. ✓ Cleanup |
|||
|
|||
**Test Coverage**: |
|||
- SparkReadWriteTest: 6 tests |
|||
- SparkSQLTest: 4 tests |
|||
|
|||
### Job 2: spark-example (Optional) |
|||
**Duration**: ~5 minutes |
|||
**Runs**: Only on push/manual trigger (not on PRs) |
|||
|
|||
Steps: |
|||
1. ✓ Checkout code |
|||
2. ✓ Setup JDK 11 |
|||
3. ✓ Download Apache Spark 3.5.0 (cached) |
|||
4. ✓ Start SeaweedFS |
|||
5. ✓ Build project |
|||
6. ✓ Run example Spark application |
|||
7. ✓ Verify output |
|||
8. ✓ Cleanup |
|||
|
|||
### Job 3: summary (Status Check) |
|||
**Duration**: < 1 minute |
|||
|
|||
Provides overall test status summary. |
|||
|
|||
## Viewing Results |
|||
|
|||
### In GitHub UI |
|||
|
|||
1. Go to the **Actions** tab in your GitHub repository |
|||
2. Click on **Spark Integration Tests** workflow |
|||
3. View individual workflow runs |
|||
4. Check test reports and logs |
|||
|
|||
### Status Badge |
|||
|
|||
Add this badge to your README.md to show the workflow status: |
|||
|
|||
```markdown |
|||
[](https://github.com/seaweedfs/seaweedfs/actions/workflows/spark-integration-tests.yml) |
|||
``` |
|||
|
|||
### Test Reports |
|||
|
|||
After each run: |
|||
- Test results are uploaded as artifacts (retained for 30 days) |
|||
- Detailed JUnit reports are published |
|||
- Logs are available for each step |
|||
|
|||
## Configuration |
|||
|
|||
### Environment Variables |
|||
|
|||
Set in the workflow: |
|||
```yaml |
|||
env: |
|||
SEAWEEDFS_TEST_ENABLED: true |
|||
SEAWEEDFS_FILER_HOST: localhost |
|||
SEAWEEDFS_FILER_PORT: 8888 |
|||
SEAWEEDFS_FILER_GRPC_PORT: 18888 |
|||
``` |
|||
|
|||
### Timeout |
|||
|
|||
- spark-tests job: 30 minutes max |
|||
- spark-example job: 20 minutes max |
|||
|
|||
## Troubleshooting CI Failures |
|||
|
|||
### SeaweedFS Connection Issues |
|||
|
|||
**Symptom**: Tests fail with connection refused |
|||
|
|||
**Check**: |
|||
1. View SeaweedFS logs in the workflow output |
|||
2. Look for "Display SeaweedFS logs on failure" step |
|||
3. Verify health check succeeded |
|||
|
|||
**Solution**: The workflow already includes retry logic and health checks |
|||
|
|||
### Test Failures |
|||
|
|||
**Symptom**: Tests pass locally but fail in CI |
|||
|
|||
**Check**: |
|||
1. Download test artifacts from the workflow run |
|||
2. Review detailed surefire reports |
|||
3. Check for timing issues or resource constraints |
|||
|
|||
**Common Issues**: |
|||
- Docker startup timing (already handled with 30 retries) |
|||
- Network issues (retry logic included) |
|||
- Resource limits (CI has sufficient memory) |
|||
|
|||
### Build Failures |
|||
|
|||
**Symptom**: Maven build fails |
|||
|
|||
**Check**: |
|||
1. Verify dependencies are available |
|||
2. Check Maven cache |
|||
3. Review build logs |
|||
|
|||
### Example Application Failures |
|||
|
|||
**Note**: This job is optional and only runs on push/manual trigger |
|||
|
|||
**Check**: |
|||
1. Verify Spark was downloaded and cached correctly |
|||
2. Check spark-submit logs |
|||
3. Verify SeaweedFS output directory |
|||
|
|||
## Manual Workflow Trigger |
|||
|
|||
To manually run the workflow: |
|||
|
|||
1. Go to **Actions** tab |
|||
2. Select **Spark Integration Tests** |
|||
3. Click **Run workflow** button |
|||
4. Select branch |
|||
5. Click **Run workflow** |
|||
|
|||
This is useful for: |
|||
- Testing changes before pushing |
|||
- Re-running failed tests |
|||
- Testing with different configurations |
|||
|
|||
## Local Testing Matching CI |
|||
|
|||
To run tests locally that match the CI environment: |
|||
|
|||
```bash |
|||
# Use the same Docker setup as CI |
|||
cd test/java/spark |
|||
docker-compose up -d seaweedfs-master seaweedfs-volume seaweedfs-filer |
|||
|
|||
# Wait for services (same as CI) |
|||
for i in {1..30}; do |
|||
curl -f http://localhost:8888/ && break |
|||
sleep 2 |
|||
done |
|||
|
|||
# Run tests (same environment variables as CI) |
|||
export SEAWEEDFS_TEST_ENABLED=true |
|||
export SEAWEEDFS_FILER_HOST=localhost |
|||
export SEAWEEDFS_FILER_PORT=8888 |
|||
export SEAWEEDFS_FILER_GRPC_PORT=18888 |
|||
mvn test -B |
|||
|
|||
# Cleanup |
|||
docker-compose down -v |
|||
``` |
|||
|
|||
## Maintenance |
|||
|
|||
### Updating Spark Version |
|||
|
|||
To update to a newer Spark version: |
|||
|
|||
1. Update `pom.xml`: Change `<spark.version>` |
|||
2. Update workflow: Change Spark download URL |
|||
3. Test locally first |
|||
4. Create PR to test in CI |
|||
|
|||
### Updating Java Version |
|||
|
|||
1. Update `pom.xml`: Change `<maven.compiler.source>` and `<target>` |
|||
2. Update workflow: Change JDK version in `setup-java` steps |
|||
3. Test locally |
|||
4. Update README with new requirements |
|||
|
|||
### Adding New Tests |
|||
|
|||
New test classes are automatically discovered and run by the workflow. |
|||
Just ensure they: |
|||
- Extend `SparkTestBase` |
|||
- Use `skipIfTestsDisabled()` |
|||
- Are in the correct package |
|||
|
|||
## CI Performance |
|||
|
|||
### Typical Run Times |
|||
|
|||
| Job | Duration | Can Fail Build? | |
|||
|-----|----------|-----------------| |
|||
| spark-tests | 5-10 min | Yes | |
|||
| spark-example | 5 min | No (optional) | |
|||
| summary | < 1 min | Only if tests fail | |
|||
|
|||
### Optimizations |
|||
|
|||
The workflow includes: |
|||
- ✓ Maven dependency caching |
|||
- ✓ Spark binary caching |
|||
- ✓ Parallel job execution |
|||
- ✓ Smart path filtering |
|||
- ✓ Docker layer caching |
|||
|
|||
### Resource Usage |
|||
|
|||
- Memory: ~4GB per job |
|||
- Disk: ~2GB (cached) |
|||
- Network: ~500MB (first run) |
|||
|
|||
## Security Considerations |
|||
|
|||
- No secrets required (tests use default ports) |
|||
- Runs in isolated Docker environment |
|||
- Clean up removes all test data |
|||
- No external services accessed |
|||
|
|||
## Future Enhancements |
|||
|
|||
Potential improvements: |
|||
- [ ] Matrix testing (multiple Spark versions) |
|||
- [ ] Performance benchmarking |
|||
- [ ] Code coverage reporting |
|||
- [ ] Integration with larger datasets |
|||
- [ ] Multi-node Spark cluster testing |
|||
|
|||
## Support |
|||
|
|||
If CI tests fail: |
|||
|
|||
1. Check workflow logs in GitHub Actions |
|||
2. Download test artifacts for detailed reports |
|||
3. Try reproducing locally using the "Local Testing" section above |
|||
4. Review recent changes in the failing paths |
|||
5. Check SeaweedFS logs in the workflow output |
|||
|
|||
For persistent issues: |
|||
- Open an issue with workflow run link |
|||
- Include test failure logs |
|||
- Note if it passes locally |
|||
|
|||
@ -0,0 +1,76 @@ |
|||
.PHONY: help build test test-local test-docker clean run-example docker-up docker-down |
|||
|
|||
help: |
|||
@echo "SeaweedFS Spark Integration Tests" |
|||
@echo "" |
|||
@echo "Available targets:" |
|||
@echo " build - Build the project" |
|||
@echo " test - Run integration tests (requires SeaweedFS running)" |
|||
@echo " test-local - Run tests against local SeaweedFS" |
|||
@echo " test-docker - Run tests in Docker with SeaweedFS" |
|||
@echo " run-example - Run the example Spark application" |
|||
@echo " docker-up - Start SeaweedFS in Docker" |
|||
@echo " docker-down - Stop SeaweedFS Docker containers" |
|||
@echo " clean - Clean build artifacts" |
|||
|
|||
build: |
|||
mvn clean package |
|||
|
|||
test: |
|||
@if [ -z "$$SEAWEEDFS_TEST_ENABLED" ]; then \
|
|||
echo "Setting SEAWEEDFS_TEST_ENABLED=true"; \
|
|||
export SEAWEEDFS_TEST_ENABLED=true; \
|
|||
fi |
|||
mvn test |
|||
|
|||
test-local: |
|||
@echo "Testing against local SeaweedFS (localhost:8888)..." |
|||
./run-tests.sh |
|||
|
|||
test-docker: |
|||
@echo "Running tests in Docker..." |
|||
docker-compose up --build --abort-on-container-exit spark-tests |
|||
docker-compose down |
|||
|
|||
docker-up: |
|||
@echo "Starting SeaweedFS in Docker..." |
|||
docker-compose up -d seaweedfs-master seaweedfs-volume seaweedfs-filer |
|||
@echo "Waiting for services to be ready..." |
|||
@sleep 5 |
|||
@echo "SeaweedFS is ready!" |
|||
@echo " Master: http://localhost:9333" |
|||
@echo " Filer: http://localhost:8888" |
|||
|
|||
docker-down: |
|||
@echo "Stopping SeaweedFS Docker containers..." |
|||
docker-compose down -v |
|||
|
|||
run-example: |
|||
@echo "Running example application..." |
|||
@if ! command -v spark-submit > /dev/null; then \
|
|||
echo "Error: spark-submit not found. Please install Apache Spark."; \
|
|||
exit 1; \
|
|||
fi |
|||
spark-submit \
|
|||
--class seaweed.spark.SparkSeaweedFSExample \
|
|||
--master local[2] \
|
|||
--conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
|
|||
--conf spark.hadoop.fs.seaweed.filer.host=localhost \
|
|||
--conf spark.hadoop.fs.seaweed.filer.port=8888 \
|
|||
--conf spark.hadoop.fs.seaweed.filer.port.grpc=18888 \
|
|||
--conf spark.hadoop.fs.seaweed.replication="" \
|
|||
target/seaweedfs-spark-integration-tests-1.0-SNAPSHOT.jar \
|
|||
seaweedfs://localhost:8888/spark-example-output |
|||
|
|||
clean: |
|||
mvn clean |
|||
@echo "Build artifacts cleaned" |
|||
|
|||
verify-seaweedfs: |
|||
@echo "Verifying SeaweedFS connection..." |
|||
@curl -f http://localhost:8888/ > /dev/null 2>&1 && \
|
|||
echo "✓ SeaweedFS filer is accessible" || \
|
|||
(echo "✗ SeaweedFS filer is not accessible at http://localhost:8888"; exit 1) |
|||
|
|||
.DEFAULT_GOAL := help |
|||
|
|||
@ -0,0 +1,361 @@ |
|||
# SeaweedFS Spark Integration Tests |
|||
|
|||
Comprehensive integration tests for Apache Spark with SeaweedFS HDFS client. |
|||
|
|||
## Overview |
|||
|
|||
This test suite validates that Apache Spark works correctly with SeaweedFS as the storage backend, covering: |
|||
|
|||
- **Data I/O**: Reading and writing data in various formats (Parquet, CSV, JSON) |
|||
- **Spark SQL**: Complex SQL queries, joins, aggregations, and window functions |
|||
- **Partitioning**: Partitioned writes and partition pruning |
|||
- **Performance**: Large dataset operations |
|||
|
|||
## Prerequisites |
|||
|
|||
### 1. Running SeaweedFS |
|||
|
|||
Start SeaweedFS with default ports: |
|||
|
|||
```bash |
|||
# Terminal 1: Start master |
|||
weed master |
|||
|
|||
# Terminal 2: Start volume server |
|||
weed volume -mserver=localhost:9333 |
|||
|
|||
# Terminal 3: Start filer |
|||
weed filer -master=localhost:9333 |
|||
``` |
|||
|
|||
Verify services are running: |
|||
- Master: http://localhost:9333 |
|||
- Filer HTTP: http://localhost:8888 |
|||
- Filer gRPC: localhost:18888 |
|||
|
|||
### 2. Java and Maven |
|||
|
|||
- Java 8 or higher |
|||
- Maven 3.6 or higher |
|||
|
|||
### 3. Apache Spark (for standalone execution) |
|||
|
|||
Download and extract Apache Spark 3.5.0: |
|||
|
|||
```bash |
|||
wget https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz |
|||
tar xzf spark-3.5.0-bin-hadoop3.tgz |
|||
export SPARK_HOME=$(pwd)/spark-3.5.0-bin-hadoop3 |
|||
export PATH=$SPARK_HOME/bin:$PATH |
|||
``` |
|||
|
|||
## Building |
|||
|
|||
```bash |
|||
mvn clean package |
|||
``` |
|||
|
|||
This creates: |
|||
- Test JAR: `target/seaweedfs-spark-integration-tests-1.0-SNAPSHOT.jar` |
|||
- Fat JAR (with dependencies): `target/original-seaweedfs-spark-integration-tests-1.0-SNAPSHOT.jar` |
|||
|
|||
## Running Integration Tests |
|||
|
|||
### Quick Test |
|||
|
|||
Run all integration tests (requires running SeaweedFS): |
|||
|
|||
```bash |
|||
# Enable integration tests |
|||
export SEAWEEDFS_TEST_ENABLED=true |
|||
|
|||
# Run all tests |
|||
mvn test |
|||
``` |
|||
|
|||
### Run Specific Test |
|||
|
|||
```bash |
|||
export SEAWEEDFS_TEST_ENABLED=true |
|||
|
|||
# Run only read/write tests |
|||
mvn test -Dtest=SparkReadWriteTest |
|||
|
|||
# Run only SQL tests |
|||
mvn test -Dtest=SparkSQLTest |
|||
``` |
|||
|
|||
### Custom SeaweedFS Configuration |
|||
|
|||
If your SeaweedFS is running on a different host or port: |
|||
|
|||
```bash |
|||
export SEAWEEDFS_TEST_ENABLED=true |
|||
export SEAWEEDFS_FILER_HOST=my-seaweedfs-host |
|||
export SEAWEEDFS_FILER_PORT=8888 |
|||
export SEAWEEDFS_FILER_GRPC_PORT=18888 |
|||
|
|||
mvn test |
|||
``` |
|||
|
|||
### Skip Tests |
|||
|
|||
By default, tests are skipped if `SEAWEEDFS_TEST_ENABLED` is not set: |
|||
|
|||
```bash |
|||
mvn test # Tests will be skipped with message |
|||
``` |
|||
|
|||
## Running the Example Application |
|||
|
|||
### Local Mode |
|||
|
|||
Run the example application in Spark local mode: |
|||
|
|||
```bash |
|||
spark-submit \ |
|||
--class seaweed.spark.SparkSeaweedFSExample \ |
|||
--master local[2] \ |
|||
--conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ |
|||
--conf spark.hadoop.fs.seaweed.filer.host=localhost \ |
|||
--conf spark.hadoop.fs.seaweed.filer.port=8888 \ |
|||
--conf spark.hadoop.fs.seaweed.filer.port.grpc=18888 \ |
|||
--conf spark.hadoop.fs.seaweed.replication="" \ |
|||
target/seaweedfs-spark-integration-tests-1.0-SNAPSHOT.jar \ |
|||
seaweedfs://localhost:8888/spark-example-output |
|||
``` |
|||
|
|||
### Cluster Mode |
|||
|
|||
For production Spark clusters: |
|||
|
|||
```bash |
|||
spark-submit \ |
|||
--class seaweed.spark.SparkSeaweedFSExample \ |
|||
--master spark://master-host:7077 \ |
|||
--deploy-mode cluster \ |
|||
--conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ |
|||
--conf spark.hadoop.fs.seaweed.filer.host=seaweedfs-filer \ |
|||
--conf spark.hadoop.fs.seaweed.filer.port=8888 \ |
|||
--conf spark.hadoop.fs.seaweed.filer.port.grpc=18888 \ |
|||
--conf spark.hadoop.fs.seaweed.replication=001 \ |
|||
--conf spark.executor.instances=4 \ |
|||
--conf spark.executor.memory=4g \ |
|||
--conf spark.executor.cores=2 \ |
|||
target/seaweedfs-spark-integration-tests-1.0-SNAPSHOT.jar \ |
|||
seaweedfs://seaweedfs-filer:8888/spark-output |
|||
``` |
|||
|
|||
## Configuration |
|||
|
|||
### SeaweedFS Configuration Options |
|||
|
|||
Configure Spark to use SeaweedFS through Hadoop configuration: |
|||
|
|||
| Property | Description | Default | Example | |
|||
|----------|-------------|---------|---------| |
|||
| `spark.hadoop.fs.seaweedfs.impl` | FileSystem implementation class | - | `seaweed.hdfs.SeaweedFileSystem` | |
|||
| `spark.hadoop.fs.seaweed.filer.host` | SeaweedFS filer hostname | `localhost` | `seaweedfs-filer` | |
|||
| `spark.hadoop.fs.seaweed.filer.port` | SeaweedFS filer HTTP port | `8888` | `8888` | |
|||
| `spark.hadoop.fs.seaweed.filer.port.grpc` | SeaweedFS filer gRPC port | `18888` | `18888` | |
|||
| `spark.hadoop.fs.seaweed.replication` | Replication strategy | (uses HDFS default) | `001`, `""` (filer default) | |
|||
| `spark.hadoop.fs.seaweed.buffer.size` | Buffer size for I/O | `4MB` | `8388608` | |
|||
|
|||
### Replication Configuration Priority |
|||
|
|||
1. **Non-empty value** (e.g., `001`) - uses that specific replication |
|||
2. **Empty string** (`""`) - uses SeaweedFS filer's default replication |
|||
3. **Not configured** - uses Hadoop/Spark's replication parameter |
|||
|
|||
## Test Coverage |
|||
|
|||
### SparkReadWriteTest |
|||
|
|||
- ✓ Write and read Parquet files |
|||
- ✓ Write and read CSV files with headers |
|||
- ✓ Write and read JSON files |
|||
- ✓ Partitioned data writes with partition pruning |
|||
- ✓ Append mode operations |
|||
- ✓ Large dataset handling (10,000+ rows) |
|||
|
|||
### SparkSQLTest |
|||
|
|||
- ✓ Create tables and run SELECT queries |
|||
- ✓ Aggregation queries (GROUP BY, SUM, AVG) |
|||
- ✓ JOIN operations between datasets |
|||
- ✓ Window functions (RANK, PARTITION BY) |
|||
|
|||
## Continuous Integration |
|||
|
|||
### GitHub Actions |
|||
|
|||
A GitHub Actions workflow is configured at `.github/workflows/spark-integration-tests.yml` that automatically: |
|||
- Runs on push/PR to `master`/`main` when Spark or HDFS code changes |
|||
- Starts SeaweedFS in Docker |
|||
- Runs all integration tests |
|||
- Runs the example application |
|||
- Uploads test reports |
|||
- Can be triggered manually via workflow_dispatch |
|||
|
|||
The workflow includes two jobs: |
|||
1. **spark-tests**: Runs all integration tests (10 tests) |
|||
2. **spark-example**: Runs the example Spark application |
|||
|
|||
View the workflow status in the GitHub Actions tab of the repository. |
|||
|
|||
### CI-Friendly Test Execution |
|||
|
|||
```bash |
|||
# In CI environment |
|||
./scripts/start-seaweedfs.sh # Start SeaweedFS in background |
|||
export SEAWEEDFS_TEST_ENABLED=true |
|||
mvn clean test |
|||
./scripts/stop-seaweedfs.sh # Cleanup |
|||
``` |
|||
|
|||
### Docker-Based Testing |
|||
|
|||
Use docker-compose for isolated testing: |
|||
|
|||
```bash |
|||
docker-compose up -d seaweedfs |
|||
export SEAWEEDFS_TEST_ENABLED=true |
|||
mvn test |
|||
docker-compose down |
|||
``` |
|||
|
|||
## Troubleshooting |
|||
|
|||
### Tests are Skipped |
|||
|
|||
**Symptom**: Tests show "Skipping test - SEAWEEDFS_TEST_ENABLED not set" |
|||
|
|||
**Solution**: |
|||
```bash |
|||
export SEAWEEDFS_TEST_ENABLED=true |
|||
mvn test |
|||
``` |
|||
|
|||
### Connection Refused Errors |
|||
|
|||
**Symptom**: `java.net.ConnectException: Connection refused` |
|||
|
|||
**Solution**: |
|||
1. Verify SeaweedFS is running: |
|||
```bash |
|||
curl http://localhost:8888/ |
|||
``` |
|||
|
|||
2. Check if ports are accessible: |
|||
```bash |
|||
netstat -an | grep 8888 |
|||
netstat -an | grep 18888 |
|||
``` |
|||
|
|||
### ClassNotFoundException: seaweed.hdfs.SeaweedFileSystem |
|||
|
|||
**Symptom**: Spark cannot find the SeaweedFS FileSystem implementation |
|||
|
|||
**Solution**: |
|||
1. Ensure the SeaweedFS HDFS client is in your classpath |
|||
2. For spark-submit, add the JAR: |
|||
```bash |
|||
spark-submit --jars /path/to/seaweedfs-hadoop3-client-*.jar ... |
|||
``` |
|||
|
|||
### Out of Memory Errors |
|||
|
|||
**Symptom**: `java.lang.OutOfMemoryError: Java heap space` |
|||
|
|||
**Solution**: |
|||
```bash |
|||
mvn test -DargLine="-Xmx4g" |
|||
``` |
|||
|
|||
For spark-submit: |
|||
```bash |
|||
spark-submit --driver-memory 4g --executor-memory 4g ... |
|||
``` |
|||
|
|||
### gRPC Version Conflicts |
|||
|
|||
**Symptom**: `java.lang.NoSuchMethodError` related to gRPC |
|||
|
|||
**Solution**: Ensure consistent gRPC versions. The project uses Spark 3.5.0 compatible versions. |
|||
|
|||
## Performance Tips |
|||
|
|||
1. **Increase buffer size** for large files: |
|||
```bash |
|||
--conf spark.hadoop.fs.seaweed.buffer.size=8388608 |
|||
``` |
|||
|
|||
2. **Use appropriate replication** based on your cluster: |
|||
```bash |
|||
--conf spark.hadoop.fs.seaweed.replication=001 |
|||
``` |
|||
|
|||
3. **Enable partition pruning** by partitioning data on commonly filtered columns |
|||
|
|||
4. **Use columnar formats** (Parquet) for better performance |
|||
|
|||
## Additional Examples |
|||
|
|||
### PySpark with SeaweedFS |
|||
|
|||
```python |
|||
from pyspark.sql import SparkSession |
|||
|
|||
spark = SparkSession.builder \ |
|||
.appName("PySparkSeaweedFS") \ |
|||
.config("spark.hadoop.fs.seaweedfs.impl", "seaweed.hdfs.SeaweedFileSystem") \ |
|||
.config("spark.hadoop.fs.seaweed.filer.host", "localhost") \ |
|||
.config("spark.hadoop.fs.seaweed.filer.port", "8888") \ |
|||
.config("spark.hadoop.fs.seaweed.filer.port.grpc", "18888") \ |
|||
.getOrCreate() |
|||
|
|||
# Write data |
|||
df = spark.range(1000) |
|||
df.write.parquet("seaweedfs://localhost:8888/pyspark-output") |
|||
|
|||
# Read data |
|||
df_read = spark.read.parquet("seaweedfs://localhost:8888/pyspark-output") |
|||
df_read.show() |
|||
``` |
|||
|
|||
### Scala with SeaweedFS |
|||
|
|||
```scala |
|||
import org.apache.spark.sql.SparkSession |
|||
|
|||
val spark = SparkSession.builder() |
|||
.appName("ScalaSeaweedFS") |
|||
.config("spark.hadoop.fs.seaweedfs.impl", "seaweed.hdfs.SeaweedFileSystem") |
|||
.config("spark.hadoop.fs.seaweed.filer.host", "localhost") |
|||
.config("spark.hadoop.fs.seaweed.filer.port", "8888") |
|||
.config("spark.hadoop.fs.seaweed.filer.port.grpc", "18888") |
|||
.getOrCreate() |
|||
|
|||
// Write data |
|||
val df = spark.range(1000) |
|||
df.write.parquet("seaweedfs://localhost:8888/scala-output") |
|||
|
|||
// Read data |
|||
val dfRead = spark.read.parquet("seaweedfs://localhost:8888/scala-output") |
|||
dfRead.show() |
|||
``` |
|||
|
|||
## Contributing |
|||
|
|||
When adding new tests: |
|||
|
|||
1. Extend `SparkTestBase` for new test classes |
|||
2. Use `skipIfTestsDisabled()` in test methods |
|||
3. Clean up test data in tearDown |
|||
4. Add documentation to this README |
|||
5. Ensure tests work in CI environment |
|||
|
|||
## License |
|||
|
|||
Same as SeaweedFS project. |
|||
|
|||
@ -0,0 +1,72 @@ |
|||
version: '3.8' |
|||
|
|||
services: |
|||
seaweedfs-master: |
|||
image: seaweedfs:test-grpc-v1.77.0 |
|||
container_name: seaweedfs-spark-master |
|||
ports: |
|||
- "9333:9333" |
|||
- "19333:19333" |
|||
command: "master -ip=seaweedfs-master -ip.bind=0.0.0.0" |
|||
networks: |
|||
- seaweedfs-spark |
|||
|
|||
seaweedfs-volume: |
|||
image: seaweedfs:test-grpc-v1.77.0 |
|||
container_name: seaweedfs-spark-volume |
|||
ports: |
|||
- "8080:8080" |
|||
- "18080:18080" |
|||
command: "volume -mserver=seaweedfs-master:9333 -ip.bind=0.0.0.0 -port=8080" |
|||
depends_on: |
|||
- seaweedfs-master |
|||
networks: |
|||
- seaweedfs-spark |
|||
|
|||
seaweedfs-filer: |
|||
image: seaweedfs:test-grpc-v1.77.0 |
|||
container_name: seaweedfs-spark-filer |
|||
ports: |
|||
- "8888:8888" |
|||
- "18888:18888" |
|||
command: ["filer", "-master=seaweedfs-master:9333", "-ip.bind=0.0.0.0", "-port=8888", "-port.grpc=18888"] |
|||
depends_on: |
|||
- seaweedfs-master |
|||
- seaweedfs-volume |
|||
networks: |
|||
- seaweedfs-spark |
|||
environment: |
|||
- GRPC_GO_LOG_VERBOSITY_LEVEL=99 |
|||
- GRPC_GO_LOG_SEVERITY_LEVEL=info |
|||
- GLOG_v=4 |
|||
- GLOG_logtostderr=true |
|||
- GODEBUG=http2debug=2 |
|||
healthcheck: |
|||
test: ["CMD", "wget", "--spider", "-q", "http://localhost:8888/"] |
|||
interval: 5s |
|||
timeout: 3s |
|||
retries: 10 |
|||
start_period: 10s |
|||
|
|||
spark-tests: |
|||
image: maven:3.9-eclipse-temurin-17 |
|||
container_name: seaweedfs-spark-tests |
|||
network_mode: "host" |
|||
volumes: |
|||
- .:/workspace |
|||
- ~/.m2:/root/.m2 |
|||
working_dir: /workspace |
|||
environment: |
|||
- SEAWEEDFS_TEST_ENABLED=true |
|||
- SEAWEEDFS_FILER_HOST=localhost |
|||
- SEAWEEDFS_FILER_PORT=8888 |
|||
- SEAWEEDFS_FILER_GRPC_PORT=18888 |
|||
- HADOOP_HOME=/tmp |
|||
command: sh -c "sleep 30 && mvn test" |
|||
mem_limit: 4g |
|||
cpus: 2 |
|||
|
|||
networks: |
|||
seaweedfs-spark: |
|||
driver: bridge |
|||
|
|||
@ -0,0 +1,159 @@ |
|||
<?xml version="1.0" encoding="UTF-8"?> |
|||
<project xmlns="http://maven.apache.org/POM/4.0.0" |
|||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" |
|||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> |
|||
<modelVersion>4.0.0</modelVersion> |
|||
|
|||
<groupId>com.seaweedfs</groupId> |
|||
<artifactId>seaweedfs-spark-integration-tests</artifactId> |
|||
<version>1.0-SNAPSHOT</version> |
|||
<packaging>jar</packaging> |
|||
|
|||
<name>SeaweedFS Spark Integration Tests</name> |
|||
<description>Integration tests for Apache Spark with SeaweedFS HDFS client</description> |
|||
|
|||
<properties> |
|||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> |
|||
<maven.compiler.source>1.8</maven.compiler.source> |
|||
<maven.compiler.target>1.8</maven.compiler.target> |
|||
<spark.version>3.5.0</spark.version> |
|||
<hadoop.version>3.3.6</hadoop.version> |
|||
<scala.binary.version>2.12</scala.binary.version> |
|||
<junit.version>4.13.2</junit.version> |
|||
</properties> |
|||
|
|||
<dependencies> |
|||
<!-- Spark Core --> |
|||
<dependency> |
|||
<groupId>org.apache.spark</groupId> |
|||
<artifactId>spark-core_${scala.binary.version}</artifactId> |
|||
<version>${spark.version}</version> |
|||
<scope>provided</scope> |
|||
</dependency> |
|||
|
|||
<!-- Spark SQL --> |
|||
<dependency> |
|||
<groupId>org.apache.spark</groupId> |
|||
<artifactId>spark-sql_${scala.binary.version}</artifactId> |
|||
<version>${spark.version}</version> |
|||
<scope>provided</scope> |
|||
</dependency> |
|||
|
|||
<!-- Hadoop Client --> |
|||
<dependency> |
|||
<groupId>org.apache.hadoop</groupId> |
|||
<artifactId>hadoop-client</artifactId> |
|||
<version>${hadoop.version}</version> |
|||
<scope>provided</scope> |
|||
</dependency> |
|||
|
|||
<!-- SeaweedFS Hadoop3 Client --> |
|||
<dependency> |
|||
<groupId>com.seaweedfs</groupId> |
|||
<artifactId>seaweedfs-hadoop3-client</artifactId> |
|||
<version>3.80</version> |
|||
</dependency> |
|||
|
|||
<!-- Testing --> |
|||
<dependency> |
|||
<groupId>junit</groupId> |
|||
<artifactId>junit</artifactId> |
|||
<version>${junit.version}</version> |
|||
<scope>test</scope> |
|||
</dependency> |
|||
|
|||
<!-- Logging --> |
|||
<dependency> |
|||
<groupId>org.slf4j</groupId> |
|||
<artifactId>slf4j-api</artifactId> |
|||
<version>1.7.36</version> |
|||
</dependency> |
|||
|
|||
<dependency> |
|||
<groupId>org.slf4j</groupId> |
|||
<artifactId>slf4j-log4j12</artifactId> |
|||
<version>1.7.36</version> |
|||
<scope>test</scope> |
|||
</dependency> |
|||
</dependencies> |
|||
|
|||
<build> |
|||
<plugins> |
|||
<plugin> |
|||
<groupId>org.apache.maven.plugins</groupId> |
|||
<artifactId>maven-compiler-plugin</artifactId> |
|||
<version>3.11.0</version> |
|||
<configuration> |
|||
<source>1.8</source> |
|||
<target>1.8</target> |
|||
</configuration> |
|||
</plugin> |
|||
|
|||
<plugin> |
|||
<groupId>org.apache.maven.plugins</groupId> |
|||
<artifactId>maven-surefire-plugin</artifactId> |
|||
<version>3.0.0</version> |
|||
<configuration> |
|||
<skipTests>${skipTests}</skipTests> |
|||
<includes> |
|||
<include>**/*Test.java</include> |
|||
</includes> |
|||
<argLine> |
|||
-Xmx2g |
|||
-Dhadoop.home.dir=/tmp |
|||
--add-opens=java.base/java.lang=ALL-UNNAMED |
|||
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED |
|||
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED |
|||
--add-opens=java.base/java.io=ALL-UNNAMED |
|||
--add-opens=java.base/java.net=ALL-UNNAMED |
|||
--add-opens=java.base/java.nio=ALL-UNNAMED |
|||
--add-opens=java.base/java.util=ALL-UNNAMED |
|||
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED |
|||
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED |
|||
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED |
|||
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED |
|||
--add-opens=java.base/sun.security.action=ALL-UNNAMED |
|||
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED |
|||
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED |
|||
</argLine> |
|||
<environmentVariables> |
|||
<HADOOP_HOME>/tmp</HADOOP_HOME> |
|||
</environmentVariables> |
|||
</configuration> |
|||
</plugin> |
|||
|
|||
<!-- Shade plugin to create fat jar for Spark submit --> |
|||
<plugin> |
|||
<groupId>org.apache.maven.plugins</groupId> |
|||
<artifactId>maven-shade-plugin</artifactId> |
|||
<version>3.5.0</version> |
|||
<executions> |
|||
<execution> |
|||
<phase>package</phase> |
|||
<goals> |
|||
<goal>shade</goal> |
|||
</goals> |
|||
<configuration> |
|||
<filters> |
|||
<filter> |
|||
<artifact>*:*</artifact> |
|||
<excludes> |
|||
<exclude>META-INF/*.SF</exclude> |
|||
<exclude>META-INF/*.DSA</exclude> |
|||
<exclude>META-INF/*.RSA</exclude> |
|||
</excludes> |
|||
</filter> |
|||
</filters> |
|||
<transformers> |
|||
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> |
|||
<mainClass>seaweed.spark.SparkSeaweedFSExample</mainClass> |
|||
</transformer> |
|||
</transformers> |
|||
</configuration> |
|||
</execution> |
|||
</executions> |
|||
</plugin> |
|||
</plugins> |
|||
</build> |
|||
</project> |
|||
|
|||
@ -0,0 +1,147 @@ |
|||
#!/bin/bash |
|||
|
|||
set -e |
|||
|
|||
echo "=== SeaweedFS Spark Integration Tests Quick Start ===" |
|||
echo "" |
|||
|
|||
# Check if SeaweedFS is running |
|||
check_seaweedfs() { |
|||
echo "Checking if SeaweedFS is running..." |
|||
if curl -f http://localhost:8888/ > /dev/null 2>&1; then |
|||
echo "✓ SeaweedFS filer is accessible at http://localhost:8888" |
|||
return 0 |
|||
else |
|||
echo "✗ SeaweedFS filer is not accessible" |
|||
return 1 |
|||
fi |
|||
} |
|||
|
|||
# Start SeaweedFS with Docker if not running |
|||
start_seaweedfs() { |
|||
echo "" |
|||
echo "Starting SeaweedFS with Docker..." |
|||
docker-compose up -d seaweedfs-master seaweedfs-volume seaweedfs-filer |
|||
|
|||
echo "Waiting for SeaweedFS to be ready..." |
|||
for i in {1..30}; do |
|||
if curl -f http://localhost:8888/ > /dev/null 2>&1; then |
|||
echo "✓ SeaweedFS is ready!" |
|||
return 0 |
|||
fi |
|||
echo -n "." |
|||
sleep 2 |
|||
done |
|||
|
|||
echo "" |
|||
echo "✗ SeaweedFS failed to start" |
|||
return 1 |
|||
} |
|||
|
|||
# Build the project |
|||
build_project() { |
|||
echo "" |
|||
echo "Building the project..." |
|||
mvn clean package -DskipTests |
|||
echo "✓ Build completed" |
|||
} |
|||
|
|||
# Run tests |
|||
run_tests() { |
|||
echo "" |
|||
echo "Running integration tests..." |
|||
export SEAWEEDFS_TEST_ENABLED=true |
|||
mvn test |
|||
echo "✓ Tests completed" |
|||
} |
|||
|
|||
# Run example |
|||
run_example() { |
|||
echo "" |
|||
echo "Running example application..." |
|||
|
|||
if ! command -v spark-submit > /dev/null; then |
|||
echo "⚠ spark-submit not found. Skipping example application." |
|||
echo "To run the example, install Apache Spark and try: make run-example" |
|||
return 0 |
|||
fi |
|||
|
|||
spark-submit \ |
|||
--class seaweed.spark.SparkSeaweedFSExample \ |
|||
--master local[2] \ |
|||
--conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ |
|||
--conf spark.hadoop.fs.seaweed.filer.host=localhost \ |
|||
--conf spark.hadoop.fs.seaweed.filer.port=8888 \ |
|||
--conf spark.hadoop.fs.seaweed.filer.port.grpc=18888 \ |
|||
target/seaweedfs-spark-integration-tests-1.0-SNAPSHOT.jar \ |
|||
seaweedfs://localhost:8888/spark-quickstart-output |
|||
|
|||
echo "✓ Example completed" |
|||
} |
|||
|
|||
# Cleanup |
|||
cleanup() { |
|||
echo "" |
|||
echo "Cleaning up..." |
|||
docker-compose down -v |
|||
echo "✓ Cleanup completed" |
|||
} |
|||
|
|||
# Main execution |
|||
main() { |
|||
# Check if Docker is available |
|||
if ! command -v docker > /dev/null; then |
|||
echo "Error: Docker is not installed or not in PATH" |
|||
exit 1 |
|||
fi |
|||
|
|||
# Check if Maven is available |
|||
if ! command -v mvn > /dev/null; then |
|||
echo "Error: Maven is not installed or not in PATH" |
|||
exit 1 |
|||
fi |
|||
|
|||
# Check if SeaweedFS is running, if not start it |
|||
if ! check_seaweedfs; then |
|||
read -p "Do you want to start SeaweedFS with Docker? (y/n) " -n 1 -r |
|||
echo |
|||
if [[ $REPLY =~ ^[Yy]$ ]]; then |
|||
start_seaweedfs || exit 1 |
|||
else |
|||
echo "Please start SeaweedFS manually and rerun this script." |
|||
exit 1 |
|||
fi |
|||
fi |
|||
|
|||
# Build project |
|||
build_project || exit 1 |
|||
|
|||
# Run tests |
|||
run_tests || exit 1 |
|||
|
|||
# Run example if Spark is available |
|||
run_example |
|||
|
|||
echo "" |
|||
echo "=== Quick Start Completed Successfully! ===" |
|||
echo "" |
|||
echo "Next steps:" |
|||
echo " - View test results in target/surefire-reports/" |
|||
echo " - Check example output at http://localhost:8888/" |
|||
echo " - Run 'make help' for more options" |
|||
echo " - Read README.md for detailed documentation" |
|||
echo "" |
|||
|
|||
read -p "Do you want to stop SeaweedFS? (y/n) " -n 1 -r |
|||
echo |
|||
if [[ $REPLY =~ ^[Yy]$ ]]; then |
|||
cleanup |
|||
fi |
|||
} |
|||
|
|||
# Handle Ctrl+C |
|||
trap cleanup INT |
|||
|
|||
# Run main |
|||
main |
|||
|
|||
@ -0,0 +1,44 @@ |
|||
#!/bin/bash |
|||
|
|||
set -e |
|||
|
|||
echo "=== SeaweedFS Spark Integration Tests Runner ===" |
|||
echo "" |
|||
|
|||
# Check if SeaweedFS is running |
|||
check_seaweedfs() { |
|||
if curl -f http://localhost:8888/ > /dev/null 2>&1; then |
|||
echo "✓ SeaweedFS filer is accessible at http://localhost:8888" |
|||
return 0 |
|||
else |
|||
echo "✗ SeaweedFS filer is not accessible" |
|||
return 1 |
|||
fi |
|||
} |
|||
|
|||
# Main |
|||
if ! check_seaweedfs; then |
|||
echo "" |
|||
echo "Please start SeaweedFS first. You can use:" |
|||
echo " cd test/java/spark && docker-compose up -d" |
|||
echo "Or:" |
|||
echo " make docker-up" |
|||
exit 1 |
|||
fi |
|||
|
|||
echo "" |
|||
echo "Running Spark integration tests..." |
|||
echo "" |
|||
|
|||
export SEAWEEDFS_TEST_ENABLED=true |
|||
export SEAWEEDFS_FILER_HOST=localhost |
|||
export SEAWEEDFS_FILER_PORT=8888 |
|||
export SEAWEEDFS_FILER_GRPC_PORT=18888 |
|||
|
|||
# Run tests |
|||
mvn test "$@" |
|||
|
|||
echo "" |
|||
echo "✓ Test run completed" |
|||
echo "View detailed reports in: target/surefire-reports/" |
|||
|
|||
@ -0,0 +1,143 @@ |
|||
package seaweed.spark; |
|||
|
|||
import org.apache.spark.sql.Dataset; |
|||
import org.apache.spark.sql.Row; |
|||
import org.apache.spark.sql.SaveMode; |
|||
import org.apache.spark.sql.SparkSession; |
|||
|
|||
/** |
|||
* Example Spark application demonstrating SeaweedFS integration. |
|||
* |
|||
* This can be submitted to a Spark cluster using spark-submit. |
|||
* |
|||
* Example usage: |
|||
* spark-submit \ |
|||
* --class seaweed.spark.SparkSeaweedFSExample \ |
|||
* --master local[2] \ |
|||
* --conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ |
|||
* --conf spark.hadoop.fs.seaweed.filer.host=localhost \ |
|||
* --conf spark.hadoop.fs.seaweed.filer.port=8888 \ |
|||
* --conf spark.hadoop.fs.seaweed.filer.port.grpc=18888 \ |
|||
* target/seaweedfs-spark-integration-tests-1.0-SNAPSHOT.jar \ |
|||
* seaweedfs://localhost:8888/output |
|||
*/ |
|||
public class SparkSeaweedFSExample { |
|||
|
|||
public static void main(String[] args) { |
|||
if (args.length < 1) { |
|||
System.err.println("Usage: SparkSeaweedFSExample <output-path>"); |
|||
System.err.println("Example: seaweedfs://localhost:8888/spark-output"); |
|||
System.exit(1); |
|||
} |
|||
|
|||
String outputPath = args[0]; |
|||
|
|||
// Create Spark session |
|||
SparkSession spark = SparkSession.builder() |
|||
.appName("SeaweedFS Spark Example") |
|||
.getOrCreate(); |
|||
|
|||
try { |
|||
System.out.println("=== SeaweedFS Spark Integration Example ===\n"); |
|||
|
|||
// Example 1: Generate data and write to SeaweedFS |
|||
System.out.println("1. Generating sample data..."); |
|||
Dataset<Row> data = spark.range(0, 1000) |
|||
.selectExpr( |
|||
"id", |
|||
"id * 2 as doubled", |
|||
"CAST(rand() * 100 AS INT) as random_value" |
|||
); |
|||
|
|||
System.out.println(" Generated " + data.count() + " rows"); |
|||
data.show(5); |
|||
|
|||
// Write as Parquet |
|||
String parquetPath = outputPath + "/data.parquet"; |
|||
System.out.println("\n2. Writing data to SeaweedFS as Parquet..."); |
|||
System.out.println(" Path: " + parquetPath); |
|||
|
|||
data.write() |
|||
.mode(SaveMode.Overwrite) |
|||
.parquet(parquetPath); |
|||
|
|||
System.out.println(" ✓ Write completed"); |
|||
|
|||
// Read back and verify |
|||
System.out.println("\n3. Reading data back from SeaweedFS..."); |
|||
Dataset<Row> readData = spark.read().parquet(parquetPath); |
|||
System.out.println(" Read " + readData.count() + " rows"); |
|||
|
|||
// Perform aggregation |
|||
System.out.println("\n4. Performing aggregation..."); |
|||
Dataset<Row> stats = readData.selectExpr( |
|||
"COUNT(*) as count", |
|||
"AVG(random_value) as avg_random", |
|||
"MAX(doubled) as max_doubled" |
|||
); |
|||
|
|||
stats.show(); |
|||
|
|||
// Write aggregation results |
|||
String statsPath = outputPath + "/stats.parquet"; |
|||
System.out.println("5. Writing stats to: " + statsPath); |
|||
stats.write() |
|||
.mode(SaveMode.Overwrite) |
|||
.parquet(statsPath); |
|||
|
|||
// Create a partitioned dataset |
|||
System.out.println("\n6. Creating partitioned dataset..."); |
|||
Dataset<Row> partitionedData = data.selectExpr( |
|||
"*", |
|||
"CAST(id % 10 AS INT) as partition_key" |
|||
); |
|||
|
|||
String partitionedPath = outputPath + "/partitioned.parquet"; |
|||
System.out.println(" Path: " + partitionedPath); |
|||
|
|||
partitionedData.write() |
|||
.mode(SaveMode.Overwrite) |
|||
.partitionBy("partition_key") |
|||
.parquet(partitionedPath); |
|||
|
|||
System.out.println(" ✓ Partitioned write completed"); |
|||
|
|||
// Read specific partition |
|||
System.out.println("\n7. Reading specific partition (partition_key=0)..."); |
|||
Dataset<Row> partition0 = spark.read() |
|||
.parquet(partitionedPath) |
|||
.filter("partition_key = 0"); |
|||
|
|||
System.out.println(" Partition 0 contains " + partition0.count() + " rows"); |
|||
partition0.show(5); |
|||
|
|||
// SQL example |
|||
System.out.println("\n8. Using Spark SQL..."); |
|||
readData.createOrReplaceTempView("seaweedfs_data"); |
|||
|
|||
Dataset<Row> sqlResult = spark.sql( |
|||
"SELECT " + |
|||
" CAST(id / 100 AS INT) as bucket, " + |
|||
" COUNT(*) as count, " + |
|||
" AVG(random_value) as avg_random " + |
|||
"FROM seaweedfs_data " + |
|||
"GROUP BY CAST(id / 100 AS INT) " + |
|||
"ORDER BY bucket" |
|||
); |
|||
|
|||
System.out.println(" Bucketed statistics:"); |
|||
sqlResult.show(); |
|||
|
|||
System.out.println("\n=== Example completed successfully! ==="); |
|||
System.out.println("Output location: " + outputPath); |
|||
|
|||
} catch (Exception e) { |
|||
System.err.println("Error: " + e.getMessage()); |
|||
e.printStackTrace(); |
|||
System.exit(1); |
|||
} finally { |
|||
spark.stop(); |
|||
} |
|||
} |
|||
} |
|||
|
|||
@ -0,0 +1,216 @@ |
|||
package seaweed.spark; |
|||
|
|||
import org.apache.spark.sql.Dataset; |
|||
import org.apache.spark.sql.Row; |
|||
import org.apache.spark.sql.SaveMode; |
|||
import org.junit.Test; |
|||
|
|||
import java.util.Arrays; |
|||
import java.util.List; |
|||
|
|||
import static org.junit.Assert.*; |
|||
|
|||
/** |
|||
* Integration tests for Spark read/write operations with SeaweedFS. |
|||
*/ |
|||
public class SparkReadWriteTest extends SparkTestBase { |
|||
|
|||
@Test |
|||
public void testWriteAndReadParquet() { |
|||
skipIfTestsDisabled(); |
|||
|
|||
// Create test data |
|||
List<Person> people = Arrays.asList( |
|||
new Person("Alice", 30), |
|||
new Person("Bob", 25), |
|||
new Person("Charlie", 35) |
|||
); |
|||
|
|||
Dataset<Row> df = spark.createDataFrame(people, Person.class); |
|||
|
|||
// Write to SeaweedFS |
|||
String outputPath = getTestPath("people.parquet"); |
|||
df.write().mode(SaveMode.Overwrite).parquet(outputPath); |
|||
|
|||
// Read back from SeaweedFS |
|||
Dataset<Row> readDf = spark.read().parquet(outputPath); |
|||
|
|||
// Verify |
|||
assertEquals(3, readDf.count()); |
|||
assertEquals(2, readDf.columns().length); |
|||
|
|||
List<Row> results = readDf.collectAsList(); |
|||
assertTrue(results.stream().anyMatch(r -> "Alice".equals(r.getAs("name")) && (Integer)r.getAs("age") == 30)); |
|||
assertTrue(results.stream().anyMatch(r -> "Bob".equals(r.getAs("name")) && (Integer)r.getAs("age") == 25)); |
|||
assertTrue(results.stream().anyMatch(r -> "Charlie".equals(r.getAs("name")) && (Integer)r.getAs("age") == 35)); |
|||
} |
|||
|
|||
@Test |
|||
public void testWriteAndReadCSV() { |
|||
skipIfTestsDisabled(); |
|||
|
|||
// Create test data |
|||
List<Person> people = Arrays.asList( |
|||
new Person("Alice", 30), |
|||
new Person("Bob", 25) |
|||
); |
|||
|
|||
Dataset<Row> df = spark.createDataFrame(people, Person.class); |
|||
|
|||
// Write to SeaweedFS as CSV |
|||
String outputPath = getTestPath("people.csv"); |
|||
df.write().mode(SaveMode.Overwrite).option("header", "true").csv(outputPath); |
|||
|
|||
// Read back from SeaweedFS |
|||
Dataset<Row> readDf = spark.read().option("header", "true").option("inferSchema", "true").csv(outputPath); |
|||
|
|||
// Verify |
|||
assertEquals(2, readDf.count()); |
|||
assertEquals(2, readDf.columns().length); |
|||
} |
|||
|
|||
@Test |
|||
public void testWriteAndReadJSON() { |
|||
skipIfTestsDisabled(); |
|||
|
|||
// Create test data |
|||
List<Person> people = Arrays.asList( |
|||
new Person("Alice", 30), |
|||
new Person("Bob", 25), |
|||
new Person("Charlie", 35) |
|||
); |
|||
|
|||
Dataset<Row> df = spark.createDataFrame(people, Person.class); |
|||
|
|||
// Write to SeaweedFS as JSON |
|||
String outputPath = getTestPath("people.json"); |
|||
df.write().mode(SaveMode.Overwrite).json(outputPath); |
|||
|
|||
// Read back from SeaweedFS |
|||
Dataset<Row> readDf = spark.read().json(outputPath); |
|||
|
|||
// Verify |
|||
assertEquals(3, readDf.count()); |
|||
assertEquals(2, readDf.columns().length); |
|||
} |
|||
|
|||
@Test |
|||
public void testWritePartitionedData() { |
|||
skipIfTestsDisabled(); |
|||
|
|||
// Create test data with multiple years |
|||
List<PersonWithYear> people = Arrays.asList( |
|||
new PersonWithYear("Alice", 30, 2020), |
|||
new PersonWithYear("Bob", 25, 2021), |
|||
new PersonWithYear("Charlie", 35, 2020), |
|||
new PersonWithYear("David", 28, 2021) |
|||
); |
|||
|
|||
Dataset<Row> df = spark.createDataFrame(people, PersonWithYear.class); |
|||
|
|||
// Write partitioned data to SeaweedFS |
|||
String outputPath = getTestPath("people_partitioned"); |
|||
df.write().mode(SaveMode.Overwrite).partitionBy("year").parquet(outputPath); |
|||
|
|||
// Read back from SeaweedFS |
|||
Dataset<Row> readDf = spark.read().parquet(outputPath); |
|||
|
|||
// Verify |
|||
assertEquals(4, readDf.count()); |
|||
|
|||
// Verify partition filtering works |
|||
Dataset<Row> filtered2020 = readDf.filter("year = 2020"); |
|||
assertEquals(2, filtered2020.count()); |
|||
|
|||
Dataset<Row> filtered2021 = readDf.filter("year = 2021"); |
|||
assertEquals(2, filtered2021.count()); |
|||
} |
|||
|
|||
@Test |
|||
public void testAppendMode() { |
|||
skipIfTestsDisabled(); |
|||
|
|||
String outputPath = getTestPath("people_append.parquet"); |
|||
|
|||
// Write first batch |
|||
List<Person> batch1 = Arrays.asList( |
|||
new Person("Alice", 30), |
|||
new Person("Bob", 25) |
|||
); |
|||
Dataset<Row> df1 = spark.createDataFrame(batch1, Person.class); |
|||
df1.write().mode(SaveMode.Overwrite).parquet(outputPath); |
|||
|
|||
// Append second batch |
|||
List<Person> batch2 = Arrays.asList( |
|||
new Person("Charlie", 35), |
|||
new Person("David", 28) |
|||
); |
|||
Dataset<Row> df2 = spark.createDataFrame(batch2, Person.class); |
|||
df2.write().mode(SaveMode.Append).parquet(outputPath); |
|||
|
|||
// Read back and verify |
|||
Dataset<Row> readDf = spark.read().parquet(outputPath); |
|||
assertEquals(4, readDf.count()); |
|||
} |
|||
|
|||
@Test |
|||
public void testLargeDataset() { |
|||
skipIfTestsDisabled(); |
|||
|
|||
// Create a larger dataset |
|||
Dataset<Row> largeDf = spark.range(0, 10000) |
|||
.selectExpr("id as value", "id * 2 as doubled"); |
|||
|
|||
String outputPath = getTestPath("large_dataset.parquet"); |
|||
largeDf.write().mode(SaveMode.Overwrite).parquet(outputPath); |
|||
|
|||
// Read back and verify |
|||
Dataset<Row> readDf = spark.read().parquet(outputPath); |
|||
assertEquals(10000, readDf.count()); |
|||
|
|||
// Verify some data |
|||
Row firstRow = readDf.first(); |
|||
assertEquals(0L, firstRow.getLong(0)); |
|||
assertEquals(0L, firstRow.getLong(1)); |
|||
} |
|||
|
|||
// Test data classes |
|||
public static class Person implements java.io.Serializable { |
|||
private String name; |
|||
private int age; |
|||
|
|||
public Person() {} |
|||
|
|||
public Person(String name, int age) { |
|||
this.name = name; |
|||
this.age = age; |
|||
} |
|||
|
|||
public String getName() { return name; } |
|||
public void setName(String name) { this.name = name; } |
|||
public int getAge() { return age; } |
|||
public void setAge(int age) { this.age = age; } |
|||
} |
|||
|
|||
public static class PersonWithYear implements java.io.Serializable { |
|||
private String name; |
|||
private int age; |
|||
private int year; |
|||
|
|||
public PersonWithYear() {} |
|||
|
|||
public PersonWithYear(String name, int age, int year) { |
|||
this.name = name; |
|||
this.age = age; |
|||
this.year = year; |
|||
} |
|||
|
|||
public String getName() { return name; } |
|||
public void setName(String name) { this.name = name; } |
|||
public int getAge() { return age; } |
|||
public void setAge(int age) { this.age = age; } |
|||
public int getYear() { return year; } |
|||
public void setYear(int year) { this.year = year; } |
|||
} |
|||
} |
|||
|
|||
@ -0,0 +1,236 @@ |
|||
package seaweed.spark; |
|||
|
|||
import org.apache.spark.sql.Dataset; |
|||
import org.apache.spark.sql.Row; |
|||
import org.apache.spark.sql.SaveMode; |
|||
import org.junit.Test; |
|||
|
|||
import java.util.Arrays; |
|||
import java.util.List; |
|||
|
|||
import static org.junit.Assert.*; |
|||
|
|||
/** |
|||
* Integration tests for Spark SQL operations with SeaweedFS. |
|||
*/ |
|||
public class SparkSQLTest extends SparkTestBase { |
|||
|
|||
@Test |
|||
public void testCreateTableAndQuery() { |
|||
skipIfTestsDisabled(); |
|||
|
|||
// Create test data |
|||
List<Employee> employees = Arrays.asList( |
|||
new Employee(1, "Alice", "Engineering", 100000), |
|||
new Employee(2, "Bob", "Sales", 80000), |
|||
new Employee(3, "Charlie", "Engineering", 120000), |
|||
new Employee(4, "David", "Sales", 75000) |
|||
); |
|||
|
|||
Dataset<Row> df = spark.createDataFrame(employees, Employee.class); |
|||
|
|||
// Write to SeaweedFS |
|||
String tablePath = getTestPath("employees"); |
|||
df.write().mode(SaveMode.Overwrite).parquet(tablePath); |
|||
|
|||
// Create temporary view |
|||
Dataset<Row> employeesDf = spark.read().parquet(tablePath); |
|||
employeesDf.createOrReplaceTempView("employees"); |
|||
|
|||
// Run SQL queries |
|||
Dataset<Row> engineeringEmployees = spark.sql( |
|||
"SELECT name, salary FROM employees WHERE department = 'Engineering'" |
|||
); |
|||
|
|||
assertEquals(2, engineeringEmployees.count()); |
|||
|
|||
Dataset<Row> highPaidEmployees = spark.sql( |
|||
"SELECT name, salary FROM employees WHERE salary > 90000" |
|||
); |
|||
|
|||
assertEquals(2, highPaidEmployees.count()); |
|||
} |
|||
|
|||
@Test |
|||
public void testAggregationQueries() { |
|||
skipIfTestsDisabled(); |
|||
|
|||
// Create sales data |
|||
List<Sale> sales = Arrays.asList( |
|||
new Sale("2024-01", "Product A", 100), |
|||
new Sale("2024-01", "Product B", 150), |
|||
new Sale("2024-02", "Product A", 120), |
|||
new Sale("2024-02", "Product B", 180), |
|||
new Sale("2024-03", "Product A", 110) |
|||
); |
|||
|
|||
Dataset<Row> df = spark.createDataFrame(sales, Sale.class); |
|||
|
|||
// Write to SeaweedFS |
|||
String tablePath = getTestPath("sales"); |
|||
df.write().mode(SaveMode.Overwrite).parquet(tablePath); |
|||
|
|||
// Create temporary view |
|||
Dataset<Row> salesDf = spark.read().parquet(tablePath); |
|||
salesDf.createOrReplaceTempView("sales"); |
|||
|
|||
// Aggregate query |
|||
Dataset<Row> monthlySales = spark.sql( |
|||
"SELECT month, SUM(amount) as total FROM sales GROUP BY month ORDER BY month" |
|||
); |
|||
|
|||
List<Row> results = monthlySales.collectAsList(); |
|||
assertEquals(3, results.size()); |
|||
assertEquals("2024-01", results.get(0).getString(0)); |
|||
assertEquals(250, results.get(0).getLong(1)); |
|||
} |
|||
|
|||
@Test |
|||
public void testJoinOperations() { |
|||
skipIfTestsDisabled(); |
|||
|
|||
// Create employee data |
|||
List<Employee> employees = Arrays.asList( |
|||
new Employee(1, "Alice", "Engineering", 100000), |
|||
new Employee(2, "Bob", "Sales", 80000) |
|||
); |
|||
|
|||
// Create department data |
|||
List<Department> departments = Arrays.asList( |
|||
new Department("Engineering", "Building Products"), |
|||
new Department("Sales", "Selling Products") |
|||
); |
|||
|
|||
Dataset<Row> empDf = spark.createDataFrame(employees, Employee.class); |
|||
Dataset<Row> deptDf = spark.createDataFrame(departments, Department.class); |
|||
|
|||
// Write to SeaweedFS |
|||
String empPath = getTestPath("employees_join"); |
|||
String deptPath = getTestPath("departments_join"); |
|||
|
|||
empDf.write().mode(SaveMode.Overwrite).parquet(empPath); |
|||
deptDf.write().mode(SaveMode.Overwrite).parquet(deptPath); |
|||
|
|||
// Read back and create views |
|||
spark.read().parquet(empPath).createOrReplaceTempView("emp"); |
|||
spark.read().parquet(deptPath).createOrReplaceTempView("dept"); |
|||
|
|||
// Join query |
|||
Dataset<Row> joined = spark.sql( |
|||
"SELECT e.name, e.salary, d.description " + |
|||
"FROM emp e JOIN dept d ON e.department = d.name" |
|||
); |
|||
|
|||
assertEquals(2, joined.count()); |
|||
|
|||
List<Row> results = joined.collectAsList(); |
|||
assertTrue(results.stream().anyMatch(r -> |
|||
"Alice".equals(r.getString(0)) && "Building Products".equals(r.getString(2)) |
|||
)); |
|||
} |
|||
|
|||
@Test |
|||
public void testWindowFunctions() { |
|||
skipIfTestsDisabled(); |
|||
|
|||
// Create employee data with salaries |
|||
List<Employee> employees = Arrays.asList( |
|||
new Employee(1, "Alice", "Engineering", 100000), |
|||
new Employee(2, "Bob", "Engineering", 120000), |
|||
new Employee(3, "Charlie", "Sales", 80000), |
|||
new Employee(4, "David", "Sales", 90000) |
|||
); |
|||
|
|||
Dataset<Row> df = spark.createDataFrame(employees, Employee.class); |
|||
|
|||
String tablePath = getTestPath("employees_window"); |
|||
df.write().mode(SaveMode.Overwrite).parquet(tablePath); |
|||
|
|||
Dataset<Row> employeesDf = spark.read().parquet(tablePath); |
|||
employeesDf.createOrReplaceTempView("employees_ranked"); |
|||
|
|||
// Window function query - rank employees by salary within department |
|||
Dataset<Row> ranked = spark.sql( |
|||
"SELECT name, department, salary, " + |
|||
"RANK() OVER (PARTITION BY department ORDER BY salary DESC) as rank " + |
|||
"FROM employees_ranked" |
|||
); |
|||
|
|||
assertEquals(4, ranked.count()); |
|||
|
|||
// Verify Bob has rank 1 in Engineering (highest salary) |
|||
List<Row> results = ranked.collectAsList(); |
|||
Row bobRow = results.stream() |
|||
.filter(r -> "Bob".equals(r.getString(0))) |
|||
.findFirst() |
|||
.orElse(null); |
|||
|
|||
assertNotNull(bobRow); |
|||
assertEquals(1, bobRow.getInt(3)); |
|||
} |
|||
|
|||
// Test data classes |
|||
public static class Employee implements java.io.Serializable { |
|||
private int id; |
|||
private String name; |
|||
private String department; |
|||
private int salary; |
|||
|
|||
public Employee() {} |
|||
|
|||
public Employee(int id, String name, String department, int salary) { |
|||
this.id = id; |
|||
this.name = name; |
|||
this.department = department; |
|||
this.salary = salary; |
|||
} |
|||
|
|||
public int getId() { return id; } |
|||
public void setId(int id) { this.id = id; } |
|||
public String getName() { return name; } |
|||
public void setName(String name) { this.name = name; } |
|||
public String getDepartment() { return department; } |
|||
public void setDepartment(String department) { this.department = department; } |
|||
public int getSalary() { return salary; } |
|||
public void setSalary(int salary) { this.salary = salary; } |
|||
} |
|||
|
|||
public static class Sale implements java.io.Serializable { |
|||
private String month; |
|||
private String product; |
|||
private int amount; |
|||
|
|||
public Sale() {} |
|||
|
|||
public Sale(String month, String product, int amount) { |
|||
this.month = month; |
|||
this.product = product; |
|||
this.amount = amount; |
|||
} |
|||
|
|||
public String getMonth() { return month; } |
|||
public void setMonth(String month) { this.month = month; } |
|||
public String getProduct() { return product; } |
|||
public void setProduct(String product) { this.product = product; } |
|||
public int getAmount() { return amount; } |
|||
public void setAmount(int amount) { this.amount = amount; } |
|||
} |
|||
|
|||
public static class Department implements java.io.Serializable { |
|||
private String name; |
|||
private String description; |
|||
|
|||
public Department() {} |
|||
|
|||
public Department(String name, String description) { |
|||
this.name = name; |
|||
this.description = description; |
|||
} |
|||
|
|||
public String getName() { return name; } |
|||
public void setName(String name) { this.name = name; } |
|||
public String getDescription() { return description; } |
|||
public void setDescription(String description) { this.description = description; } |
|||
} |
|||
} |
|||
|
|||
@ -0,0 +1,126 @@ |
|||
package seaweed.spark; |
|||
|
|||
import org.apache.hadoop.conf.Configuration; |
|||
import org.apache.spark.SparkConf; |
|||
import org.apache.spark.sql.SparkSession; |
|||
import org.junit.After; |
|||
import org.junit.Before; |
|||
|
|||
import java.io.IOException; |
|||
|
|||
/** |
|||
* Base class for Spark integration tests with SeaweedFS. |
|||
* |
|||
* These tests require a running SeaweedFS cluster. |
|||
* Set environment variable SEAWEEDFS_TEST_ENABLED=true to enable these tests. |
|||
*/ |
|||
public abstract class SparkTestBase { |
|||
|
|||
protected SparkSession spark; |
|||
protected static final String TEST_ROOT = "/test-spark"; |
|||
protected static final boolean TESTS_ENABLED = |
|||
"true".equalsIgnoreCase(System.getenv("SEAWEEDFS_TEST_ENABLED")); |
|||
|
|||
// SeaweedFS connection settings |
|||
protected static final String SEAWEEDFS_HOST = |
|||
System.getenv().getOrDefault("SEAWEEDFS_FILER_HOST", "localhost"); |
|||
protected static final String SEAWEEDFS_PORT = |
|||
System.getenv().getOrDefault("SEAWEEDFS_FILER_PORT", "8888"); |
|||
protected static final String SEAWEEDFS_GRPC_PORT = |
|||
System.getenv().getOrDefault("SEAWEEDFS_FILER_GRPC_PORT", "18888"); |
|||
|
|||
@Before |
|||
public void setUpSpark() throws IOException { |
|||
if (!TESTS_ENABLED) { |
|||
return; |
|||
} |
|||
|
|||
SparkConf sparkConf = new SparkConf() |
|||
.setAppName("SeaweedFS Integration Test") |
|||
.setMaster("local[1]") // Single thread to avoid concurrent gRPC issues |
|||
.set("spark.driver.host", "localhost") |
|||
.set("spark.sql.warehouse.dir", getSeaweedFSPath("/spark-warehouse")) |
|||
// SeaweedFS configuration |
|||
.set("spark.hadoop.fs.defaultFS", String.format("seaweedfs://%s:%s", SEAWEEDFS_HOST, SEAWEEDFS_PORT)) |
|||
.set("spark.hadoop.fs.seaweedfs.impl", "seaweed.hdfs.SeaweedFileSystem") |
|||
.set("spark.hadoop.fs.seaweed.impl", "seaweed.hdfs.SeaweedFileSystem") |
|||
.set("spark.hadoop.fs.seaweed.filer.host", SEAWEEDFS_HOST) |
|||
.set("spark.hadoop.fs.seaweed.filer.port", SEAWEEDFS_PORT) |
|||
.set("spark.hadoop.fs.seaweed.filer.port.grpc", SEAWEEDFS_GRPC_PORT) |
|||
.set("spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl", "seaweed.hdfs.SeaweedAbstractFileSystem") |
|||
// Set replication to empty string to use filer default |
|||
.set("spark.hadoop.fs.seaweed.replication", "") |
|||
// Smaller buffer to reduce load |
|||
.set("spark.hadoop.fs.seaweed.buffer.size", "1048576") // 1MB |
|||
// Reduce parallelism |
|||
.set("spark.default.parallelism", "1") |
|||
.set("spark.sql.shuffle.partitions", "1") |
|||
// Simpler output committer |
|||
.set("spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version", "2") |
|||
.set("spark.sql.sources.commitProtocolClass", "org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol") |
|||
// Disable speculative execution to reduce load |
|||
.set("spark.speculation", "false"); |
|||
|
|||
spark = SparkSession.builder() |
|||
.config(sparkConf) |
|||
.getOrCreate(); |
|||
|
|||
// Clean up test directory |
|||
cleanupTestDirectory(); |
|||
} |
|||
|
|||
@After |
|||
public void tearDownSpark() { |
|||
if (!TESTS_ENABLED || spark == null) { |
|||
return; |
|||
} |
|||
|
|||
try { |
|||
// Try to cleanup but don't fail if it doesn't work |
|||
cleanupTestDirectory(); |
|||
} catch (Exception e) { |
|||
System.err.println("Cleanup failed: " + e.getMessage()); |
|||
} finally { |
|||
try { |
|||
spark.stop(); |
|||
} catch (Exception e) { |
|||
System.err.println("Spark stop failed: " + e.getMessage()); |
|||
} |
|||
spark = null; |
|||
} |
|||
} |
|||
|
|||
protected String getSeaweedFSPath(String path) { |
|||
return String.format("seaweedfs://%s:%s%s", SEAWEEDFS_HOST, SEAWEEDFS_PORT, path); |
|||
} |
|||
|
|||
protected String getTestPath(String subPath) { |
|||
return getSeaweedFSPath(TEST_ROOT + "/" + subPath); |
|||
} |
|||
|
|||
private void cleanupTestDirectory() { |
|||
if (spark != null) { |
|||
try { |
|||
Configuration conf = spark.sparkContext().hadoopConfiguration(); |
|||
org.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.FileSystem.get( |
|||
java.net.URI.create(getSeaweedFSPath("/")), conf); |
|||
org.apache.hadoop.fs.Path testPath = new org.apache.hadoop.fs.Path(TEST_ROOT); |
|||
if (fs.exists(testPath)) { |
|||
fs.delete(testPath, true); |
|||
} |
|||
} catch (Exception e) { |
|||
// Suppress cleanup errors - they shouldn't fail tests |
|||
// Common in distributed systems with eventual consistency |
|||
System.err.println("Warning: cleanup failed (non-critical): " + e.getMessage()); |
|||
} |
|||
} |
|||
} |
|||
|
|||
protected void skipIfTestsDisabled() { |
|||
if (!TESTS_ENABLED) { |
|||
System.out.println("Skipping test - SEAWEEDFS_TEST_ENABLED not set"); |
|||
org.junit.Assume.assumeTrue("SEAWEEDFS_TEST_ENABLED not set", false); |
|||
} |
|||
} |
|||
} |
|||
|
|||
@ -0,0 +1,18 @@ |
|||
# Set root logger level |
|||
log4j.rootLogger=WARN, console |
|||
|
|||
# Console appender |
|||
log4j.appender.console=org.apache.log4j.ConsoleAppender |
|||
log4j.appender.console.target=System.err |
|||
log4j.appender.console.layout=org.apache.log4j.PatternLayout |
|||
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n |
|||
|
|||
# Set log levels for specific packages |
|||
log4j.logger.org.apache.spark=WARN |
|||
log4j.logger.org.apache.hadoop=WARN |
|||
log4j.logger.seaweed=INFO |
|||
|
|||
# Suppress unnecessary warnings |
|||
log4j.logger.org.apache.spark.util.Utils=ERROR |
|||
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR |
|||
|
|||
@ -0,0 +1,40 @@ |
|||
#!/bin/bash |
|||
|
|||
# Run a single test method for quick iteration |
|||
|
|||
set -e |
|||
|
|||
if [ $# -eq 0 ]; then |
|||
echo "Usage: ./test-one.sh <TestClass>#<methodName>" |
|||
echo "" |
|||
echo "Examples:" |
|||
echo " ./test-one.sh SparkReadWriteTest#testWriteAndReadParquet" |
|||
echo " ./test-one.sh SparkSQLTest#testCreateTableAndQuery" |
|||
echo "" |
|||
exit 1 |
|||
fi |
|||
|
|||
# Check if SeaweedFS is running |
|||
if ! curl -f http://localhost:8888/ > /dev/null 2>&1; then |
|||
echo "✗ SeaweedFS filer is not accessible at http://localhost:8888" |
|||
echo "" |
|||
echo "Please start SeaweedFS first:" |
|||
echo " docker-compose up -d" |
|||
echo "" |
|||
exit 1 |
|||
fi |
|||
|
|||
echo "✓ SeaweedFS filer is accessible" |
|||
echo "" |
|||
echo "Running test: $1" |
|||
echo "" |
|||
|
|||
# Set environment variables |
|||
export SEAWEEDFS_TEST_ENABLED=true |
|||
export SEAWEEDFS_FILER_HOST=localhost |
|||
export SEAWEEDFS_FILER_PORT=8888 |
|||
export SEAWEEDFS_FILER_GRPC_PORT=18888 |
|||
|
|||
# Run the specific test |
|||
mvn test -Dtest="$1" |
|||
|
|||
Write
Preview
Loading…
Cancel
Save
Reference in new issue