Upgrading from Parquet 1.13.1 (bundled with Spark 3.5.0) to 1.16.0.
Root cause analysis showed:
- Parquet writes 684/696 bytes total (confirmed via totalBytesWritten)
- But Parquet's footer claims file should be 762/774 bytes
- Consistent 78-byte discrepancy across all files
- This is a Parquet writer bug in file size calculation
Parquet 1.16.0 changelog includes:
- Multiple fixes for compressed file handling
- Improved footer metadata accuracy
- Better handling of column statistics
- Fixes for Snappy compression edge cases
Test approach:
1. Keep Spark 3.5.0 (stable, known good)
2. Override transitive Parquet dependencies to 1.16.0
3. If this fixes the issue, great!
4. If not, consider upgrading Spark to 4.0.1
References:
- Latest Parquet: https://downloads.apache.org/parquet/apache-parquet-1.16.0/
- Parquet format: 2.12.0 (latest)
This should resolve the 'Still have: 78 bytes left' EOFException.
ROOT CAUSE: Maven was downloading seaweedfs-client:3.80 from Maven Central
instead of using the locally built version in CI!
Changes:
- Changed all versions from 3.80 to 3.80.1-SNAPSHOT
- other/java/client/pom.xml: 3.80 → 3.80.1-SNAPSHOT
- other/java/hdfs2/pom.xml: property 3.80 → 3.80.1-SNAPSHOT
- other/java/hdfs3/pom.xml: property 3.80 → 3.80.1-SNAPSHOT
- test/java/spark/pom.xml: property 3.80 → 3.80.1-SNAPSHOT
Maven behavior:
- Release versions (3.80): Downloaded from remote repos if available
- SNAPSHOT versions: Prefer local builds, can be updated
This ensures the CI uses the locally built JARs with our debug logging!
Also added unique [DEBUG-2024] markers to verify in logs.
Maven warning:
'The artifact org.slf4j:slf4j-log4j12:jar:1.7.36 has been relocated
to org.slf4j:slf4j-reload4j:jar:1.7.36'
slf4j-log4j12 was replaced by slf4j-reload4j due to log4j vulnerabilities.
The reload4j project is a fork of log4j 1.2.17 with security fixes.
This is a drop-in replacement with the same API.
- Update nimbus-jose-jwt from 9.37.4 to 10.0.2
- Fixes CVE: GHSA-xwmg-2g98-w7v9 (DoS via deeply nested JSON)
- 9.38.0 doesn't exist in Maven Central; 10.0.2 is the patched version
- Remove Jetty dependency management (12.0.12 doesn't exist)
- Verified with mvn -U clean verify that all dependencies resolve correctly
- Build succeeds with all security patches applied
- Update from 9.37.2 to 9.37.4 to address CVE
- 9.37.2 is vulnerable, 9.37.4 is the patched version for 9.x line
- Verified with mvn dependency:tree that override is applied
- Restore explicit Jetty version management in dependencyManagement
- Pin Jetty 12.0.12 for transitive dependencies from Spark/Hadoop
- Remove misleading comment about Jetty versions availability
- Include jetty-server, jetty-http, jetty-servlet, jetty-util, jetty-io, jetty-security
- Use jetty.version property for consistency across all Jetty artifacts
- Update Netty to 4.1.125.Final (latest security patch)
- Jetty 12.0.x versions greater than 12.0.9 do not exist in Maven Central
- Attempted 12.0.10, 12.0.12, 12.0.16 - none are available
- Next available versions are in 12.1.x series
- Remove Jetty dependency management to rely on transitive resolution
- Allows build to proceed with Jetty versions from Spark/Hadoop dependencies
- Can revisit with explicit version pinning if CVE concerns arise
- Upgrade from 9.4.53.v20231009 to 12.0.16 (meets requirement >12.0.9)
- Addresses security vulnerabilities in older Jetty versions
- Externalized version to jetty.version property for easier maintenance
- Added jetty-util, jetty-io, jetty-security to dependencyManagement
- Ensures all Jetty transitive dependencies use secure version
- Upgrade from 3.9.3 to 3.9.4 (latest stable)
- Ensures all known security vulnerabilities are patched
- Fixes GHSA-g93m-8x6h-g5gv, GHSA-r978-9m6m-6gm6, GHSA-2hmj-97jw-28jh
- Upgrade from 3.9.1 to 3.9.3
- Fixes GHSA-g93m-8x6h-g5gv: Authentication bypass in Admin Server
- Fixes GHSA-r978-9m6m-6gm6: Information disclosure in persistent watchers
- Fixes GHSA-2hmj-97jw-28jh: Insufficient permission check in snapshot/restore
- Addresses high and moderate severity vulnerabilities
- Set Parquet I/O loggers to OFF (completely disabled)
- Add log4j.configuration system property to ensure config is used
- Override Spark's default log4j configuration
- Prevents thousands of record-level DEBUG messages in CI logs
- Upgrade from 4.1.118.Final to 4.1.124.Final
- Fixes GHSA-prj3-ccx8-p6x4: MadeYouReset HTTP/2 DDoS vulnerability
- 4.1.124.Final is the confirmed patched version per GitHub advisory
- All versions <= 4.1.123.Final are vulnerable
- Upgrade from 4.1.115.Final to 4.1.118.Final
- Fixes CVE-2025-24970: improper validation in SslHandler
- Fixes CVE-2024-47535: unsafe environment file reading on Windows
- Fixes CVE-2024-29025: HttpPostRequestDecoder resource exhaustion
- Addresses GHSA-prj3-ccx8-p6x4 and related vulnerabilities
- Add seaweedfs.hadoop3.client.version property set to 3.80
- Replace hardcoded version with ${seaweedfs.hadoop3.client.version}
- Enables easier version management from single location
- Follows Maven best practices for dependency versioning
- Change compiler plugin source/target from hardcoded 1.8 to use properties
- Ensures consistency with maven.compiler.source/target set to 11
- Prevents version mismatch between properties and plugin configuration
- Aligns with surefire Java 9+ module arguments
- Fix inconsistency between maven.compiler.source/target (1.8) and
surefire JVM args (Java 9+ module flags like --add-opens)
- Update to Java 11 to match CI environment (GitHub Actions uses Java 11)
- Docker environment uses Java 17 which is also compatible
- Java 11+ is required for the --add-opens/--add-exports flags used
in the surefire configuration
chrislu