chrislu
6 days ago
7 changed files with 388 additions and 177 deletions
-
24other/java/client/src/main/java/seaweedfs/client/SeaweedRead.java
-
177test/java/spark/EOF_EXCEPTION_ANALYSIS.md
-
15test/java/spark/src/main/java/seaweed/spark/SparkSeaweedFSExample.java
-
130test/java/spark/src/test/java/seaweed/spark/SparkSQLTest.java
-
17test/java/spark/src/test/java/seaweed/spark/SparkTestBase.java
@ -0,0 +1,177 @@ |
|||||
|
# EOFException Analysis: "Still have: 78 bytes left" |
||||
|
|
||||
|
## Problem Summary |
||||
|
|
||||
|
Spark Parquet writes succeed, but subsequent reads fail with: |
||||
|
``` |
||||
|
java.io.EOFException: Reached the end of stream. Still have: 78 bytes left |
||||
|
``` |
||||
|
|
||||
|
## What the Logs Tell Us |
||||
|
|
||||
|
### Write Phase ✅ (Everything looks correct) |
||||
|
|
||||
|
**year=2020 file:** |
||||
|
``` |
||||
|
🔧 Created stream: position=0 bufferSize=1048576 |
||||
|
🔒 close START: position=0 buffer.position()=696 totalBytesWritten=696 |
||||
|
→ Submitted 696 bytes, new position=696 |
||||
|
✅ close END: finalPosition=696 totalBytesWritten=696 |
||||
|
Calculated file size: 696 (chunks: 696, attr: 696, #chunks: 1) |
||||
|
``` |
||||
|
|
||||
|
**year=2021 file:** |
||||
|
``` |
||||
|
🔧 Created stream: position=0 bufferSize=1048576 |
||||
|
🔒 close START: position=0 buffer.position()=684 totalBytesWritten=684 |
||||
|
→ Submitted 684 bytes, new position=684 |
||||
|
✅ close END: finalPosition=684 totalBytesWritten=684 |
||||
|
Calculated file size: 684 (chunks: 684, attr: 684, #chunks: 1) |
||||
|
``` |
||||
|
|
||||
|
**Key observations:** |
||||
|
- ✅ `totalBytesWritten == position == buffer == chunks == attr` |
||||
|
- ✅ All bytes received through `write()` are flushed and stored |
||||
|
- ✅ File metadata is consistent |
||||
|
- ✅ No bytes lost in SeaweedFS layer |
||||
|
|
||||
|
### Read Phase ❌ (Parquet expects more bytes) |
||||
|
|
||||
|
**Consistent pattern:** |
||||
|
- year=2020: wrote 696 bytes, **expects 774 bytes** → missing 78 |
||||
|
- year=2021: wrote 684 bytes, **expects 762 bytes** → missing 78 |
||||
|
|
||||
|
The **78-byte discrepancy is constant across both files**, suggesting it's not random data loss. |
||||
|
|
||||
|
## Hypotheses |
||||
|
|
||||
|
### H1: Parquet Footer Not Fully Written |
||||
|
Parquet file structure: |
||||
|
``` |
||||
|
[Magic "PAR1" 4B] [Data pages] [Footer] [Footer length 4B] [Magic "PAR1" 4B] |
||||
|
``` |
||||
|
|
||||
|
**Possible scenario:** |
||||
|
1. Parquet writes 684 bytes of data pages |
||||
|
2. Parquet **intends** to write 78 bytes of footer metadata |
||||
|
3. Our `SeaweedOutputStream.close()` is called |
||||
|
4. Only data pages (684 bytes) make it to the file |
||||
|
5. Footer (78 bytes) is lost or never written |
||||
|
|
||||
|
**Evidence for:** |
||||
|
- 78 bytes is a reasonable size for a Parquet footer with minimal metadata |
||||
|
- Files say "snappy.parquet" → compressed, so footer would be small |
||||
|
- Consistent 78-byte loss across files |
||||
|
|
||||
|
**Evidence against:** |
||||
|
- Our `close()` logs show all bytes received via `write()` were processed |
||||
|
- If Parquet wrote footer to stream, we'd see `totalBytesWritten=762` |
||||
|
|
||||
|
### H2: FSDataOutputStream Position Tracking Mismatch |
||||
|
Hadoop wraps our stream: |
||||
|
```java |
||||
|
new FSDataOutputStream(seaweedOutputStream, statistics) |
||||
|
``` |
||||
|
|
||||
|
**Possible scenario:** |
||||
|
1. Parquet writes 684 bytes → `FSDataOutputStream` increments position to 684 |
||||
|
2. Parquet writes 78-byte footer → `FSDataOutputStream` increments position to 762 |
||||
|
3. **BUT** only 684 bytes reach our `SeaweedOutputStream.write()` |
||||
|
4. Parquet queries `FSDataOutputStream.getPos()` → returns 762 |
||||
|
5. Parquet writes "file size: 762" in its footer |
||||
|
6. Actual file only has 684 bytes |
||||
|
|
||||
|
**Evidence for:** |
||||
|
- Would explain why our logs show 684 but Parquet expects 762 |
||||
|
- FSDataOutputStream might have its own buffering |
||||
|
|
||||
|
**Evidence against:** |
||||
|
- FSDataOutputStream is well-tested Hadoop core component |
||||
|
- Unlikely to lose bytes |
||||
|
|
||||
|
### H3: Race Condition During File Rename |
||||
|
Files are written to `_temporary/` then renamed to final location. |
||||
|
|
||||
|
**Possible scenario:** |
||||
|
1. Write completes successfully (684 bytes) |
||||
|
2. `close()` flushes and updates metadata |
||||
|
3. File is renamed while metadata is propagating |
||||
|
4. Read happens before metadata sync completes |
||||
|
5. Reader gets stale file size or incomplete footer |
||||
|
|
||||
|
**Evidence for:** |
||||
|
- Distributed systems often have eventual consistency issues |
||||
|
- Rename might not sync metadata immediately |
||||
|
|
||||
|
**Evidence against:** |
||||
|
- We added `fs.seaweed.write.flush.sync=true` to force sync |
||||
|
- Error is consistent, not intermittent |
||||
|
|
||||
|
### H4: Compression-Related Size Confusion |
||||
|
Files use Snappy compression (`*.snappy.parquet`). |
||||
|
|
||||
|
**Possible scenario:** |
||||
|
1. Parquet tracks uncompressed size internally |
||||
|
2. Writes compressed data to stream |
||||
|
3. Size mismatch between compressed file and uncompressed metadata |
||||
|
|
||||
|
**Evidence against:** |
||||
|
- Parquet handles compression internally and consistently |
||||
|
- Would affect all Parquet users, not just SeaweedFS |
||||
|
|
||||
|
## Next Debugging Steps |
||||
|
|
||||
|
### Added: getPos() Logging |
||||
|
```java |
||||
|
public synchronized long getPos() { |
||||
|
long currentPos = position + buffer.position(); |
||||
|
LOG.info("[DEBUG-2024] 📍 getPos() called: flushedPosition={} bufferPosition={} returning={}", |
||||
|
position, buffer.position(), currentPos); |
||||
|
return currentPos; |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
**Will reveal:** |
||||
|
- If/when Parquet queries position |
||||
|
- What value is returned vs what was actually written |
||||
|
- If FSDataOutputStream bypasses our position tracking |
||||
|
|
||||
|
### Next Steps if getPos() is NOT called: |
||||
|
→ Parquet is not using position tracking |
||||
|
→ Focus on footer write completion |
||||
|
|
||||
|
### Next Steps if getPos() returns 762 but we only wrote 684: |
||||
|
→ FSDataOutputStream has buffering issue or byte loss |
||||
|
→ Need to investigate Hadoop wrapper behavior |
||||
|
|
||||
|
### Next Steps if getPos() returns 684 (correct): |
||||
|
→ Issue is in footer metadata or read path |
||||
|
→ Need to examine Parquet footer contents |
||||
|
|
||||
|
## Parquet File Format Context |
||||
|
|
||||
|
Typical small Parquet file (~700 bytes): |
||||
|
``` |
||||
|
Offset Content |
||||
|
0-3 Magic "PAR1" |
||||
|
4-650 Row group data (compressed) |
||||
|
651-728 Footer metadata (schema, row group pointers) |
||||
|
729-732 Footer length (4 bytes, value: 78) |
||||
|
733-736 Magic "PAR1" |
||||
|
Total: 737 bytes |
||||
|
``` |
||||
|
|
||||
|
If footer length field says "78" but only data exists: |
||||
|
- File ends at byte 650 |
||||
|
- Footer starts at byte 651 (but doesn't exist) |
||||
|
- Reader tries to read 78 bytes, gets EOFException |
||||
|
|
||||
|
This matches our error pattern perfectly. |
||||
|
|
||||
|
## Recommended Fix Directions |
||||
|
|
||||
|
1. **Ensure footer is fully written before close returns** |
||||
|
2. **Add explicit fsync/hsync before metadata write** |
||||
|
3. **Verify FSDataOutputStream doesn't buffer separately** |
||||
|
4. **Check if Parquet needs special OutputStreamAdapter** |
||||
|
|
||||
Write
Preview
Loading…
Cancel
Save
Reference in new issue