Browse Source
The issue: We're not finding the correct file because: 1. Error mentions: test-spark/employees/part-00000-xxx.parquet 2. But we downloaded chunk from employees_window (different file!) The problem: - File is already written when error occurs - Error happens during READ, not write - Need to find when SeaweedInputStream opens this file for reading New approach: 1. Extract filename from EOF error message 2. Search for 'new path:' + filename (when file is opened for read) 3. Get chunk info from the entry details logged at that point 4. Download the ACTUAL failing chunk This should finally get us the right file with the 78-byte issue!pull/7526/head
1 changed files with 25 additions and 3 deletions
Loading…
Reference in new issue