seaweedfs

History

Chris Lu 7f3f61ea28 fix: resolve Kafka gateway response deadlocks causing Sarama client hangs (#8762 ) * fix: resolve Kafka gateway response deadlocks causing Sarama client hangs Fix three bugs in the Kafka protocol handler that caused sequential clients (notably Sarama) to hang during E2E tests: 1. Race condition in correlation queue ordering: the correlation ID was added to the response ordering queue AFTER sending the request to the processing channel. A fast processor (e.g. ApiVersions) could finish and send its response before the ID was in the queue, causing the response writer to miss it — permanently deadlocking the connection. Now the ID is added BEFORE the channel send, with error response injection on send failure. 2. Silent error response drops: when processRequestSync returned an error, the response writer logged it but never sent anything back to the client. The client would block forever waiting for bytes that never arrived. Now sends a Kafka UNKNOWN_SERVER_ERROR response. 3. Produce V0/V1 missing timeout_ms parsing: the handler skipped the 4-byte timeout field, reading it as topicsCount instead. This caused incorrect parsing of the entire produce request for V0/V1 clients. * fix: API-versioned error responses, unsupported-version queue fix, V0V1 header alignment 1. errors.go — BuildAPIErrorResponse: emits a minimal-but-valid error body whose layout matches the schema the client expects for each API key and version (throttle_time position, array fields, etc.). The old 2-byte generic body corrupted the protocol stream for APIs whose response begins with throttle_time_ms or an array. 2. handler.go — unsupported-version path: the correlationID was never added to correlationQueue before sending to responseChan, so the response writer could never match it and the client hung. Now appends the ID under correlationQueueMu before the send. 3. produce.go — handleProduceV0V1: requestBody is already post-header (HandleConn strips client_id). The handler was erroneously parsing acks bytes as a client_id length, misaligning all subsequent field reads. Removed the client_id parsing; offset now starts at 0 with acks(2) + timeout_ms(4) + topicsCount(4), matching handleProduceV2Plus. * fix: free pooled message buffer per-iteration instead of deferring The read loop allocated messageBuf via mem.Allocate and deferred mem.Free. Since the defer only runs when HandleConn returns, pool buffers accumulated for the entire connection lifetime — one per request. Worse, the deferred frees ran in LIFO order before wg.Wait(), so processing goroutines could read from already-freed pool buffers. Now: read into a pooled buffer, immediately copy to Go-managed memory, and return the pool buffer. messageBuf is a regular slice safe for async goroutine access with no defer accumulation. * fix: cancel context before wg.Wait and on worker response-send timeout Two related issues: 1. Cleanup defer ordering deadlock: defers run LIFO — the cleanup defer (close channels, wg.Wait) ran before the cancel() defer. The response writer is in the WaitGroup and exits only on ctx.Done() or responseChan close, but both signals came after wg.Wait(). Deadlock on every normal connection close (EOF, read error, queue-full). Fix: call cancel() at the start of the cleanup defer, before wg.Wait(). 2. Worker 5s response-send timeout: when the timeout fired, the response was silently dropped but the correlationID remained in the ordered queue. The response writer could never advance past it, stalling all subsequent responses permanently. Fix: call cancel() to tear down the connection — if we cannot deliver a response in 5s the connection is irrecoverable. * chore: remove empty no-op ListOffsets conditional The `if apiKey == 2 {}` block had no body — leftover debug code. ListOffsets routing is handled by isDataPlaneAPI (returns false, sending it to the control channel). No behavior change.		2 days ago
..
src	HDFS: Java client replication configuration (#7526)	4 months ago
.gitignore	HDFS: Java client replication configuration (#7526)	4 months ago
Makefile	HDFS: Java client replication configuration (#7526)	4 months ago
docker-compose.yml	Update docker-compose.yml	1 month ago
pom.xml	build(deps): bump org.apache.spark:spark-core_2.12 from 3.5.0 to 3.5.7 in /test/java/spark (#8674)	1 week ago
quick-start.sh	HDFS: Java client replication configuration (#7526)	4 months ago
run-tests.sh	HDFS: Java client replication configuration (#7526)	4 months ago