Browse Source

fix heartbeat deadlock: send initial messages before awaiting stream

The bidirectional gRPC stream had a deadlock: send_heartbeat().await
waits for response headers from the server, but the Go master won't
send response headers until it receives the first heartbeat message.
Pre-populating the channel before creating the stream resolves this.

Tested end-to-end: Rust volume server successfully registers with Go
master, receives volume assignments, and handles uploads/downloads.
rust-volume-server
Chris Lu 6 days ago
parent
commit
3f9d8b93db
  1. 11
      seaweed-volume/src/server/heartbeat.rs

11
seaweed-volume/src/server/heartbeat.rs

@ -98,15 +98,16 @@ async fn do_heartbeat(
let mut client = SeaweedClient::new(channel);
let (tx, rx) = tokio::sync::mpsc::channel::<master_pb::Heartbeat>(32);
let stream = tokio_stream::wrappers::ReceiverStream::new(rx);
let mut response_stream = client.send_heartbeat(stream).await?.into_inner();
// Send initial volume heartbeat
// Send initial heartbeats BEFORE calling send_heartbeat to avoid deadlock:
// the server won't send response headers until it receives the first message,
// but send_heartbeat().await waits for response headers.
tx.send(collect_heartbeat(config, state)).await?;
// Send initial EC shards heartbeat
tx.send(collect_ec_heartbeat(state)).await?;
let stream = tokio_stream::wrappers::ReceiverStream::new(rx);
let mut response_stream = client.send_heartbeat(stream).await?.into_inner();
info!("Heartbeat stream established with {}", grpc_addr);
let mut volume_tick = tokio::time::interval(pulse);

Loading…
Cancel
Save