seaweedfs/docker/compose/BLOCK_GETTING_STARTED.md

SeaweedFS Block Storage -- Getting Started

Block storage exposes SeaweedFS volumes as /dev/sdX block devices via iSCSI. You can format them with ext4/xfs, mount them, and use them like any disk.

Prerequisites

• Linux host with open-iscsi installed
• Docker with compose plugin (docker compose)

# Install iSCSI initiator (Ubuntu/Debian)
sudo apt-get install -y open-iscsi

# Verify
sudo systemctl start iscsid

Quick Start (5 minutes)

1. Build the image

# From the seaweedfs repo root
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o docker/compose/weed ./weed
cd docker
docker build -f Dockerfile.local -t seaweedfs-block:local .

2. Start the cluster

cd docker/compose

# Set HOST_IP to your machine's IP (for remote iSCSI clients)
# Use 127.0.0.1 for local-only testing
HOST_IP=127.0.0.1 docker compose -f local-block-compose.yml up -d

Wait ~5 seconds for the volume server to register with the master.

3. Create a block volume

curl -s -X POST http://localhost:9333/block/volume \
  -H "Content-Type: application/json" \
  -d '{"name":"myvolume","size_bytes":1073741824}'

This creates a 1GB block volume, auto-assigns it as primary, and starts the iSCSI target. The response includes the IQN and iSCSI address.

4. Connect via iSCSI

# Discover targets
sudo iscsiadm -m discovery -t sendtargets -p 127.0.0.1:3260

# Login
sudo iscsiadm -m node -T iqn.2024-01.com.seaweedfs:vol.myvolume \
  -p 127.0.0.1:3260 --login

# Find the new device
lsblk | grep sd

5. Format and mount

# Format with ext4
sudo mkfs.ext4 /dev/sdX

# Mount
sudo mkdir -p /mnt/myvolume
sudo mount /dev/sdX /mnt/myvolume

# Use it like any filesystem
echo "hello" | sudo tee /mnt/myvolume/test.txt

6. Cleanup

sudo umount /mnt/myvolume
sudo iscsiadm -m node -T iqn.2024-01.com.seaweedfs:vol.myvolume \
  -p 127.0.0.1:3260 --logout
docker compose -f local-block-compose.yml down -v

API Reference

All endpoints are on the master server (default: port 9333).

Create volume

POST /block/volume
Content-Type: application/json

{
  "name": "myvolume",
  "size_bytes": 1073741824,
  "disk_type": "ssd",
  "replica_placement": "001",
  "durability_mode": "best_effort"
}

Field	Required	Default	Description
name	yes	--	Volume name (alphanumeric + hyphens)
size_bytes	yes	--	Volume size in bytes
disk_type	no	""	Disk type hint: ssd, hdd
replica_placement	no	000	SeaweedFS placement: 000 (no replica), 001 (1 replica same rack)
durability_mode	no	best_effort	best_effort, sync_all, sync_quorum
replica_factor	no	2	Number of copies: 1, 2, or 3

List volumes

GET /block/volumes

Returns JSON array of all block volumes with status, role, epoch, IQN, etc.

Lookup volume

GET /block/volume/{name}

Delete volume

DELETE /block/volume/{name}

Assign role

POST /block/assign
Content-Type: application/json

{
  "name": "myvolume",
  "epoch": 2,
  "role": "primary",
  "lease_ttl_ms": 30000
}

Roles: primary, replica, stale, rebuilding.

Cluster status

GET /block/status

Returns volume count, server count, failover stats, queue depth.

Remote Client Setup

To connect from a remote machine (not the Docker host):

1. Set HOST_IP to the Docker host's network-reachable IP:

   HOST_IP=192.168.1.100 docker compose -f local-block-compose.yml up -d
   

2. On the client machine:

   sudo iscsiadm -m discovery -t sendtargets -p 192.168.1.100:3260
   sudo iscsiadm -m node -T iqn.2024-01.com.seaweedfs:vol.myvolume \
     -p 192.168.1.100:3260 --login
   

Volume Lifecycle

create  -->  primary (serving I/O via iSCSI)
                |
           unmount/remount OK (lease auto-renewed by master)
                |
         assign replica  -->  WAL shipping active
                |
          kill primary   -->  promote replica  -->  new primary
                |
          old primary    -->  rebuild from new primary

Key points:

• Lease renewal is automatic. The master continuously renews the primary's write lease via the heartbeat stream. Unmount/remount works without manual intervention.
• Epoch fencing. Each role change bumps the epoch. Old primaries cannot write after being demoted -- even if they still have the lease.
• Volumes survive container restart. Data is stored in the Docker volume at /data/blocks/. The volume server re-registers with the master on restart.

Troubleshooting

iSCSI login fails with "No records found"

• Run discovery first: sudo iscsiadm -m discovery -t sendtargets -p HOST:3260

Device not appearing after login

• Check dmesg | tail for SCSI errors
• Verify the volume is assigned as primary: curl http://HOST:9333/block/volumes

I/O errors on write

• Check volume role is primary (not none or stale)
• Check master is running (lease renewal requires master heartbeat)

Stuck iSCSI session after container restart

• Force logout: sudo iscsiadm -m node -T IQN -p HOST:PORT --logout
• If stuck: sudo ss -K dst HOST dport = 3260 to kill the TCP connection
• Then re-discover and login

Docker Compose Reference

# local-block-compose.yml
services:
  master:
    image: seaweedfs-block:local
    ports:
      - "9333:9333"      # HTTP API
      - "19333:19333"     # gRPC
    command: ["master", "-ip=master", "-ip.bind=0.0.0.0", "-mdir=/data"]

  volume:
    image: seaweedfs-block:local
    ports:
      - "8280:8080"       # Volume HTTP
      - "18280:18080"     # Volume gRPC
      - "3260:3260"       # iSCSI target
    command: >
      volume -ip=volume -master=master:9333 -dir=/data
      -block.dir=/data/blocks
      -block.listen=0.0.0.0:3260
      -block.portal=${HOST_IP:-127.0.0.1}:3260,1      

Key flags:

• -block.dir: Directory for .blk volume files
• -block.listen: iSCSI target listen address (inside container)
• -block.portal: iSCSI portal address reported to clients (must be reachable)