seaweedfs/learn/projects/sw-block/phases/phase-6-progress.md

Phase 6 Progress

Status

• CP6-1 complete. 54 CSI tests (25 dev + 30 QA - 1 removed).
• CP6-2 complete. 172 CP6-2 tests (118 dev/review + 54 QA). 1 QA bug found and fixed.
• Phase 6 cumulative: 226 tests, all PASS.

Completed

• CP6-1 Task 0: Extracted BlockVolAdapter to shared blockvol/adapter.go, added DisconnectVolume to TargetServer, added Session.TargetIQN().

• CP6-1 Task 1: VolumeManager (multi-volume BlockVol + shared TargetServer lifecycle). 10 tests.

• CP6-1 Task 2: CSI Identity service (GetPluginInfo, GetPluginCapabilities, Probe). 3 tests.

• CP6-1 Task 3: CSI Controller service (CreateVolume, DeleteVolume, ValidateVolumeCapabilities). 4 tests.

• CP6-1 Task 4: CSI Node service (NodeStageVolume, NodeUnstageVolume, NodePublishVolume, NodeUnpublishVolume). 7 tests.

• CP6-1 Task 5: gRPC server + binary entry point (csi/cmd/block-csi/main.go).

• CP6-1 Task 6: K8s manifests (DaemonSet, StorageClass, RBAC, example PVC) + smoke-test.sh.

• CP6-1 Review fixes: 5 findings + 2 open questions resolved, 3 new tests added.

  • Finding 1: CreateVolume idempotency after restart (adopts existing .blk files on disk).
  • Finding 2: NodePublishVolume validates empty StagingTargetPath.
  • Finding 3: Resource leak cleanup on error paths (success flag + deferred CloseVolume).
  • Finding 4: Synchronous listener creation (bind errors surface immediately).
  • Finding 5: IQN collision avoidance (SHA256 hash suffix on truncation).

• CP6-1 QA adversarial: 30 tests in qa_csi_test.go. 5 bugs found and fixed:

  • BUG-QA-1 (Medium): DeleteVolume leaked .snap.* delta files. Fixed: glob+remove snapshot files.
  • BUG-QA-2 (High): Start not retryable after failure (sync.Once). Fixed: state machine.
  • BUG-QA-3 (High): Stop then Start broken (sync.Once already fired). Fixed: same state machine.
  • BUG-QA-4 (Low): CreateVolume ignored LimitBytes. Fixed: validate and cap size.
  • BUG-QA-5 (Medium): sanitizeFilename case divergence with SanitizeIQN. Fixed: lowercase both.
  • Additional: goroutine captured m.target by reference (nil after Stop). Fixed: local capture.

• CP6-2 complete. All 7 tasks done. 63 CSI tests + 48 server block tests = 111 CP6-2 tests, all PASS.

CP6-2: Control-Plane Integration

Completed Tasks

• Task 0: Proto Extension + Code Generation — block volume messages in master.proto/volume_server.proto, Go stubs regenerated, conversion helpers + 5 tests.
• Task 1: Master Block Volume Registry — in-memory registry with Pending→Active status tracking, full/delta heartbeat reconciliation, per-name inflight lock (TOCTOU prevention), placement (fewest volumes), block-capable server tracking. 11 tests.
• Task 2: Volume Server Block Volume gRPC — AllocateBlockVolume/DeleteBlockVolume gRPC handlers on VolumeServer, CreateBlockVol/DeleteBlockVol on BlockService, shared naming (blockvol/naming.go). 5 tests.
• Task 3: Master Block Volume RPC Handlers — CreateBlockVolume (idempotent, inflight lock, retry up to 3 servers), DeleteBlockVolume (idempotent), LookupBlockVolume. Mock VS call injection for testability. 9 tests.
• Task 4: Heartbeat Wiring — block volume fields in heartbeat stream, volume server sends initial full heartbeat + deltas, master processes via UpdateFullHeartbeat/UpdateDeltaHeartbeat.
• Task 5: CSI Controller Refactor — VolumeBackend interface (LocalVolumeBackend + MasterVolumeClient), controller uses backend instead of VolumeManager, returns volume_context with iscsiAddr+iqn, mode flag (controller/node/all). 5 backend tests.
• Task 6: CSI Node Refactor + K8s Manifests — Node reads volume_context for remote targets, staged volume tracking with IQN derivation fallback on restart, split K8s manifests (csi-driver.yaml, csi-controller.yaml Deployment, csi-node.yaml DaemonSet). 4 new node tests (11 total).

New Files (CP6-2)

File	Description
blockvol/naming.go	Shared SanitizeIQN + SanitizeFilename
blockvol/naming_test.go	4 naming tests
blockvol/block_heartbeat_proto.go	Go wire type ↔ proto conversion
blockvol/block_heartbeat_proto_test.go	5 conversion tests
server/master_block_registry.go	Block volume registry + placement
server/master_block_registry_test.go	11 registry tests
server/volume_grpc_block.go	VS block volume gRPC handlers
server/volume_grpc_block_test.go	5 VS tests
server/master_grpc_server_block.go	Master block volume RPC handlers
server/master_grpc_server_block_test.go	9 master handler tests
csi/volume_backend.go	VolumeBackend interface + clients
csi/volume_backend_test.go	5 backend tests
csi/deploy/csi-controller.yaml	Controller Deployment manifest
csi/deploy/csi-node.yaml	Node DaemonSet manifest

Modified Files (CP6-2)

File	Changes
pb/master.proto	Block volume messages, Heartbeat fields 24-27, RPCs
pb/volume_server.proto	AllocateBlockVolume, VolumeServerDeleteBlockVolume
server/master_server.go	BlockVolumeRegistry + VS call fields
server/master_grpc_server.go	Block volume heartbeat processing
server/volume_grpc_client_to_master.go	Block volume in heartbeat stream
server/volume_server_block.go	CreateBlockVol/DeleteBlockVol on BlockService
csi/controller.go	VolumeBackend instead of VolumeManager
csi/controller_test.go	Updated for VolumeBackend
csi/node.go	Remote target support + staged volume tracking
csi/node_test.go	4 new remote target tests
csi/server.go	Mode flag, MasterAddr, VolumeBackend config
csi/cmd/block-csi/main.go	--master, --mode flags
csi/deploy/csi-driver.yaml	CSIDriver object only (split out workloads)
csi/qa_csi_test.go	Updated for VolumeBackend

CP6-2 Review Fixes

All findings from both reviewers addressed. 4 new tests added (118 total CP6-2 tests).

#	Finding	Severity	Fix
R1-F1	DeleteBlockVol doesn't terminate active sessions	High	Use DisconnectVolume instead of RemoveVolume
R1-F2	Block registry server list never pruned	Medium	UnmarkBlockCapable on VS disconnect in SendHeartbeat defer
R1-F3	Block volume status never updates after create	Medium	Mark StatusActive immediately after successful VS allocate
R1-F4	IQN generation on startup scan doesn't sanitize	Low	Apply blockvol.SanitizeIQN(name) in scan path
R1-F5/R2-F3	CreateBlockVol idempotent path skips TargetServer	Medium	Re-add adapter to TargetServer on idempotent path
R2-F1	UpdateFullHeartbeat doesn't update SizeBytes	Low	Copy info.VolumeSize to existing.SizeBytes
R2-F2	inflightEntry.done channel is dead code	Low	Removed done channel, simplified to empty struct
R2-F4	CreateBlockVolume idempotent check doesn't validate size	Medium	Return error if existing size < requested size
R2-F5	Full + delta heartbeat can fire on same message	Low	Changed second if to else if + comment
R2-F6	NodeUnstageVolume deletes staged entry before cleanup	Medium	Delete from staged map only after successful cleanup

New tests: TestMaster_CreateIdempotentSizeMismatch, TestRegistry_UnmarkDeadServer, TestRegistry_FullHeartbeatUpdatesSizeBytes, TestNode_UnstageRetryKeepsStagedEntry.

CP6-2 QA Adversarial Tests

54 tests across 2 files. 1 bug found and fixed.

File	Tests	Areas
server/qa_block_cp62_test.go	22	Registry (8), Master RPCs (8), VS BlockService (6)
csi/qa_cp62_test.go	32	Node remote (6), Controller backend (5), Backend (2), Naming (2), Lifecycle (4), Server/Driver (2), VolumeManager (4), Edge cases (7)

BUG-QA-CP62-1 (Medium): NewCSIDriver accepts invalid mode strings.

• NewCSIDriver(DriverConfig{Mode: "invalid"}) returns nil error. Driver runs with only identity server — no controller, no node. K8s reports capabilities but all operations fail Unimplemented.
• Fix: Added switch validation after mode defaulting. Returns "csi: invalid mode %q, must be controller/node/all".
• Test: TestQA_ModeInvalid.

Final CP6-2 test count: 118 dev/review + 54 QA = 172 CP6-2 tests, all PASS.

Cumulative Phase 6 test count: 54 CP6-1 + 172 CP6-2 = 226 tests.

CSI Testing Ladder

Level	What	Tools	Status
1. Unit tests	Mock iscsiadm/mount. Confirm idempotency, error handling, edge cases.	go test	DONE (226 tests)
2. gRPC conformance	csi-sanity tool validates all CSI RPCs against spec. No K8s needed.	csi-sanity	DONE (33 pass, 58 skip)
3. Integration smoke	Full iSCSI lifecycle with real filesystem (via csi-sanity "should work" tests).	csi-sanity + iscsiadm	DONE (489 SCSI cmds)
4. Single-node K8s (k3s)	Deploy CSI DaemonSet on k3s. PVC → Pod → write data → delete/recreate → verify persistence.	k3s v1.34.4	DONE
5. Failure/chaos	Kill CSI controller pod; ensure no IO outage for existing volumes. Node restart with staged volumes.	chaos-mesh or manual	TODO
6. K8s E2E suite	SIG-Storage tests validate provisioning, attach/detach, resize, snapshots.	e2e.test binary	TODO

Level 2: csi-sanity Conformance (M02)

Result: 33 Passed, 0 Failed, 58 Skipped, 1 Pending.

Run on M02 (192.168.1.184) with block-csi in local mode. Used helper scripts for staging/target path management.

Bugs found and fixed during csi-sanity:

#	Bug	Severity	Fix
BUG-SANITY-1	CreateVolume accepted empty VolumeCapabilities	Medium	Added len(req.VolumeCapabilities) == 0 check
BUG-SANITY-2	ValidateVolumeCapabilities accepted empty VolumeCapabilities	Medium	Same check added
BUG-SANITY-3	NodeStageVolume accepted nil VolumeCapability	Medium	Added nil check
BUG-SANITY-4	NodePublishVolume used mount -t ext4 instead of bind mount	High	Added BindMount method to MountUtil interface
BUG-SANITY-5	NodeUnpublishVolume didn't remove target path	Medium	Added os.RemoveAll per CSI spec
BUG-SANITY-6	NodeUnpublishVolume failed on unmounted path	Medium	Added IsMounted check before unmount

All existing unit tests updated with VolumeCapabilities/VolumeCapability in test requests.

Level 3: Integration Smoke (M02)

Verified through csi-sanity's full lifecycle tests which exercised real iSCSI:

• 489 real SCSI commands processed (READ_10, WRITE_10, SYNC_CACHE, INQUIRY, etc.)
• Full cycle: CreateVolume → NodeStageVolume (iSCSI login + mkfs.ext4 + mount) → NodePublishVolume → NodeUnpublishVolume → NodeUnstageVolume (unmount + iSCSI logout) → DeleteVolume
• Clean state verified: no leftover iSCSI sessions, mounts, or volume files

Level 4: k3s PVC→Pod (M02)

Result: PASS — data persists across pod deletion/recreation.

k3s v1.34.4 single-node on M02. CSI deployed as DaemonSet with 3 containers:

1. block-csi (privileged, nsenter wrappers for host iscsiadm/mount/umount/mkfs/blkid/mountpoint)
2. csi-provisioner (v5.1.0, --node-deployment for single-node)
3. csi-node-driver-registrar (v2.12.0)

Test sequence:

1. Created PVC (100Mi, sw-block StorageClass) → Bound
2. Created pod → wrote "hello sw-block" to /data/test.txt → md5: 7be761488cf480c966077c7aca4ea3ed
3. Deleted pod (PVC retained) → iSCSI session cleanly closed
4. Recreated pod with same PVC → read "hello sw-block" → same md5 verified
5. Appended "persistence works!" → confirmed read-write

Additional bug fixed during k3s testing:

#	Bug	Severity	Fix
BUG-K3S-1	IsLoggedIn didn't handle iscsiadm exit code 21 (nsenter suppresses output)	Medium	Added exitErr.ExitCode() == 21 check

DaemonSet manifest: learn/projects/sw-block/test/csi-k3s-node.yaml

• CP6-3 complete. 67 CP6-3 tests. All PASS.

CP6-3: Failover + Rebuild in Kubernetes

Completed Tasks

• Task 0: Proto Extension + Wire Type Updates — Added replica_data_addr, replica_ctrl_addr to BlockVolumeInfoMessage/BlockVolumeAssignment; rebuild_addr to BlockVolumeAssignment; replica_server to Create/LookupBlockVolumeResponse; replica fields to AllocateBlockVolumeResponse. Updated wire types and converters. 8 tests.
• Task 1: Master Assignment Queue + Delivery — BlockAssignmentQueue with Enqueue/Peek/Confirm/ConfirmFromHeartbeat. Retain-until-confirmed pattern (F1): assignments resent on every heartbeat until VS confirms via matching (path, epoch, role). Stale epoch pruning during Peek. Wired into HeartbeatResponse delivery. 11 tests.
• Task 2: VS Assignment Receiver Wiring — VS extracts block_volume_assignments from HeartbeatResponse and calls BlockService.ProcessAssignments.
• Task 3: BlockService Replication Support — ProcessAssignments dispatches to HandleAssignment + setupPrimaryReplication/setupReplicaReceiver/startRebuild per role. ReplicationPorts deterministic hash (F3). Heartbeat reports replica addresses (F5). 9 tests.
• Task 4: Registry Replica Tracking + CreateVolume — Added SetReplica/ClearReplica/SwapPrimaryReplica to registry. CreateBlockVolume creates on 2 servers (primary + replica), enqueues assignments. Single-copy mode if only 1 server or replica fails (F4). LookupBlockVolume returns ReplicaServer. 10 tests.
• Task 5: Master Failover Detection — failoverBlockVolumes on VS disconnect. Lease-aware promotion (F2): promote only after LastLeaseGrant + LeaseTTL expires. Deferred promotion via time.AfterFunc for unexpired leases. promoteReplica swaps primary/replica, bumps epoch, enqueues new primary assignment. 11 tests.
• Task 6: ControllerPublishVolume/UnpublishVolume — ControllerPublishVolume calls backend.LookupVolume, returns publish_context{iscsiAddr, iqn}. ControllerUnpublishVolume is no-op. Added PUBLISH_UNPUBLISH_VOLUME capability. NodeStageVolume prefers publish_context over volume_context (reflects current primary after failover). 8 tests.
• Task 7: Rebuild on Recovery — recoverBlockVolumes on VS reconnect drains pendingRebuilds, sets reconnected server as replica, enqueues Rebuilding assignments. 10 tests (shared with Task 5 test file).

Design Review Findings Addressed

#	Finding	Severity	Resolution
F1	Assignment delivery can be dropped	Critical	Retain-until-confirmed: Peek+Confirm pattern, assignments resent every heartbeat
F2	Failover without lease check → split-brain	Critical	Gate promotion on now > lastLeaseGrant + leaseTTL; deferred promotion for unexpired leases
F3	Replication ports change on VS restart	Critical	Deterministic port = FNV hash of path, offset from base iSCSI port
F4	Partial create (replica fails)	Medium	Single-copy mode with ReplicaServer="", skip replica assignments
F5	UpdateFullHeartbeat ignores replica addresses	Medium	VS includes replica_data/ctrl in InfoMessage; registry updates on heartbeat

Code Review 1 Findings Addressed

#	Finding	Severity	Resolution
R1-1	AllocateBlockVolume missing repl addrs	High	AllocateBlockVolume now returns ReplicaDataAddr/CtrlAddr/RebuildListenAddr from ReplicationPorts()
R1-2	Primary never starts rebuild server	High	setupPrimaryReplication now calls vol.StartRebuildServer(rebuildAddr)
R1-3	Assignment queue never confirms after startup	High	VS sends periodic full block heartbeat (5×sleepInterval tick) enabling master confirmation
R1-4	Replica addresses not reported in heartbeat	Medium	BlockService.CollectBlockVolumeHeartbeat wraps store's collector, fills ReplicaDataAddr/CtrlAddr from replStates
R1-5	Lease never refreshed after create	Medium	UpdateFullHeartbeat refreshes LastLeaseGrant on every heartbeat; periodic block heartbeats keep it current

Code Review 2 Findings Addressed

#	Finding	Severity	Resolution
R2-F1	LastLeaseGrant set AFTER Register → stale-lease race	High	Moved to entry initializer BEFORE Register
R2-F2	Deferred promotion timer has no cancellation	Medium	Timers stored in blockFailoverState.deferredTimers; cancelled in recoverBlockVolumes on reconnect
R2-F3	SwapPrimaryReplica hardcodes uint32(1)	Medium	Changed to blockvol.RoleToWire(blockvol.RolePrimary)
R2-F4	DeleteBlockVolume doesn't delete replica	Medium	Added best-effort replica delete (non-fatal if replica VS is down)
R2-F5	promoteReplica reads epoch without lock	Medium	SwapPrimaryReplica now computes epoch+1 atomically inside lock, returns newEpoch
R2-F6	Redundant string(server) casts	Low	Removed — servers already typed as string
R2-F7	startRebuild goroutine has no feedback path	Low	Documented as future work (VS could report via heartbeat)

New Files (CP6-3)

File	Description
server/master_block_assignment_queue.go	Assignment queue with retain-until-confirmed
server/master_block_assignment_queue_test.go	11 queue tests
server/master_block_failover.go	Failover detection + rebuild on recovery
server/master_block_failover_test.go	21 failover + rebuild tests

Modified Files (CP6-3)

File	Changes
pb/master.proto	Replica/rebuild fields on assignment/info/response messages
pb/volume_server.proto	Replica/rebuild fields on AllocateBlockVolumeResponse
pb/master_pb/master.pb.go	New fields + getters
pb/volume_server_pb/volume_server.pb.go	New fields + getters
storage/blockvol/block_heartbeat.go	ReplicaDataAddr/CtrlAddr on InfoMessage, RebuildAddr on Assignment
storage/blockvol/block_heartbeat_proto.go	Updated converters + AssignmentsToProto
server/master_server.go	blockAssignmentQueue, blockFailover, blockAllocResult struct
server/master_grpc_server.go	Assignment delivery in heartbeat, failover on disconnect, recovery on reconnect
server/master_grpc_server_block.go	Replica creation, assignment enqueueing, tryCreateReplica; R2-F1 LastLeaseGrant fix; R2-F4 replica delete; R2-F6 cast cleanup
server/master_block_registry.go	Replica fields, lease fields, SetReplica/ClearReplica/SwapPrimaryReplica; R2-F3 RoleToWire; R2-F5 atomic epoch; R1-5 lease refresh
server/volume_grpc_client_to_master.go	Assignment processing from HeartbeatResponse; R1-3 periodic block heartbeat tick
server/volume_grpc_block.go	R1-1 replication ports in AllocateBlockVolumeResponse
server/volume_server_block.go	ProcessAssignments, replication setup, ReplicationPorts; R1-2 StartRebuildServer; R1-4 CollectBlockVolumeHeartbeat with repl addrs
server/master_block_failover.go	R2-F2 deferred timer cancellation; R2-F5 new SwapPrimaryReplica API; R2-F7 rebuild feedback comment
storage/store_blockvol.go	WithVolume (exported)
csi/controller.go	ControllerPublishVolume/UnpublishVolume, PUBLISH_UNPUBLISH capability
csi/node.go	Prefer publish_context over volume_context

CP6-3 Test Count

File	New Tests
blockvol/block_heartbeat_proto_test.go	7
server/master_block_assignment_queue_test.go	11
server/volume_server_block_test.go	9
server/master_block_registry_test.go	5
server/master_grpc_server_block_test.go	6
server/master_block_failover_test.go	21
csi/controller_test.go	6
csi/node_test.go	2
Total CP6-3	67

Cumulative Phase 6 test count: 54 CP6-1 + 172 CP6-2 + 67 CP6-3 = 293 tests.

CP6-3 QA Adversarial Tests

48 tests in server/qa_block_cp63_test.go. 1 bug found and fixed.

Group	Tests	Areas
Assignment Queue	8	Wrong epoch confirm, partial heartbeat confirm, same-path different roles, concurrent ops
Registry	7	Double swap, swap no-replica, concurrent swap+lookup, SetReplica replace, heartbeat clobber
Failover	7	Deferred cancel on reconnect, double disconnect, mixed lease states, volume deleted during timer
Create+Delete	5	Lease non-zero after create, replica delete on vol delete, replica delete failure
Rebuild	3	Double reconnect, nil failover state, full cycle
Integration	2	Failover enqueues assignment, heartbeat confirms failover assignment
Edge Cases	5	Epoch monotonic, cancel timers no rebuilds, replica server dies, empty batch
Master-level	5	Delete VS unreachable, sanitized name, concurrent create/delete, all VS fail, slow allocate
VS-level	6	Concurrent create, concurrent create/delete, delete cleans snapshots, sanitization collision, idempotent re-add, nil block service

BUG-QA-CP63-1 (Medium): SetReplica leaks old replica server in byServer index.

• SetReplica didn't remove old replica server from byServer when replacing with a new one.
• Fix: Added removeFromServer(oldReplicaServer, name) before setting new replica (3 lines).
• Test: TestQA_Reg_SetReplicaTwice_ReplacesOld.

Final CP6-3 test count: 67 dev/review + 48 QA = 115 CP6-3 tests, all PASS.

CP6-3 Integration Tests

8 tests in server/integration_block_test.go. Full cross-component flows.

#	Test	What it proves
1	FailoverCSIPublish	LookupBlockVolume returns new iSCSI addr after failover
2	RebuildOnRecovery	Rebuilding assignment enqueued + heartbeat confirms it
3	AssignmentDeliveryConfirmation	Queue retains until heartbeat confirms matching (path, epoch)
4	LeaseAwarePromotion	Promotion deferred until lease TTL expires
5	ReplicaFailureSingleCopy	Single-copy mode: no replica assignments, failover is no-op
6	TransientDisconnectNoSplitBrain	Deferred timer cancelled on reconnect, no split-brain
7	FullLifecycle	11-phase lifecycle: create→publish→confirm→failover→re-publish→recover→rebuild→delete
8	DoubleFailover	Two successive failovers: epoch 1→2→3
9	MultiVolumeFailoverRebuild	3 volumes, kill 1 server, rebuild all affected

Final CP6-3 test count: 67 dev/review + 48 QA + 8 mock integration + 3 real integration = 126 CP6-3 tests, all PASS.

Cumulative Phase 6 with QA: 54 CP6-1 + 172 CP6-2 + 126 CP6-3 = 352 tests.

CP6-3 Real Integration Tests (M02)

3 tests in blockvol/test/cp63_test.go, run on M02 (192.168.1.184) with real iSCSI.

Bug found: RoleNone → RoleRebuilding transition not allowed. After VS restart, volume is RoleNone. Master sends Rebuilding assignment, but both validTransitions (role.go) and HandleAssignment (promotion.go) rejected this path.

• Fix: Added RoleRebuilding: true to validTransitions[RoleNone] in role.go. Added RoleNone → RoleRebuilding case in HandleAssignment with SetEpoch + SetRole.
• Admin API: Added action:"connect" to /rebuild endpoint (starts rebuild client).

#	Test	Time	What it proves
1	FailoverCSIAddressSwitch	3.2s	Write A → kill primary → promote replica → re-discover at new iSCSI address → verify A → write B → verify A+B. Simulates CSI ControllerPublishVolume address-switch.
2	RebuildDataConsistency	5.3s	Write A (replicated) → kill replica → write B (missed) → restart replica as Rebuilding → rebuild server + client → wait role→Replica → kill primary → promote rebuilt → verify A+B. Full end-to-end rebuild with data verification.
3	FullLifecycleFailoverRebuild	6.4s	Write A → kill primary → promote → write B → rebuild old primary → write C → kill new primary → promote old → verify A+B. 11-phase lifecycle: failover→recoverBlockVolumes→rebuild.

All 7 existing HA tests: PASS (no regression). Total real integration: 10 tests on M02.

In Progress

• None.

Blockers

• None.

Next Steps

• CP6-4: Soak testing, lease renewal timers, monitoring dashboards.

Notes

• CSI spec dependency: github.com/container-storage-interface/spec v1.10.0.
• Architecture: CSI binary embeds TargetServer + BlockVol in-process (loopback iSCSI).
• Interface-based ISCSIUtil/MountUtil for unit testing without real iscsiadm/mount.
• k3s deployment requires: hostNetwork, hostPID, privileged, /dev mount, nsenter wrappers for host commands.
• Known pre-existing flaky: TestQAPhase4ACP1/role_concurrent_transitions (unrelated to CSI).

CP6-1 Test Catalog

VolumeManager (csi/volume_manager_test.go) — 10 tests

#	Test	What it proves
1	CreateOpenClose	Create, verify IQN, close, reopen lifecycle
2	DeleteRemovesFile	.blk file removed on delete
3	DuplicateCreate	Same size idempotent; different size returns ErrVolumeSizeMismatch
4	ListenAddr	Non-empty listen address after start
5	OpenNonExistent	Error on opening non-existent volume
6	CloseAlreadyClosed	Idempotent close of non-tracked volume
7	ConcurrentCreateDelete	10 parallel create+delete, no races
8	SanitizeIQN	Special char replacement, truncation to 64 chars
9	CreateIdempotentAfterRestart	Existing .blk file adopted on restart
10	IQNCollision	Long names with same prefix get distinct IQNs via hash suffix

Identity (csi/identity_test.go) — 3 tests

#	Test	What it proves
1	GetPluginInfo	Returns correct driver name + version
2	GetPluginCapabilities	Returns CONTROLLER_SERVICE capability
3	Probe	Returns ready=true

Controller (csi/controller_test.go) — 4 tests

#	Test	What it proves
1	CreateVolume	Volume created and tracked
2	CreateIdempotent	Same name+size succeeds, different size returns AlreadyExists
3	DeleteVolume	Volume removed after delete
4	DeleteNotFound	Delete non-existent returns success (CSI spec)

Node (csi/node_test.go) — 7 tests

#	Test	What it proves
1	StageUnstage	Full stage flow (discovery+login+mount) and unstage (unmount+logout+close)
2	PublishUnpublish	Bind mount from staging to target path
3	StageIdempotent	Already-mounted staging path returns OK without side effects
4	StageLoginFailure	iSCSI login error propagated as Internal
5	StageMkfsFailure	mkfs error propagated as Internal
6	StageLoginFailureCleanup	Volume closed after login failure (no resource leak)
7	PublishMissingStagingPath	Empty StagingTargetPath returns InvalidArgument

Adapter (blockvol/adapter_test.go) — 3 tests

#	Test	What it proves
1	AdapterALUAProvider	ALUAState/TPGroupID/DeviceNAA correct values
2	RoleToALUA	All role→ALUA state mappings
3	UUIDToNAA	NAA-6 byte layout from UUID

CP6-2 Test Catalog

Registry (server/master_block_registry_test.go) — 11 tests

#	Test	What it proves
1	RegisterLookup	Register + Lookup returns entry
2	DuplicateRegister	Second register same name errors
3	Unregister	Unregister removes entry
4	ListByServer	Returns only entries for given server
5	FullHeartbeat	Marks active, removes stale, adds new
6	DeltaHeartbeat	Add/remove deltas applied correctly
7	PickServer	Fewest-volumes placement
8	Inflight	AcquireInflight blocks duplicate, ReleaseInflight unblocks
9	BlockCapable	MarkBlockCapable / UnmarkBlockCapable tracking
10	UnmarkDeadServer	R1-F2 regression test
11	FullHeartbeatUpdatesSizeBytes	R2-F1 regression test

Master RPCs (server/master_grpc_server_block_test.go) — 9 tests

#	Test	What it proves
1	CreateHappyPath	Create → register → lookup works
2	CreateIdempotent	Same name+size returns same entry
3	CreateIdempotentSizeMismatch	Same name, smaller size → error
4	CreateInflightBlock	Concurrent create same name → one fails
5	Delete	Delete → VS called → unregistered
6	DeleteNotFound	Delete non-existent → success
7	Lookup	Lookup returns entry
8	LookupNotFound	Lookup non-existent → NotFound
9	CreateRetryNextServer	First VS fails → retries on next

VS Block gRPC (server/volume_grpc_block_test.go) — 5 tests

#	Test	What it proves
1	Allocate	Create via gRPC returns path+iqn+addr
2	AllocateEmptyName	Empty name → error
3	AllocateZeroSize	Zero size → error
4	Delete	Delete via gRPC succeeds
5	DeleteNilService	Nil blockService → error

Naming (blockvol/naming_test.go) — 4 tests

#	Test	What it proves
1	SanitizeFilename	Lowercases, replaces invalid chars
2	SanitizeIQN	Lowercases, replaces, truncates with hash
3	IQNMaxLength	64-char names pass through unchanged
4	IQNHashDeterministic	Same input → same hash suffix

Proto conversion (blockvol/block_heartbeat_proto_test.go) — 5 tests

#	Test	What it proves
1	RoundTrip	Go→proto→Go preserves all fields
2	NilSafe	Nil input → nil output
3	ShortRoundTrip	Short info round-trip
4	AssignmentRoundTrip	Assignment round-trip
5	SliceHelpers	Slice conversion helpers

Backend (csi/volume_backend_test.go) — 5 tests

#	Test	What it proves
1	LocalCreate	LocalVolumeBackend.CreateVolume creates + returns info
2	LocalDelete	LocalVolumeBackend.DeleteVolume removes volume
3	LocalLookup	LocalVolumeBackend.LookupVolume returns info
4	LocalLookupNotFound	Lookup non-existent returns not-found
5	LocalDeleteNotFound	Delete non-existent returns success

Node remote (csi/node_test.go additions) — 4 tests

#	Test	What it proves
1	StageRemoteTarget	volume_context drives iSCSI instead of local mgr
2	UnstageRemoteTarget	Staged map IQN used for logout
3	UnstageAfterRestart	IQN derived from iqnPrefix when staged map empty
4	UnstageRetryKeepsStagedEntry	R2-F6 regression: staged entry preserved on failure

QA Server (server/qa_block_cp62_test.go) — 22 tests

#	Test	What it proves
1	Reg_FullHeartbeatCrossTalk	Heartbeat from s2 doesn't remove s1 volumes
2	Reg_FullHeartbeatEmptyServer	Empty heartbeat marks server block-capable
3	Reg_ConcurrentHeartbeatAndRegister	10 goroutines heartbeat+register, no races
4	Reg_DeltaHeartbeatUnknownPath	Delta for unknown path is no-op
5	Reg_PickServerTiebreaker	PickServer returns first server on tie
6	Reg_ReregisterDifferentServer	Re-register same name on different server fails
7	Reg_InflightIndependence	Inflight lock for vol-a doesn't block vol-b
8	Reg_BlockCapableServersAfterUnmark	Unmark removes from block-capable list
9	Master_DeleteVSUnreachable	Delete fails if VS delete fails (no orphan)
10	Master_CreateSanitizedName	Names with special chars go through
11	Master_ConcurrentCreateDelete	Concurrent create+delete on same name, no panic
12	Master_AllVSFailNoOrphan	All 3 servers fail → error, no registry entry
13	Master_SlowAllocateBlocksSecond	Inflight lock blocks concurrent same-name create
14	Master_CreateZeroSize	Zero size → InvalidArgument
15	Master_CreateEmptyName	Empty name → InvalidArgument
16	Master_EmptyNameValidation	Whitespace-only name → InvalidArgument
17	VS_ConcurrentCreate	20 goroutines create same vol, no crash
18	VS_ConcurrentCreateDelete	20 goroutines create+delete interleaved
19	VS_DeleteCleansSnapshots	Delete removes .snap.* files
20	VS_SanitizationCollision	Idempotent create after sanitization matches
21	VS_CreateIdempotentReaddTarget	Idempotent create re-adds adapter to TargetServer
22	VS_GrpcNilBlockService	Nil blockService returns error (not panic)

QA CSI (csi/qa_cp62_test.go) — 32 tests

#	Test	What it proves
1	Node_RemoteUnstageNoCloseVolume	Remote unstage doesn't call CloseVolume
2	Node_RemoteUnstageFailPreservesStaged	Failed unstage preserves staged entry
3	Node_ConcurrentStageUnstage	20 concurrent stage+unstage, no races
4	Node_RemotePortalUsedCorrectly	Remote portal used for discovery (not local)
5	Node_PartialVolumeContext	Missing iqn falls back to local mgr
6	Node_UnstageNoMgrNoPrefix	No mgr + no prefix → empty IQN (graceful)
7	Ctrl_VolumeContextPresent	CreateVolume returns iscsiAddr+iqn in context
8	Ctrl_ValidateUsesBackend	ValidateVolumeCapabilities uses backend lookup
9	Ctrl_CreateLargerSizeRejected	Existing vol + larger size → AlreadyExists
10	Ctrl_ExactBlockSizeBoundary	Exact 4MB boundary succeeds
11	Ctrl_ConcurrentCreate	10 concurrent creates, one succeeds
12	Backend_LookupAfterRestart	Volume found after VolumeManager restart
13	Backend_DeleteThenLookup	Lookup after delete → not found
14	Naming_CrossLayerConsistency	CSI and blockvol SanitizeIQN produce same result
15	Naming_LongNameHashCollision	Two 70-char names → distinct IQNs
16	RemoteLifecycleFull	Full remote stage→publish→unpublish→unstage→delete
17	ModeControllerNoMgr	Controller mode with masterAddr, no local mgr
18	ModeNodeOnly	Node mode creates mgr but no controller
19	ModeInvalid	Invalid mode → error (BUG-QA-CP62-1)
20	Srv_AllModeLocalBackend	All mode without master uses local backend
21	Srv_DoubleStop	Double Stop doesn't panic
22	VM_CreateAfterStop	Create after stop returns error
23	VM_OpenNonExistent	Open non-existent returns error
24	VM_ListenAddrAfterStop	ListenAddr after stop returns empty
25	VM_VolumeIQNSanitized	VolumeIQN applies sanitization
26	Edge_MinSize	Minimum 4MB volume succeeds
27	Edge_BelowMinSize	Below minimum → error
28	Edge_RequiredEqualsLimit	Required == limit succeeds
29	Edge_RoundingExceedsLimit	Rounding up exceeds limit → error
30	Edge_EmptyVolumeIDNode	Empty volumeID → InvalidArgument
31	Node_PublishWithoutStaging	Publish unstaged vol → still works (mock)
32	Node_DoubleUnstage	Double unstage → idempotent success