Mirror
/
seaweedfs
mirror of https://github.com/seaweedfs/seaweedfs.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
13089 Commits
121 Branches
330 Tags
366 MiB
 
 
 
 
 
 
Tree: 512bb5bcf6
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '512bb5bcf6'
${ noResults }
seaweedfs/learn/projects/sw-block/phases/phase-6-dev-log.md

14 KiB
Raw Blame History

Phase 6 Dev Log

Append-only communication between agents. Newest entries at bottom. Each entry: [date] [role] message

Roles: DEV, REVIEWER, TESTER, ARCHITECT

[2026-03-03] [DEV] CP6-1 implementation complete. 7 tasks (Task 0-6) delivered:

  • Task 0: Extracted BlockVolAdapter to shared blockvol/adapter.go (exported fields, ALUA methods, RoleToALUA/UUIDToNAA helpers). Added DisconnectVolume(iqn) to TargetServer (kills sessions + removes volume). Added Session.TargetIQN() accessor. Updated standalone binary to use shared adapter. 3 adapter tests.
  • Task 1: VolumeManager -- multi-volume BlockVol + shared TargetServer lifecycle. sync.Once start guard, atomic ready flag, IQN sanitization with hash suffix for collision avoidance. 10 tests.
  • Task 2: CSI Identity service (GetPluginInfo, GetPluginCapabilities, Probe). 3 tests.
  • Task 3: CSI Controller service (CreateVolume with idempotency + size mismatch, DeleteVolume). 4 tests.
  • Task 4: CSI Node service (Stage/Unstage/Publish/Unpublish) with ISCSIUtil/MountUtil interfaces. 7 tests.
  • Task 5: gRPC server + binary entry point (unix/tcp socket, signal handler, graceful shutdown).
  • Task 6: K8s manifests (DaemonSet, StorageClass, RBAC, example PVC) + smoke-test.sh. Total: 12 new Go files, 2 modified, 4 YAML, 1 shell script, 25+3=28 tests. CSI spec v1.10.0 added.

[2026-03-03] [REVIEWER] CP6-1 review returned 5 findings:

  1. (High) CreateVolume not idempotent after restart -- only checks in-memory map, misses existing .blk files.
  2. (Medium) NodePublishVolume doesn't validate empty StagingTargetPath.
  3. (Medium) NodeStageVolume resource leak -- OpenVolume not cleaned up on discovery/login/mount failure.
  4. (Medium) Target start race -- ListenAndServe in goroutine, ready=true set before bind confirmed.
  5. (Low) IQN collision -- truncation without hash suffix causes identical IQNs for long names. Open Q1: How should CreateVolume handle pre-existing .blk files on disk? Open Q2: What happens in NodeUnstageVolume if unmount succeeds but logout fails?

[2026-03-03] [DEV] All 5 review findings + 2 open questions resolved:

  • Finding 1: CreateVolume now checks os.Stat for existing .blk files, adopts via OpenBlockVol. Added ErrVolumeSizeMismatch. Controller maps it to codes.AlreadyExists.
  • Finding 2: Added stagingPath=="" check in NodePublishVolume returning InvalidArgument.
  • Finding 3: Added success flag + deferred CloseVolume after OpenVolume in NodeStageVolume.
  • Finding 4: Listener created synchronously via net.Listen before ready=true. Serve in goroutine.
  • Finding 5: SanitizeIQN appends SHA256 hash suffix (8 hex chars) when truncating to 64.
  • Open Q1: Pre-existing files adopted as idempotent success if size >= requested.
  • Open Q2: NodeUnstageVolume uses best-effort cleanup (firstErr pattern), always attempts CloseVolume. 3 new tests: CreateIdempotentAfterRestart, IQNCollision, StageLoginFailureCleanup, PublishMissingStagingPath. All 25 CSI tests + full regression PASS.

[2026-03-03] [TESTER] CP6-1 QA adversarial suite: 30 tests in qa_csi_test.go. 26 PASS, 4 FAIL confirming 5 bugs. Groups: QA-VM (8), QA-CTRL (5), QA-NODE (7), QA-SRV (3), QA-ID (1), QA-IQN (5), QA-X (1). Bugs: BUG-QA-1 snapshot leak, BUG-QA-2/3 sync.Once restart, BUG-QA-4 LimitBytes ignored, BUG-QA-5 case divergence.

[2026-03-03] [DEV] All 5 QA bugs fixed:

  • BUG-QA-1: DeleteVolume now globs+removes volPath+".snap.*" (both tracked and untracked paths).
  • BUG-QA-2+3: Replaced sync.Once+atomic.Bool with managerState enum (stopped/starting/ready/failed). Start() retryable after failure or Stop(). Stop() sets state=stopped, nils target. Goroutine captures target locally before launch (prevents nil deref after Stop).
  • BUG-QA-4: Controller CreateVolume validates LimitBytes. When RequiredBytes=0 and LimitBytes set, uses LimitBytes as target size. Rejects RequiredBytes > LimitBytes and post-rounding overflow.
  • BUG-QA-5: sanitizeFilename now lowercases (matching SanitizeIQN). "VolA" and "vola" produce same file and same IQN — treated as same volume via file adoption path.
  • QA-CTRL-4 test updated from bug-detection to behavior-documentation (NotFound is by design; volumes re-tracked via CreateVolume after restart). All 54 CSI tests + full regression PASS (blockvol 63s, iscsi 2.3s, csi 0.4s).

[2026-03-03] [DEV] CP6-2 complete. See separate CP6-2 entries in progress.md.

[2026-03-04] [TESTER] CSI Testing Ladder Levels 2-4 complete on M02 (192.168.1.184):

Level 2: csi-sanity gRPC Conformance

  • cross-compiled block-csi (linux/amd64), installed csi-sanity on M02
  • Result: 33 Passed, 0 Failed, 58 Skipped (optional RPCs), 1 Pending
  • 6 bugs found and fixed: empty VolumeCapabilities validation (3 RPCs), bind mount for NodePublish, target path removal in NodeUnpublish, IsMounted check before unmount
  • All 226 unit tests updated with VolumeCapabilities/VolumeCapability in requests

Level 3: Integration Smoke

  • Verified via csi-sanity's "should work" tests exercising real iSCSI on M02
  • 489 real SCSI commands processed (READ_10, WRITE_10, SYNC_CACHE, INQUIRY, etc.)
  • Full lifecycle: Create → Stage (discovery+login+mkfs+mount) → Publish → Unpublish → Unstage (unmount+logout) → Delete
  • Clean state: no leftover sessions, mounts, or volume files

Level 4: k3s PVC→Pod

  • Installed k3s v1.34.4 on M02, deployed CSI DaemonSet (block-csi + csi-provisioner + registrar)
  • DaemonSet uses nsenter wrappers for host iscsiadm/mount/umount/blkid/mountpoint/mkfs.ext4
  • Test: PVC (100Mi) → Pod writes "hello sw-block" → md5 7be761488cf480c966077c7aca4ea3ed → Pod deleted → PVC retained → New pod reads same data → PASS
  • 1 additional bug: IsLoggedIn didn't handle iscsiadm exit code 21 (nsenter suppresses output) → Fixed by checking ExitError.ExitCode() == 21 directly

Code changes from Levels 2-4:

  • controller.go: +VolumeCapabilities validation in CreateVolume, ValidateVolumeCapabilities
  • node.go: +VolumeCapability nil check, BindMount for publish, IsMounted+RemoveAll in unpublish
  • iscsi_util.go: +BindMount interface+impl (real+mock), IsLoggedIn exit code 21 handling
  • controller_test.go, node_test.go, qa_csi_test.go, qa_cp62_test.go: testVolCaps()/testVolCap() helpers

[2026-03-04] [DEV] CP6-3 Review 1+2 findings fixed (12 total, 5 High, 5 Medium, 2 Low):

  • R1-1 (High): AllocateBlockVolume now returns ReplicaDataAddr/CtrlAddr/RebuildListenAddr from ReplicationPorts().
  • R1-2 (High): setupPrimaryReplication now calls vol.StartRebuildServer(rebuildAddr) with deterministic port.
  • R1-3 (High): VS sends periodic full block heartbeat (5×sleepInterval) enabling assignment confirmation.
  • R2-F1 (High): LastLeaseGrant moved to entry initializer before Register (was after → stale-lease race).
  • R1-4 (Medium): BlockService.CollectBlockVolumeHeartbeat fills ReplicaDataAddr/CtrlAddr from replStates.
  • R1-5 (Medium): UpdateFullHeartbeat refreshes LastLeaseGrant on every heartbeat.
  • R2-F2 (Medium): Deferred promotion timers stored and cancelled on VS reconnect (prevents split-brain).
  • R2-F3 (Medium): SwapPrimaryReplica uses blockvol.RoleToWire(blockvol.RolePrimary) instead of uint32(1).
  • R2-F4 (Medium): DeleteBlockVolume now deletes replica (best-effort, non-fatal).
  • R2-F5 (Medium): SwapPrimaryReplica computes epoch+1 atomically inside lock, returns newEpoch.
  • R2-F6 (Low): Removed redundant string(server) casts.
  • R2-F7 (Low): Documented rebuild feedback as future work. All 293 tests PASS: blockvol (24s), csi (1.6s), iscsi (2.6s), server (3.3s).

[2026-03-04] [DEV] CP6-3 implementation complete. 8 tasks (Task 0-7) delivered:

  • Task 0: Proto extension — replica/rebuild address fields in master.proto, volume_server.proto, generated pb.go files, wire types, converters. AssignmentsToProto batch helper. 8 tests.
  • Task 1: Assignment queue — BlockAssignmentQueue with retain-until-confirmed (F1). Enqueue/Peek/Confirm/ConfirmFromHeartbeat. Stale epoch pruning. Wired into HeartbeatResponse. 11 tests.
  • Task 2: VS assignment receiver — extracts block_volume_assignments from HeartbeatResponse, calls BlockService.ProcessAssignments.
  • Task 3: BlockService replication — ProcessAssignments dispatches HandleAssignment + setupPrimaryReplication/setupReplicaReceiver/startRebuild. Deterministic ports via FNV hash (F3). Heartbeat reports replica addresses (F5). 9 tests.
  • Task 4: Registry replica + CreateVolume — SetReplica/ClearReplica/SwapPrimaryReplica. CreateBlockVolume creates primary + replica, enqueues assignments. Single-copy mode (F4). 10 tests.
  • Task 5: Failover — failoverBlockVolumes on VS disconnect. Lease-aware promotion (F2): promote only after lease expires, deferred via time.AfterFunc. SwapPrimaryReplica + epoch bump. 11 failover tests.
  • Task 6: ControllerPublish — ControllerPublishVolume returns fresh primary address via LookupVolume. ControllerUnpublishVolume no-op. PUBLISH_UNPUBLISH_VOLUME capability. NodeStageVolume prefers publish_context over volume_context. 8 tests.
  • Task 7: Rebuild on recovery — recoverBlockVolumes on VS reconnect drains pendingRebuilds, enqueues Rebuilding assignments. 10 tests (shared file with Task 5). Total: 4 new files, ~15 modified, 67 new tests. All 5 review findings (F1-F5) addressed. All tests PASS: blockvol (43s), csi (1.4s), iscsi (2.5s), server (3.2s). Cumulative Phase 6: 293 tests.

[2026-03-04] [TESTER] CP6-3 QA adversarial suite: 48 tests in qa_block_cp63_test.go. 47 PASS, 1 FAIL confirming 1 bug. Groups: QA-Queue (8), QA-Reg (7), QA-Failover (7), QA-Create (5), QA-Rebuild (3), QA-Integration (2), QA-Edge (5), QA-Master (5), QA-VS (6).

BUG-QA-CP63-1 (Medium): SetReplica leaks old replica server in byServer index.

  • When calling SetReplica("vol1", "vs3", ...) on a volume whose replica was previously vs2, vs2 remains in the byServer index. ListByServer("vs2") still returns vol1.
  • Impact: PickServer over-counts old replica server's volume count (wrong placement). Failover could trigger on stale index entries.
  • Fix: Added removeFromServer(oldReplicaServer, name) before setting new replica in SetReplica().
  • File: master_block_registry.go:285 (3 lines added).
  • Test: TestQA_Reg_SetReplicaTwice_ReplacesOld.

All 48 QA tests + full regression PASS: blockvol (23s), csi (1.1s), iscsi (2.5s), server (4.8s). Cumulative Phase 6: 293 + 48 = 341 tests.

[2026-03-04] [TESTER] CP6-3 integration tests: 8 tests in integration_block_test.go. All 8 PASS.

Required Tests:

  1. TestIntegration_FailoverCSIPublish — Create replicated vol → kill primary → verify LookupBlockVolume (CSI ControllerPublishVolume path) returns promoted replica's iSCSI addr.
  2. TestIntegration_RebuildOnRecovery — Failover → reconnect old primary → verify Rebuilding assignment enqueued with correct epoch → confirm via heartbeat.
  3. TestIntegration_AssignmentDeliveryConfirmation — Create replicated vol → verify pending assignments → wrong epoch doesn't confirm → correct heartbeat confirms → queue cleared.

Nice-to-have Tests: 4. TestIntegration_LeaseAwarePromotion — Lease not expired → promotion deferred → after TTL → promoted. 5. TestIntegration_ReplicaFailureSingleCopy — Replica alloc fails → single-copy mode → no replica assignments → failover is no-op (no replica to promote). 6. TestIntegration_TransientDisconnectNoSplitBrain — VS disconnects with active lease → deferred timer → VS reconnects → timer cancelled → no promotion (split-brain prevented).

Extra coverage: 7. TestIntegration_FullLifecycle — Create → publish → confirm assignments → failover → re-publish → confirm → recover → rebuild → confirm → delete. Full 11-phase lifecycle. 8. TestIntegration_DoubleFailover — Primary dies → promoted → promoted replica also dies → original server re-promoted (epoch=3). 9. TestIntegration_MultiVolumeFailoverRebuild — 3 volumes across 2 servers → kill one server → all primaries promoted → reconnect → rebuild assignments for each.

All 349 server+QA+integration tests PASS (6.8s). Cumulative Phase 6: 293 + 48 + 8 = 349 tests.

[2026-03-05] [TESTER] CP6-3 real integration tests on M02 (192.168.1.184): 3 tests, all PASS.

Bug found during testing: RoleNone → RoleRebuilding transition not allowed.

  • After VS restart, volume is RoleNone. Master sends Rebuilding assignment, but both validTransitions (role.go) and HandleAssignment (promotion.go) rejected this path.
  • Fix: Added RoleRebuilding: true to validTransitions[RoleNone] in role.go. Added RoleNone → RoleRebuilding case in HandleAssignment (promotion.go) with SetEpoch + SetMasterEpoch + SetRole.
  • Infrastructure: Added action:"connect" to admin.go /rebuild endpoint to start rebuild client (calls blockvol.StartRebuild in background goroutine). Added StartRebuildClient method to ha_target.go.

Tests (cp63_test.go, //go:build integration):

  1. FailoverCSIAddressSwitch (3.2s) — Write data A → kill primary → promote replica → client re-discovers at new iSCSI address → verify data A → write data B → verify A+B. Simulates CSI ControllerPublishVolume address-switch flow.
  2. RebuildDataConsistency (5.3s) — Write A (replicated) → kill replica → write B (missed) → restart replica as Rebuilding → start rebuild server on primary → connect rebuild client → wait for role→replica → kill primary → promote rebuilt replica → verify A+B intact. Full end-to-end rebuild with data verification.
  3. FullLifecycleFailoverRebuild (6.4s) — Write A → kill primary → promote replica → write B → start rebuild server → restart old primary as Rebuilding → rebuild → write C → kill new primary → promote rebuilt old-primary → verify A+B intact. 11-phase lifecycle simulating master's failover→recoverBlockVolumes→rebuild flow.

Existing 7 HA tests: all PASS (no regression). Total real integration: 10 tests on M02. Code changes: role.go (+1 line), promotion.go (+7 lines), admin.go (+15 lines), ha_target.go (+20 lines), cp63_test.go (new, ~350 lines).