seaweedfs

Commit Graph

Author	SHA1	Message	Date
pingqiu	47238df0d7	fix: add RecoveryOrchestrator as real integrated entry path New: orchestrator.go — RecoveryOrchestrator drives recovery lifecycle from assignment through execution to completion/escalation: - ProcessAssignment: reconcile + session creation + auto-log - ExecuteRecovery: connect → handshake from RetainedHistory → outcome - CompleteCatchUp: begin catch-up → progress → complete + auto-log - CompleteRebuild: connect → handshake → history-driven source → transfer → tail replay → complete + auto-log - InvalidateEpoch: invalidate stale sessions + auto-log All integration tests rewritten to use orchestrator as entry path. No direct sender API calls in recovery lifecycle. SessionSnapshot now includes: TruncateRequired/ToLSN/Recorded, RebuildSource, RebuildPhase. RecoveryLog is auto-populated by orchestrator at every transition. 7 integration tests via orchestrator: - ChangedAddress, NeedsRebuild→Rebuild, EpochBump, MultiReplica - Observability: session snapshot, rebuild snapshot, auto-populated log Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2 days ago
pingqiu	7436b3b79c	feat: add integration closure and observability (Phase 05 Slice 4) New files: - observe.go: RegistryStatus, SenderStatus, RecoveryLog for debugging - integration_test.go: V2-boundary integration tests through real engine entry path Observability: - Registry.Status() returns full snapshot: per-sender state, session snapshots, counts by category (InSync, Recovering, Rebuilding) - RecoveryLog: append-only event log for recovery lifecycle debugging Integration tests (6): - ChangedAddress_FullFlow: initial recovery → address change → sender preserved → new session → recovery with proof - NeedsRebuild_ThenRebuildAssignment: catch-up fails → NeedsRebuild → rebuild assignment → history-driven source → InSync - EpochBump_DuringRecovery: mid-recovery epoch bump → old session rejected → new assignment at new epoch → InSync - MultiReplica_MixedOutcomes: 3 replicas, 3 outcomes via RetainedHistory proofs, registry status verified - RegistryStatus_Snapshot: observability snapshot structure - RecoveryLog: event recording and filtering Engine module at 54 tests (12 + 18 + 18 + 6). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2 days ago
pingqiu	4d06622c01	fix: add nil check for RetainedHistory in sender APIs RecordHandshakeFromHistory and SelectRebuildFromHistory now return an error instead of panicking on nil history input. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2 days ago
pingqiu	cc8c529962	fix: connect recovery decisions to RetainedHistory, fix rebuild source RetainedHistory as engine input: - RecordHandshakeFromHistory: sender-level API consuming RetainedHistory directly, returns RecoverabilityProof alongside outcome - SelectRebuildFromHistory: sender-level API consuming RetainedHistory for rebuild-source decision RebuildSourceDecision soundness: - Now requires BOTH trusted checkpoint AND replayable tail (CheckpointLSN >= TailLSN and CommittedLSN <= HeadLSN) - Trusted checkpoint with unreplayable tail falls back to full_base 4 new tests: - TrustedCheckpoint_UnreplayableTail (the regression case) - SenderDriven_CatchUp (history → proof → outcome → complete) - SenderDriven_Rebuild_SnapshotTail (history → source → rebuild) - SenderDriven_Rebuild_FallsBackToFullBase (unreplayable tail) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2 days ago
pingqiu	ff7ea41099	feat: add engine data/recoverability core (Phase 05 Slice 3) New file: history.go — RetainedHistory connects recovery decisions to actual WAL retention state: - IsRecoverable: checks gap against tail/head boundaries - MakeHandshakeResult: generates HandshakeResult from retention state - RebuildSourceDecision: chooses snapshot+tail vs full base from checkpoint state (trusted vs untrusted) - ProveRecoverability: generates explicit proof explaining why recovery is or is not allowed 14 new tests (recoverability_test.go): - Recoverable/unrecoverable gap (exact boundary, beyond head) - Trusted/untrusted/no checkpoint → rebuild source selection - Handshake from retained history → outcome classification - Recoverability proofs (zero-gap, ahead, within retention, beyond) - E2E: two replicas driven by retained history (catch-up + rebuild) - Truncation required for replica ahead of committed Engine module at 44 tests (12 + 18 + 14). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2 days ago
pingqiu	368a956aee	fix: correct catch-up entry counting and rebuild transfer gate Entry counting: - Session.setRange now initializes recoveredTo = startLSN - RecordCatchUpProgress delta counts only actual catch-up work (recoveredTo - startLSN), not the replica's pre-existing prefix Rebuild transfer gate: - BeginTailReplay requires TransferredTo >= SnapshotLSN - Prevents tail replay on incomplete base transfer 3 new regression tests: - BudgetEntries_NonZeroStart_CountsOnlyDelta (30 entries within 50 budget) - BudgetEntries_NonZeroStart_ExceedsBudget (30 entries exceeds 20 budget) - Rebuild_PartialTransfer_BlocksTailReplay Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	3 days ago
pingqiu	930de4ba78	feat: add Slice 2 recovery execution tests (Phase 05) 15 new engine-level recovery execution tests: - Zero-gap / catch-up / needs-rebuild branching (3 tests) - Stale execution rejection during active recovery (2 tests) - Bounded catch-up: frozen target, duration, entries, stall (5 tests) - Completion before convergence rejected - Rebuild exclusivity: catch-up APIs excluded (1 test) - Rebuild lifecycle: snapshot+tail, full base, stale ID (3 tests) - Assignment-driven recovery flow Engine module now at 27 tests (12 Slice 1 + 15 Slice 2). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	3 days ago
pingqiu	61e9408261	fix: separate stable ReplicaID from Endpoint in registry Registry is now keyed by stable ReplicaID, not by address. DataAddr changes preserve sender identity — the core V2 invariant. Changes: - ReplicaAssignment{ReplicaID, Endpoint} replaces map[string]Endpoint - AssignmentIntent.Replicas uses []ReplicaAssignment - Registry.Reconcile takes []ReplicaAssignment - Tests use stable IDs ("replica-1", "r1") independent of addresses New test: ChangedDataAddr_PreservesSenderIdentity - Same ReplicaID, different DataAddr (10.0.0.1 → 10.0.0.2) - Sender pointer preserved, session invalidated, new session attached - This is the exact V1/V1.5 regression that V2 must fix doc.go: clarified Slice 1 core vs carried-forward files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	3 days ago
pingqiu	bb24b4b039	fix: encapsulate engine sender/session authority state All mutable state on Sender and Session is now unexported: - Sender.state, .epoch, .endpoint, .session, .stopped → accessors - Session.id, .phase, .kind, etc. → read-only accessors - Session() replaced by SessionSnapshot() (returns disconnected copy) - SessionID() and HasActiveSession() for common queries - AttachSession returns (sessionID, error) not (Session, error) - SupersedeSession returns sessionID not Session Budget configuration via SessionOption: - WithBudget(CatchUpBudget) passed to AttachSession - No direct field mutation on session from external code New test: Encapsulation_SnapshotIsReadOnly proves snapshot mutation does not leak back to sender state. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	3 days ago
pingqiu	20d70f9fb6	feat: add V2 engine replication core (Phase 05 Slice 1) Creates sw-block/engine/replication/ — the real V2 engine ownership core, promoted from sw-block/prototype/enginev2/ with all accepted invariants. Files: - types.go: Endpoint, ReplicaState, SessionKind, SessionPhase, FSM transitions - sender.go: per-replica Sender with full execution + rebuild APIs - session.go: Session with identity, phases, frozen target, truncation, budget - registry.go: Registry with reconcile + assignment intent + epoch invalidation - budget.go: CatchUpBudget (duration, entries, stall detection) - rebuild.go: RebuildState FSM (snapshot+tail vs full base) - outcome.go: HandshakeResult + ClassifyRecoveryOutcome Tests (ownership_test.go, 13 tests): - Changed-address invalidation (A10) - Stale session ID rejected at all APIs (A3) - Stale completion after supersede (A3) - Epoch bump invalidates all sessions (A3) - Stale assignment epoch rejected - Rebuild exclusivity (catch-up APIs rejected) - Rebuild full lifecycle - Frozen target rejects chase (A5) - Budget violation escalates (A5) - E2E: 3 replicas, 3 outcomes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	3 days ago
pingqiu	26a1b33c2e	feat: add A5-A8 acceptance traceability and rebuild-source evidence Cleanup: removed redundant TargetLSNAtStart from CatchUpBudget. FrozenTargetLSN on RecoverySession is the single source of truth. Acceptance traceability (acceptance_test.go): - A5: 3 evidence tests (unrecoverable gap, budget escalation, frozen target) - A6: 2 evidence tests (exact boundary, contiguity required) - A7: 3 evidence tests (snapshot history, catch-up replay, truncation) - A8: 2 evidence tests (convergence required, truncation required) Rebuild-source decision evidence: - snapshot_tail when trusted base exists - full_base when no snapshot or untrusted - 3 explicit tests 13 new tests total. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	3 days ago
pingqiu	8f5070679c	fix: make frozen target intrinsic and rebuild completion exclusive Frozen target is now unconditional: - FrozenTargetLSN field on RecoverySession, set by BeginCatchUp - RecordCatchUpProgress enforces FrozenTargetLSN regardless of Budget - Catch-up is always a bounded (R, H0] contract Rebuild completion exclusivity: - CompleteSessionByID explicitly rejects SessionRebuild by kind - Rebuild sessions can ONLY complete via CompleteRebuild Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	3 days ago
pingqiu	8e4028758f	fix: make rebuild path exclusive, enforce phase discipline, require tick for stall budget Rebuild exclusivity: - BeginCatchUp rejects SessionRebuild ("must use rebuild APIs") - RecordCatchUpProgress rejects SessionRebuild - Rebuild sessions can only be completed via CompleteRebuild - All legacy rebuild-through-catch-up paths in tests converted Phase discipline: - SelectRebuildSource requires session.Phase == PhaseHandshake - Cannot skip BeginConnect + RecordHandshake Stall budget: - RecordCatchUpProgress requires tick parameter when ProgressDeadlineTicks > 0 (no silent stall budget bypass) 3 new tests: rebuild exclusivity (catch-up APIs rejected), rebuild source requires handshake phase, stall budget requires tick. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	3 days ago
pingqiu	5b66a85f92	fix: wire rebuild FSM into sender, enforce frozen target, fix entry counting Rebuild execution path: - newRecoverySession auto-initializes RebuildState for SessionRebuild - Sender rebuild APIs: SelectRebuildSource, BeginRebuildTransfer, RecordRebuildTransferProgress, BeginRebuildTailReplay, RecordRebuildTailProgress, CompleteRebuild - All rebuild APIs are sender-authority-gated by sessionID - E2E rebuild test now drives through rebuild FSM, not catch-up APIs Bounded CatchUp enforcement: - BeginCatchUp freezes TargetLSNAtStart from session.TargetLSN - RecordCatchUpProgress rejects progress beyond frozen target - Entry counting uses LSN delta (recoveredTo - previous), not call count - Merged RecordCatchUpProgressAt into RecordCatchUpProgress (tick param) 5 new tests: target-frozen enforcement, sender-level rebuild via rebuild APIs, reject non-rebuild, reject stale ID on rebuild. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	3 days ago
pingqiu	3f0048cbd9	feat: add bounded CatchUp budget and Rebuild mode state machine (Phase 4.5 P0) Bounded CatchUp: - CatchUpBudget: MaxDurationTicks, MaxEntries, ProgressDeadlineTicks - BudgetCheck: runtime consumption tracker (StartTick, EntriesReplayed, LastProgressTick) - Sender.CheckBudget: evaluates budget, escalates to NeedsRebuild on violation - RecordCatchUpProgressAt: tracks progress tick for stall detection - BeginCatchUp accepts optional startTick for budget tracking Rebuild state machine: - RebuildSource: snapshot_tail (preferred) vs full_base (fallback) - RebuildPhase: init → source_select → transfer → tail_replay → completed\|aborted - SelectSource: chooses based on snapshot availability - Phase ordering enforced, transfer regression rejected - ReadyToComplete validates target reached 13 new tests: budget enforcement (duration, entries, stall, no-budget), sender budget integration, rebuild lifecycle (snapshot+tail, full base, abort, phase order, regression), E2E bounded catch-up → rebuild. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	3 days ago
pingqiu	90c39b549d	feat: add prototype scenario closure (Phase 04 P4) Maps V2 acceptance criteria A1-A7, A10 to enginev2 prototype evidence. Adds 4 V2-boundary scenarios against the prototype. Scenario tests: - A1: committed data survives promotion (WAL truncation boundary) - A2: uncommitted data truncated, not revived - A3: stale epoch fenced at sender + session + assignment layers - A4: short-gap catch-up with WAL-backed proof + data verification - A5: unrecoverable gap escalates to NeedsRebuild with proof - A6: recoverability boundary exact (tail +/- 1 LSN) - A7: historical data correct after tail advancement (snapshot) - A10: changed-address → invalidation → new assignment → recovery V2-boundary scenarios: - NeedsRebuild persists across topology update - catch-up does not overwrite safe data - 5 disconnect/reconnect cycles preserve sender identity - full V2 harness: 3 replicas, 3 outcomes (zero-gap, catch-up, rebuild) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	3 days ago
pingqiu	942a0b7da7	fix: strengthen IsRecoverable contiguity check and StateAt snapshot correctness IsRecoverable now verifies three conditions: - startExclusive >= tailLSN (not recycled) - endInclusive <= headLSN (within WAL) - all LSNs in range exist contiguously (no holes) StateAt now uses base snapshot captured during AdvanceTail: - returns nil for LSNs before snapshot boundary (unreconstructable) - correctly includes block state from recycled entries via snapshot 5 new tests: end-beyond-head, missing entries, state after tail advance, nil before snapshot, block last written before tail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	4 days ago
pingqiu	c89709e47e	feat: add WAL history model and recoverability proof (Phase 04 P3) Adds minimal historical-data prototype to enginev2: - WALHistory: retained-prefix model with Append, Commit, AdvanceTail, Truncate, EntriesInRange, IsRecoverable, StateAt - MakeHandshakeResult connects WAL state to outcome classification - RecordTruncation execution API for divergent tail cleanup - CompleteSessionByID gates on truncation when required - Zero-gap requires exact equality (FlushedLSN == CommittedLSN) - Replica-ahead classified as CatchUp with mandatory truncation 15 new tests: WAL basics, provable recoverability, unprovable gap, exact boundary, truncation enforcement, WAL-backed end-to-end recovery with data verification. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	4 days ago
pingqiu	edec7098e8	feat: add V2 protocol simulator and enginev2 sender/session prototype Adds sw-block/ directory with: - distsim: protocol correctness simulator (96 tests) - cluster model with epoch fencing, barrier semantics, commit modes - endpoint identity, control-plane flow, candidate eligibility - timeout events, timer races, same-tick ordering - session ownership tracking with ID-based stale fencing - enginev2: standalone V2 sender/session implementation (63 tests) - per-replica Sender with identity-preserving reconciliation - RecoverySession with FSM phase transitions and session ID - execution APIs: BeginConnect, RecordHandshake, BeginCatchUp, RecordCatchUpProgress, CompleteSessionByID — all sender-authority-gated - recovery outcome branching: zero-gap, catch-up, needs-rebuild - assignment-intent orchestration with epoch fencing - design docs: acceptance criteria, open questions, first-slice spec, protocol development process Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	4 days ago

19 Commits (47238df0d71682f47c8ec646bf1bd865a1487f20)