PR #4432
Sections
Review

fix: reconcile interrupted shutdown delivery without replay ↗

main ← fix/durable-shutdown-recovery 56 files +1240 −380 PR #4432 ↗

Interrupted shutdown prompts now settle and retire from journal proof instead of replaying payloads.

Why this PR exists

An interrupted prompt has unclear outcome at shutdown. A naive restart can resend it or stall the session.

Who is affected: Users who resume a session after backend shutdown or restart.

Result: Sessions restore from proof and accept new prompts. Old prompts never send twice.

What it does

Impact at a glance

Breaking changes

Affected users / triggerBeforeAfterRequired actionSource
Users who restart during an interrupted prompt with incomplete evidenceRestart could replay the prompt payload or proceed without full proof.Resume refuses with a dependency error when proof is missing or identity changed.Provide exact journal and SQL evidence before resume, or keep the session parked.apps/backend/internal/agent/runtime/lifecycle/native_resume_delivery.go

Feature flags and rollout

The recovery fix ships without a flag. Shutdown settlement, retirement checks, and restore admission run for every session.

No UX changes with flags off

No flag gates this fix. The disabled path is the same as the enabled path.

Data and storage

The proof carries session identity and cursor state. It adds no new tables.

FieldTypeNotes
NativeResumeDeliveryEvidencestructsession, incarnation, generation, stream, and native session proof
SubmissionSummarystructbounded non-secret identity of one retained prompt
RecoveryDescriptorstructowner snapshot with stream watermark and submission list

Architecture, end to end

Shutdown settles the journal first. Lifecycle proves the dead owner. Orchestrator admits restore only with proof.

flowchart LR
  Shutdown[Process shutdown settlement] --> Journal[agentctl journal bbolt]
  Journal --> Lifecycle[Lifecycle native resume]
  Lifecycle --> Orchestrator[Orchestrator restart recovery]
  Orchestrator --> SQL[(SQLite delivery rows)]
  Lifecycle -- retire exact unknown --> Journal

Key code changes

Drag to pan. Use the + and − buttons to zoom. Click a node to open the full code. The arrows show how the parts interact.

drag to pan · +/− to zoom · click a node for details

func HasUnprovenOutcome(submission Submission) bool
Click for details →

The helper marks only history without a proven terminal result as eligible for restore.

Outcome gate
func HasUnprovenOutcome(submission Submission) bool {
  if submission.State == SubmissionInterruptedUnknown {
    return true
  }
  if submission.TerminalEventRetained || submission.TerminalSequence != 0 {
    return false
  }
  switch submission.State {
  case SubmissionFailed, SubmissionCompleted, SubmissionCancelled:
    return true
  default:
    return false
  }
}
Retired history stays out of recoveryapps/backend/internal/agentctl/journal/recovery.go ↗
func submissionNeedsRecovery(submission Submission) bool
Click for details →

The descriptor hides retired uncertain history so repeat adoption uses the same rule.

Recovery filter
func submissionNeedsRecovery(submission Submission) bool {
  if submission.Retired && (submission.State == SubmissionInterruptedUnknown || HasUnprovenOutcome(submission)) {
    return false
  }
  switch submission.State {
  case SubmissionPrepared, SubmissionAccepted, SubmissionDispatching, SubmissionInterruptedUnknown:
    return true
  case SubmissionCompleted:
    return !submission.TerminalEventRetained
  default:
    return false
  }
}
func nativeResumeRetirementIDs(execution *AgentExecution, status *agentctl.DeliveryStatus, submissions []journal.Submission, rows []*models.AgentDeliverySubmission) ([]string, error)
Click for details →

The selector retires a prompt only when journal and SQL agree on identity and uncertainty.

Retirement selection
case journal.SubmissionInterruptedUnknown, journal.SubmissionCompleted, journal.SubmissionFailed, journal.SubmissionCancelled:
  if !journal.HasUnprovenOutcome(submission) {
    continue
  }
  sqlSubmission, found := backend[submission.ID]
  if !found || sqlSubmission.State != models.DeliverySubmissionInterruptedUnknown {
    if submission.State == journal.SubmissionFailed || submission.State == journal.SubmissionCancelled {
      continue
    }
    return nil, fmt.Errorf("%w: native resume requires uncertain SQL history", ErrDurableAdoptionBlocked)
  }
func (m *Manager) ReconcileLoadedNativeDelivery(ctx context.Context, sessionID string) (*models.NativeResumeDeliveryEvidence, error)
Click for details →

The path finishes retirement for an idle native successor from the stored prompt binding.

Loaded successor
func (m *Manager) ReconcileLoadedNativeDelivery(ctx context.Context, sessionID string) (*models.NativeResumeDeliveryEvidence, error) {
  execution, exists := m.GetExecutionBySessionID(sessionID)
  if !exists || execution == nil {
    return nil, ErrDurableAdoptionEvidenceUnavailable
  }
  if !execution.promptMu.TryLock() {
    return nil, ErrDeliveryRecoveryBlocked
  }
  defer execution.promptMu.Unlock()
  if execution.Status != v1.AgentStatusReady || execution.dispatchedPromptPending.Load() {
    return nil, ErrDeliveryRecoveryBlocked
  }
Legacy failed history rejoins native resumeapps/backend/internal/orchestrator/legacy_native_restart.go ↗
func (s *Service) resumeLegacyNativeRestart(ctx context.Context, session *models.TaskSession) (bool, error)
Click for details →

The entry rechecks retained evidence and stream projection before it resumes a legacy session.

Legacy entry
func (s *Service) resumeLegacyNativeRestart(ctx context.Context, session *models.TaskSession) (bool, error) {
  if session == nil || session.State != models.TaskSessionStateWaitingForInput || !silentRestoreSessionRouteEligible(session) {
    return false, nil
  }
  source, err := s.legacyNativeRestartSource(ctx, session)
  if err != nil || source == nil {
    return false, err
  }
Read the changes as a list

Unproven outcome gate

apps/backend/internal/agentctl/journal/submission_outcome.go

The helper marks only history without a proven terminal result as eligible for restore.

Outcome gate
func HasUnprovenOutcome(submission Submission) bool {
  if submission.State == SubmissionInterruptedUnknown {
    return true
  }
  if submission.TerminalEventRetained || submission.TerminalSequence != 0 {
    return false
  }
  switch submission.State {
  case SubmissionFailed, SubmissionCompleted, SubmissionCancelled:
    return true
  default:
    return false
  }
}

Retired history stays out of recovery

apps/backend/internal/agentctl/journal/recovery.go

The descriptor hides retired uncertain history so repeat adoption uses the same rule.

Recovery filter
func submissionNeedsRecovery(submission Submission) bool {
  if submission.Retired && (submission.State == SubmissionInterruptedUnknown || HasUnprovenOutcome(submission)) {
    return false
  }
  switch submission.State {
  case SubmissionPrepared, SubmissionAccepted, SubmissionDispatching, SubmissionInterruptedUnknown:
    return true
  case SubmissionCompleted:
    return !submission.TerminalEventRetained
  default:
    return false
  }
}

Retire exact unknown prompts only

apps/backend/internal/agent/runtime/lifecycle/native_resume_delivery.go

The selector retires a prompt only when journal and SQL agree on identity and uncertainty.

Retirement selection
case journal.SubmissionInterruptedUnknown, journal.SubmissionCompleted, journal.SubmissionFailed, journal.SubmissionCancelled:
  if !journal.HasUnprovenOutcome(submission) {
    continue
  }
  sqlSubmission, found := backend[submission.ID]
  if !found || sqlSubmission.State != models.DeliverySubmissionInterruptedUnknown {
    if submission.State == journal.SubmissionFailed || submission.State == journal.SubmissionCancelled {
      continue
    }
    return nil, fmt.Errorf("%w: native resume requires uncertain SQL history", ErrDurableAdoptionBlocked)
  }

Reuse an already loaded successor

apps/backend/internal/agent/runtime/lifecycle/loaded_native_reconciliation.go

The path finishes retirement for an idle native successor from the stored prompt binding.

Loaded successor
func (m *Manager) ReconcileLoadedNativeDelivery(ctx context.Context, sessionID string) (*models.NativeResumeDeliveryEvidence, error) {
  execution, exists := m.GetExecutionBySessionID(sessionID)
  if !exists || execution == nil {
    return nil, ErrDurableAdoptionEvidenceUnavailable
  }
  if !execution.promptMu.TryLock() {
    return nil, ErrDeliveryRecoveryBlocked
  }
  defer execution.promptMu.Unlock()
  if execution.Status != v1.AgentStatusReady || execution.dispatchedPromptPending.Load() {
    return nil, ErrDeliveryRecoveryBlocked
  }

Legacy failed history rejoins native resume

apps/backend/internal/orchestrator/legacy_native_restart.go

The entry rechecks retained evidence and stream projection before it resumes a legacy session.

Legacy entry
func (s *Service) resumeLegacyNativeRestart(ctx context.Context, session *models.TaskSession) (bool, error) {
  if session == nil || session.State != models.TaskSessionStateWaitingForInput || !silentRestoreSessionRouteEligible(session) {
    return false, nil
  }
  source, err := s.legacyNativeRestartSource(ctx, session)
  if err != nil || source == nil {
    return false, err
  }

Risk

6 / 10 Medium
1 low5 medium10 high

Why this score

  • Recovery touches shutdown, journal, lifecycle, orchestrator, and SQL in one change.
  • A wrong retirement choice could hide work or block a valid resume.
  • New matrix, shutdown, and binding tests cover the main paths, but live shutdown timing is hard to reproduce.

Trade-offs and review notes

Where to look first

  1. Read native_resume_delivery.go for retirement selection and the no-replay rule.
  2. Read loaded_native_reconciliation.go for the already-loaded successor path.
  3. Read submission_outcome.go and recovery.go for the shared outcome gate.
  4. Read legacy_native_restart.go for the legacy failed-history entry.
  5. Check the state matrix and shutdown tests for refused versus restored cases.