main ← fix/durable-shutdown-recovery
56 files
+1240
−380
PR #4432 ↗
Interrupted shutdown prompts now settle and retire from journal proof instead of replaying payloads.
Why this PR exists
An interrupted prompt has unclear outcome at shutdown. A naive restart can resend it or stall the session.
Who is affected: Users who resume a session after backend shutdown or restart.
Result: Sessions restore from proof and accept new prompts. Old prompts never send twice.
What it does
- Settle shutdown prompts as uncertain before the adapter closes.
- Retire only journal and SQL records with exact matching identity.
- Reuse an idle loaded successor from the stored prompt binding.
Impact at a glance
- Breaking changes: Breaking changes detected (1)
- User experience: No changes found
- Kandev plugin interfaces: No changes found
- MCP tools: No changes found
- Database changes: No changes found
Breaking changes
| Affected users / trigger | Before | After | Required action | Source |
|---|
| Users who restart during an interrupted prompt with incomplete evidence | Restart could replay the prompt payload or proceed without full proof. | Resume refuses with a dependency error when proof is missing or identity changed. | Provide exact journal and SQL evidence before resume, or keep the session parked. | apps/backend/internal/agent/runtime/lifecycle/native_resume_delivery.go |
Feature flags and rollout
- Coverage: Feature behavior ships without flags
- Flag-off UX: No UX changes with flags off
The recovery fix ships without a flag. Shutdown settlement, retirement checks, and restore admission run for every session.
No UX changes with flags off
No flag gates this fix. The disabled path is the same as the enabled path.
Data and storage
The proof carries session identity and cursor state. It adds no new tables.
| Field | Type | Notes |
|---|
| NativeResumeDeliveryEvidence | struct | session, incarnation, generation, stream, and native session proof |
| SubmissionSummary | struct | bounded non-secret identity of one retained prompt |
| RecoveryDescriptor | struct | owner snapshot with stream watermark and submission list |
Architecture, end to end
Shutdown settles the journal first. Lifecycle proves the dead owner. Orchestrator admits restore only with proof.
flowchart LR
Shutdown[Process shutdown settlement] --> Journal[agentctl journal bbolt]
Journal --> Lifecycle[Lifecycle native resume]
Lifecycle --> Orchestrator[Orchestrator restart recovery]
Orchestrator --> SQL[(SQLite delivery rows)]
Lifecycle -- retire exact unknown --> Journal
Key code changes
Drag to pan. Use the + and − buttons to zoom. Click a node to open the full code. The arrows show how the parts interact.
drag to pan · +/− to zoom · click a node for details
func HasUnprovenOutcome(submission Submission) bool
Click for details →
The helper marks only history without a proven terminal result as eligible for restore.
Outcome gate
func HasUnprovenOutcome(submission Submission) bool {
if submission.State == SubmissionInterruptedUnknown {
return true
}
if submission.TerminalEventRetained || submission.TerminalSequence != 0 {
return false
}
switch submission.State {
case SubmissionFailed, SubmissionCompleted, SubmissionCancelled:
return true
default:
return false
}
}
func submissionNeedsRecovery(submission Submission) bool
Click for details →
The descriptor hides retired uncertain history so repeat adoption uses the same rule.
Recovery filter
func submissionNeedsRecovery(submission Submission) bool {
if submission.Retired && (submission.State == SubmissionInterruptedUnknown || HasUnprovenOutcome(submission)) {
return false
}
switch submission.State {
case SubmissionPrepared, SubmissionAccepted, SubmissionDispatching, SubmissionInterruptedUnknown:
return true
case SubmissionCompleted:
return !submission.TerminalEventRetained
default:
return false
}
}
func nativeResumeRetirementIDs(execution *AgentExecution, status *agentctl.DeliveryStatus, submissions []journal.Submission, rows []*models.AgentDeliverySubmission) ([]string, error)
Click for details →
The selector retires a prompt only when journal and SQL agree on identity and uncertainty.
Retirement selection
case journal.SubmissionInterruptedUnknown, journal.SubmissionCompleted, journal.SubmissionFailed, journal.SubmissionCancelled:
if !journal.HasUnprovenOutcome(submission) {
continue
}
sqlSubmission, found := backend[submission.ID]
if !found || sqlSubmission.State != models.DeliverySubmissionInterruptedUnknown {
if submission.State == journal.SubmissionFailed || submission.State == journal.SubmissionCancelled {
continue
}
return nil, fmt.Errorf("%w: native resume requires uncertain SQL history", ErrDurableAdoptionBlocked)
}
func (m *Manager) ReconcileLoadedNativeDelivery(ctx context.Context, sessionID string) (*models.NativeResumeDeliveryEvidence, error)
Click for details →
The path finishes retirement for an idle native successor from the stored prompt binding.
Loaded successor
func (m *Manager) ReconcileLoadedNativeDelivery(ctx context.Context, sessionID string) (*models.NativeResumeDeliveryEvidence, error) {
execution, exists := m.GetExecutionBySessionID(sessionID)
if !exists || execution == nil {
return nil, ErrDurableAdoptionEvidenceUnavailable
}
if !execution.promptMu.TryLock() {
return nil, ErrDeliveryRecoveryBlocked
}
defer execution.promptMu.Unlock()
if execution.Status != v1.AgentStatusReady || execution.dispatchedPromptPending.Load() {
return nil, ErrDeliveryRecoveryBlocked
}
func (s *Service) resumeLegacyNativeRestart(ctx context.Context, session *models.TaskSession) (bool, error)
Click for details →
The entry rechecks retained evidence and stream projection before it resumes a legacy session.
Legacy entry
func (s *Service) resumeLegacyNativeRestart(ctx context.Context, session *models.TaskSession) (bool, error) {
if session == nil || session.State != models.TaskSessionStateWaitingForInput || !silentRestoreSessionRouteEligible(session) {
return false, nil
}
source, err := s.legacyNativeRestartSource(ctx, session)
if err != nil || source == nil {
return false, err
}
Read the changes as a list
Unproven outcome gate
apps/backend/internal/agentctl/journal/submission_outcome.go ↗The helper marks only history without a proven terminal result as eligible for restore.
Outcome gate
func HasUnprovenOutcome(submission Submission) bool {
if submission.State == SubmissionInterruptedUnknown {
return true
}
if submission.TerminalEventRetained || submission.TerminalSequence != 0 {
return false
}
switch submission.State {
case SubmissionFailed, SubmissionCompleted, SubmissionCancelled:
return true
default:
return false
}
}
Retired history stays out of recovery
apps/backend/internal/agentctl/journal/recovery.go ↗The descriptor hides retired uncertain history so repeat adoption uses the same rule.
Recovery filter
func submissionNeedsRecovery(submission Submission) bool {
if submission.Retired && (submission.State == SubmissionInterruptedUnknown || HasUnprovenOutcome(submission)) {
return false
}
switch submission.State {
case SubmissionPrepared, SubmissionAccepted, SubmissionDispatching, SubmissionInterruptedUnknown:
return true
case SubmissionCompleted:
return !submission.TerminalEventRetained
default:
return false
}
}
Retire exact unknown prompts only
apps/backend/internal/agent/runtime/lifecycle/native_resume_delivery.go ↗The selector retires a prompt only when journal and SQL agree on identity and uncertainty.
Retirement selection
case journal.SubmissionInterruptedUnknown, journal.SubmissionCompleted, journal.SubmissionFailed, journal.SubmissionCancelled:
if !journal.HasUnprovenOutcome(submission) {
continue
}
sqlSubmission, found := backend[submission.ID]
if !found || sqlSubmission.State != models.DeliverySubmissionInterruptedUnknown {
if submission.State == journal.SubmissionFailed || submission.State == journal.SubmissionCancelled {
continue
}
return nil, fmt.Errorf("%w: native resume requires uncertain SQL history", ErrDurableAdoptionBlocked)
}
Reuse an already loaded successor
apps/backend/internal/agent/runtime/lifecycle/loaded_native_reconciliation.go ↗The path finishes retirement for an idle native successor from the stored prompt binding.
Loaded successor
func (m *Manager) ReconcileLoadedNativeDelivery(ctx context.Context, sessionID string) (*models.NativeResumeDeliveryEvidence, error) {
execution, exists := m.GetExecutionBySessionID(sessionID)
if !exists || execution == nil {
return nil, ErrDurableAdoptionEvidenceUnavailable
}
if !execution.promptMu.TryLock() {
return nil, ErrDeliveryRecoveryBlocked
}
defer execution.promptMu.Unlock()
if execution.Status != v1.AgentStatusReady || execution.dispatchedPromptPending.Load() {
return nil, ErrDeliveryRecoveryBlocked
}
Legacy failed history rejoins native resume
apps/backend/internal/orchestrator/legacy_native_restart.go ↗The entry rechecks retained evidence and stream projection before it resumes a legacy session.
Legacy entry
func (s *Service) resumeLegacyNativeRestart(ctx context.Context, session *models.TaskSession) (bool, error) {
if session == nil || session.State != models.TaskSessionStateWaitingForInput || !silentRestoreSessionRouteEligible(session) {
return false, nil
}
source, err := s.legacyNativeRestartSource(ctx, session)
if err != nil || source == nil {
return false, err
}
Risk
6
/ 10
Medium
Why this score
- Recovery touches shutdown, journal, lifecycle, orchestrator, and SQL in one change.
- A wrong retirement choice could hide work or block a valid resume.
- New matrix, shutdown, and binding tests cover the main paths, but live shutdown timing is hard to reproduce.
Trade-offs and review notes
- Strict proof refuses more resumes, but it stops duplicate prompt sends.
- Persisted bindings allow a loaded successor to finish, but they add a second recovery entry.
- Legacy failed labels stay visible as history, but reviewers must read two labels.
Where to look first
- Read native_resume_delivery.go for retirement selection and the no-replay rule.
- Read loaded_native_reconciliation.go for the already-loaded successor path.
- Read submission_outcome.go and recovery.go for the shared outcome gate.
- Read legacy_native_restart.go for the legacy failed-history entry.
- Check the state matrix and shutdown tests for refused versus restored cases.