PR #3636
Sections
Review

fix: fence cancelled session recovery attempts

main ← feature/investigate-task-mes-65f 57 files +1842 −412 PR #3636 ↗

Cancelled resume attempts can no longer dispatch prompts, replace conversation identity, or stop a replacement execution; inconclusive session load errors now preserve the saved identity instead of falling back to a new session.

Why this change

A cancelled startup still dispatched its prompt and replaced the provider conversation. An inconclusive session/load timeout fell back to session/new and lost the saved identity. Late callbacks from the old attempt could stop a new attempt that reused the same execution.

What it does

Architecture, end to end

Resume now owns one attempt per session. The registry fences every callback and the lifecycle binds the attempt to the execution generation.

flowchart LR
  User[User retry or cancel] --> Orchestrator[Orchestrator resumeAttempt registry]
  Orchestrator --> Lifecycle[Lifecycle Manager]
  Lifecycle --> Agentctl[agentctl ACP]
  Agentctl --> Provider[Provider session]
  Orchestrator -- "invalidate + cancel" --> Lifecycle
  Lifecycle -- "AttemptID on events" --> Orchestrator
  Orchestrator -- "resumeAttemptAllowsExecution" --> Handlers[Event handlers]
  Handlers -- "cleanupCancelledResumeAttempt" --> Executor[Executor]

Key code changes

Drag to pan. Use the + and − buttons to zoom. Click a node to open the full code. The arrows show how the parts interact.

drag to pan · +/− to zoom · click a node for details

type AgentEventPayload struct
Click for details →

Every lifecycle payload now carries the immutable recovery attempt that owns the callback.

Payloads
type AgentEventPayload struct {
  AgentExecutionID   string `json:"agent_execution_id"`
  AttemptID          string `json:"attempt_id,omitempty"`
  RunID              string `json:"run_id,omitempty"`
  TaskID             string `json:"task_id"`
  SessionID          string `json:"session_id,omitempty"`
}

type AgentctlEventPayload struct {
  TaskID            string `json:"task_id"`
  SessionID         string `json:"session_id"`
  AgentExecutionID  string `json:"agent_execution_id"`
  AttemptID         string `json:"attempt_id,omitempty"`
}

type ACPSessionCreatedPayload struct {
  TaskID           string `json:"task_id"`
  SessionID        string `json:"session_id"`
  AgentExecutionID string `json:"agent_execution_id"`
  AttemptID        string `json:"attempt_id,omitempty"`
  ACPSessionID     string `json:"acp_session_id"`
}

type AgentStreamEventPayload struct {
  Type           string `json:"type"`
  AgentID        string `json:"agent_id"`
  ExecutionID    string `json:"execution_id"`
  AttemptID      string `json:"attempt_id,omitempty"`
  AgentProfileID string `json:"agent_profile_id,omitempty"`
  TaskID         string `json:"task_id"`
  SessionID      string `json:"session_id"`
  Data           *AgentStreamEventData `json:"data"`
}
type resumeAttemptRegistry struct
Click for details →

The registry owns one cancellable attempt per session and retains tombstones so late callbacks fail closed.

Registry
type resumeAttempt struct {
  id        uint64
  taskID    string
  sessionID string
  ctx       context.Context
  cancel    context.CancelFunc
  done      chan struct{}
  executionID string
  retained bool
}

type resumeAttemptRegistry struct {
  mu              sync.Mutex
  nextID          uint64
  attempts        map[string]*resumeAttempt
  tombstones      map[string][]resumeAttemptTombstone
  recoveryHistory map[string]struct{}
  latestExecution map[string]map[string]uint64
}

func (r *resumeAttemptRegistry) begin(parent context.Context, taskID, sessionID string) (*resumeAttempt, bool) {
  current := r.attempts[sessionID]
  if current != nil && current.ctx.Err() == nil {
    return current, false
  }
  if current != nil {
    r.retainLocked(current)
    delete(r.attempts, sessionID)
  }
  r.nextID++
  attemptCtx, cancel := context.WithCancel(parent)
  attempt := &resumeAttempt{id: r.nextID, taskID: taskID, sessionID: sessionID, ctx: attemptCtx, cancel: cancel, done: make(chan struct{})}
  r.attempts[sessionID] = attempt
  return attempt, true
}

func (s *Service) resumeAttemptAllowsExecution(sessionID, executionID string, origin ...string) bool {
  originID := resumeAttemptOrigin(origin)
  registry := s.resumeAttemptStore()
  registry.mu.Lock()
  defer registry.mu.Unlock()
  if current := registry.attempts[sessionID]; current != nil {
    return activeResumeAttemptAllowsExecutionLocked(registry, current, executionID, originID)
  }
  return resumeAttemptTombstoneAllowsExecution(registry, sessionID, executionID, originID)
}
Lifecycle binds attempt to execution generationapps/backend/internal/agent/runtime/lifecycle/types.go ↗
func (e *AgentExecution) withStartupAttempt(generation uint64, callback func(attemptID string)) bool
Click for details →

The execution keeps a per-generation attempt map and leases callbacks so a replacement cannot be mutated by an old attempt.

Execution state
type AgentExecution struct {
  ID              string
  ResumeAttemptID string
  startupAttemptGeneration uint64
  startupRecoveryStarted   bool
  startupAttemptIDs  map[uint64]string
  startupLifecycleMu sync.Mutex
  startupCallbackMu sync.RWMutex
}

func (e *AgentExecution) beginStartupAttemptWithID(attemptID string) uint64 {
  e.startupCallbackMu.Lock()
  defer e.startupCallbackMu.Unlock()
  e.startupLifecycleMu.Lock()
  defer e.startupLifecycleMu.Unlock()
  e.startupAttemptGeneration++
  e.startupRecoveryStarted = false
  e.recordStartupAttemptIDLocked(e.startupAttemptGeneration, attemptID)
  return e.startupAttemptGeneration
}

func (e *AgentExecution) withStartupAttempt(generation uint64, callback func(attemptID string)) bool {
  e.startupCallbackMu.RLock()
  defer e.startupCallbackMu.RUnlock()
  e.startupLifecycleMu.Lock()
  if e.startupAttemptGeneration != generation {
    e.startupLifecycleMu.Unlock()
    return false
  }
  attemptID := e.startupAttemptIDs[generation]
  if attemptID == "" {
    attemptID = e.ResumeAttemptID
  }
  e.startupLifecycleMu.Unlock()
  callback(attemptID)
  return true
}
Inconclusive load failures preserve identityapps/backend/internal/agent/runtime/lifecycle/session.go ↗
func (sm *SessionManager) createOrLoadSession(ctx context.Context, client *agentctl.Client, agentConfig agents.Agent, existingSessionID string, workspacePath string, mcpServers []agentctltypes.McpServer) (string, error)
Click for details →

Only confirmed unsupported or unknown session errors may fall back to session/new; timeouts and internal errors return without replacing the token.

Load decision
if rt.SessionConfig.NativeSessionResume && existingSessionID != "" {
  sessionID, err := sm.loadSession(ctx, client, agentConfig, existingSessionID, mcpServers)
  if err == nil {
    return sessionID, nil
  }
  if isTransportDeadErr(err) {
    return "", err
  }
  if !isSessionLoadFallbackErr(err) {
    sm.logger.Warn("session/load failed with an inconclusive error, preserving session identity",
      zap.String("existing_session_id", existingSessionID),
      zap.String("reason", err.Error()))
    return "", err
  }
  sm.logger.Warn("session/load failed, falling back to session/new",
    zap.String("existing_session_id", existingSessionID),
    zap.String("reason", err.Error()))
  return sm.createNewSession(ctx, client, agentConfig, workspacePath, mcpServers)
}
return sm.createNewSession(ctx, client, agentConfig, workspacePath, mcpServers)
func (s *Service) handleAgentBootReady(ctx context.Context, data watcher.AgentEventData)
Click for details →

Boot-ready and ready handlers reject stale attempts and cleanup removes only the exact cancelled execution.

Handler fence
func (s *Service) handleAgentBootReady(ctx context.Context, data watcher.AgentEventData) {
  if !s.resumeAttemptAllowsExecution(data.SessionID, data.AgentExecutionID, data.AttemptID) {
    s.logger.Debug("ignoring agent.boot_ready from a stale resume attempt",
      zap.String("session_id", data.SessionID),
      zap.String("attempt_id", data.AttemptID))
    return
  }
  // ... session state checks and drain
}

func (s *Service) cleanupCancelledResumeAttempt(attempt *resumeAttempt) {
  executionID := attempt.execution()
  if executionID == "" || !s.resumeAttemptStore().canCleanup(attempt) ||
    !s.claimForcedExecutionCleanup(attempt.sessionID, executionID) {
    return
  }
  cleanupCtx, cancel := context.WithTimeout(context.Background(), cancellationOperationTTL)
  defer cancel()
  s.executor.StopExecution(cleanupCtx, executionID, "cancelled resume startup", true)
}

func (s *Service) cleanupStaleResumeExecution(executionID, taskID, sessionID, attemptID string) {
  if !s.resumeAttemptStore().canCleanupIdentity(sessionID, executionID, attemptID) {
    return
  }
  go s.cleanupAgentExecution(executionID, taskID, sessionID)
}
type SessionRecoveryFailure struct
Click for details →

Pre-dispatch resume errors carry attempt, execution, and stamp identity so the UI shows one recovery card and suppresses duplicate banners.

Failure identity
type SessionRecoveryIdentity struct {
  AttemptID   string
  ExecutionID string
  ErrorStamp  string
}

type SessionRecoveryFailure struct {
  Err      error
  Identity SessionRecoveryIdentity
}

func (s *Service) withSessionRecoveryFailureIdentity(ctx context.Context, taskID, sessionID string, attempt *resumeAttempt, executionID string, err error) error {
  if err == nil || errors.Is(err, ErrResumeAttemptCancelled) {
    return err
  }
  identity := sessionRecoveryIdentity(attempt, executionID)
  identity.ErrorStamp = s.matchingSessionRecoveryErrorStamp(ctx, taskID, sessionID, identity)
  return &SessionRecoveryFailure{Err: err, Identity: identity}
}

func (s *Service) HasActiveSessionRecoveryForFailure(ctx context.Context, taskID, sessionID string, failure error) bool {
  identity, ok := RecoveryFailureIdentity(failure)
  if !ok {
    return false
  }
  lastError, ok := models.LoadLastAgentError(session.Metadata)
  return ok && !lastError.IsDismissed() && recoveryIdentityMatchesLastError(identity, lastError)
}
Read the changes as a list

AttemptID on every lifecycle event

apps/backend/internal/agent/runtime/lifecycle/event_types.go

Every lifecycle payload now carries the immutable recovery attempt that owns the callback.

Payloads
type AgentEventPayload struct {
  AgentExecutionID   string `json:"agent_execution_id"`
  AttemptID          string `json:"attempt_id,omitempty"`
  RunID              string `json:"run_id,omitempty"`
  TaskID             string `json:"task_id"`
  SessionID          string `json:"session_id,omitempty"`
}

type AgentctlEventPayload struct {
  TaskID            string `json:"task_id"`
  SessionID         string `json:"session_id"`
  AgentExecutionID  string `json:"agent_execution_id"`
  AttemptID         string `json:"attempt_id,omitempty"`
}

type ACPSessionCreatedPayload struct {
  TaskID           string `json:"task_id"`
  SessionID        string `json:"session_id"`
  AgentExecutionID string `json:"agent_execution_id"`
  AttemptID        string `json:"attempt_id,omitempty"`
  ACPSessionID     string `json:"acp_session_id"`
}

type AgentStreamEventPayload struct {
  Type           string `json:"type"`
  AgentID        string `json:"agent_id"`
  ExecutionID    string `json:"execution_id"`
  AttemptID      string `json:"attempt_id,omitempty"`
  AgentProfileID string `json:"agent_profile_id,omitempty"`
  TaskID         string `json:"task_id"`
  SessionID      string `json:"session_id"`
  Data           *AgentStreamEventData `json:"data"`
}

Process-local resume attempt registry

apps/backend/internal/orchestrator/resume_attempt.go

The registry owns one cancellable attempt per session and retains tombstones so late callbacks fail closed.

Registry
type resumeAttempt struct {
  id        uint64
  taskID    string
  sessionID string
  ctx       context.Context
  cancel    context.CancelFunc
  done      chan struct{}
  executionID string
  retained bool
}

type resumeAttemptRegistry struct {
  mu              sync.Mutex
  nextID          uint64
  attempts        map[string]*resumeAttempt
  tombstones      map[string][]resumeAttemptTombstone
  recoveryHistory map[string]struct{}
  latestExecution map[string]map[string]uint64
}

func (r *resumeAttemptRegistry) begin(parent context.Context, taskID, sessionID string) (*resumeAttempt, bool) {
  current := r.attempts[sessionID]
  if current != nil && current.ctx.Err() == nil {
    return current, false
  }
  if current != nil {
    r.retainLocked(current)
    delete(r.attempts, sessionID)
  }
  r.nextID++
  attemptCtx, cancel := context.WithCancel(parent)
  attempt := &resumeAttempt{id: r.nextID, taskID: taskID, sessionID: sessionID, ctx: attemptCtx, cancel: cancel, done: make(chan struct{})}
  r.attempts[sessionID] = attempt
  return attempt, true
}

func (s *Service) resumeAttemptAllowsExecution(sessionID, executionID string, origin ...string) bool {
  originID := resumeAttemptOrigin(origin)
  registry := s.resumeAttemptStore()
  registry.mu.Lock()
  defer registry.mu.Unlock()
  if current := registry.attempts[sessionID]; current != nil {
    return activeResumeAttemptAllowsExecutionLocked(registry, current, executionID, originID)
  }
  return resumeAttemptTombstoneAllowsExecution(registry, sessionID, executionID, originID)
}

Lifecycle binds attempt to execution generation

apps/backend/internal/agent/runtime/lifecycle/types.go

The execution keeps a per-generation attempt map and leases callbacks so a replacement cannot be mutated by an old attempt.

Execution state
type AgentExecution struct {
  ID              string
  ResumeAttemptID string
  startupAttemptGeneration uint64
  startupRecoveryStarted   bool
  startupAttemptIDs  map[uint64]string
  startupLifecycleMu sync.Mutex
  startupCallbackMu sync.RWMutex
}

func (e *AgentExecution) beginStartupAttemptWithID(attemptID string) uint64 {
  e.startupCallbackMu.Lock()
  defer e.startupCallbackMu.Unlock()
  e.startupLifecycleMu.Lock()
  defer e.startupLifecycleMu.Unlock()
  e.startupAttemptGeneration++
  e.startupRecoveryStarted = false
  e.recordStartupAttemptIDLocked(e.startupAttemptGeneration, attemptID)
  return e.startupAttemptGeneration
}

func (e *AgentExecution) withStartupAttempt(generation uint64, callback func(attemptID string)) bool {
  e.startupCallbackMu.RLock()
  defer e.startupCallbackMu.RUnlock()
  e.startupLifecycleMu.Lock()
  if e.startupAttemptGeneration != generation {
    e.startupLifecycleMu.Unlock()
    return false
  }
  attemptID := e.startupAttemptIDs[generation]
  if attemptID == "" {
    attemptID = e.ResumeAttemptID
  }
  e.startupLifecycleMu.Unlock()
  callback(attemptID)
  return true
}

Inconclusive load failures preserve identity

apps/backend/internal/agent/runtime/lifecycle/session.go

Only confirmed unsupported or unknown session errors may fall back to session/new; timeouts and internal errors return without replacing the token.

Load decision
if rt.SessionConfig.NativeSessionResume && existingSessionID != "" {
  sessionID, err := sm.loadSession(ctx, client, agentConfig, existingSessionID, mcpServers)
  if err == nil {
    return sessionID, nil
  }
  if isTransportDeadErr(err) {
    return "", err
  }
  if !isSessionLoadFallbackErr(err) {
    sm.logger.Warn("session/load failed with an inconclusive error, preserving session identity",
      zap.String("existing_session_id", existingSessionID),
      zap.String("reason", err.Error()))
    return "", err
  }
  sm.logger.Warn("session/load failed, falling back to session/new",
    zap.String("existing_session_id", existingSessionID),
    zap.String("reason", err.Error()))
  return sm.createNewSession(ctx, client, agentConfig, workspacePath, mcpServers)
}
return sm.createNewSession(ctx, client, agentConfig, workspacePath, mcpServers)

Fenced event handlers and exact cleanup

apps/backend/internal/orchestrator/event_handlers_agent.go

Boot-ready and ready handlers reject stale attempts and cleanup removes only the exact cancelled execution.

Handler fence
func (s *Service) handleAgentBootReady(ctx context.Context, data watcher.AgentEventData) {
  if !s.resumeAttemptAllowsExecution(data.SessionID, data.AgentExecutionID, data.AttemptID) {
    s.logger.Debug("ignoring agent.boot_ready from a stale resume attempt",
      zap.String("session_id", data.SessionID),
      zap.String("attempt_id", data.AttemptID))
    return
  }
  // ... session state checks and drain
}

func (s *Service) cleanupCancelledResumeAttempt(attempt *resumeAttempt) {
  executionID := attempt.execution()
  if executionID == "" || !s.resumeAttemptStore().canCleanup(attempt) ||
    !s.claimForcedExecutionCleanup(attempt.sessionID, executionID) {
    return
  }
  cleanupCtx, cancel := context.WithTimeout(context.Background(), cancellationOperationTTL)
  defer cancel()
  s.executor.StopExecution(cleanupCtx, executionID, "cancelled resume startup", true)
}

func (s *Service) cleanupStaleResumeExecution(executionID, taskID, sessionID, attemptID string) {
  if !s.resumeAttemptStore().canCleanupIdentity(sessionID, executionID, attemptID) {
    return
  }
  go s.cleanupAgentExecution(executionID, taskID, sessionID)
}

Correlated recovery failure presentation

apps/backend/internal/orchestrator/session_recovery_feedback.go

Pre-dispatch resume errors carry attempt, execution, and stamp identity so the UI shows one recovery card and suppresses duplicate banners.

Failure identity
type SessionRecoveryIdentity struct {
  AttemptID   string
  ExecutionID string
  ErrorStamp  string
}

type SessionRecoveryFailure struct {
  Err      error
  Identity SessionRecoveryIdentity
}

func (s *Service) withSessionRecoveryFailureIdentity(ctx context.Context, taskID, sessionID string, attempt *resumeAttempt, executionID string, err error) error {
  if err == nil || errors.Is(err, ErrResumeAttemptCancelled) {
    return err
  }
  identity := sessionRecoveryIdentity(attempt, executionID)
  identity.ErrorStamp = s.matchingSessionRecoveryErrorStamp(ctx, taskID, sessionID, identity)
  return &SessionRecoveryFailure{Err: err, Identity: identity}
}

func (s *Service) HasActiveSessionRecoveryForFailure(ctx context.Context, taskID, sessionID string, failure error) bool {
  identity, ok := RecoveryFailureIdentity(failure)
  if !ok {
    return false
  }
  lastError, ok := models.LoadLastAgentError(session.Metadata)
  return ok && !lastError.IsDismissed() && recoveryIdentityMatchesLastError(identity, lastError)
}

Data and storage

Attempt identity is process-local and never persisted. Tombstones and stamps correlate the durable error record.

FieldTypeNotes
resumeAttempt.iduint64monotonic per session, forms resume-<id> identity
resumeAttempt.executionIDstringbound on first callback while registry lock is held
resumeAttemptTombstonestructretained after finish, max 16 per session
AgentExecution.startupAttemptIDsmap[uint64]stringper-generation attempt, bounded to last 8
SessionRecoveryIdentitystructAttemptID + ExecutionID + ErrorStamp for UI correlation
LastAgentError.Stampstringdurable error stamp matched to recovery identity

Risk

6 / 10 Medium
1 low5 medium10 high

Why this score

  • Touches startup, cancellation, and event paths that run concurrently across orchestrator and lifecycle.
  • Fencing is fail-closed; a misclassified load error or missing AttemptID could block a valid resume.
  • Covered by barrier tests for timeout, cancellation, late callbacks, browser disconnect, and mobile/desktop E2E.

Trade-offs and review notes

Where to look first

  1. Verify resume_attempt.go fencing: begin, isCurrent, canCleanup, and tombstone fail-closed logic.
  2. Check lifecycle generation lease: withStartupAttempt, startupCallbackMu ordering, and AttemptID propagation in events.go.
  3. Confirm session.go load classification: isSessionLoadFallbackErr covers only confirmed fallback cases.
  4. Review event_handlers_agent.go and task_operations.go for resumeAttemptAllowsExecution and cancellation guard ordering.