PR #3529
Sections
Review

fix(backend): prevent duplicate or stranded launches when tasks auto-start on create

main ← feature/harden-auto-start-re-q57 6 files +412 −78 PR #3529 ↗

Fix duplicate and stranded auto-start launches by claiming the create-time opt-in token synchronously and restoring it on every early failure, and by separating dependency gate errors from genuine blocks.

Why this change

Tasks created with auto_start_on_create and a queue promotion token could launch twice or strand with no retry marker. A stale read in the startup sweep and missing restores on early failures left tasks with no durable token for the next sweep.

What it does

Architecture, end to end

Create-time auto-start now flows through one chokepoint. The token is claimed before any async work, and every early exit restores it.

flowchart LR
  Created[task.created] --> HandleCreated[handleTaskCreated]
  HandleCreated --> Claim1[claim MetaKeyAutoStartOnCreate]
  Claim1 --> AutoStart[autoStartTaskForStep]
  AutoStart --> Gate{dependency gate}
  Gate -- blocked genuine --> Wait[wait for predecessor]
  Gate -- read error --> Restore1[restore token]
  Gate -- ok --> CheckPromo{has promotion?}
  CheckPromo -- yes --> Promo[handleTaskQueuePromotedWithClaimed]
  CheckPromo -- no --> Claim2[claimAutoStartOnCreateForLaunch]
  Claim2 --> Launch[async StartTask / queueOfficeRun]
  Launch -- fail --> Restore2[handleAutoStartFailure restores tokens]
  Launch -- ok --> Session[(session or run)]
  Sweep[reconcileTaskLifecycleTokens] --> Recover[recoverTaskLifecycleAttempt]
  Recover --> Promo
  Recover --> Claim2

Key code changes

Drag to pan. Use the + and − buttons to zoom. Click a node to open the full code. The arrows show how the parts interact.

drag to pan · +/− to zoom · click a node for details

Split dependency gate into blocked and gateErroredapps/backend/internal/orchestrator/event_handlers_dependencies.go ↗
func (s *Service) dependencyBlocksAutoStart(ctx context.Context, taskID, eventName string) (blocked, gateErrored bool)
Click for details →

The gate now tells callers whether a block came from a read error or a real dependency, so only errors restore the one-shot token.

New signature and comment
func (s *Service) dependencyBlocksAutoStart(ctx context.Context, taskID, eventName string) (blocked, gateErrored bool) {
  if s.dependencyReader == nil {
    return false, false
  }
  isBlocked, reason, err := s.dependencyReader.DependencyGate(ctx, taskID)
  if err != nil {
    s.logger.Warn(eventName+": dependency lookup failed; skipping auto-start",
      zap.String("task_id", taskID), zap.Error(err))
    return true, true
  }
  if isBlocked {
    s.logger.Debug(eventName+": task has unresolved dependencies; skipping auto-start",
      zap.String("task_id", taskID), zap.String("blocked_reason", reason))
  }
  return isBlocked, false
}
Carry claimed token through auto-start chokepointapps/backend/internal/orchestrator/event_handlers_workflow.go ↗
func (s *Service) autoStartTaskForStep(ctx context.Context, taskID, stepID, eventName string, stepTransitionID int64, autoStartOnCreateClaimed bool)
Click for details →

The chokepoint now knows if the caller already claimed the create token, and it restores the token on every early return that would otherwise strand the task.

Signature and early restores
func (s *Service) autoStartTaskForStep(ctx context.Context, taskID, stepID, eventName string, stepTransitionID int64, autoStartOnCreateClaimed bool) {
  task, err := s.repo.GetTask(ctx, taskID)
  if err != nil {
    s.logger.Warn(eventName+": failed to load task for auto-start", zap.String("task_id", taskID), zap.Error(err))
    if autoStartOnCreateClaimed {
      s.restoreTaskLifecycleToken(ctx, taskID, models.MetaKeyAutoStartOnCreate, true, eventName)
    }
    return
  }
  if blocked, gateErrored := s.dependencyBlocksAutoStart(ctx, taskID, eventName); blocked {
    if gateErrored && autoStartOnCreateClaimed {
      s.restoreTaskLifecycleToken(ctx, taskID, models.MetaKeyAutoStartOnCreate, true, eventName)
    }
    return
  }
  if hasQueuePromotionPending(task) {
    s.handleTaskQueuePromotedWithAutoStartOnCreateClaimed(ctx, watcher.TaskEventData{TaskID: task.ID, StepTransitionID: stepTransitionID}, autoStartOnCreateClaimed)
    return
  }
  step, err := s.workflowStepGetter.GetStep(ctx, stepID)
  if err != nil {
    if autoStartOnCreateClaimed {
      s.restoreTaskLifecycleToken(ctx, taskID, models.MetaKeyAutoStartOnCreate, true, eventName)
    }
    return
  }
  s.autoStartTaskForLoadedStep(ctx, task, step, eventName, false, stepTransitionID, autoStartOnCreateClaimed)
}
Synchronous claim and structured restore on failureapps/backend/internal/orchestrator/event_handlers_workflow.go ↗
func (s *Service) claimAutoStartOnCreateForLaunch(ctx context.Context, task *models.Task, alreadyClaimed bool) bool
Click for details →

The launch claims the token before the goroutine starts, and the failure handler restores it together with the promotion token.

Claim helper
func (s *Service) claimAutoStartOnCreateForLaunch(ctx context.Context, task *models.Task, alreadyClaimed bool) bool {
  if alreadyClaimed {
    return true
  }
  if !models.HasAutoStartOnCreateIntent(task.Metadata) {
    return false
  }
  return s.claimTaskEventMetadata(ctx, task, models.MetaKeyAutoStartOnCreate)
}
Tokens struct and failure handler
type autoStartLaunchTokens struct {
  hasGuard                 bool
  restoreQueuePromotion    bool
  queuePromotionToken      interface{}
  restoreAutoStartOnCreate bool
}

func (s *Service) handleAutoStartFailure(ctx context.Context, taskID, eventName string, tokens autoStartLaunchTokens) {
  if tokens.hasGuard {
    s.restoreAutoStartClaim(ctx, taskID, eventName)
  }
  if tokens.restoreQueuePromotion {
    s.restoreTaskLifecycleToken(ctx, taskID, models.MetaKeyQueuePromotionPending, tokens.queuePromotionToken, eventName)
  }
  if tokens.restoreAutoStartOnCreate {
    s.restoreTaskLifecycleToken(ctx, taskID, models.MetaKeyAutoStartOnCreate, true, eventName)
  }
  s.setTaskAutoStartFailedMarker(ctx, taskID, eventName)
}
Use in loaded step
func (s *Service) autoStartTaskForLoadedStep(ctx context.Context, task *models.Task, step *wfmodels.WorkflowStep, eventName string, restoreQueuePromotion bool, stepTransitionID int64, autoStartOnCreateClaimed bool) {
  // ...
  restoreAutoStartOnCreate := s.claimAutoStartOnCreateForLaunch(ctx, task, autoStartOnCreateClaimed)
  if s.isOfficeTask(ctx, task.ID) {
    s.autoStartOfficeTaskForLoadedStep(ctx, task, step, eventName, restoreQueuePromotion, stepTransitionID, restoreAutoStartOnCreate)
    return
  }
  // ...
  go func() {
    // ...
    _, err := s.StartTask(asyncCtx, task.ID, startAgentProfileID, executorID, executorProfileID, "", task.Description, step.ID, planMode, true, nil)
    if err != nil {
      s.handleAutoStartFailure(asyncCtx, task.ID, eventName, autoStartLaunchTokens{
        hasGuard: hasGuard, restoreQueuePromotion: restoreQueuePromotion, queuePromotionToken: queuePromotionToken, restoreAutoStartOnCreate: restoreAutoStartOnCreate,
      })
    }
  }()
}
Promotion handler that preserves create-token ownershipapps/backend/internal/orchestrator/event_handlers_workflow.go ↗
func (s *Service) handleTaskQueuePromotedWithAutoStartOnCreateClaimed(ctx context.Context, data watcher.TaskEventData, autoStartOnCreateClaimed bool)
Click for details →

The promotion path now receives the already-claimed flag so a task with both tokens does not lose the create token on redirect.

Split and carry flag
func (s *Service) handleTaskQueuePromoted(ctx context.Context, data watcher.TaskEventData) {
  s.handleTaskQueuePromotedWithAutoStartOnCreateClaimed(ctx, data, false)
}

func (s *Service) loadQueuePromotedTaskAndTargetStep(ctx context.Context, taskID string) (*models.Task, *wfmodels.WorkflowStep, bool) {
  task, err := s.repo.GetTask(ctx, taskID)
  if err != nil {
    s.logger.Warn("task.queue_promoted: failed to load task", zap.String("task_id", taskID), zap.Error(err))
    return nil, nil, false
  }
  if task.QueuedForStepID != "" || !task.WIPAdmitted {
    return nil, nil, false
  }
  if queuedMoveExitPending(task) || manualMoveLifecyclePending(task) {
    return nil, nil, false
  }
  targetStep, err := s.workflowStepGetter.GetStep(ctx, task.WorkflowStepID)
  if err != nil || targetStep == nil {
    return nil, nil, false
  }
  return task, targetStep, true
}

func (s *Service) handleTaskQueuePromotedWithAutoStartOnCreateClaimed(ctx context.Context, data watcher.TaskEventData, autoStartOnCreateClaimed bool) {
  task, targetStep, ok := s.loadQueuePromotedTaskAndTargetStep(ctx, data.TaskID)
  if !ok {
    return
  }
  // ...
  s.autoStartTaskForLoadedStep(ctx, task, targetStep, "task.queue_promoted", true, data.StepTransitionID, autoStartOnCreateClaimed)
}
Re-fetch after promotion to close stale-read raceapps/backend/internal/orchestrator/event_handlers_workflow.go ↗
func (s *Service) recoverTaskLifecycleAttempt(ctx context.Context, taskID string) bool
Click for details →

The sweep now reloads the task after the promotion branch so it sees the synchronous claim and does not schedule a second launch.

Re-fetch after promotion
if _, pending := task.Metadata[models.MetaKeyQueuePromotionPending]; pending {
  s.handleTaskQueuePromoted(ctx, watcher.TaskEventData{TaskID: taskID})
  // handleTaskQueuePromoted's no-session branch schedules a launch via
  // autoStartTaskForLoadedStep, which synchronously claims
  // MetaKeyAutoStartOnCreate when the task still carries it before this call returns.
  task, err = s.repo.GetTask(ctx, taskID)
  if err != nil || task == nil {
    return false
  }
}
if autoStartOnCreateActionable(task) {
  s.recoverAutoStartOnCreate(ctx, task)
}
func (s *Service) blocksAutoStartLaunch(ctx context.Context, req *LaunchSessionRequest) bool
Click for details →

The session launch path now uses one helper that checks both step policy and dependency gate, without token-restore side effects.

Helper and use
func (s *Service) blocksAutoStartLaunch(ctx context.Context, req *LaunchSessionRequest) bool {
  if s.shouldBlockAutoStart(ctx, req) {
    return true
  }
  blocked, _ := s.dependencyBlocksAutoStart(ctx, req.TaskID, "session.launch")
  return blocked
}

func (s *Service) launchStart(ctx context.Context, req *LaunchSessionRequest) (*LaunchSessionResponse, error) {
  if req.AutoStart && s.blocksAutoStartLaunch(ctx, req) {
    req.LaunchWorkspace = true
    return s.launchPrepare(ctx, req)
  }
  // ...
}
Read the changes as a list

Split dependency gate into blocked and gateErrored

apps/backend/internal/orchestrator/event_handlers_dependencies.go

The gate now tells callers whether a block came from a read error or a real dependency, so only errors restore the one-shot token.

New signature and comment
func (s *Service) dependencyBlocksAutoStart(ctx context.Context, taskID, eventName string) (blocked, gateErrored bool) {
  if s.dependencyReader == nil {
    return false, false
  }
  isBlocked, reason, err := s.dependencyReader.DependencyGate(ctx, taskID)
  if err != nil {
    s.logger.Warn(eventName+": dependency lookup failed; skipping auto-start",
      zap.String("task_id", taskID), zap.Error(err))
    return true, true
  }
  if isBlocked {
    s.logger.Debug(eventName+": task has unresolved dependencies; skipping auto-start",
      zap.String("task_id", taskID), zap.String("blocked_reason", reason))
  }
  return isBlocked, false
}

Carry claimed token through auto-start chokepoint

apps/backend/internal/orchestrator/event_handlers_workflow.go

The chokepoint now knows if the caller already claimed the create token, and it restores the token on every early return that would otherwise strand the task.

Signature and early restores
func (s *Service) autoStartTaskForStep(ctx context.Context, taskID, stepID, eventName string, stepTransitionID int64, autoStartOnCreateClaimed bool) {
  task, err := s.repo.GetTask(ctx, taskID)
  if err != nil {
    s.logger.Warn(eventName+": failed to load task for auto-start", zap.String("task_id", taskID), zap.Error(err))
    if autoStartOnCreateClaimed {
      s.restoreTaskLifecycleToken(ctx, taskID, models.MetaKeyAutoStartOnCreate, true, eventName)
    }
    return
  }
  if blocked, gateErrored := s.dependencyBlocksAutoStart(ctx, taskID, eventName); blocked {
    if gateErrored && autoStartOnCreateClaimed {
      s.restoreTaskLifecycleToken(ctx, taskID, models.MetaKeyAutoStartOnCreate, true, eventName)
    }
    return
  }
  if hasQueuePromotionPending(task) {
    s.handleTaskQueuePromotedWithAutoStartOnCreateClaimed(ctx, watcher.TaskEventData{TaskID: task.ID, StepTransitionID: stepTransitionID}, autoStartOnCreateClaimed)
    return
  }
  step, err := s.workflowStepGetter.GetStep(ctx, stepID)
  if err != nil {
    if autoStartOnCreateClaimed {
      s.restoreTaskLifecycleToken(ctx, taskID, models.MetaKeyAutoStartOnCreate, true, eventName)
    }
    return
  }
  s.autoStartTaskForLoadedStep(ctx, task, step, eventName, false, stepTransitionID, autoStartOnCreateClaimed)
}

Synchronous claim and structured restore on failure

apps/backend/internal/orchestrator/event_handlers_workflow.go

The launch claims the token before the goroutine starts, and the failure handler restores it together with the promotion token.

Claim helper
func (s *Service) claimAutoStartOnCreateForLaunch(ctx context.Context, task *models.Task, alreadyClaimed bool) bool {
  if alreadyClaimed {
    return true
  }
  if !models.HasAutoStartOnCreateIntent(task.Metadata) {
    return false
  }
  return s.claimTaskEventMetadata(ctx, task, models.MetaKeyAutoStartOnCreate)
}
Tokens struct and failure handler
type autoStartLaunchTokens struct {
  hasGuard                 bool
  restoreQueuePromotion    bool
  queuePromotionToken      interface{}
  restoreAutoStartOnCreate bool
}

func (s *Service) handleAutoStartFailure(ctx context.Context, taskID, eventName string, tokens autoStartLaunchTokens) {
  if tokens.hasGuard {
    s.restoreAutoStartClaim(ctx, taskID, eventName)
  }
  if tokens.restoreQueuePromotion {
    s.restoreTaskLifecycleToken(ctx, taskID, models.MetaKeyQueuePromotionPending, tokens.queuePromotionToken, eventName)
  }
  if tokens.restoreAutoStartOnCreate {
    s.restoreTaskLifecycleToken(ctx, taskID, models.MetaKeyAutoStartOnCreate, true, eventName)
  }
  s.setTaskAutoStartFailedMarker(ctx, taskID, eventName)
}
Use in loaded step
func (s *Service) autoStartTaskForLoadedStep(ctx context.Context, task *models.Task, step *wfmodels.WorkflowStep, eventName string, restoreQueuePromotion bool, stepTransitionID int64, autoStartOnCreateClaimed bool) {
  // ...
  restoreAutoStartOnCreate := s.claimAutoStartOnCreateForLaunch(ctx, task, autoStartOnCreateClaimed)
  if s.isOfficeTask(ctx, task.ID) {
    s.autoStartOfficeTaskForLoadedStep(ctx, task, step, eventName, restoreQueuePromotion, stepTransitionID, restoreAutoStartOnCreate)
    return
  }
  // ...
  go func() {
    // ...
    _, err := s.StartTask(asyncCtx, task.ID, startAgentProfileID, executorID, executorProfileID, "", task.Description, step.ID, planMode, true, nil)
    if err != nil {
      s.handleAutoStartFailure(asyncCtx, task.ID, eventName, autoStartLaunchTokens{
        hasGuard: hasGuard, restoreQueuePromotion: restoreQueuePromotion, queuePromotionToken: queuePromotionToken, restoreAutoStartOnCreate: restoreAutoStartOnCreate,
      })
    }
  }()
}

Promotion handler that preserves create-token ownership

apps/backend/internal/orchestrator/event_handlers_workflow.go

The promotion path now receives the already-claimed flag so a task with both tokens does not lose the create token on redirect.

Split and carry flag
func (s *Service) handleTaskQueuePromoted(ctx context.Context, data watcher.TaskEventData) {
  s.handleTaskQueuePromotedWithAutoStartOnCreateClaimed(ctx, data, false)
}

func (s *Service) loadQueuePromotedTaskAndTargetStep(ctx context.Context, taskID string) (*models.Task, *wfmodels.WorkflowStep, bool) {
  task, err := s.repo.GetTask(ctx, taskID)
  if err != nil {
    s.logger.Warn("task.queue_promoted: failed to load task", zap.String("task_id", taskID), zap.Error(err))
    return nil, nil, false
  }
  if task.QueuedForStepID != "" || !task.WIPAdmitted {
    return nil, nil, false
  }
  if queuedMoveExitPending(task) || manualMoveLifecyclePending(task) {
    return nil, nil, false
  }
  targetStep, err := s.workflowStepGetter.GetStep(ctx, task.WorkflowStepID)
  if err != nil || targetStep == nil {
    return nil, nil, false
  }
  return task, targetStep, true
}

func (s *Service) handleTaskQueuePromotedWithAutoStartOnCreateClaimed(ctx context.Context, data watcher.TaskEventData, autoStartOnCreateClaimed bool) {
  task, targetStep, ok := s.loadQueuePromotedTaskAndTargetStep(ctx, data.TaskID)
  if !ok {
    return
  }
  // ...
  s.autoStartTaskForLoadedStep(ctx, task, targetStep, "task.queue_promoted", true, data.StepTransitionID, autoStartOnCreateClaimed)
}

Re-fetch after promotion to close stale-read race

apps/backend/internal/orchestrator/event_handlers_workflow.go

The sweep now reloads the task after the promotion branch so it sees the synchronous claim and does not schedule a second launch.

Re-fetch after promotion
if _, pending := task.Metadata[models.MetaKeyQueuePromotionPending]; pending {
  s.handleTaskQueuePromoted(ctx, watcher.TaskEventData{TaskID: taskID})
  // handleTaskQueuePromoted's no-session branch schedules a launch via
  // autoStartTaskForLoadedStep, which synchronously claims
  // MetaKeyAutoStartOnCreate when the task still carries it before this call returns.
  task, err = s.repo.GetTask(ctx, taskID)
  if err != nil || task == nil {
    return false
  }
}
if autoStartOnCreateActionable(task) {
  s.recoverAutoStartOnCreate(ctx, task)
}

Downgrade helper for session.launch

apps/backend/internal/orchestrator/session_launch.go

The session launch path now uses one helper that checks both step policy and dependency gate, without token-restore side effects.

Helper and use
func (s *Service) blocksAutoStartLaunch(ctx context.Context, req *LaunchSessionRequest) bool {
  if s.shouldBlockAutoStart(ctx, req) {
    return true
  }
  blocked, _ := s.dependencyBlocksAutoStart(ctx, req.TaskID, "session.launch")
  return blocked
}

func (s *Service) launchStart(ctx context.Context, req *LaunchSessionRequest) (*LaunchSessionResponse, error) {
  if req.AutoStart && s.blocksAutoStartLaunch(ctx, req) {
    req.LaunchWorkspace = true
    return s.launchPrepare(ctx, req)
  }
  // ...
}

Data and storage

One-shot metadata keys drive the lifecycle. Each key is claimed atomically and restored only on failure.

FieldTypeNotes
MetaKeyAutoStartOnCreateboolcreate-time opt-in; claimed synchronously before launch
MetaKeyQueuePromotionPendingmap[from_step_id]WIP promotion token; restored on launch failure
MetaKeyAutoStartGuard / Claimedboolreview watcher dedup; only restored when guard present
MetaKeyDeferredLaunchjsondeferred agent intent; separate from auto-start tokens
MetaKeyAutoStartFailedboolfailure marker surfaced on task card

Risk

5 / 10 Medium
1 low5 medium10 high

Why this score

  • Touches the single auto-start chokepoint used by task.created, task.moved, promotion, and dependency resolution.
  • Token claim is synchronous and atomic, but a missed restore would strand tasks until next sweep.
  • Covered by five new tests for double-launch and stranded-token cases, plus existing promotion tests.

Trade-offs and review notes

Where to look first

  1. Check claimAutoStartOnCreateForLaunch is called before any goroutine and that handleAutoStartFailure restores both tokens.
  2. Verify autoStartTaskForStep restores on GetTask, GetStep, and gateErrored, and that genuine blocks do not restore.
  3. Confirm recoverTaskLifecycleAttempt re-fetches after promotion and after manualMoveLifecycleCompleted.
  4. Review handleTaskQueuePromotedWithAutoStartOnCreateClaimed carries the flag into autoStartTaskForLoadedStep.