PR #3613
Sections
Review

feat(office): make the unattended loop legible end-to-end

main ← feature/office-loop-liveness-0sn 109 files +2847 −612 PR #3613 ↗

The PR makes the Office unattended loop observable by adding causation correlation, per-hop counters, terminal-shape classification, and a single workspace health verdict that answers whether the loop is alive.

Why this change

The unattended loop has no single surface that answers whether it ran. A dead schedule, a stuck queue, and a silent success all read as healthy because no counter, no correlation id, and no verdict exists.

What it does

Architecture, end to end

A cron tick claims a trigger, creates a routine run, enqueues a wakeup, creates a runs row, the scheduler claims and launches it, and the run reaches a terminal shape. Counters and the health read observe each hop without changing scheduling.

flowchart LR
  Cron["Cron tick\n30s loop"] --> Claim["Claim trigger\nIncLoopTriggerClaimed"]
  Claim --> Run["Routine run\ncausation_id + last_run_at"]
  Run --> Wake["Wakeup request\ncausation_id"]
  Wake --> Queue["runs row\ncausation_id"]
  Queue --> Sched["Scheduler claim\nIncLoopRunClaimed"]
  Sched --> Launch["Agent launch\nsession_id persist"]
  Launch --> Term["Terminal shape\nIncLoopTerminal"]
  Term --> Health["GET loop-health\nverdict + evidence"]
  Cron -.-> Counters["GET loop-counters\nworkspace filtered"]
  Health --> Verdict["unknown → dead → degraded → not_armed → healthy"]

Key code changes

Drag to pan. Use the + and − buttons to zoom. Click a node to open the full code. The arrows show how the parts interact.

drag to pan · +/− to zoom · click a node for details

func (r *Repository) migrateLoopLivenessCausationID()
Click for details →

Adds causation_id to three tables with partial indexes and publishes an activation instant only after all three columns exist.

Migration
func (r *Repository) migrateLoopLivenessCausationID() {
  _ = r.migrate.Apply("office_routine_runs.causation_id",
    `ALTER TABLE office_routine_runs ADD COLUMN causation_id TEXT NOT NULL DEFAULT ''`)
  _ = r.migrate.Apply("idx_office_routine_runs_causation_id",
    `CREATE INDEX IF NOT EXISTS idx_office_routine_runs_causation_id
      ON office_routine_runs(causation_id) WHERE causation_id != ''`)
  _ = r.migrate.Apply("agent_wakeup_requests.causation_id",
    `ALTER TABLE agent_wakeup_requests ADD COLUMN causation_id TEXT NOT NULL DEFAULT ''`)
  _ = r.migrate.Apply("idx_agent_wakeup_requests_causation_id",
    `CREATE INDEX IF NOT EXISTS idx_agent_wakeup_requests_causation_id
      ON agent_wakeup_requests(causation_id) WHERE causation_id != ''`)
  _ = r.migrate.Apply("runs.causation_id",
    `ALTER TABLE runs ADD COLUMN causation_id TEXT NOT NULL DEFAULT ''`)
  _ = r.migrate.Apply("idx_runs_causation_id",
    `CREATE INDEX IF NOT EXISTS idx_runs_causation_id
      ON runs(causation_id) WHERE causation_id != ''`)
}
Activation
func (r *Repository) activateLoopLiveness() {
  for _, probe := range []struct{ table, column string }{
    {"office_routine_runs", causationIDColumn},
    {"agent_wakeup_requests", causationIDColumn},
    {"runs", causationIDColumn},
  } {
    exists, err := columnExists(r.db, probe.table, probe.column)
    if err != nil || !exists {
      return
    }
  }
  persistence.WriteMetaKeyIfAbsent(r.db, loopLivenessActivationKey, time.Now().UTC().Format(time.RFC3339))
}
Routine fire recency and causation mintapps/backend/internal/office/routines/service.go ↗
func (s *RoutineService) dispatchRoutineRun(...)
Click for details →

Mints one causation id per fire, advances last_run_at before concurrency policy, and counts the fire by source and disposition.

Mint and persist
  run := &RoutineRun{
    RoutineID: routine.ID,
    TriggerID: triggerID,
    Source: source,
    Status: models.RoutineRunStatusReceived,
    StartedAt: &now,
    CausationID: uuid.New().String(),
  }
  if err := s.repo.CreateRoutineRun(ctx, run); err != nil {
    return nil, fmt.Errorf("create run: %w", err)
  }
  disposition := string(models.RoutineRunStatusFailed)
  defer func() {
    service.IncLoopRoutineRun(routine.WorkspaceID, source, disposition)
  }()
  if err := s.repo.TouchRoutineLastRun(ctx, routine.ID, *run.StartedAt); err != nil {
    service.IncLoopLastRunAtWriteFailed(routine.WorkspaceID)
    s.logger.Warn("touch routine last_run_at", zap.String("routine_id", routine.ID), zap.Error(err))
  }
Monotonic touch
UPDATE office_routines
SET last_run_at = ?, updated_at = ?
WHERE id = ? AND (last_run_at IS NULL OR last_run_at < ?)
func (a *routineWakeupAdapter) CreateWakeupRequest(...)
Click for details →

Copies the fire's causation id onto the wakeup request so the lightweight path stays correlated.

Adapter
func (a *routineWakeupAdapter) CreateWakeupRequest(ctx context.Context, req *officeroutines.WakeupRequest) error {
  row := &officesqlite.WakeupRequest{
    ID: req.ID,
    AgentProfileID: req.AgentProfileID,
    Source: req.Source,
    Reason: req.Reason,
    Payload: req.Payload,
    RequestedAt: req.RequestedAt,
    CausationID: req.CausationID,
  }
  if err := a.repo.CreateWakeupRequest(ctx, row); err != nil {
    if errors.Is(err, officesqlite.ErrWakeupIdempotencyConflict) {
      return officeroutines.ErrWakeupAlreadyRequested
    }
    return err
  }
  return nil
}
Lightweight materialize
  req := &WakeupRequest{
    ID: uuid.New().String(),
    AgentProfileID: routine.AssigneeAgentProfileID,
    Source: "routine",
    Payload: payloadStr,
    RequestedAt: time.Now().UTC(),
    CausationID: run.CausationID,
  }
  if err := s.wakeup.CreateWakeupRequest(ctx, req); err != nil {
    return s.finalizeLightweightRun(ctx, run, models.RoutineRunStatusFailed)
  }
  service.IncLoopWakeupCreated(routine.WorkspaceID, req.Source)
Session hop persisted on both launch pathsapps/backend/internal/backendapp/main.go ↗
func (a *officeOrchestratorTaskStarter) StartTaskWithLaunchContextReturningSession(...)
Click for details →

Returns the orchestrator's session id and persists it on the runs row so a success claim carries evidence an agent ran.

Direct launch adapter
type officeOrchestratorTaskStarter struct { orch *orchestrator.Service }
var _ officeservice.TaskStarterWithLaunchContextSession = (*officeOrchestratorTaskStarter)(nil)
func (a *officeOrchestratorTaskStarter) StartTaskWithLaunchContextReturningSession(ctx context.Context, taskID, agentProfileID string, launch officeservice.LaunchContext) (string, error) {
  execution, err := a.startTaskWithEnvAndSkills(ctx, taskID, agentProfileID, launch.ExecutorID, launch.ExecutorProfileID, launch.Priority, launch.Prompt, launch.WorkflowStepID, launch.PlanMode, launch.Attachments, launch.Env, launch.AdditionalSkillSlugs)
  if err != nil || execution == nil {
    return "", err
  }
  return execution.SessionID, nil
}
Routed launch adapter
var _ officescheduler.TaskStarterWithSession = (*schedulerTaskStarterAdapter)(nil)
func (a *schedulerTaskStarterAdapter) StartTaskWithRouteReturningSession(ctx context.Context, taskID, agentProfileID string, launch officescheduler.LaunchContext, route officescheduler.RouteOverride) (string, error) {
  execution, err := a.startTaskWithRoute(ctx, taskID, agentProfileID, launch, route)
  if err != nil || execution == nil {
    return "", err
  }
  return execution.SessionID, nil
}
Persist with guard
UPDATE runs SET session_id = ? WHERE id = ? AND ? != ''
func ClassifyTerminalRun(status string, outcome *string, sessionID string, ...) TerminalShape
Click for details →

Maps every terminal run to one of seven shapes so silent success and unlaunched skip are distinct from real success.

Shapes
const (
  ShapePreActivation TerminalShape = "pre_activation"
  ShapeLaunchedCompleted TerminalShape = "launched_completed"
  ShapeLaunchedFailed TerminalShape = "launched_failed"
  ShapeSilentSuccess TerminalShape = "silent_success"
  ShapeUnlaunchedSkipped TerminalShape = "unlaunched_skipped"
  ShapeUnlaunchedFailed TerminalShape = "unlaunched_failed"
  ShapeUnclassified TerminalShape = "unclassified"
)
Classifier
func ClassifyTerminalRun(status string, outcome *string, sessionID string, requestedAt time.Time, activationInstant time.Time, activationPublished bool) TerminalShape {
  if !activationPublished || requestedAt.Before(activationInstant) {
    return ShapePreActivation
  }
  outcomeVal := ""
  if outcome != nil { outcomeVal = *outcome }
  return classifyByOutcome(status, outcomeVal, sessionID != "")
}
func classifyByOutcome(status, outcomeVal string, launched bool) TerminalShape {
  switch {
  case status == "finished" && outcomeVal == "processed" && launched:
    return ShapeLaunchedCompleted
  case status == "finished" && outcomeVal == "processed" && !launched:
    return ShapeSilentSuccess
  case status == "finished" && !launched && skipOutcomes[outcomeVal]:
    return ShapeUnlaunchedSkipped
  case terminalFailedStatuses[status] && launched:
    return ShapeLaunchedFailed
  case terminalFailedStatuses[status] && !launched:
    return ShapeUnlaunchedFailed
  default:
    return ShapeUnclassified
  }
}
func EvaluateLoopHealth(ctx context.Context, repo LoopHealthRepo, workspaceID string, now time.Time)
Click for details →

Evaluates one verdict from overdue, stuck, and silent-success evidence and exposes workspace-filtered counters.

Verdict precedence
func decideLoopHealthVerdict(triggerTotal, eligibleCount, stuckTotal, silentTotal int) string {
  switch {
  case triggerTotal > 0:
    return VerdictDead
  case stuckTotal > 0 || silentTotal > 0:
    return VerdictDegraded
  case eligibleCount == 0:
    return VerdictNotArmed
  default:
    return VerdictHealthy
  }
}
Counters
var (
  loopCronTickTotal = expvar.NewMap("office_loop_cron_tick_total")
  loopTriggerClaimedTotal = expvar.NewMap("office_loop_trigger_claimed_total")
  loopRoutineRunTotal = expvar.NewMap("office_loop_routine_run_total")
  loopWakeupCreatedTotal = expvar.NewMap("office_loop_wakeup_created_total")
  loopRunClaimedTotal = expvar.NewMap("office_loop_run_claimed_total")
  loopLaunchTotal = expvar.NewMap("office_loop_launch_total")
  loopTerminalTotal = expvar.NewMap("office_loop_terminal_total")
  loopCronTickAt = expvar.NewString("office_loop_cron_tick_at")
  loopProcessStartedAt = expvar.NewString("office_loop_process_started_at")
)
func IncLoopCronTick(atRFC3339 string) { loopCronTickTotal.Add("", 1); loopCronTickAt.Set(atRFC3339) }
func RecordLoopProcessStarted(atRFC3339 string) { loopProcessStartedAt.Set(atRFC3339) }
Routes
func registerLoopHealthRoutes(api *gin.RouterGroup, h *Handler) {
  api.GET("/workspaces/:wsId/loop-health", h.getLoopHealth)
  api.GET("/workspaces/:wsId/loop-counters", h.getLoopCounters)
}
Read the changes as a list

Causation columns and activation probe

apps/backend/internal/office/repository/sqlite/base_migrations.go

Adds causation_id to three tables with partial indexes and publishes an activation instant only after all three columns exist.

Migration
func (r *Repository) migrateLoopLivenessCausationID() {
  _ = r.migrate.Apply("office_routine_runs.causation_id",
    `ALTER TABLE office_routine_runs ADD COLUMN causation_id TEXT NOT NULL DEFAULT ''`)
  _ = r.migrate.Apply("idx_office_routine_runs_causation_id",
    `CREATE INDEX IF NOT EXISTS idx_office_routine_runs_causation_id
      ON office_routine_runs(causation_id) WHERE causation_id != ''`)
  _ = r.migrate.Apply("agent_wakeup_requests.causation_id",
    `ALTER TABLE agent_wakeup_requests ADD COLUMN causation_id TEXT NOT NULL DEFAULT ''`)
  _ = r.migrate.Apply("idx_agent_wakeup_requests_causation_id",
    `CREATE INDEX IF NOT EXISTS idx_agent_wakeup_requests_causation_id
      ON agent_wakeup_requests(causation_id) WHERE causation_id != ''`)
  _ = r.migrate.Apply("runs.causation_id",
    `ALTER TABLE runs ADD COLUMN causation_id TEXT NOT NULL DEFAULT ''`)
  _ = r.migrate.Apply("idx_runs_causation_id",
    `CREATE INDEX IF NOT EXISTS idx_runs_causation_id
      ON runs(causation_id) WHERE causation_id != ''`)
}
Activation
func (r *Repository) activateLoopLiveness() {
  for _, probe := range []struct{ table, column string }{
    {"office_routine_runs", causationIDColumn},
    {"agent_wakeup_requests", causationIDColumn},
    {"runs", causationIDColumn},
  } {
    exists, err := columnExists(r.db, probe.table, probe.column)
    if err != nil || !exists {
      return
    }
  }
  persistence.WriteMetaKeyIfAbsent(r.db, loopLivenessActivationKey, time.Now().UTC().Format(time.RFC3339))
}

Routine fire recency and causation mint

apps/backend/internal/office/routines/service.go

Mints one causation id per fire, advances last_run_at before concurrency policy, and counts the fire by source and disposition.

Mint and persist
  run := &RoutineRun{
    RoutineID: routine.ID,
    TriggerID: triggerID,
    Source: source,
    Status: models.RoutineRunStatusReceived,
    StartedAt: &now,
    CausationID: uuid.New().String(),
  }
  if err := s.repo.CreateRoutineRun(ctx, run); err != nil {
    return nil, fmt.Errorf("create run: %w", err)
  }
  disposition := string(models.RoutineRunStatusFailed)
  defer func() {
    service.IncLoopRoutineRun(routine.WorkspaceID, source, disposition)
  }()
  if err := s.repo.TouchRoutineLastRun(ctx, routine.ID, *run.StartedAt); err != nil {
    service.IncLoopLastRunAtWriteFailed(routine.WorkspaceID)
    s.logger.Warn("touch routine last_run_at", zap.String("routine_id", routine.ID), zap.Error(err))
  }
Monotonic touch
UPDATE office_routines
SET last_run_at = ?, updated_at = ?
WHERE id = ? AND (last_run_at IS NULL OR last_run_at < ?)

Wakeup adapter carries causation

apps/backend/internal/backendapp/adapters_office.go

Copies the fire's causation id onto the wakeup request so the lightweight path stays correlated.

Adapter
func (a *routineWakeupAdapter) CreateWakeupRequest(ctx context.Context, req *officeroutines.WakeupRequest) error {
  row := &officesqlite.WakeupRequest{
    ID: req.ID,
    AgentProfileID: req.AgentProfileID,
    Source: req.Source,
    Reason: req.Reason,
    Payload: req.Payload,
    RequestedAt: req.RequestedAt,
    CausationID: req.CausationID,
  }
  if err := a.repo.CreateWakeupRequest(ctx, row); err != nil {
    if errors.Is(err, officesqlite.ErrWakeupIdempotencyConflict) {
      return officeroutines.ErrWakeupAlreadyRequested
    }
    return err
  }
  return nil
}
Lightweight materialize
  req := &WakeupRequest{
    ID: uuid.New().String(),
    AgentProfileID: routine.AssigneeAgentProfileID,
    Source: "routine",
    Payload: payloadStr,
    RequestedAt: time.Now().UTC(),
    CausationID: run.CausationID,
  }
  if err := s.wakeup.CreateWakeupRequest(ctx, req); err != nil {
    return s.finalizeLightweightRun(ctx, run, models.RoutineRunStatusFailed)
  }
  service.IncLoopWakeupCreated(routine.WorkspaceID, req.Source)

Session hop persisted on both launch paths

apps/backend/internal/backendapp/main.go

Returns the orchestrator's session id and persists it on the runs row so a success claim carries evidence an agent ran.

Direct launch adapter
type officeOrchestratorTaskStarter struct { orch *orchestrator.Service }
var _ officeservice.TaskStarterWithLaunchContextSession = (*officeOrchestratorTaskStarter)(nil)
func (a *officeOrchestratorTaskStarter) StartTaskWithLaunchContextReturningSession(ctx context.Context, taskID, agentProfileID string, launch officeservice.LaunchContext) (string, error) {
  execution, err := a.startTaskWithEnvAndSkills(ctx, taskID, agentProfileID, launch.ExecutorID, launch.ExecutorProfileID, launch.Priority, launch.Prompt, launch.WorkflowStepID, launch.PlanMode, launch.Attachments, launch.Env, launch.AdditionalSkillSlugs)
  if err != nil || execution == nil {
    return "", err
  }
  return execution.SessionID, nil
}
Routed launch adapter
var _ officescheduler.TaskStarterWithSession = (*schedulerTaskStarterAdapter)(nil)
func (a *schedulerTaskStarterAdapter) StartTaskWithRouteReturningSession(ctx context.Context, taskID, agentProfileID string, launch officescheduler.LaunchContext, route officescheduler.RouteOverride) (string, error) {
  execution, err := a.startTaskWithRoute(ctx, taskID, agentProfileID, launch, route)
  if err != nil || execution == nil {
    return "", err
  }
  return execution.SessionID, nil
}
Persist with guard
UPDATE runs SET session_id = ? WHERE id = ? AND ? != ''

Terminal shape classification

apps/backend/internal/office/service/terminal_shapes.go

Maps every terminal run to one of seven shapes so silent success and unlaunched skip are distinct from real success.

Shapes
const (
  ShapePreActivation TerminalShape = "pre_activation"
  ShapeLaunchedCompleted TerminalShape = "launched_completed"
  ShapeLaunchedFailed TerminalShape = "launched_failed"
  ShapeSilentSuccess TerminalShape = "silent_success"
  ShapeUnlaunchedSkipped TerminalShape = "unlaunched_skipped"
  ShapeUnlaunchedFailed TerminalShape = "unlaunched_failed"
  ShapeUnclassified TerminalShape = "unclassified"
)
Classifier
func ClassifyTerminalRun(status string, outcome *string, sessionID string, requestedAt time.Time, activationInstant time.Time, activationPublished bool) TerminalShape {
  if !activationPublished || requestedAt.Before(activationInstant) {
    return ShapePreActivation
  }
  outcomeVal := ""
  if outcome != nil { outcomeVal = *outcome }
  return classifyByOutcome(status, outcomeVal, sessionID != "")
}
func classifyByOutcome(status, outcomeVal string, launched bool) TerminalShape {
  switch {
  case status == "finished" && outcomeVal == "processed" && launched:
    return ShapeLaunchedCompleted
  case status == "finished" && outcomeVal == "processed" && !launched:
    return ShapeSilentSuccess
  case status == "finished" && !launched && skipOutcomes[outcomeVal]:
    return ShapeUnlaunchedSkipped
  case terminalFailedStatuses[status] && launched:
    return ShapeLaunchedFailed
  case terminalFailedStatuses[status] && !launched:
    return ShapeUnlaunchedFailed
  default:
    return ShapeUnclassified
  }
}

Health verdict and per-hop counters

apps/backend/internal/office/dashboard/loop_health.go

Evaluates one verdict from overdue, stuck, and silent-success evidence and exposes workspace-filtered counters.

Verdict precedence
func decideLoopHealthVerdict(triggerTotal, eligibleCount, stuckTotal, silentTotal int) string {
  switch {
  case triggerTotal > 0:
    return VerdictDead
  case stuckTotal > 0 || silentTotal > 0:
    return VerdictDegraded
  case eligibleCount == 0:
    return VerdictNotArmed
  default:
    return VerdictHealthy
  }
}
Counters
var (
  loopCronTickTotal = expvar.NewMap("office_loop_cron_tick_total")
  loopTriggerClaimedTotal = expvar.NewMap("office_loop_trigger_claimed_total")
  loopRoutineRunTotal = expvar.NewMap("office_loop_routine_run_total")
  loopWakeupCreatedTotal = expvar.NewMap("office_loop_wakeup_created_total")
  loopRunClaimedTotal = expvar.NewMap("office_loop_run_claimed_total")
  loopLaunchTotal = expvar.NewMap("office_loop_launch_total")
  loopTerminalTotal = expvar.NewMap("office_loop_terminal_total")
  loopCronTickAt = expvar.NewString("office_loop_cron_tick_at")
  loopProcessStartedAt = expvar.NewString("office_loop_process_started_at")
)
func IncLoopCronTick(atRFC3339 string) { loopCronTickTotal.Add("", 1); loopCronTickAt.Set(atRFC3339) }
func RecordLoopProcessStarted(atRFC3339 string) { loopProcessStartedAt.Set(atRFC3339) }
Routes
func registerLoopHealthRoutes(api *gin.RouterGroup, h *Handler) {
  api.GET("/workspaces/:wsId/loop-health", h.getLoopHealth)
  api.GET("/workspaces/:wsId/loop-counters", h.getLoopCounters)
}

Data and storage

Three new causation_id columns, one monotonic timestamp, and derived shapes drive the health read. All new columns default to empty string so legacy rows read as uncorrelated.

FieldTypeNotes
office_routine_runs.causation_idTEXT NOT NULL DEFAULT ''UUID minted per fire, partial index where != ''
agent_wakeup_requests.causation_idTEXT NOT NULL DEFAULT ''Copied from fire, partial index where != ''
runs.causation_idTEXT NOT NULL DEFAULT ''Copied from wakeup request, partial index where != ''
office_routines.last_run_atTIMESTAMPMonotonic touch only via TouchRoutineLastRun, not UpdateRoutine
runs.session_idTEXT NOT NULL DEFAULT ''Persisted on launch, last non-empty wins by claimed_at
kandev_meta.telemetry.office_loop_liveness.activated_atTEXT RFC3339Published once after probe, gates pre_activation shape
TerminalShapeenum(7)pre_activation, launched_completed, launched_failed, silent_success, unlaunched_skipped, unlaunched_failed, unclassified

Risk

5 / 10 Medium
1 low5 medium10 high

Why this score

  • Detection only: no trigger re-arm, no run requeue, and no agent launch changes, so a bug cannot repair or break scheduling.
  • Three additive columns with default '' and partial indexes: rollback is a no-op for old rows, but a failed migration leaves activation unpublished and health returns unknown.
  • Large surface with 30+ new tests, but stuck-run and trigger predicates mirror scheduler filters and must stay in sync with future scheduler changes.

Trade-offs and review notes

Where to look first

  1. Verify TouchRoutineLastRun monotonicity and that UpdateRoutine no longer writes last_run_at.
  2. Check causation id is minted once per fire and never changes on coalesce, skip, promotion, retry, or park.
  3. Confirm session persist uses last non-empty wins and counts distinct failures for empty launch versus persist failure.
  4. Review terminal shape ordering: pre_activation first, silent_success below launched_completed, and total coverage including no_agent_launched.
  5. Check stuck-run predicate mirrors ClaimNextEligibleRun exclusions and that trigger predicate excludes claiming triggers.