PR #3566
Sections
Review

feat(office): bound run history growth with a scheduled retention sweep

main ← feature/retention-and-gc-for-tak 69 files +2847 −312 PR #3566 ↗

Adds a scheduled retention sweep that bounds office_routine_runs, runs, and their satellites by age and per-owner floor, with preview, census, health, and System UI.

Why this change

Office writes routine-run and run history on every firing and never deletes it. A 5-minute routine creates 105k rows per year, so an install that never deletes a routine grows without bound.

What it does

Architecture, end to end

Retention runs on its own schedule, separate from the Office tick. Settings flow from the shared store to both scheduler and sweep, and results flow to health and UI.

flowchart LR
  SettingsStore[(settings store)] --> Scheduler
  SettingsStore --> Sweeper
  PreviewMarker[(preview marker)] --> Sweeper
  Scheduler -- RunSweep --> Sweeper
  Scheduler -- RunCensus --> Sweeper
  Sweeper --> Store
  Store --> DB[(office_routine_runs / runs / satellites)]
  Sweeper --> CensusTracker
  Sweeper --> LastSweep
  CensusTracker --> Checker
  LastSweep --> Checker
  Checker --> Health[/health]
  Sweeper --> Handler
  Handler --> API[GET/PUT /api/v1/system/retention]
  API --> Frontend[RetentionSettingsCard]
  Scheduler -.-> Handler
  DB -. indexes .-> Store

Key code changes

Drag to pan. Use the + and − buttons to zoom. Click a node to open the full code. The arrows show how the parts interact.

drag to pan · +/− to zoom · click a node for details

func NewRuntime(pool *db.Pool, settingsStore *systemsettings.Store, logError func(string, error)) *Runtime
Click for details →

Composes every retention component behind one Start/Stop pair and wires scheduler re-arm to the HTTP handler.

Runtime
type Runtime struct {
  Store         *Store
  SettingsStore *SettingsStore
  PreviewMarker *PreviewMarkerStore
  Sweeper       *Sweeper
  Scheduler     *Scheduler
  Checker       *Checker
  Handler       *Handler
}

func NewRuntime(pool *db.Pool, settingsStore *systemsettings.Store, logError func(string, error)) *Runtime {
  store := NewStore(pool)
  retentionSettings := NewSettingsStore(settingsStore)
  previewMarker := NewPreviewMarkerStore(settingsStore)
  sweeper := NewSweeper(pool, store, retentionSettings, previewMarker)
  scheduler := NewScheduler(retentionSettings, sweeper, SchedulerOptions{})
  checker := NewChecker(retentionSettings, sweeper, previewMarker)
  handler := NewHandler(HandlerConfig{
    SettingsStore:     retentionSettings,
    Sweeper:           sweeper,
    OnSettingsChanged: scheduler.ApplySettings,
    LogError:          logError,
  })
  return &Runtime{Store: store, SettingsStore: retentionSettings, PreviewMarker: previewMarker, Sweeper: sweeper, Scheduler: scheduler, Checker: checker, Handler: handler}
}
func (s *Sweeper) RunSweep(ctx context.Context)
Click for details →

Runs one sweep in fixed order, handles preview, backlog, satellite deletes, and lock loss between tables.

RunSweep
func (s *Sweeper) RunSweep(ctx context.Context) {
  if !s.beginSweep() {
    s.recordSkip()
    return
  }
  defer s.endSweep()

  settings, err := s.settingsStore.GetSettingsForSweep(ctx)
  if err != nil {
    s.recordSkip()
    return
  }
  if !settings.Enabled {
    return
  }

  q, session, ok := s.acquireQueryer(ctx)
  if !ok {
    s.recordSkip()
    return
  }
  if session != nil {
    defer session.release()
  }

  now := time.Now().UTC()
  report := LastSweep{StartedAt: now}
  report.OfficeRoutineRuns = s.sweepRoutineRuns(ctx, q, settings.RoutineRuns, now, settings.BatchLimit)

  if session != nil && !session.alive(ctx) {
    s.recordSkip()
    return
  }

  runsResult, satellites := s.sweepRuns(ctx, q, settings.Runs, now, settings.BatchLimit)
  report.Runs = runsResult
  report.RunEvents = satellites.RunEvents
  report.RouteAttempts = satellites.RouteAttempts
  report.RunSkills = satellites.RunSkills

  report.FinishedAt = time.Now().UTC()
  s.mu.Lock()
  s.lastSweep = &report
  s.mu.Unlock()
  incSweepCompleted()
}
func (s *Scheduler) run(ctx context.Context, settings Settings, wake <-chan struct{})
Click for details →

Owns census and sweep timers on one goroutine and re-arms both on settings change with fixed-delay semantics.

Scheduler loop
func (s *Scheduler) run(ctx context.Context, settings Settings, wake <-chan struct{}) {
  defer s.wg.Done()
  s.sweeper.RunCensus(ctx)
  census := s.after(sweepInterval(settings))

  var sweep <-chan time.Time
  if settings.Enabled {
    sweep = s.after(firstSweepDelay)
  }

  for {
    select {
    case <-ctx.Done():
      return
    case <-wake:
      wasEnabled := settings.Enabled
      settings = s.latestSettings()
      sweep = nil
      if settings.Enabled {
        if wasEnabled {
          sweep = s.after(sweepInterval(settings))
        } else {
          sweep = s.after(firstSweepDelay)
        }
      }
      census = s.after(sweepInterval(settings))
    case <-census:
      s.sweeper.RunCensus(ctx)
      census = s.after(sweepInterval(settings))
    case <-sweep:
      s.sweeper.RunSweep(ctx)
      sweep = s.after(sweepInterval(settings))
    }
  }
}
func (s *Store) DeleteRunBatch(ctx context.Context, q queryer, cutoff time.Time, floor, batchLimit int) (RunBatchResult, error)
Click for details →

Selects eligible runs by status, age, and floor, then deletes satellites and runs in one transaction with retry on resurrection.

Eligibility subquery
func runEligibleSubquery() string {
  return `
    SELECT id,
           COALESCE(finished_at, requested_at) AS completion_time,
           ROW_NUMBER() OVER (
             PARTITION BY agent_profile_id
             ORDER BY COALESCE(finished_at, requested_at) DESC, id DESC
           ) AS rn
    FROM runs
    WHERE status IN (?)
      AND NOT EXISTS (
        SELECT 1 FROM office_agent_pause_recoveries
        WHERE office_agent_pause_recoveries.failed_run_id = runs.id
      )`
}
Batch delete
func (s *Store) DeleteRunBatch(ctx context.Context, q queryer, cutoff time.Time, floor, batchLimit int) (RunBatchResult, error) {
  for attempt := 0; attempt < 2; attempt++ {
    ids, err := s.selectEligibleRunIDs(ctx, q, cutoff, floor, batchLimit)
    if err != nil {
      return RunBatchResult{}, err
    }
    if len(ids) == 0 {
      return RunBatchResult{}, nil
    }
    result, matched, err := s.deleteRunBatchOnce(ctx, q, ids, cutoff, floor)
    if err != nil {
      return RunBatchResult{}, err
    }
    if matched {
      return result, nil
    }
  }
  return RunBatchResult{Abandoned: true}, nil
}
func (c *Checker) Check(ctx context.Context) []health.Issue
Click for details →

Derives preview, backlog, failure, threshold, disabled, unknown-status, and count-failed issues from live sweep and census state.

Checker
func (c *Checker) Check(ctx context.Context) []health.Issue {
  var issues []health.Issue
  settings, err := c.settingsStore.GetSettings(ctx)
  if err != nil {
    issues = append(issues, issue("office_retention_settings_invalid", "Retention settings unreadable", fmt.Sprintf("Stored retention settings could not be read; using the documented defaults. (%s)", err.Error())))
  }
  if _, readable := c.previewMarker.Get(ctx); !readable {
    issues = append(issues, issue("office_retention_preview_unreadable", "Retention preview marker unreadable", "The retention preview marker could not be read; office_routine_runs and runs will be previewed again on the next sweep rather than deleting."))
  }
  if lastSweep, ok := c.sweeper.LastSweepSnapshot(); ok {
    issues = append(issues, sweptTableIssues(lastSweep, settings)...)
    issues = append(issues, failedTableIssues(lastSweep)...)
  }
  issues = append(issues, c.censusIssues(settings)...)
  sort.Slice(issues, func(i, j int) bool { return issues[i].ID < issues[j].ID })
  return issues
}
function RetentionSettingsCard()
Click for details →

Renders policy controls and live sweep and census status, and saves through the new retention API.

Settings card
export function RetentionSettingsCard() {
  const remote = useRetentionSettings();
  const isAdmin = useIsAdmin();
  const { draft, setDraft, canEdit } = useRetentionDraft(remote, isAdmin);
  if (remote.isLoading && !remote.status) return <RetentionSettingsLoading />;
  if (remote.error && !remote.status) return <RetentionSettingsLoadError error={remote.error} />;
  return (
    <div className="min-w-0 space-y-4" data-testid="retention-settings">
      {draft && <RetentionPolicyCard draft={draft} canEdit={canEdit} onChange={setDraft} />}
      <RetentionStatusCard status={remote.status} />
    </div>
  );
}
API client
export function fetchRetentionStatus(options?: ApiRequestOptions): Promise<RetentionStatus> {
  return fetchJson<RetentionStatus>(`${SYSTEM_BASE}/retention`, {
    ...options,
    cache: "no-store",
  });
}

export function saveRetentionSettings(settings: RetentionSettings, options?: ApiRequestOptions): Promise<RetentionSettings> {
  return fetchJson<RetentionSettings>(`${SYSTEM_BASE}/retention`, {
    ...options,
    init: { ...(options?.init ?? {}), method: "PUT", body: JSON.stringify(settings) },
  });
}
Read the changes as a list

Runtime composition

apps/backend/internal/office/retention/runtime.go

Composes every retention component behind one Start/Stop pair and wires scheduler re-arm to the HTTP handler.

Runtime
type Runtime struct {
  Store         *Store
  SettingsStore *SettingsStore
  PreviewMarker *PreviewMarkerStore
  Sweeper       *Sweeper
  Scheduler     *Scheduler
  Checker       *Checker
  Handler       *Handler
}

func NewRuntime(pool *db.Pool, settingsStore *systemsettings.Store, logError func(string, error)) *Runtime {
  store := NewStore(pool)
  retentionSettings := NewSettingsStore(settingsStore)
  previewMarker := NewPreviewMarkerStore(settingsStore)
  sweeper := NewSweeper(pool, store, retentionSettings, previewMarker)
  scheduler := NewScheduler(retentionSettings, sweeper, SchedulerOptions{})
  checker := NewChecker(retentionSettings, sweeper, previewMarker)
  handler := NewHandler(HandlerConfig{
    SettingsStore:     retentionSettings,
    Sweeper:           sweeper,
    OnSettingsChanged: scheduler.ApplySettings,
    LogError:          logError,
  })
  return &Runtime{Store: store, SettingsStore: retentionSettings, PreviewMarker: previewMarker, Sweeper: sweeper, Scheduler: scheduler, Checker: checker, Handler: handler}
}

Sweep orchestration

apps/backend/internal/office/retention/sweep.go

Runs one sweep in fixed order, handles preview, backlog, satellite deletes, and lock loss between tables.

RunSweep
func (s *Sweeper) RunSweep(ctx context.Context) {
  if !s.beginSweep() {
    s.recordSkip()
    return
  }
  defer s.endSweep()

  settings, err := s.settingsStore.GetSettingsForSweep(ctx)
  if err != nil {
    s.recordSkip()
    return
  }
  if !settings.Enabled {
    return
  }

  q, session, ok := s.acquireQueryer(ctx)
  if !ok {
    s.recordSkip()
    return
  }
  if session != nil {
    defer session.release()
  }

  now := time.Now().UTC()
  report := LastSweep{StartedAt: now}
  report.OfficeRoutineRuns = s.sweepRoutineRuns(ctx, q, settings.RoutineRuns, now, settings.BatchLimit)

  if session != nil && !session.alive(ctx) {
    s.recordSkip()
    return
  }

  runsResult, satellites := s.sweepRuns(ctx, q, settings.Runs, now, settings.BatchLimit)
  report.Runs = runsResult
  report.RunEvents = satellites.RunEvents
  report.RouteAttempts = satellites.RouteAttempts
  report.RunSkills = satellites.RunSkills

  report.FinishedAt = time.Now().UTC()
  s.mu.Lock()
  s.lastSweep = &report
  s.mu.Unlock()
  incSweepCompleted()
}

Scheduler with dual timers

apps/backend/internal/office/retention/scheduler.go

Owns census and sweep timers on one goroutine and re-arms both on settings change with fixed-delay semantics.

Scheduler loop
func (s *Scheduler) run(ctx context.Context, settings Settings, wake <-chan struct{}) {
  defer s.wg.Done()
  s.sweeper.RunCensus(ctx)
  census := s.after(sweepInterval(settings))

  var sweep <-chan time.Time
  if settings.Enabled {
    sweep = s.after(firstSweepDelay)
  }

  for {
    select {
    case <-ctx.Done():
      return
    case <-wake:
      wasEnabled := settings.Enabled
      settings = s.latestSettings()
      sweep = nil
      if settings.Enabled {
        if wasEnabled {
          sweep = s.after(sweepInterval(settings))
        } else {
          sweep = s.after(firstSweepDelay)
        }
      }
      census = s.after(sweepInterval(settings))
    case <-census:
      s.sweeper.RunCensus(ctx)
      census = s.after(sweepInterval(settings))
    case <-sweep:
      s.sweeper.RunSweep(ctx)
      sweep = s.after(sweepInterval(settings))
    }
  }
}

Store SQL and batch deletes

apps/backend/internal/office/retention/store.go

Selects eligible runs by status, age, and floor, then deletes satellites and runs in one transaction with retry on resurrection.

Eligibility subquery
func runEligibleSubquery() string {
  return `
    SELECT id,
           COALESCE(finished_at, requested_at) AS completion_time,
           ROW_NUMBER() OVER (
             PARTITION BY agent_profile_id
             ORDER BY COALESCE(finished_at, requested_at) DESC, id DESC
           ) AS rn
    FROM runs
    WHERE status IN (?)
      AND NOT EXISTS (
        SELECT 1 FROM office_agent_pause_recoveries
        WHERE office_agent_pause_recoveries.failed_run_id = runs.id
      )`
}
Batch delete
func (s *Store) DeleteRunBatch(ctx context.Context, q queryer, cutoff time.Time, floor, batchLimit int) (RunBatchResult, error) {
  for attempt := 0; attempt < 2; attempt++ {
    ids, err := s.selectEligibleRunIDs(ctx, q, cutoff, floor, batchLimit)
    if err != nil {
      return RunBatchResult{}, err
    }
    if len(ids) == 0 {
      return RunBatchResult{}, nil
    }
    result, matched, err := s.deleteRunBatchOnce(ctx, q, ids, cutoff, floor)
    if err != nil {
      return RunBatchResult{}, err
    }
    if matched {
      return result, nil
    }
  }
  return RunBatchResult{Abandoned: true}, nil
}

Health checker

apps/backend/internal/office/retention/health.go

Derives preview, backlog, failure, threshold, disabled, unknown-status, and count-failed issues from live sweep and census state.

Checker
func (c *Checker) Check(ctx context.Context) []health.Issue {
  var issues []health.Issue
  settings, err := c.settingsStore.GetSettings(ctx)
  if err != nil {
    issues = append(issues, issue("office_retention_settings_invalid", "Retention settings unreadable", fmt.Sprintf("Stored retention settings could not be read; using the documented defaults. (%s)", err.Error())))
  }
  if _, readable := c.previewMarker.Get(ctx); !readable {
    issues = append(issues, issue("office_retention_preview_unreadable", "Retention preview marker unreadable", "The retention preview marker could not be read; office_routine_runs and runs will be previewed again on the next sweep rather than deleting."))
  }
  if lastSweep, ok := c.sweeper.LastSweepSnapshot(); ok {
    issues = append(issues, sweptTableIssues(lastSweep, settings)...)
    issues = append(issues, failedTableIssues(lastSweep)...)
  }
  issues = append(issues, c.censusIssues(settings)...)
  sort.Slice(issues, func(i, j int) bool { return issues[i].ID < issues[j].ID })
  return issues
}

System UI and API

apps/web/components/settings/system/retention-settings-card.tsx

Renders policy controls and live sweep and census status, and saves through the new retention API.

Settings card
export function RetentionSettingsCard() {
  const remote = useRetentionSettings();
  const isAdmin = useIsAdmin();
  const { draft, setDraft, canEdit } = useRetentionDraft(remote, isAdmin);
  if (remote.isLoading && !remote.status) return <RetentionSettingsLoading />;
  if (remote.error && !remote.status) return <RetentionSettingsLoadError error={remote.error} />;
  return (
    <div className="min-w-0 space-y-4" data-testid="retention-settings">
      {draft && <RetentionPolicyCard draft={draft} canEdit={canEdit} onChange={setDraft} />}
      <RetentionStatusCard status={remote.status} />
    </div>
  );
}
API client
export function fetchRetentionStatus(options?: ApiRequestOptions): Promise<RetentionStatus> {
  return fetchJson<RetentionStatus>(`${SYSTEM_BASE}/retention`, {
    ...options,
    cache: "no-store",
  });
}

export function saveRetentionSettings(settings: RetentionSettings, options?: ApiRequestOptions): Promise<RetentionSettings> {
  return fetchJson<RetentionSettings>(`${SYSTEM_BASE}/retention`, {
    ...options,
    init: { ...(options?.init ?? {}), method: "PUT", body: JSON.stringify(settings) },
  });
}

Data and storage

Retention adds policy, sweep result, and census types. Settings persist as one JSON document; sweep and census are in-memory views.

FieldTypeNotes
Settings.enabledboolmaster switch; when false no sweep runs
Settings.sweep_interval_hoursint 1..168fixed-delay interval between sweeps
Settings.batch_limitint 100..100000max rows deleted per table per sweep
TableSettings.window_daysint 1..3650age past which history becomes eligible
TableSettings.floor_per_ownerint 0..10000newest rows kept per routine or agent
TableSettings.warn_rowsint >=0threshold for retained-count warning
LastSweep.office_routine_runsSweptTableResultdeleted, backlog, previewed, would_delete, error
LastSweep.runsSweptTableResultsame, plus satellites run_events, route_attempts, run_skills
TableCensus.stateenum fresh/stale/not_computedtri-state freshness of retained count
TableCensus.retained_countint64current rows in table
TableCensus.unknown_statuses[]UnknownStatusCountunrecognized statuses treated as live
PreviewMarkermap TableName -> timeper-table first-preview completion, never cleared

Risk

6 / 10 Medium
1 low5 medium10 high

Why this score

  • Deletes production history rows; a window or floor mistake removes data that cannot be restored without a backup.
  • Touches both SQLite and PostgreSQL with advisory locks, chunked deletes, and expression indexes that must stay in sync.
  • Large new surface with 69 files but strong test coverage and preview gate that deletes nothing on first sweep.

Trade-offs and review notes

Where to look first

  1. Verify eligibility re-assertion in store.go: status, age, floor, and pause-recovery exclusion must all appear in the DELETE.
  2. Check scheduler fixed-delay and firstSweepDelay logic, including disabled-to-enabled re-arm and census refresh of settings.
  3. Confirm health issue ids and messages match the spec, especially preview_pending, backlog, threshold vs disabled, and unknown status.
  4. Review preview marker GetWith/MarkCompletedWith on the sweep connection to avoid pool deadlock on PostgreSQL.