PR #3290
Sections
Review

fix(office): stop parent tasks from stalling when a reopened child finishes again

main ← feature/office-reopened-chil-0jr 11 files +412 −38 PR #3290 ↗

Parent tasks now wake again after a child is reopened and recompleted with the same terminal state, by storing a child generation and normalizing timestamp text across SQLite and Postgres.

Why this change

A child that completes, is reopened, and completes again with the same state produces the same child_set_key. The reconciler and the edge path both treat the second completion as already delivered, so the parent never wakes and stalls.

What it does

Architecture, end to end

Two producers race for the same completion wave. The fix makes both derive the same generation-aware operation id and lets the level-triggered sweep detect a same-key reopen.

flowchart LR
  Child[Child task state change] --> Edge[queueChildrenCompletedRun\nedge path]
  Child --> Tasks[(tasks)]
  Tasks --> Sweep[ListStuckParents\nsweep every 30s]
  Sweep --> Receipt[(parent_child_wake_receipts)]
  Sweep --> Reconciler[ParentWakeReconciler]
  Edge --> Engine[Workflow engine]
  Reconciler --> Engine
  Engine --> Runs[(runs)]
  Receipt -- child_set_key + child_generation --> Sweep
  Runs -- queued/claimed/terminal --> Sweep

Key code changes

Drag to pan. Use the + and − buttons to zoom. Click a node to open the full code. The arrows show how the parts interact.

drag to pan · +/− to zoom · click a node for details

Portable second-precision timestamp textapps/backend/internal/db/dialect/time.go ↗
func SecondPrecisionText(driver, expr string) string
Click for details →

Normalizes timestamp rendering so SQLite whole-second text and Postgres microsecond text compare as equal strings.

New helper
func SecondPrecisionText(driver, expr string) string {
	if IsPostgres(driver) {
		return fmt.Sprintf("to_char(%s, 'YYYY-MM-DD HH24:MI:SS')", expr)
	}
	return fmt.Sprintf("strftime('%%Y-%%m-%%d %%H:%%M:%%S', %s)", expr)
}
type WakeReceipt struct
Click for details →

Stores the newest child updated_at at delivery time so a same-key reopen is still distinguishable.

Struct
type WakeReceipt struct {
	ParentTaskID        string    `db:"parent_task_id"`
	ChildSetKey         string    `db:"child_set_key"`
	DeliveredRunID      string    `db:"delivered_run_id"`
	DeliveryOperationID string    `db:"delivery_operation_id"`
	DeliveredAt         time.Time `db:"delivered_at"`
	ChildGeneration string `db:"child_generation"`
}
Upsert
func (r *Repository) UpsertWakeReceiptTx(
	ctx context.Context, tx *sqlx.Tx,
	parentTaskID, childSetKey, deliveredRunID, deliveryOperationID, childGeneration string,
	deliveredAt time.Time,
) error {
	_, err := tx.ExecContext(ctx, tx.Rebind(`
		INSERT INTO parent_child_wake_receipts (
			parent_task_id, child_set_key, delivered_run_id,
			delivery_operation_id, delivered_at, child_generation
		) VALUES (?, ?, ?, ?, ?, ?)
		ON CONFLICT (parent_task_id) DO UPDATE SET
			child_set_key = excluded.child_set_key,
			delivered_run_id = excluded.delivered_run_id,
			delivery_operation_id = excluded.delivery_operation_id,
			delivered_at = excluded.delivered_at,
			child_generation = excluded.child_generation
	`), parentTaskID, childSetKey, deliveredRunID, deliveryOperationID, deliveredAt, childGeneration)
	return err
}
Generation helper
func (r *Repository) GetChildSetKeyAndGeneration(ctx context.Context, parentTaskID string) (string, string, error) {
	var rows []childSetKeyRow
	if err := r.ro.SelectContext(ctx, &rows, r.ro.Rebind(`
		SELECT id, state FROM tasks WHERE parent_id = ? AND archived_at IS NULL ORDER BY id
	`), parentTaskID); err != nil {
		return "", "", err
	}
	driver := r.ro.DriverName()
	generationText := dialect.SecondPrecisionText(driver, "MAX(updated_at)")
	var generation sql.NullString
	if err := r.ro.GetContext(ctx, &generation, r.ro.Rebind(`
		SELECT `+generationText+` FROM tasks WHERE parent_id = ? AND archived_at IS NULL
	`), parentTaskID); err != nil {
		return "", "", err
	}
	return formatChildSetKey(rows), generation.String, nil
}
func (r *Repository) ListStuckParents(ctx context.Context, reason string, limit int) ([]StuckParentCandidate, error)
Click for details →

Adds a generation equality check so a byte-identical child_set_key with a newer updated_at is swept again.

Candidate
type StuckParentCandidate struct {
	ParentTaskID           string `db:"parent_task_id"`
	AssigneeAgentProfileID string `db:"assignee_agent_profile_id"`
	WorkflowStepID         string `db:"workflow_step_id"`
	ChildSetKey            string `db:"child_set_key"`
	NewestChildUpdatedAt string `db:"newest_child_updated_at"`
}
Query
driver := r.ro.DriverName()
newestChildUpdatedAtText := dialect.SecondPrecisionText(driver, "MAX(c.updated_at)")
requestedAtText := dialect.SecondPrecisionText(driver, "w.requested_at")
// in CTE:
// SELECT strftime('%Y-%m-%d %H:%M:%S', MAX(c.updated_at)) AS newest_child_updated_at
// WHERE clause:
// AND (
//   r.child_set_key IS DISTINCT FROM s.child_set_key
//   OR (NOT EXISTS (SELECT 1 FROM runs delivered WHERE delivered.id = r.delivered_run_id) AND COALESCE(r.delivery_operation_id, '') = '')
//   OR s.newest_child_updated_at != COALESCE(r.child_generation, '')
// )
// AND NOT EXISTS (
//   SELECT 1 FROM runs w WHERE json_extract(w.payload, '$.task_id') = s.parent_task_id
//     AND w.reason = ? AND (w.status IN ('queued','claimed') OR (w.status IN ('finished','failed','cancelled') AND to_char(w.requested_at,'YYYY-MM-DD HH24:MI:SS') >= s.newest_child_updated_at))
// )
func wakeOperationID(parentTaskID, childSetKey, generation string) string
Click for details →

Hashes parent, child ids, and generation so a new generation mints a new id and the engine does not swallow it as already applied.

Operation id
func wakeOperationID(parentTaskID, childSetKey, generation string) string {
	childIDs := make([]string, 0)
	for _, child := range strings.Split(childSetKey, ",") {
		if child == "" { continue }
		if separator := strings.LastIndexByte(child, ':'); separator >= 0 {
			child = child[:separator]
		}
		childIDs = append(childIDs, child)
	}
	canonicalChildSet := strings.Join(childIDs, ",")
	sum := sha256.Sum256([]byte(parentTaskID + "\x00" + canonicalChildSet + "\x00" + generation))
	return fmt.Sprintf("task_children_completed:%s:%s", parentTaskID, hex.EncodeToString(sum[:]))
}
Reconciler use
currentKey, err := svc.repo.GetChildSetKey(ctx, c.ParentTaskID)
if currentKey != c.ChildSetKey { return }
operationID := wakeOperationID(c.ParentTaskID, c.ChildSetKey, c.NewestChildUpdatedAt)
accepted, err := svc.dispatchEngineTriggerForRecovery(ctx, c.ParentTaskID, engine.TriggerOnChildrenCompleted, payload, operationID)
if !accepted { return }
h.recordReceipt(ctx, svc, c, operationID)
// recordReceipt persists c.NewestChildUpdatedAt as child_generation
if err := svc.repo.UpsertWakeReceiptTx(ctx, tx, c.ParentTaskID, c.ChildSetKey, "", operationID, c.NewestChildUpdatedAt, deliveredAt); err != nil { return }
Edge path uses same generation and operation idapps/backend/internal/office/service/event_subscribers.go ↗
func (s *Service) queueChildrenCompletedRun(ctx context.Context, parentID string) error
Click for details →

Makes the edge-triggered dispatch and the reconciler derive the identical id for the same wave, so idempotency collapses the race.

Handler
func (s *Service) queueChildrenCompletedRun(ctx context.Context, parentID string) error {
	allDone, err := s.repo.AreAllChildrenTerminal(ctx, parentID)
	if err != nil || !allDone { return err }
	childSetKey, generation, err := s.repo.GetChildSetKeyAndGeneration(ctx, parentID)
	if err != nil { return fmt.Errorf("get child set key: %w", err) }
	children, _, err := s.repo.GetChildSummaries(ctx, parentID)
	key := wakeOperationID(parentID, childSetKey, generation)
	summaries := make([]engine.ChildSummary, 0, len(children))
	prsByTask := s.lookupChildPRLinks(ctx, children)
	for _, c := range children {
		summaries = append(summaries, engine.ChildSummary{TaskID: c.TaskID, Status: c.State, Summary: c.LastComment, PRLinks: prsByTask[c.TaskID]})
	}
	return s.dispatchEngineTrigger(ctx, parentID, engine.TriggerOnChildrenCompleted, engine.OnChildrenCompletedPayload{ChildSummaries: summaries}, key)
}
func (r *Repository) createParentChildWakeReceiptsTable() error
Click for details →

Creates the column on fresh databases and migrates existing ones so the generation check has storage.

Create table
CREATE TABLE IF NOT EXISTS parent_child_wake_receipts (
	parent_task_id        TEXT PRIMARY KEY,
	child_set_key         TEXT NOT NULL,
	delivered_run_id      TEXT NOT NULL DEFAULT '',
	delivery_operation_id TEXT NOT NULL DEFAULT '',
	delivered_at          TIMESTAMP NOT NULL,
	child_generation      TEXT NOT NULL DEFAULT ''
);
Migration
func (r *Repository) migrateParentWakeReceiptColumns() {
	r.migrate.Apply("parent_child_wake_receipts.delivery_operation_id", `ALTER TABLE parent_child_wake_receipts ADD COLUMN delivery_operation_id TEXT NOT NULL DEFAULT ''`)
	r.migrate.Apply("parent_child_wake_receipts.child_generation", `ALTER TABLE parent_child_wake_receipts ADD COLUMN child_generation TEXT NOT NULL DEFAULT ''`)
}
Read the changes as a list

Portable second-precision timestamp text

apps/backend/internal/db/dialect/time.go

Normalizes timestamp rendering so SQLite whole-second text and Postgres microsecond text compare as equal strings.

New helper
func SecondPrecisionText(driver, expr string) string {
	if IsPostgres(driver) {
		return fmt.Sprintf("to_char(%s, 'YYYY-MM-DD HH24:MI:SS')", expr)
	}
	return fmt.Sprintf("strftime('%%Y-%%m-%%d %%H:%%M:%%S', %s)", expr)
}

Receipt now stores child generation

apps/backend/internal/office/repository/sqlite/wake_receipts.go

Stores the newest child updated_at at delivery time so a same-key reopen is still distinguishable.

Struct
type WakeReceipt struct {
	ParentTaskID        string    `db:"parent_task_id"`
	ChildSetKey         string    `db:"child_set_key"`
	DeliveredRunID      string    `db:"delivered_run_id"`
	DeliveryOperationID string    `db:"delivery_operation_id"`
	DeliveredAt         time.Time `db:"delivered_at"`
	ChildGeneration string `db:"child_generation"`
}
Upsert
func (r *Repository) UpsertWakeReceiptTx(
	ctx context.Context, tx *sqlx.Tx,
	parentTaskID, childSetKey, deliveredRunID, deliveryOperationID, childGeneration string,
	deliveredAt time.Time,
) error {
	_, err := tx.ExecContext(ctx, tx.Rebind(`
		INSERT INTO parent_child_wake_receipts (
			parent_task_id, child_set_key, delivered_run_id,
			delivery_operation_id, delivered_at, child_generation
		) VALUES (?, ?, ?, ?, ?, ?)
		ON CONFLICT (parent_task_id) DO UPDATE SET
			child_set_key = excluded.child_set_key,
			delivered_run_id = excluded.delivered_run_id,
			delivery_operation_id = excluded.delivery_operation_id,
			delivered_at = excluded.delivered_at,
			child_generation = excluded.child_generation
	`), parentTaskID, childSetKey, deliveredRunID, deliveryOperationID, deliveredAt, childGeneration)
	return err
}
Generation helper
func (r *Repository) GetChildSetKeyAndGeneration(ctx context.Context, parentTaskID string) (string, string, error) {
	var rows []childSetKeyRow
	if err := r.ro.SelectContext(ctx, &rows, r.ro.Rebind(`
		SELECT id, state FROM tasks WHERE parent_id = ? AND archived_at IS NULL ORDER BY id
	`), parentTaskID); err != nil {
		return "", "", err
	}
	driver := r.ro.DriverName()
	generationText := dialect.SecondPrecisionText(driver, "MAX(updated_at)")
	var generation sql.NullString
	if err := r.ro.GetContext(ctx, &generation, r.ro.Rebind(`
		SELECT `+generationText+` FROM tasks WHERE parent_id = ? AND archived_at IS NULL
	`), parentTaskID); err != nil {
		return "", "", err
	}
	return formatChildSetKey(rows), generation.String, nil
}

Sweep re-admits same-key reopen

apps/backend/internal/office/repository/sqlite/wake_receipts.go

Adds a generation equality check so a byte-identical child_set_key with a newer updated_at is swept again.

Candidate
type StuckParentCandidate struct {
	ParentTaskID           string `db:"parent_task_id"`
	AssigneeAgentProfileID string `db:"assignee_agent_profile_id"`
	WorkflowStepID         string `db:"workflow_step_id"`
	ChildSetKey            string `db:"child_set_key"`
	NewestChildUpdatedAt string `db:"newest_child_updated_at"`
}
Query
driver := r.ro.DriverName()
newestChildUpdatedAtText := dialect.SecondPrecisionText(driver, "MAX(c.updated_at)")
requestedAtText := dialect.SecondPrecisionText(driver, "w.requested_at")
// in CTE:
// SELECT strftime('%Y-%m-%d %H:%M:%S', MAX(c.updated_at)) AS newest_child_updated_at
// WHERE clause:
// AND (
//   r.child_set_key IS DISTINCT FROM s.child_set_key
//   OR (NOT EXISTS (SELECT 1 FROM runs delivered WHERE delivered.id = r.delivered_run_id) AND COALESCE(r.delivery_operation_id, '') = '')
//   OR s.newest_child_updated_at != COALESCE(r.child_generation, '')
// )
// AND NOT EXISTS (
//   SELECT 1 FROM runs w WHERE json_extract(w.payload, '$.task_id') = s.parent_task_id
//     AND w.reason = ? AND (w.status IN ('queued','claimed') OR (w.status IN ('finished','failed','cancelled') AND to_char(w.requested_at,'YYYY-MM-DD HH24:MI:SS') >= s.newest_child_updated_at))
// )

Generation-aware operation id and receipt

apps/backend/internal/office/service/scheduler_wake_reconciler.go

Hashes parent, child ids, and generation so a new generation mints a new id and the engine does not swallow it as already applied.

Operation id
func wakeOperationID(parentTaskID, childSetKey, generation string) string {
	childIDs := make([]string, 0)
	for _, child := range strings.Split(childSetKey, ",") {
		if child == "" { continue }
		if separator := strings.LastIndexByte(child, ':'); separator >= 0 {
			child = child[:separator]
		}
		childIDs = append(childIDs, child)
	}
	canonicalChildSet := strings.Join(childIDs, ",")
	sum := sha256.Sum256([]byte(parentTaskID + "\x00" + canonicalChildSet + "\x00" + generation))
	return fmt.Sprintf("task_children_completed:%s:%s", parentTaskID, hex.EncodeToString(sum[:]))
}
Reconciler use
currentKey, err := svc.repo.GetChildSetKey(ctx, c.ParentTaskID)
if currentKey != c.ChildSetKey { return }
operationID := wakeOperationID(c.ParentTaskID, c.ChildSetKey, c.NewestChildUpdatedAt)
accepted, err := svc.dispatchEngineTriggerForRecovery(ctx, c.ParentTaskID, engine.TriggerOnChildrenCompleted, payload, operationID)
if !accepted { return }
h.recordReceipt(ctx, svc, c, operationID)
// recordReceipt persists c.NewestChildUpdatedAt as child_generation
if err := svc.repo.UpsertWakeReceiptTx(ctx, tx, c.ParentTaskID, c.ChildSetKey, "", operationID, c.NewestChildUpdatedAt, deliveredAt); err != nil { return }

Edge path uses same generation and operation id

apps/backend/internal/office/service/event_subscribers.go

Makes the edge-triggered dispatch and the reconciler derive the identical id for the same wave, so idempotency collapses the race.

Handler
func (s *Service) queueChildrenCompletedRun(ctx context.Context, parentID string) error {
	allDone, err := s.repo.AreAllChildrenTerminal(ctx, parentID)
	if err != nil || !allDone { return err }
	childSetKey, generation, err := s.repo.GetChildSetKeyAndGeneration(ctx, parentID)
	if err != nil { return fmt.Errorf("get child set key: %w", err) }
	children, _, err := s.repo.GetChildSummaries(ctx, parentID)
	key := wakeOperationID(parentID, childSetKey, generation)
	summaries := make([]engine.ChildSummary, 0, len(children))
	prsByTask := s.lookupChildPRLinks(ctx, children)
	for _, c := range children {
		summaries = append(summaries, engine.ChildSummary{TaskID: c.TaskID, Status: c.State, Summary: c.LastComment, PRLinks: prsByTask[c.TaskID]})
	}
	return s.dispatchEngineTrigger(ctx, parentID, engine.TriggerOnChildrenCompleted, engine.OnChildrenCompletedPayload{ChildSummaries: summaries}, key)
}

Schema adds child_generation column

apps/backend/internal/office/repository/sqlite/base.go

Creates the column on fresh databases and migrates existing ones so the generation check has storage.

Create table
CREATE TABLE IF NOT EXISTS parent_child_wake_receipts (
	parent_task_id        TEXT PRIMARY KEY,
	child_set_key         TEXT NOT NULL,
	delivered_run_id      TEXT NOT NULL DEFAULT '',
	delivery_operation_id TEXT NOT NULL DEFAULT '',
	delivered_at          TIMESTAMP NOT NULL,
	child_generation      TEXT NOT NULL DEFAULT ''
);
Migration
func (r *Repository) migrateParentWakeReceiptColumns() {
	r.migrate.Apply("parent_child_wake_receipts.delivery_operation_id", `ALTER TABLE parent_child_wake_receipts ADD COLUMN delivery_operation_id TEXT NOT NULL DEFAULT ''`)
	r.migrate.Apply("parent_child_wake_receipts.child_generation", `ALTER TABLE parent_child_wake_receipts ADD COLUMN child_generation TEXT NOT NULL DEFAULT ''`)
}

Data and storage

One new TEXT column and one new candidate field let the sweep distinguish a same-key reopen without changing the child_set_key format.

FieldTypeNotes
parent_child_wake_receipts.child_generationTEXTnewest_child_updated_at at delivery, rendered via SecondPrecisionText; empty for legacy rows
StuckParentCandidate.NewestChildUpdatedAtstringMAX(c.updated_at) rendered via SecondPrecisionText, compared to child_generation
StuckParentCandidate.ChildSetKeystringGROUP_CONCAT id:state, unchanged; still compared via IS DISTINCT FROM
WakeReceipt.ChildGenerationstringpersisted generation, read by GetWakeReceipt and written by UpsertWakeReceiptTx

Risk

5 / 10 Medium
1 low5 medium10 high

Why this score

  • Touches the 30-second reconciler sweep and its SQL, which runs on every tick for every parent.
  • Adds a schema migration that must be idempotent on both SQLite and Postgres and re-admits legacy receipts once.
  • Well covered by new and existing tests, including a Postgres twin and same-second edge cases, but the fix accepts a same-second reopen suppressed by an edge run as a residual.

Trade-offs and review notes

Where to look first

  1. Verify ListStuckParents third OR arm and the SecondPrecisionText use on both newest_child_updated_at and requested_at.
  2. Check wakeOperationID now includes generation and that both edge and reconciler paths call it with the same inputs.
  3. Confirm UpsertWakeReceiptTx and GetChildSetKeyAndGeneration wire generation correctly and the migration is replay-safe.
  4. Review the accepted residual: same-second reopen after an edge run is not recovered, later second is.