PR #3486
Sections
Review

test(office): prove a dropped children-completed wake gets recovered

main ← test/parent-wake-reconciler-recovers-failed-edge-dispatch 1 files +227 −2 PR #3486 ↗

This PR adds two regression tests that prove ParentWakeReconciler recovers a failed on_children_completed wake and does not double-queue a successful one.

Why this change

queueChildrenCompletedRun can fail after AreAllChildrenTerminal succeeds, and finalizeDone only logs the failure at Warn because ParentWakeReconciler is the documented recovery path. No test proved that recovery actually happens.

What it does

Architecture, end to end

The edge path fires on task.moved. The level path sweeps stuck parents on Tick. Both use the same operation id so the second dispatch dedupes.

flowchart LR
  Child[Child task enters Done] --> Event[task.moved event]
  Event --> Finalize[finalizeDone]
  Finalize --> Edge[queueChildrenCompletedRun]
  Edge --> Dispatcher[Workflow dispatcher]
  Dispatcher -- fail --> Warn[Warn log]
  Tick[ParentWakeReconciler Tick] --> List[ListStuckParents]
  List --> Reconcile[reconcileOne]
  Reconcile --> Dispatcher
  Dispatcher --> Run[(runs + wake receipt)]
  Edge -. same operation id .-> Reconcile

Key code changes

Drag to pan. Use the + and − buttons to zoom. Click a node to open the full code. The arrows show how the parts interact.

drag to pan · +/− to zoom · click a node for details

func (d *oneShotFailureDispatcher) HandleTrigger(...) error
Click for details →

This helper fails the first on_children_completed dispatch and records every call so the test can compare operation ids.

Helper
type oneShotFailureDispatcher struct {
	inner       *queueRunDispatcher
	failTrigger engine.Trigger
	failArmed   bool
	failErr     error

	mu    sync.Mutex
	calls []dispatcherCall
}

func (d *oneShotFailureDispatcher) HandleTrigger(
	ctx context.Context, taskID string, trigger engine.Trigger, payload any, opID string,
) error {
	d.mu.Lock()
	d.calls = append(d.calls, dispatcherCall{taskID, trigger, payload, opID})
	d.mu.Unlock()
	if d.failArmed && trigger == d.failTrigger {
		d.failArmed = false
		return d.failErr
	}
	return d.inner.HandleTrigger(ctx, taskID, trigger, payload, opID)
}
func (d *countingDispatcher) HandleTrigger(...) error
Click for details →

This helper counts dispatch attempts so the control test can prove the reconciler never tries to dispatch behind a successful edge delivery.

Helper
type countingDispatcher struct {
	inner *queueRunDispatcher

	mu sync.Mutex
	n  int
}

func (d *countingDispatcher) HandleTrigger(
	ctx context.Context, taskID string, trigger engine.Trigger, payload any, opID string,
) error {
	d.mu.Lock()
	d.n++
	d.mu.Unlock()
	return d.inner.HandleTrigger(ctx, taskID, trigger, payload, opID)
}

func (d *countingDispatcher) Count() int {
	d.mu.Lock()
	defer d.mu.Unlock()
	return d.n
}
func TestParentWakeReconciler_RecoversFailedEdgeDispatch(t *testing.T)
Click for details →

This test injects a failure on the edge path, then proves Tick re-delivers the same wake with the same operation id as a real run.

Setup and edge failure
func TestParentWakeReconciler_RecoversFailedEdgeDispatch(t *testing.T) {
	svc, eb := newTestServiceWithBus(t)
	ctx := context.Background()

	dispatcher := &oneShotFailureDispatcher{
		inner:       &queueRunDispatcher{svc: svc},
		failTrigger: engine.TriggerOnChildrenCompleted,
		failArmed:   true,
		failErr:     fmt.Errorf("injected on_children_completed dispatch failure"),
	}
	svc.SetWorkflowEngineDispatcher(dispatcher)

	adoptOffice(t, svc, "ws-1")
	seedStuckParent(t, svc, "ws-1", "parent-1", "worker-1")

	publishChildDone(t, eb, "parent-1", "worker-1")

	calls := dispatcher.Calls()
	if len(calls) != 1 {
		t.Fatalf("want exactly one dispatch attempt after the failed edge publish, got %d", len(calls))
	}
	edgeOpID := calls[0].opID
	runs, _ := svc.ListRuns(ctx, "ws-1")
	if len(runs) != 0 {
		t.Fatalf("failed edge dispatch left a queued run: %#v", runs)
	}
}
Reconciler recovery
handler := service.NewParentWakeReconciler(service.NewSchedulerIntegration(svc, 0))
	if err := handler.Tick(ctx); err != nil {
		t.Fatalf("tick: %v", err)
	}

	calls = dispatcher.Calls()
	if len(calls) != 2 {
		t.Fatalf("want a second dispatch attempt from the reconciler, got %d", len(calls))
	}
	if calls[1].opID != edgeOpID {
		t.Fatalf("reconciler recovery used a different operation id: edge=%q reconciler=%q", edgeOpID, calls[1].opID)
	}

	runsAfter, _ := svc.ListRuns(ctx, "ws-1")
	if len(runsAfter) != 1 {
		t.Fatalf("want exactly one queued run after recovery, got %d", len(runsAfter))
	}
	if runsAfter[0].Reason != service.RunReasonTaskChildrenCompleted {
		t.Fatalf("recovered run reason = %q", runsAfter[0].Reason)
	}
Control test: successful edge is not double-queuedapps/backend/internal/office/service/scheduler_wake_reconciler_test.go ↗
func TestParentWakeReconciler_DoesNotDoubleQueueASuccessfulEdgeDispatch(t *testing.T)
Click for details →

This control uses the same setup but lets the edge succeed, then proves Tick adds nothing.

Control
func TestParentWakeReconciler_DoesNotDoubleQueueASuccessfulEdgeDispatch(t *testing.T) {
	svc, eb := newTestServiceWithBus(t)
	ctx := context.Background()

	dispatcher := &countingDispatcher{inner: &queueRunDispatcher{svc: svc}}
	svc.SetWorkflowEngineDispatcher(dispatcher)

	adoptOffice(t, svc, "ws-1")
	seedStuckParent(t, svc, "ws-1", "parent-1", "worker-1")

	publishChildDone(t, eb, "parent-1", "worker-1")

	runs, _ := svc.ListRuns(ctx, "ws-1")
	if len(runs) != 1 {
		t.Fatalf("successful edge dispatch did not queue a run: %#v", runs)
	}

	handler := service.NewParentWakeReconciler(service.NewSchedulerIntegration(svc, 0))
	if err := handler.Tick(ctx); err != nil {
		t.Fatalf("tick: %v", err)
	}

	if dispatcher.Count() != 1 {
		t.Fatalf("want the reconciler to skip dispatch entirely, got %d attempts", dispatcher.Count())
	}
	runsAfter, _ := svc.ListRuns(ctx, "ws-1")
	if len(runsAfter) != 1 {
		t.Fatalf("tick added a run behind a successful edge dispatch: %#v", runsAfter)
	}
}
func publishChildDone(t *testing.T, eb bus.EventBus, parentID, workerID string)
Click for details →

This helper publishes the exact task.moved shape that triggers finalizeDone, so both tests use the same edge path.

Helper
func publishChildDone(t *testing.T, eb bus.EventBus, parentID, workerID string) {
	t.Helper()
	ctx := context.Background()
	moveEvent := bus.NewEvent("task.moved", "test", map[string]string{
		"task_id":                   parentID + "-child-0",
		"workspace_id":              "ws-1",
		"from_step_id":              "step-1",
		"to_step_id":                "step-done",
		"to_step_name":              "Done",
		"from_step_name":            "In Progress",
		"assignee_agent_profile_id": workerID,
		"parent_id":                 parentID,
	})
	if err := eb.Publish(ctx, "task.moved", moveEvent); err != nil {
		t.Fatalf("publish task.moved: %v", err)
	}
}
Read the changes as a list

One-shot failure dispatcher

apps/backend/internal/office/service/scheduler_wake_reconciler_test.go

This helper fails the first on_children_completed dispatch and records every call so the test can compare operation ids.

Helper
type oneShotFailureDispatcher struct {
	inner       *queueRunDispatcher
	failTrigger engine.Trigger
	failArmed   bool
	failErr     error

	mu    sync.Mutex
	calls []dispatcherCall
}

func (d *oneShotFailureDispatcher) HandleTrigger(
	ctx context.Context, taskID string, trigger engine.Trigger, payload any, opID string,
) error {
	d.mu.Lock()
	d.calls = append(d.calls, dispatcherCall{taskID, trigger, payload, opID})
	d.mu.Unlock()
	if d.failArmed && trigger == d.failTrigger {
		d.failArmed = false
		return d.failErr
	}
	return d.inner.HandleTrigger(ctx, taskID, trigger, payload, opID)
}

Counting dispatcher

apps/backend/internal/office/service/scheduler_wake_reconciler_test.go

This helper counts dispatch attempts so the control test can prove the reconciler never tries to dispatch behind a successful edge delivery.

Helper
type countingDispatcher struct {
	inner *queueRunDispatcher

	mu sync.Mutex
	n  int
}

func (d *countingDispatcher) HandleTrigger(
	ctx context.Context, taskID string, trigger engine.Trigger, payload any, opID string,
) error {
	d.mu.Lock()
	d.n++
	d.mu.Unlock()
	return d.inner.HandleTrigger(ctx, taskID, trigger, payload, opID)
}

func (d *countingDispatcher) Count() int {
	d.mu.Lock()
	defer d.mu.Unlock()
	return d.n
}

Recovery test: failed edge is re-delivered

apps/backend/internal/office/service/scheduler_wake_reconciler_test.go

This test injects a failure on the edge path, then proves Tick re-delivers the same wake with the same operation id as a real run.

Setup and edge failure
func TestParentWakeReconciler_RecoversFailedEdgeDispatch(t *testing.T) {
	svc, eb := newTestServiceWithBus(t)
	ctx := context.Background()

	dispatcher := &oneShotFailureDispatcher{
		inner:       &queueRunDispatcher{svc: svc},
		failTrigger: engine.TriggerOnChildrenCompleted,
		failArmed:   true,
		failErr:     fmt.Errorf("injected on_children_completed dispatch failure"),
	}
	svc.SetWorkflowEngineDispatcher(dispatcher)

	adoptOffice(t, svc, "ws-1")
	seedStuckParent(t, svc, "ws-1", "parent-1", "worker-1")

	publishChildDone(t, eb, "parent-1", "worker-1")

	calls := dispatcher.Calls()
	if len(calls) != 1 {
		t.Fatalf("want exactly one dispatch attempt after the failed edge publish, got %d", len(calls))
	}
	edgeOpID := calls[0].opID
	runs, _ := svc.ListRuns(ctx, "ws-1")
	if len(runs) != 0 {
		t.Fatalf("failed edge dispatch left a queued run: %#v", runs)
	}
}
Reconciler recovery
handler := service.NewParentWakeReconciler(service.NewSchedulerIntegration(svc, 0))
	if err := handler.Tick(ctx); err != nil {
		t.Fatalf("tick: %v", err)
	}

	calls = dispatcher.Calls()
	if len(calls) != 2 {
		t.Fatalf("want a second dispatch attempt from the reconciler, got %d", len(calls))
	}
	if calls[1].opID != edgeOpID {
		t.Fatalf("reconciler recovery used a different operation id: edge=%q reconciler=%q", edgeOpID, calls[1].opID)
	}

	runsAfter, _ := svc.ListRuns(ctx, "ws-1")
	if len(runsAfter) != 1 {
		t.Fatalf("want exactly one queued run after recovery, got %d", len(runsAfter))
	}
	if runsAfter[0].Reason != service.RunReasonTaskChildrenCompleted {
		t.Fatalf("recovered run reason = %q", runsAfter[0].Reason)
	}

Control test: successful edge is not double-queued

apps/backend/internal/office/service/scheduler_wake_reconciler_test.go

This control uses the same setup but lets the edge succeed, then proves Tick adds nothing.

Control
func TestParentWakeReconciler_DoesNotDoubleQueueASuccessfulEdgeDispatch(t *testing.T) {
	svc, eb := newTestServiceWithBus(t)
	ctx := context.Background()

	dispatcher := &countingDispatcher{inner: &queueRunDispatcher{svc: svc}}
	svc.SetWorkflowEngineDispatcher(dispatcher)

	adoptOffice(t, svc, "ws-1")
	seedStuckParent(t, svc, "ws-1", "parent-1", "worker-1")

	publishChildDone(t, eb, "parent-1", "worker-1")

	runs, _ := svc.ListRuns(ctx, "ws-1")
	if len(runs) != 1 {
		t.Fatalf("successful edge dispatch did not queue a run: %#v", runs)
	}

	handler := service.NewParentWakeReconciler(service.NewSchedulerIntegration(svc, 0))
	if err := handler.Tick(ctx); err != nil {
		t.Fatalf("tick: %v", err)
	}

	if dispatcher.Count() != 1 {
		t.Fatalf("want the reconciler to skip dispatch entirely, got %d attempts", dispatcher.Count())
	}
	runsAfter, _ := svc.ListRuns(ctx, "ws-1")
	if len(runsAfter) != 1 {
		t.Fatalf("tick added a run behind a successful edge dispatch: %#v", runsAfter)
	}
}

Shared event helper

apps/backend/internal/office/service/scheduler_wake_reconciler_test.go

This helper publishes the exact task.moved shape that triggers finalizeDone, so both tests use the same edge path.

Helper
func publishChildDone(t *testing.T, eb bus.EventBus, parentID, workerID string) {
	t.Helper()
	ctx := context.Background()
	moveEvent := bus.NewEvent("task.moved", "test", map[string]string{
		"task_id":                   parentID + "-child-0",
		"workspace_id":              "ws-1",
		"from_step_id":              "step-1",
		"to_step_id":                "step-done",
		"to_step_name":              "Done",
		"from_step_name":            "In Progress",
		"assignee_agent_profile_id": workerID,
		"parent_id":                 parentID,
	})
	if err := eb.Publish(ctx, "task.moved", moveEvent); err != nil {
		t.Fatalf("publish task.moved: %v", err)
	}
}

Data and storage

The wake uses a deterministic operation id derived from parent id and child set. The receipt stores it so ListStuckParents can skip already-delivered waves.

FieldTypeNotes
operation_idstringtask_children_completed:<parent>:<sha256(parent+childIds)> — same for edge and reconciler
child_set_keystringcomma-joined childId:state pairs, state stripped for operation id
wake_receiptrowparent_task_id + child_set_key + delivery_operation_id + delivered_at
runs.idempotency_keystringunique index idx_run_idempotency dedupes the second dispatch

Risk

2 / 10 Low
1 low5 medium10 high

Why this score

  • Test-only change: no production code, no migration, no runtime behavior change.
  • New helpers are test-local and do not affect other packages.
  • Failure injection is scoped to one trigger and one call, then auto-disarms.

Trade-offs and review notes

Where to look first

  1. Check that publishChildDone matches the shape TestWakeOperationID_UnifiedAcrossEdgeAndReconcilerPaths uses.
  2. Confirm the failed edge leaves no run and no receipt before Tick runs.
  3. Verify the reconciler re-uses the exact edge operation id and creates one RunReasonTaskChildrenCompleted run.
  4. Confirm the control test proves the opposite: a successful edge queues one run and Tick adds none.