PR #3271
Sections
Review

fix(office): stop parent tasks waking twice when children finish

main ← feature/make-wake-delivery-a-pt6 15 files +312 −48 PR #3271 ↗

Parent tasks no longer wake twice when children finish: terminal-to-terminal moves are ignored and the edge and reconciler paths now share one idempotency key that the runs queue dedupes durably.

Why this change

A parent task wakes twice for one child completion wave. A terminal-to-terminal edit fires a second wake, and the edge path and the reconciler race with different operation IDs so the queue cannot dedupe them.

What it does

Architecture, end to end

Two producers race for the same parent wake. The fix unifies their operation ID and makes the queue dedupe on the durable index.

flowchart LR
  ChildDone[Child task enters Done] --> Moved[task.moved event]
  Moved --> Edge[handleTaskMoved / finalizeDone]
  Edge --> QCC[queueChildrenCompletedRun]
  QCC --> OpID[wakeOperationID parent + child IDs]
  OpID --> Engine[Workflow engine HandleTrigger]
  Engine --> Queue[runs Service QueueRun]
  Tick[ParentWakeReconciler Tick] --> List[ListStuckParents]
  List --> Recon[reconcileOne]
  Recon --> OpID
  Queue --> Check{CheckIdempotencyKey 24h?}
  Check -- miss --> Insert[CreateRun]
  Insert --> Index[(idx_run_idempotency UNIQUE)]
  Index -- duplicate --> Deduped[QueueOutcomeDeduped]
  Index -- new --> Queued[QueueOutcomeQueued]

Key code changes

Drag to pan. Use the + and − buttons to zoom. Click a node to open the full code. The arrows show how the parts interact.

drag to pan · +/− to zoom · click a node for details

func (s *Service) handleTaskMoved(ctx context.Context, event *bus.Event) error
Click for details →

The handler now fires finalizeDone only when the task enters Done from a non-Done step.

Before and after
	}

	if categorizeStep(data.ToStepName) == stepCategoryDone {
	if categorizeStep(data.ToStepName) == stepCategoryDone &&
		categorizeStep(data.FromStepName) != stepCategoryDone {
		return s.finalizeDone(ctx, data)
	}
	return nil
}
func (s *Service) queueChildrenCompletedRun(ctx context.Context, parentID string) error
Click for details →

The edge path now fetches the child set key and derives the same operation ID as the reconciler.

Shared key derivation
	if err != nil || !allDone {
		return err
	}

	// Derived the same way as ParentWakeReconciler's recovery dispatch
	// (wakeOperationID) so both producers land on the identical operation
	// id for the same parent + child set. That shared id is what lets
	// idx_run_idempotency actually dedupe the pair when the reconciler
	// races this edge-triggered path for the same completion wave.
	childSetKey, err := s.repo.GetChildSetKey(ctx, parentID)
	if err != nil {
		return fmt.Errorf("get child set key: %w", err)
	}

	children, _, err := s.repo.GetChildSummaries(ctx, parentID)
	if err != nil {
		s.logger.Error("get child summaries failed", zap.Error(err))
		children = nil
	}

	key := fmt.Sprintf("children_completed:%s", parentID)
	key := wakeOperationID(parentID, childSetKey)
	summaries := make([]engine.ChildSummary, 0, len(children))
	prsByTask := s.lookupChildPRLinks(ctx, children)
	for _, c := range children {
func wakeOperationID(parentTaskID, childSetKey string) string
Click for details →

The function strips terminal state suffixes so one completion wave keeps one ID.

State-agnostic hash
}

func wakeOperationID(parentTaskID, childSetKey string) string {
	sum := sha256.Sum256([]byte(parentTaskID + "\x00" + childSetKey))
	// A completion wave is identified by the child IDs, not by the terminal
	// state each child reached. A terminal-to-terminal edit (for example,
	// CANCELLED to COMPLETED) must not create a new parent wake. The receipt
	// keeps the full state-aware key for recovery, so strip only the state
	// suffix when deriving the operation identity.
	childIDs := make([]string, 0)
	for _, child := range strings.Split(childSetKey, ",") {
		if child == "" {
			continue
		}
		if separator := strings.LastIndexByte(child, ':'); separator >= 0 {
			child = child[:separator]
		}
		childIDs = append(childIDs, child)
	}
	canonicalChildSet := strings.Join(childIDs, ",")
	sum := sha256.Sum256([]byte(parentTaskID + "\x00" + canonicalChildSet))
	return fmt.Sprintf("task_children_completed:%s:%s", parentTaskID, hex.EncodeToString(sum[:]))
}
func (s *Service) QueueRun(ctx context.Context, req QueueRunRequest) (QueueOutcome, error)
Click for details →

QueueRun now treats a unique-index conflict as a deduped outcome instead of an error.

QueueRun race handling

	row, err := s.insertRun(ctx, agentInstanceID, req, payload)
	if err != nil {
		// idx_run_idempotency has no time bound, so a conflict here can
		// come from a row older than IdempotencyWindowHours, not just the
		// windowed race CheckIdempotencyKey guards against above. Either
		// way the existing row is definitionally the same operation this
		// key identifies, so treat it as a no-op dedupe rather than a hard
		// error (see errIdempotencyKeyConflict's doc comment).
		if errors.Is(err, errIdempotencyKeyConflict) {
			s.log.Debug("run skipped (idempotency index race)",
				zap.String("key", req.IdempotencyKey))
			return QueueOutcomeDeduped, nil
		}
		return "", err
	}
insertRun maps violation
func (s *Service) insertRun(ctx context.Context, agentInstanceID string, req QueueRunRequest, payload string) (*models.Run, error) {
    var idemKeyPtr *string
    if req.IdempotencyKey != "" {
        k := req.IdempotencyKey
        idemKeyPtr = &k
    }
    row := &models.Run{
        ID: uuid.New().String(),
        AgentProfileID: agentInstanceID,
        Reason: req.Reason,
        Payload: payload,
        Status: "queued",
        CoalescedCount: 1,
        IdempotencyKey: idemKeyPtr,
        RequestedAt: time.Now().UTC(),
    }
    if err := s.repo.CreateRun(ctx, row); err != nil {
        if runssqlite.IsIdempotencyKeyUniqueViolation(err) {
            return nil, errIdempotencyKeyConflict
        }
        return nil, fmt.Errorf("enqueue run: %w", err)
    }
    return row, nil
}
func IsIdempotencyKeyUniqueViolation(err error) bool
Click for details →

The helper detects idx_run_idempotency violations on both PostgreSQL and SQLite.

New file
package sqlite

import (
    "errors"
    "strings"
    "github.com/jackc/pgx/v5/pgconn"
)

const runIdempotencyIndexName = "idx_run_idempotency"
const sqliteRunIdempotencyViolationMessage = "UNIQUE constraint failed: runs.idempotency_key"

func IsIdempotencyKeyUniqueViolation(err error) bool {
    if err == nil {
        return false
    }
    var pgErr *pgconn.PgError
    if errors.As(err, &pgErr) {
        return pgErr.Code == "23505" && pgErr.ConstraintName == runIdempotencyIndexName
    }
    return strings.Contains(err.Error(), sqliteRunIdempotencyViolationMessage)
}
Read the changes as a list

Guard terminal-to-terminal moves

apps/backend/internal/office/service/event_subscribers.go

The handler now fires finalizeDone only when the task enters Done from a non-Done step.

Before and after
	}

	if categorizeStep(data.ToStepName) == stepCategoryDone {
	if categorizeStep(data.ToStepName) == stepCategoryDone &&
		categorizeStep(data.FromStepName) != stepCategoryDone {
		return s.finalizeDone(ctx, data)
	}
	return nil
}

Unify operation ID for edge path

apps/backend/internal/office/service/event_subscribers.go

The edge path now fetches the child set key and derives the same operation ID as the reconciler.

Shared key derivation
	if err != nil || !allDone {
		return err
	}

	// Derived the same way as ParentWakeReconciler's recovery dispatch
	// (wakeOperationID) so both producers land on the identical operation
	// id for the same parent + child set. That shared id is what lets
	// idx_run_idempotency actually dedupe the pair when the reconciler
	// races this edge-triggered path for the same completion wave.
	childSetKey, err := s.repo.GetChildSetKey(ctx, parentID)
	if err != nil {
		return fmt.Errorf("get child set key: %w", err)
	}

	children, _, err := s.repo.GetChildSummaries(ctx, parentID)
	if err != nil {
		s.logger.Error("get child summaries failed", zap.Error(err))
		children = nil
	}

	key := fmt.Sprintf("children_completed:%s", parentID)
	key := wakeOperationID(parentID, childSetKey)
	summaries := make([]engine.ChildSummary, 0, len(children))
	prsByTask := s.lookupChildPRLinks(ctx, children)
	for _, c := range children {

Canonicalize wake operation ID

apps/backend/internal/office/service/scheduler_wake_reconciler.go

The function strips terminal state suffixes so one completion wave keeps one ID.

State-agnostic hash
}

func wakeOperationID(parentTaskID, childSetKey string) string {
	sum := sha256.Sum256([]byte(parentTaskID + "\x00" + childSetKey))
	// A completion wave is identified by the child IDs, not by the terminal
	// state each child reached. A terminal-to-terminal edit (for example,
	// CANCELLED to COMPLETED) must not create a new parent wake. The receipt
	// keeps the full state-aware key for recovery, so strip only the state
	// suffix when deriving the operation identity.
	childIDs := make([]string, 0)
	for _, child := range strings.Split(childSetKey, ",") {
		if child == "" {
			continue
		}
		if separator := strings.LastIndexByte(child, ':'); separator >= 0 {
			child = child[:separator]
		}
		childIDs = append(childIDs, child)
	}
	canonicalChildSet := strings.Join(childIDs, ",")
	sum := sha256.Sum256([]byte(parentTaskID + "\x00" + canonicalChildSet))
	return fmt.Sprintf("task_children_completed:%s:%s", parentTaskID, hex.EncodeToString(sum[:]))
}

Dedupe on durable unique index

apps/backend/internal/runs/service/service.go

QueueRun now treats a unique-index conflict as a deduped outcome instead of an error.

QueueRun race handling

	row, err := s.insertRun(ctx, agentInstanceID, req, payload)
	if err != nil {
		// idx_run_idempotency has no time bound, so a conflict here can
		// come from a row older than IdempotencyWindowHours, not just the
		// windowed race CheckIdempotencyKey guards against above. Either
		// way the existing row is definitionally the same operation this
		// key identifies, so treat it as a no-op dedupe rather than a hard
		// error (see errIdempotencyKeyConflict's doc comment).
		if errors.Is(err, errIdempotencyKeyConflict) {
			s.log.Debug("run skipped (idempotency index race)",
				zap.String("key", req.IdempotencyKey))
			return QueueOutcomeDeduped, nil
		}
		return "", err
	}
insertRun maps violation
func (s *Service) insertRun(ctx context.Context, agentInstanceID string, req QueueRunRequest, payload string) (*models.Run, error) {
    var idemKeyPtr *string
    if req.IdempotencyKey != "" {
        k := req.IdempotencyKey
        idemKeyPtr = &k
    }
    row := &models.Run{
        ID: uuid.New().String(),
        AgentProfileID: agentInstanceID,
        Reason: req.Reason,
        Payload: payload,
        Status: "queued",
        CoalescedCount: 1,
        IdempotencyKey: idemKeyPtr,
        RequestedAt: time.Now().UTC(),
    }
    if err := s.repo.CreateRun(ctx, row); err != nil {
        if runssqlite.IsIdempotencyKeyUniqueViolation(err) {
            return nil, errIdempotencyKeyConflict
        }
        return nil, fmt.Errorf("enqueue run: %w", err)
    }
    return row, nil
}

Cross-database violation detector

apps/backend/internal/runs/repository/sqlite/idempotency_violation.go

The helper detects idx_run_idempotency violations on both PostgreSQL and SQLite.

New file
package sqlite

import (
    "errors"
    "strings"
    "github.com/jackc/pgx/v5/pgconn"
)

const runIdempotencyIndexName = "idx_run_idempotency"
const sqliteRunIdempotencyViolationMessage = "UNIQUE constraint failed: runs.idempotency_key"

func IsIdempotencyKeyUniqueViolation(err error) bool {
    if err == nil {
        return false
    }
    var pgErr *pgconn.PgError
    if errors.As(err, &pgErr) {
        return pgErr.Code == "23505" && pgErr.ConstraintName == runIdempotencyIndexName
    }
    return strings.Contains(err.Error(), sqliteRunIdempotencyViolationMessage)
}

Data and storage

The runs queue keeps one durable identity per operation. The 24h check is a fast path; the unique index is the source of truth.

FieldTypeNotes
runs.idempotency_keyTEXT UNIQUEdurable identity; idx_run_idempotency enforces one row per key forever
runs.statusTEXTqueued, claimed, finished; deduped requests never insert
IdempotencyWindowHoursconst 24fast-path lookback for CheckIdempotencyKey; index has no window
wakeOperationIDsha256(parent + child IDs)canonical child set without terminal state suffix
childSetKeyTEXTfull key with states kept in receipt for recovery

Risk

4 / 10 Medium
1 low5 medium10 high

Why this score

  • Change is narrow: three Go files plus a small helper, no migration.
  • Covered by new tests for terminal-to-terminal guard, operation ID stability, and index-race dedupe on both SQLite and PostgreSQL.
  • Queue dedupe change is defensive: a missed detection would surface as a spurious error log, not data loss.

Trade-offs and review notes

Where to look first

  1. Verify handleTaskMoved guard: Done from Done must not call finalizeDone.
  2. Confirm wakeOperationID canonicalization: same child IDs with different terminal states produce the same hash.
  3. Check QueueRun error handling: IsIdempotencyKeyUniqueViolation maps to QueueOutcomeDeduped, not an error.
  4. Review IsIdempotencyKeyUniqueViolation for both PostgreSQL typed error and SQLite message match.