PR #3483
Sections
Review

fix(agents): recover managed runtime capability probes

main ← feature/investigate-agent-wa-273 37 files +642 −38 PR #3483 ↗

Host capability probes now recover from stale npm ETARGET the same way task sessions do: classify the failure in agentctl, repair the exact _npx tree, and retry once with online metadata.

Why this change

Host capability probes publish failure immediately when stale offline npm metadata hides a valid exact package version. Task sessions already recover from this ETARGET, but probes do not, so every profile for that agent shows as failed until another path refreshes the cache.

What it does

Architecture, end to end

Probe recovery reuses the existing executor-local repair contract. agentctl classifies, backend decides, same warm instance repairs and retries.

flowchart LR
  Probe[Host utility probe\n--prefer-offline] --> Agentctl[agentctl ACP executor]
  Agentctl -- stderr ETARGET --> Classifier[Probe failure classifier]
  Classifier -- failure_code --> Manager[hostutility.Manager]
  Manager -- acquire exclusive --> Gate[Operation gate]
  Gate --> Repair[agentctl cache repair\nRemove _npx tree]
  Repair --> Retry[Retry probe\n--prefer-online]
  Retry --> Cache[(Capability cache)]
  Cache -- success --> Publish[Publish recovered catalogue]
  Cache -- failure --> PublishFail[Publish failed status]

Key code changes

Drag to pan. Use the + and − buttons to zoom. Click a node to open the full code. The arrows show how the parts interact.

drag to pan · +/− to zoom · click a node for details

Backend host utility
agentctl
Exclusive operation gate for host utility instancesapps/backend/internal/agent/hostutility/manager.go ↗
func (i *instance) acquireOperation(ctx context.Context, exclusive bool) (func(), error)
Click for details →

The gate lets probes and prompts run together but forces cache repair to wait for all users of the tree and block new ones.

Gate definition
// Shared probes and prompts each take one slot. Repair takes every slot so it
// cannot remove an npm execution tree while another utility process uses it.
const hostUtilityOperationCapacity int64 = 1 << 20

func (i *instance) acquireOperation(ctx context.Context, exclusive bool) (func(), error) {
  i.operationGateOnce.Do(func() {
    i.operationGate = semaphore.NewWeighted(hostUtilityOperationCapacity)
  })
  weight := int64(1)
  if exclusive {
    weight = hostUtilityOperationCapacity
  }
  if err := i.operationGate.Acquire(ctx, weight); err != nil {
    return nil, err
  }
  return func() { i.operationGate.Release(weight) }, nil
}
Instance fields
type instance struct {
  agentType         string
  instanceID        string
  workDir           string
  client            *agentctlclient.Client
  operationGateOnce sync.Once
  operationGate     *semaphore.Weighted
}
func (m *Manager) probeManagedRuntime(ctx context.Context, inst *instance, ia agents.InferenceAgent, command agents.Command, req *agentctlutil.ProbeRequest) (*agentctlutil.ProbeResponse, error)
Click for details →

The manager runs the probe, checks the stable failure code, takes exclusive access, repairs the tree, and retries once with an online-preferred command.

Probe with recovery
func (m *Manager) probeManagedRuntime(
  ctx context.Context,
  inst *instance,
  ia agents.InferenceAgent,
  command agents.Command,
  req *agentctlutil.ProbeRequest,
) (*agentctlutil.ProbeResponse, error) {
  release, err := inst.acquireOperation(ctx, false)
  if err != nil {
    return nil, err
  }
  resp, err := inst.client.Probe(ctx, req)
  release()
  if err != nil || resp.Success || resp.FailureCode != agentctlutil.ProbeFailureManagedRuntimeNPMResolution {
    return resp, err
  }
  release, err = inst.acquireOperation(ctx, true)
  if err != nil {
    return resp, nil
  }
  defer release()
  return m.recoverManagedRuntimeProbe(ctx, inst, ia, command, req, resp), nil
}
Retry validation
func managedRuntimeProbeRetry(
  command agents.Command,
  spec agents.ManagedNPMRuntimeSpec,
) (agents.Command, string, bool) {
  args := command.Args()
  if len(args) < 4 || args[0] != "npx" || args[1] != "--yes" || args[2] != "--prefer-offline" {
    return agents.Command{}, "", false
  }
  packageSpec := args[3]
  if err := managedruntime.ValidateExactPackageSpec(packageSpec); err != nil {
    return agents.Command{}, "", false
  }
  prefix := spec.Package + "@"
  if !strings.HasPrefix(packageSpec, prefix) {
    return agents.Command{}, "", false
  }
  version := strings.TrimPrefix(packageSpec, prefix)
  want := spec.ACPCommandWithNpmPreference(version, false).Args()
  if !slices.Equal(args, want) {
    return agents.Command{}, "", false
  }
  return spec.ACPCommandWithNpmPreference(version, true), packageSpec, true
}
Repair and retry
func (m *Manager) recoverManagedRuntimeProbe(
  ctx context.Context,
  inst *instance,
  ia agents.InferenceAgent,
  failedCommand agents.Command,
  failedRequest *agentctlutil.ProbeRequest,
  initial *agentctlutil.ProbeResponse,
) *agentctlutil.ProbeResponse {
  managed, ok := ia.(agents.ManagedNPMRuntimeAgent)
  if !ok {
    return initial
  }
  spec := managed.ManagedNPMRuntime()
  retryCommand, packageSpec, ok := managedRuntimeProbeRetry(failedCommand, spec)
  if !ok {
    return initial
  }
  failedConfig := failedRequest.InferenceConfig
  if err := inst.client.RepairManagedRuntimeCacheWithEnvironment(
    ctx, packageSpec, failedConfig.Env, failedConfig.StripEnv,
  ); err != nil {
    return initial
  }
  response, err := inst.client.Probe(ctx, cloneProbeRequestWithCommand(failedRequest, retryCommand))
  if err != nil {
    return &agentctlutil.ProbeResponse{Success: false, Error: err.Error()}
  }
  return response
}
func managedRuntimeProbeFailureCode(command []string, stderr string) ProbeFailureCode
Click for details →

agentctl matches bounded stderr against the trusted package spec and returns only a stable code, never raw paths or registry output.

Failure code
func managedRuntimeProbeFailureCode(command []string, stderr string) ProbeFailureCode {
  packageSpec, ok := managedRuntimeProbePackageSpec(command)
  if !ok || !npmresolution.MatchesExactPackage(stderr, packageSpec) {
    return ""
  }
  return ProbeFailureManagedRuntimeNPMResolution
}

func managedRuntimeProbePackageSpec(command []string) (string, bool) {
  if len(command) < 4 || command[0] != "npx" || command[1] != "--yes" || command[2] != "--prefer-offline" {
    return "", false
  }
  packageSpec := command[3]
  if err := managedruntime.ValidateExactPackageSpec(packageSpec); err != nil {
    return "", false
  }
  return packageSpec, true
}
Shared matcher
// Package npmresolution classifies bounded npm version-resolution diagnostics.
var etargetCodePattern = regexp.MustCompile(`(?im)^\s*npm\s+(?:ERR!|error)\s+code\s+ETARGET\b`)

func MatchesExactPackage(stderr, packageSpec string) bool {
  if packageSpec == "" || strings.TrimSpace(packageSpec) != packageSpec {
    return false
  }
  notargetPattern := regexp.MustCompile(
    `(?im)^\s*npm\s+(?:ERR!|error)\s+notarget\s+No matching version found for\s+` +
      regexp.QuoteMeta(packageSpec) + `(?:\.\s*)?$`,
  )
  return etargetCodePattern.MatchString(stderr) && notargetPattern.MatchString(stderr)
}
Probe response field
type ProbeResponse struct {
  Success     bool             `json:"success"`
  Error       string           `json:"error,omitempty"`
  FailureCode ProbeFailureCode `json:"failure_code,omitempty"`
  DurationMs  int              `json:"duration_ms,omitempty"`
  // ... agent info, models, modes, config options
}

const ProbeFailureManagedRuntimeNPMResolution ProbeFailureCode = "managed_runtime_npm_resolution"
func (m *Manager) RepairManagedRuntimeCacheWithEnvironment(ctx context.Context, packageSpec string, overrides map[string]string, stripEnv []string) error
Click for details →

The same agentctl instance resolves npm cache with the failed probe environment and removes only the deterministic _npx tree for that exact spec.

Client request
type RepairManagedRuntimeCacheRequest struct {
  PackageSpec string            `json:"package_spec"`
  Env         map[string]string `json:"env,omitempty"`
  StripEnv    []string          `json:"strip_env,omitempty"`
}

func (c *Client) RepairManagedRuntimeCacheWithEnvironment(
  ctx context.Context,
  packageSpec string,
  env map[string]string,
  stripEnv []string,
) error {
  body, err := json.Marshal(RepairManagedRuntimeCacheRequest{
    PackageSpec: packageSpec,
    Env:         env,
    StripEnv:    stripEnv,
  })
  // POST /api/v1/agent/managed-runtime/cache-repair
  return c.doRepair(ctx, body)
}
Server handler
func (s *Server) handleManagedRuntimeCacheRepair(c *gin.Context) {
  var req ManagedRuntimeCacheRepairRequest
  if err := c.ShouldBindJSON(&req); err != nil {
    c.JSON(http.StatusBadRequest, ManagedRuntimeCacheRepairResponse{Error: "invalid managed runtime cache repair request"})
    return
  }
  if err := managedruntime.ValidateExactPackageSpec(req.PackageSpec); err != nil {
    c.JSON(http.StatusBadRequest, ManagedRuntimeCacheRepairResponse{Error: "invalid managed runtime package specification"})
    return
  }
  if err := s.procMgr.RepairManagedRuntimeCacheWithEnvironment(
    c.Request.Context(), req.PackageSpec, req.Env, req.StripEnv,
  ); err != nil {
    c.JSON(http.StatusInternalServerError, ManagedRuntimeCacheRepairResponse{Error: "managed runtime cache repair failed"})
    return
  }
  c.JSON(http.StatusOK, ManagedRuntimeCacheRepairResponse{Success: true})
}
Process repair
func (m *Manager) RepairManagedRuntimeCacheWithEnvironment(
  ctx context.Context,
  packageSpec string,
  overrides map[string]string,
  stripEnv []string,
) error {
  env, err := mergeAgentEnvIntoShellConfigWithError(m.agentEnvSnapshot(), overrides)
  if err != nil {
    return errors.New("resolve agent environment for managed runtime repair")
  }
  output, err := m.Output(ctx, tools.CommandSpec{
    Path:     "npm",
    Args:     []string{"config", "get", "cache"},
    Dir:      m.cfg.WorkDir,
    Env:      env,
    StripEnv: stripEnv,
  })
  cacheRoot, err := npmCacheRootFromOutput(output)
  if err != nil {
    return err
  }
  return managedruntime.RemoveNpxExecutionTree(cacheRoot, packageSpec)
}
Guarded public probe and prompt pathsapps/backend/internal/agent/hostutility/public.go ↗
func (m *Manager) ResolveModelConfig(ctx context.Context, agentType string, req ModelConfigResolutionRequest) (ModelConfigResolution, error)
Click for details →

Model config resolution and one-shot prompts now share the same recovery and gating, so a stale cache does not block profile status either.

Model config uses recovery
probeReq := buildProbeRequest(inst, ia, req.Refresh, command)
probeReq.Model = req.Model
probeReq.Mode = req.Mode
probeReq.ConfigOptions = cloneStringMap(req.ConfigOptions)
resp, err := m.probeManagedRuntime(probeCtx, inst, ia, command, probeReq)
if err != nil {
  return nil, err
}
Prompt gating
release, err := inst.acquireOperation(ctx, false)
if err != nil {
  return nil, err
}
resp, err := inst.client.InferencePrompt(ctx, req)
release()
if err != nil {
  return nil, err
}
Read the changes as a list

Exclusive operation gate for host utility instances

apps/backend/internal/agent/hostutility/manager.go

The gate lets probes and prompts run together but forces cache repair to wait for all users of the tree and block new ones.

Gate definition
// Shared probes and prompts each take one slot. Repair takes every slot so it
// cannot remove an npm execution tree while another utility process uses it.
const hostUtilityOperationCapacity int64 = 1 << 20

func (i *instance) acquireOperation(ctx context.Context, exclusive bool) (func(), error) {
  i.operationGateOnce.Do(func() {
    i.operationGate = semaphore.NewWeighted(hostUtilityOperationCapacity)
  })
  weight := int64(1)
  if exclusive {
    weight = hostUtilityOperationCapacity
  }
  if err := i.operationGate.Acquire(ctx, weight); err != nil {
    return nil, err
  }
  return func() { i.operationGate.Release(weight) }, nil
}
Instance fields
type instance struct {
  agentType         string
  instanceID        string
  workDir           string
  client            *agentctlclient.Client
  operationGateOnce sync.Once
  operationGate     *semaphore.Weighted
}

Probe recovery pipeline

apps/backend/internal/agent/hostutility/manager.go

The manager runs the probe, checks the stable failure code, takes exclusive access, repairs the tree, and retries once with an online-preferred command.

Probe with recovery
func (m *Manager) probeManagedRuntime(
  ctx context.Context,
  inst *instance,
  ia agents.InferenceAgent,
  command agents.Command,
  req *agentctlutil.ProbeRequest,
) (*agentctlutil.ProbeResponse, error) {
  release, err := inst.acquireOperation(ctx, false)
  if err != nil {
    return nil, err
  }
  resp, err := inst.client.Probe(ctx, req)
  release()
  if err != nil || resp.Success || resp.FailureCode != agentctlutil.ProbeFailureManagedRuntimeNPMResolution {
    return resp, err
  }
  release, err = inst.acquireOperation(ctx, true)
  if err != nil {
    return resp, nil
  }
  defer release()
  return m.recoverManagedRuntimeProbe(ctx, inst, ia, command, req, resp), nil
}
Retry validation
func managedRuntimeProbeRetry(
  command agents.Command,
  spec agents.ManagedNPMRuntimeSpec,
) (agents.Command, string, bool) {
  args := command.Args()
  if len(args) < 4 || args[0] != "npx" || args[1] != "--yes" || args[2] != "--prefer-offline" {
    return agents.Command{}, "", false
  }
  packageSpec := args[3]
  if err := managedruntime.ValidateExactPackageSpec(packageSpec); err != nil {
    return agents.Command{}, "", false
  }
  prefix := spec.Package + "@"
  if !strings.HasPrefix(packageSpec, prefix) {
    return agents.Command{}, "", false
  }
  version := strings.TrimPrefix(packageSpec, prefix)
  want := spec.ACPCommandWithNpmPreference(version, false).Args()
  if !slices.Equal(args, want) {
    return agents.Command{}, "", false
  }
  return spec.ACPCommandWithNpmPreference(version, true), packageSpec, true
}
Repair and retry
func (m *Manager) recoverManagedRuntimeProbe(
  ctx context.Context,
  inst *instance,
  ia agents.InferenceAgent,
  failedCommand agents.Command,
  failedRequest *agentctlutil.ProbeRequest,
  initial *agentctlutil.ProbeResponse,
) *agentctlutil.ProbeResponse {
  managed, ok := ia.(agents.ManagedNPMRuntimeAgent)
  if !ok {
    return initial
  }
  spec := managed.ManagedNPMRuntime()
  retryCommand, packageSpec, ok := managedRuntimeProbeRetry(failedCommand, spec)
  if !ok {
    return initial
  }
  failedConfig := failedRequest.InferenceConfig
  if err := inst.client.RepairManagedRuntimeCacheWithEnvironment(
    ctx, packageSpec, failedConfig.Env, failedConfig.StripEnv,
  ); err != nil {
    return initial
  }
  response, err := inst.client.Probe(ctx, cloneProbeRequestWithCommand(failedRequest, retryCommand))
  if err != nil {
    return &agentctlutil.ProbeResponse{Success: false, Error: err.Error()}
  }
  return response
}

Stable probe failure classification

apps/backend/internal/agentctl/server/utility/acp_executor.go

agentctl matches bounded stderr against the trusted package spec and returns only a stable code, never raw paths or registry output.

Failure code
func managedRuntimeProbeFailureCode(command []string, stderr string) ProbeFailureCode {
  packageSpec, ok := managedRuntimeProbePackageSpec(command)
  if !ok || !npmresolution.MatchesExactPackage(stderr, packageSpec) {
    return ""
  }
  return ProbeFailureManagedRuntimeNPMResolution
}

func managedRuntimeProbePackageSpec(command []string) (string, bool) {
  if len(command) < 4 || command[0] != "npx" || command[1] != "--yes" || command[2] != "--prefer-offline" {
    return "", false
  }
  packageSpec := command[3]
  if err := managedruntime.ValidateExactPackageSpec(packageSpec); err != nil {
    return "", false
  }
  return packageSpec, true
}
Shared matcher
// Package npmresolution classifies bounded npm version-resolution diagnostics.
var etargetCodePattern = regexp.MustCompile(`(?im)^\s*npm\s+(?:ERR!|error)\s+code\s+ETARGET\b`)

func MatchesExactPackage(stderr, packageSpec string) bool {
  if packageSpec == "" || strings.TrimSpace(packageSpec) != packageSpec {
    return false
  }
  notargetPattern := regexp.MustCompile(
    `(?im)^\s*npm\s+(?:ERR!|error)\s+notarget\s+No matching version found for\s+` +
      regexp.QuoteMeta(packageSpec) + `(?:\.\s*)?$`,
  )
  return etargetCodePattern.MatchString(stderr) && notargetPattern.MatchString(stderr)
}
Probe response field
type ProbeResponse struct {
  Success     bool             `json:"success"`
  Error       string           `json:"error,omitempty"`
  FailureCode ProbeFailureCode `json:"failure_code,omitempty"`
  DurationMs  int              `json:"duration_ms,omitempty"`
  // ... agent info, models, modes, config options
}

const ProbeFailureManagedRuntimeNPMResolution ProbeFailureCode = "managed_runtime_npm_resolution"

Executor-local cache repair with environment

apps/backend/internal/agentctl/server/process/managed_runtime.go

The same agentctl instance resolves npm cache with the failed probe environment and removes only the deterministic _npx tree for that exact spec.

Client request
type RepairManagedRuntimeCacheRequest struct {
  PackageSpec string            `json:"package_spec"`
  Env         map[string]string `json:"env,omitempty"`
  StripEnv    []string          `json:"strip_env,omitempty"`
}

func (c *Client) RepairManagedRuntimeCacheWithEnvironment(
  ctx context.Context,
  packageSpec string,
  env map[string]string,
  stripEnv []string,
) error {
  body, err := json.Marshal(RepairManagedRuntimeCacheRequest{
    PackageSpec: packageSpec,
    Env:         env,
    StripEnv:    stripEnv,
  })
  // POST /api/v1/agent/managed-runtime/cache-repair
  return c.doRepair(ctx, body)
}
Server handler
func (s *Server) handleManagedRuntimeCacheRepair(c *gin.Context) {
  var req ManagedRuntimeCacheRepairRequest
  if err := c.ShouldBindJSON(&req); err != nil {
    c.JSON(http.StatusBadRequest, ManagedRuntimeCacheRepairResponse{Error: "invalid managed runtime cache repair request"})
    return
  }
  if err := managedruntime.ValidateExactPackageSpec(req.PackageSpec); err != nil {
    c.JSON(http.StatusBadRequest, ManagedRuntimeCacheRepairResponse{Error: "invalid managed runtime package specification"})
    return
  }
  if err := s.procMgr.RepairManagedRuntimeCacheWithEnvironment(
    c.Request.Context(), req.PackageSpec, req.Env, req.StripEnv,
  ); err != nil {
    c.JSON(http.StatusInternalServerError, ManagedRuntimeCacheRepairResponse{Error: "managed runtime cache repair failed"})
    return
  }
  c.JSON(http.StatusOK, ManagedRuntimeCacheRepairResponse{Success: true})
}
Process repair
func (m *Manager) RepairManagedRuntimeCacheWithEnvironment(
  ctx context.Context,
  packageSpec string,
  overrides map[string]string,
  stripEnv []string,
) error {
  env, err := mergeAgentEnvIntoShellConfigWithError(m.agentEnvSnapshot(), overrides)
  if err != nil {
    return errors.New("resolve agent environment for managed runtime repair")
  }
  output, err := m.Output(ctx, tools.CommandSpec{
    Path:     "npm",
    Args:     []string{"config", "get", "cache"},
    Dir:      m.cfg.WorkDir,
    Env:      env,
    StripEnv: stripEnv,
  })
  cacheRoot, err := npmCacheRootFromOutput(output)
  if err != nil {
    return err
  }
  return managedruntime.RemoveNpxExecutionTree(cacheRoot, packageSpec)
}

Guarded public probe and prompt paths

apps/backend/internal/agent/hostutility/public.go

Model config resolution and one-shot prompts now share the same recovery and gating, so a stale cache does not block profile status either.

Model config uses recovery
probeReq := buildProbeRequest(inst, ia, req.Refresh, command)
probeReq.Model = req.Model
probeReq.Mode = req.Mode
probeReq.ConfigOptions = cloneStringMap(req.ConfigOptions)
resp, err := m.probeManagedRuntime(probeCtx, inst, ia, command, probeReq)
if err != nil {
  return nil, err
}
Prompt gating
release, err := inst.acquireOperation(ctx, false)
if err != nil {
  return nil, err
}
resp, err := inst.client.InferencePrompt(ctx, req)
release()
if err != nil {
  return nil, err
}

Risk

5 / 10 Medium
1 low5 medium10 high

Why this score

  • Repair removes only one deterministic _npx tree, but a bug in path derivation could delete the wrong directory.
  • Exclusive gate blocks all probes and prompts for one agent type during repair; a stuck repair delays capability refresh.
  • Failure classification must stay strict; a loose match would retry non-recoverable auth or provider errors.

Trade-offs and review notes

Where to look first

  1. Verify managedRuntimeProbeRetry rejects untrusted commands and only flips --prefer-offline to --prefer-online.
  2. Check that MatchesExactPackage requires both ETARGET and notarget for the exact spec, not a substring.
  3. Confirm RepairManagedRuntimeCacheWithEnvironment uses the failed probe Env and StripEnv when resolving npm cache.
  4. Review the semaphore gate: shared probes take 1 slot, repair takes all slots, and cancellation returns the initial failure.
  5. Ensure recovered probes publish the catalogue without changing persisted profile model, mode, or enabled state.