PR #3482
Sections
Review

docs: replace tracked document catalogs

main ← feature/reduce-decision-inde-e64 48 files +1240 −890 PR #3482 ↗

Replace tracked decision and specification tables with an on-demand catalog command that derives listings from source files.

Why this change

Tracked catalog tables duplicate source data and cause merge conflicts. Every new document edits a shared index, so unrelated pull requests conflict and rows become stale.

What it does

Architecture, end to end

Source files remain the truth. The catalog command derives views on demand. Validation runs in pre-commit and CI.

flowchart LR
  Decisions[docs/decisions/*.md] --> Catalog[scripts/list-docs.py]
  Specs[docs/specs/**/*.md] --> Catalog
  Catalog --> Markdown[markdown table]
  Catalog --> Paths[path list]
  Catalog --> JSON[JSON for tools]
  Catalog --> Validate[validate]
  Validate --> PreCommit[pre-commit hook]
  Validate --> CI[CI lint-harness-files.yml]

Key code changes

Drag to pan. Use the + and − buttons to zoom. Click a node to open the full code. The arrows show how the parts interact.

drag to pan · +/− to zoom · click a node for details

On-demand catalog commandscripts/list-docs.py ↗
def build_parser() -> argparse.ArgumentParser
Click for details →

The command lists decisions and specifications on demand with filters and three output formats.

CLI entry
def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="List and validate Kandev documentation sources."
    )
    parser.add_argument("--root", type=Path, default=Path.cwd())
    subparsers = parser.add_subparsers(dest="command", required=True)

    decisions = subparsers.add_parser("decisions", help="list architecture decisions")
    decisions.add_argument("--status")
    decisions.add_argument("--area")
    decisions.add_argument("--text")
    add_common_arguments(decisions)

    specs = subparsers.add_parser("specs", help="list specifications")
    specs.add_argument("--system")
    specs.add_argument("--kind", choices=SPEC_KINDS)
    specs.add_argument("--status")
    specs.add_argument("--text")
    add_common_arguments(specs)

    subparsers.add_parser("validate", help="validate both documentation catalogs")
    return parser
Run and validate
def run(args: argparse.Namespace) -> int:
    root = args.root.resolve()
    if args.command == "decisions":
        documents, errors = load_documents(root, decision_files(root), parse_decision)
        if errors:
            for error in errors:
                print(f"error: {error}", file=sys.stderr)
            return 1
        selected = filter_decisions(documents, args.status, args.area, args.text)
        print(format_documents(selected, args.format, "decision"), end="")
        return 0
    if args.command == "specs":
        documents, errors = load_documents(root, spec_files(root), parse_spec)
        if errors:
            for error in errors:
                print(f"error: {error}", file=sys.stderr)
            return 1
        selected = filter_specs(documents, args.system, args.kind, args.status, args.text)
        print(format_documents(selected, args.format, "specification"), end="")
        return 0
    decisions, decision_errors = load_documents(root, decision_files(root), parse_decision)
    specs, spec_errors = load_documents(root, spec_files(root), parse_spec)
    errors = sorted(decision_errors + spec_errors)
    if errors:
        for error in errors:
            print(f"error: {error}", file=sys.stderr)
        return 1
    print(f"Validated {len(decisions)} decisions and {len(specs)} specifications.")
    return 0
Shared metadata helperscripts/spec_metadata.py ↗
def validate_metadata(kind, system, metadata, has_frontmatter) -> list[MetadataIssue]
Click for details →

The helper owns path classification and frontmatter rules so the catalog and the linter stay in sync.

Shared validation
"""Shared specification path, frontmatter, and status metadata rules."""

VALID_REQUIREMENT_STATUSES = {"draft", "active", "deprecated"}
VALID_DESIGN_STATUSES = {"draft", "current", "superseded"}
VALID_SYSTEM_STATUSES = {"draft", "active", "retired"}
VALID_MIGRATION_STATUSES = {"in_progress", "complete"}

def validate_metadata(kind, system, metadata, has_frontmatter=True):
    contract_kind = "system-index" if kind == "system" else kind
    if contract_kind not in {"requirement", "system-design", "system-index"}:
        return []
    if not has_frontmatter:
        return [MetadataIssue("frontmatter", "new specification document must start with YAML frontmatter")]
    issues = []
    if system is not None and metadata_text(metadata, "system") != system:
        issues.append(MetadataIssue("system-owner", f"frontmatter system must be `{system}`"))
    status = metadata_text(metadata, "status")
    if contract_kind == "requirement" and status not in VALID_REQUIREMENT_STATUSES:
        issues.append(MetadataIssue("requirement-status", f"requirement status must be one of {sorted(VALID_REQUIREMENT_STATUSES)}"))
    return issues
Path classification
def classify_path(relative: Path) -> tuple[str, str | None]:
    parts = relative.parts
    spec_index = parts.index("specs")
    inside = parts[spec_index + 1 :]
    if inside[0] == "guide":
        return "guide", None
    if inside[0] == "templates":
        return "template", None
    if inside[0] == "product":
        return "product", None
    if len(inside) >= 3 and inside[1] == "requirements":
        return "requirement", inside[0]
    if len(inside) >= 3 and inside[1] == "system-design":
        return "system-design", inside[0]
    if len(inside) == 2 and inside[1] in {"README.md", "glossary.md"}:
        return "system-index", inside[0]
    return "legacy", None
Static entry pages replace tracked tablesdocs/decisions/INDEX.md ↗
Decision Log
Click for details →

The index pages no longer store rows. They describe the model and show catalog commands.

Decisions entry page
# Decision Log

Architecture Decision Records (ADRs) describe durable architecture, ownership,
boundary, contract, and repository decisions for Kandev.

This page is a static entry page. It does not contain a generated decision
table. The decision files are the source of truth.

## Find decisions

Run the catalog command from the repository root:

    python3 scripts/list-docs.py decisions --format markdown

Use filters when you need a smaller result:

    python3 scripts/list-docs.py decisions --status accepted --format paths
    python3 scripts/list-docs.py decisions --area backend --format markdown
    python3 scripts/list-docs.py decisions --text restart --format json
Specifications entry page
# Specification Catalog

Kandev specifications describe product intent, required behavior, and system
design. System README files define durable boundaries.

This page is a static entry page. It does not contain a generated list of
systems or documents. The specification files are the source of truth.

## Find specifications

Run the catalog command from the repository root:

    python3 scripts/list-docs.py specs --format markdown

Use filters for common discovery tasks:

    python3 scripts/list-docs.py specs --system ui --kind requirement --format paths
    python3 scripts/list-docs.py specs --status active --format markdown
    python3 scripts/list-docs.py specs --text workflow --format json
System READMEs keep boundaries onlydocs/specs/agents/README.md ↗
Agent system
Click for details →

Each system README keeps purpose, ownership, and migration record. It no longer repeats derived file lists.

New boundary document
---
status: draft
system: agents
specification_version: 1
migration: in_progress
owners:
  - kandev
---

# Agent system

## Purpose

The agent system owns configured agent identities, profiles, roles, permissions,
provider capabilities, and agent-facing runtime contracts.

## Ownership

This system owns agent profile data, role governance, profile-backed utility
agents, provider model options, agent permissions, and the agent capability
surface shared by task and Office consumers.

## Migration record

Migration remains in progress while legacy source detail is extracted from the
canonical requirement and system-design documents. Use the catalog command to find them.

## Related systems

- [Tasks](../tasks/README.md): consumes agent profiles for task execution.
- [Office](../office/README.md): consumes agent identities for autonomous work.
Validation wired into pre-commit and CI.pre-commit-config.yaml ↗
docs-catalog hook
Click for details →

Validation runs on every commit and in CI. It reports errors but never rewrites files.

Pre-commit hook
  - id: docs-catalog
    name: Documentation catalog validation
    entry: python3 scripts/list-docs.py validate
    language: system
    files: ^docs/(decisions|specs)/.*\.md$|^scripts/(list-docs|spec_metadata|lint-spec-files)\.py$
    pass_filenames: false
CI job
      - name: Test documentation catalog
        run: python3 scripts/list-docs.test.py

      - name: Validate documentation catalog
        run: python3 scripts/list-docs.py validate

      - name: Test specification linter
        run: python3 scripts/lint-spec-files.test.py

      - name: Lint specifications
        run: python3 scripts/lint-spec-files.py --all
Skills and guides use the catalog command.agents/skills/spec/SKILL.md ↗
Locate the owning system
Click for details →

Authoring skills now point to list-docs.py instead of tracked index tables.

Updated skill
### 1. Locate the owning system

Read `docs/specs/README.md` and the likely system `README.md`. If the system
has not migrated, run this command to locate the legacy source:

    python3 scripts/list-docs.py specs --kind legacy --format paths

Search the catalog, requirements, and designs for the capability name and its
main nouns:

    python3 scripts/list-docs.py specs --text <capability-term> --format paths
AGENTS.md guidance
- **Specifications:** Durable product context, requirements, and system designs
  live under `docs/specs/**`. Use `python3
  scripts/list-docs.py specs --system <system> --format paths` to find current
  documents, including legacy sources with `--kind legacy`. Run `python3
  scripts/list-docs.py validate` and `python3 scripts/lint-spec-files.py --all`
  after specification changes.
- **Decisions:** Architecture decisions are recorded in `docs/decisions/`. Use `python3 scripts/list-docs.py decisions --format markdown` to find relevant ADRs.
Read the changes as a list

On-demand catalog command

scripts/list-docs.py

The command lists decisions and specifications on demand with filters and three output formats.

CLI entry
def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="List and validate Kandev documentation sources."
    )
    parser.add_argument("--root", type=Path, default=Path.cwd())
    subparsers = parser.add_subparsers(dest="command", required=True)

    decisions = subparsers.add_parser("decisions", help="list architecture decisions")
    decisions.add_argument("--status")
    decisions.add_argument("--area")
    decisions.add_argument("--text")
    add_common_arguments(decisions)

    specs = subparsers.add_parser("specs", help="list specifications")
    specs.add_argument("--system")
    specs.add_argument("--kind", choices=SPEC_KINDS)
    specs.add_argument("--status")
    specs.add_argument("--text")
    add_common_arguments(specs)

    subparsers.add_parser("validate", help="validate both documentation catalogs")
    return parser
Run and validate
def run(args: argparse.Namespace) -> int:
    root = args.root.resolve()
    if args.command == "decisions":
        documents, errors = load_documents(root, decision_files(root), parse_decision)
        if errors:
            for error in errors:
                print(f"error: {error}", file=sys.stderr)
            return 1
        selected = filter_decisions(documents, args.status, args.area, args.text)
        print(format_documents(selected, args.format, "decision"), end="")
        return 0
    if args.command == "specs":
        documents, errors = load_documents(root, spec_files(root), parse_spec)
        if errors:
            for error in errors:
                print(f"error: {error}", file=sys.stderr)
            return 1
        selected = filter_specs(documents, args.system, args.kind, args.status, args.text)
        print(format_documents(selected, args.format, "specification"), end="")
        return 0
    decisions, decision_errors = load_documents(root, decision_files(root), parse_decision)
    specs, spec_errors = load_documents(root, spec_files(root), parse_spec)
    errors = sorted(decision_errors + spec_errors)
    if errors:
        for error in errors:
            print(f"error: {error}", file=sys.stderr)
        return 1
    print(f"Validated {len(decisions)} decisions and {len(specs)} specifications.")
    return 0

Shared metadata helper

scripts/spec_metadata.py

The helper owns path classification and frontmatter rules so the catalog and the linter stay in sync.

Shared validation
"""Shared specification path, frontmatter, and status metadata rules."""

VALID_REQUIREMENT_STATUSES = {"draft", "active", "deprecated"}
VALID_DESIGN_STATUSES = {"draft", "current", "superseded"}
VALID_SYSTEM_STATUSES = {"draft", "active", "retired"}
VALID_MIGRATION_STATUSES = {"in_progress", "complete"}

def validate_metadata(kind, system, metadata, has_frontmatter=True):
    contract_kind = "system-index" if kind == "system" else kind
    if contract_kind not in {"requirement", "system-design", "system-index"}:
        return []
    if not has_frontmatter:
        return [MetadataIssue("frontmatter", "new specification document must start with YAML frontmatter")]
    issues = []
    if system is not None and metadata_text(metadata, "system") != system:
        issues.append(MetadataIssue("system-owner", f"frontmatter system must be `{system}`"))
    status = metadata_text(metadata, "status")
    if contract_kind == "requirement" and status not in VALID_REQUIREMENT_STATUSES:
        issues.append(MetadataIssue("requirement-status", f"requirement status must be one of {sorted(VALID_REQUIREMENT_STATUSES)}"))
    return issues
Path classification
def classify_path(relative: Path) -> tuple[str, str | None]:
    parts = relative.parts
    spec_index = parts.index("specs")
    inside = parts[spec_index + 1 :]
    if inside[0] == "guide":
        return "guide", None
    if inside[0] == "templates":
        return "template", None
    if inside[0] == "product":
        return "product", None
    if len(inside) >= 3 and inside[1] == "requirements":
        return "requirement", inside[0]
    if len(inside) >= 3 and inside[1] == "system-design":
        return "system-design", inside[0]
    if len(inside) == 2 and inside[1] in {"README.md", "glossary.md"}:
        return "system-index", inside[0]
    return "legacy", None

Static entry pages replace tracked tables

docs/decisions/INDEX.md

The index pages no longer store rows. They describe the model and show catalog commands.

Decisions entry page
# Decision Log

Architecture Decision Records (ADRs) describe durable architecture, ownership,
boundary, contract, and repository decisions for Kandev.

This page is a static entry page. It does not contain a generated decision
table. The decision files are the source of truth.

## Find decisions

Run the catalog command from the repository root:

    python3 scripts/list-docs.py decisions --format markdown

Use filters when you need a smaller result:

    python3 scripts/list-docs.py decisions --status accepted --format paths
    python3 scripts/list-docs.py decisions --area backend --format markdown
    python3 scripts/list-docs.py decisions --text restart --format json
Specifications entry page
# Specification Catalog

Kandev specifications describe product intent, required behavior, and system
design. System README files define durable boundaries.

This page is a static entry page. It does not contain a generated list of
systems or documents. The specification files are the source of truth.

## Find specifications

Run the catalog command from the repository root:

    python3 scripts/list-docs.py specs --format markdown

Use filters for common discovery tasks:

    python3 scripts/list-docs.py specs --system ui --kind requirement --format paths
    python3 scripts/list-docs.py specs --status active --format markdown
    python3 scripts/list-docs.py specs --text workflow --format json

System READMEs keep boundaries only

docs/specs/agents/README.md

Each system README keeps purpose, ownership, and migration record. It no longer repeats derived file lists.

New boundary document
---
status: draft
system: agents
specification_version: 1
migration: in_progress
owners:
  - kandev
---

# Agent system

## Purpose

The agent system owns configured agent identities, profiles, roles, permissions,
provider capabilities, and agent-facing runtime contracts.

## Ownership

This system owns agent profile data, role governance, profile-backed utility
agents, provider model options, agent permissions, and the agent capability
surface shared by task and Office consumers.

## Migration record

Migration remains in progress while legacy source detail is extracted from the
canonical requirement and system-design documents. Use the catalog command to find them.

## Related systems

- [Tasks](../tasks/README.md): consumes agent profiles for task execution.
- [Office](../office/README.md): consumes agent identities for autonomous work.

Validation wired into pre-commit and CI

.pre-commit-config.yaml

Validation runs on every commit and in CI. It reports errors but never rewrites files.

Pre-commit hook
  - id: docs-catalog
    name: Documentation catalog validation
    entry: python3 scripts/list-docs.py validate
    language: system
    files: ^docs/(decisions|specs)/.*\.md$|^scripts/(list-docs|spec_metadata|lint-spec-files)\.py$
    pass_filenames: false
CI job
      - name: Test documentation catalog
        run: python3 scripts/list-docs.test.py

      - name: Validate documentation catalog
        run: python3 scripts/list-docs.py validate

      - name: Test specification linter
        run: python3 scripts/lint-spec-files.test.py

      - name: Lint specifications
        run: python3 scripts/lint-spec-files.py --all

Skills and guides use the catalog command

.agents/skills/spec/SKILL.md

Authoring skills now point to list-docs.py instead of tracked index tables.

Updated skill
### 1. Locate the owning system

Read `docs/specs/README.md` and the likely system `README.md`. If the system
has not migrated, run this command to locate the legacy source:

    python3 scripts/list-docs.py specs --kind legacy --format paths

Search the catalog, requirements, and designs for the capability name and its
main nouns:

    python3 scripts/list-docs.py specs --text <capability-term> --format paths
AGENTS.md guidance
- **Specifications:** Durable product context, requirements, and system designs
  live under `docs/specs/**`. Use `python3
  scripts/list-docs.py specs --system <system> --format paths` to find current
  documents, including legacy sources with `--kind legacy`. Run `python3
  scripts/list-docs.py validate` and `python3 scripts/lint-spec-files.py --all`
  after specification changes.
- **Decisions:** Architecture decisions are recorded in `docs/decisions/`. Use `python3 scripts/list-docs.py decisions --format markdown` to find relevant ADRs.

Risk

3 / 10 Low
1 low5 medium10 high

Why this score

  • Docs-only change with no product runtime or data migration.
  • Validation is read-only and fails fast on malformed metadata.
  • Revert restores tracked tables without data loss because source files stay unchanged.

Trade-offs and review notes

Where to look first

  1. Check scripts/list-docs.py parsing for all ADR metadata variants and deterministic sorting.
  2. Confirm scripts/spec_metadata.py classification covers all six spec kinds and frontmatter rules.
  3. Verify system READMEs keep durable boundary text and no derived lists remain.
  4. Confirm pre-commit and CI run validate without rewriting files.