Implementation backlog¶
The project assessment explains the gaps; the roadmap defines release gates. This page owns actionable work. IDs stay stable so report feedback and benchmark failures can refer to the component that needs to change.
“Implemented” below means a foundation exists, not that its quality or portability is independently established. Preserve published captures and campaign results; corrections create new versions.
Next work¶
| Order | Deliverable | Items | Completion evidence |
|---|---|---|---|
| 1 | Reliable collection and truthful audit completion | B01, B15, B35, B36 | Controlled failure/recovery tests; missing roles cannot produce a completed-audit receipt |
| 2 | Install-to-report experience | B28, B29, B37 | Fresh installation, unrelated working directory, portable report, interruption and update checks |
| 3 | Complete agent execution benchmark | B07, B16, B17, B22, B27, B32 | Reviewed task contracts, controls, paired Sol/high and Neuralwatt DeepSeek/high runs |
| 4 | Deeper qualitative methods and verified coverage | B09, B25, B33, B34 | Criterion-clause mappings, evidence-discovery tasks, owner-reviewed useful recommendations |
| Parallel pilot | Jev/Clef claim-support review | B10 | Independently labeled cases, preserved probabilities, error/cost comparison; no automatic outcome changes |
| Later | Broader portability and calibration | B02–B06, B11, B14, B20, B38 | Matched protocols, held-out evidence and cross-harness execution |
The first milestone is one reliable complete audit, not every row in this backlog. Do not make optional energy tooling or a decision-model service prerequisites for the baseline.
Measurement and collection¶
| ID | Current foundation | Remaining work and acceptance |
|---|---|---|
| B01 | Browser request accounting preserves partial transfers | Compare matching Chromium states with an independent collector, including frames, workers, streaming and aborted/204 responses. Resolve unfinished worker records, frame gaps and NHS blob-script records without inventing zero bytes or suppressing genuine unfinished network traffic. |
| B02 | co2-runner adapter and energy research exist | Match co2-runner/sitespeed.io browser, hardware, duration, journey, counter domains and first-sample treatment before comparing their outputs. |
| B03 | Raw hardware counters retained | Validate idle baselines and phase boundaries after B02; retain startup/shutdown separately and never add overlapping counters. |
| B04 | CrUX rank-to-pageviews mapping implemented | Calibrate against owner analytics with matched service, period and held-out sites beyond the original US government population; report dispersion. |
| B05 | Annual SWDM and client-energy scenarios implemented | Validate page mix, views versus visits, repeat visits and electricity assumptions. Keep scenarios distinct from measured totals; calibration depends on B04. |
| B09 | Purposeful page/task sampling implemented | Record inventory, templates, complete processes, audiences and states. Explain departure from STAR's sampling advice; add random sampling only within an explicit broader evaluation plan. |
| B15 | Consent, scroll and journey actions implemented | Add controlled task-readiness, search-fallback, form completion, high-DPR legibility and usable media-activation cases. Wait for settled focus after print/media changes; test NHS settings-page radios separately from homepage banner buttons. Measure successful task cost separately from first load. |
| B18 | Traffic prerequisites and shared-host scope checks implemented | Maintain regressions for incomplete/stale rank lookup, wrong origin and subpath services; an unranked service needs justified inputs, not a guessed traffic row. |
| B21 | Media recommendations have bounded evidence | Include CSS background images in media inventories. Compare eager instructional images with decorative media, task context, lazy-loading thresholds and legibility. Do not turn offscreen loading into a universal defect. |
| B23 | GreenFrame captures repeated client/network estimates | Validate matched interactions, browser versions and sensitivity to screen/CPU coefficients in quiet runs. Handle terminal scenario failures without leaving a bridge process alive; preserve W3C unsuccessful-response attempts. Its estimates are not device-independent electricity measurements. |
| B24 | carbon.txt discovery and declaration interpretation implemented | Add schema validation, DNS/header delegation and independently scoped provider evidence; test conflicts, expiry and unknown versions. Valid syntax cannot certify green hosting. |
| B35 | Live failures preserved in the documentation/exporter reports | Build challenge-page identity, readiness, pre-action navigation failure, failed-zero-byte, worker-accounting and shared-host fixtures. Retry only within explicit bounds; preserve the original first visit and stop repeated rate-limited requests. |
See energy experiments and the latest reports for observed limitations. Collection fixtures should precede another broad crawl.
Evidence, skills and scoring¶
| ID | Current foundation | Remaining work and acceptance |
|---|---|---|
| B06 | Selected source files captured without executing project code | Compare public-only, selected excerpts and autonomous repository discovery with known deployment provenance and held-out expected locations. |
| B10 | Jev/Clef research and offline request preview | Run claim-support review alongside existing reviewers. Compare TypeSafe Jev, Neuralwatt Clef, Sol/high and Neuralwatt DeepSeek/high on the same independently labeled evidence. Preserve probabilities, abstention, service errors and total review cost; no outcome changes in the pilot. |
| B13 | Deterministic scoped and experimental weighted scores | Review applicability, clause coverage, weights and sampling sensitivity independently. Test owner understanding with unknown-heavy reports; no site-quality ranking from unmatched scopes. |
| B25 | 43 checks and a 197-criterion planning inventory | Expand completed public tasks, keyboard/forms, media/font tradeoffs and document alternatives, then evidence-dependent source and organizational methods. Manual/consulting classes can still be agent-assisted. |
| B26 | Source hashes, screenshot manifests and contextual evidence implemented | Validate independently retrieved proof documents and their relation to the frozen packet. Keep deployment identity explicit; metadata-only file capture is not content review. |
| B30 | Cross-service disclosure discovery gap recorded | Add linked supplements for relevant later evidence, such as the main-site statement naming the guide. Preserve scope and retrieval provenance; explicitly re-review affected checks without silently transferring parent claims. |
| B33 | General guideline, evidence, claims and context roles | Compare experience/content and governance/lifecycle references with current roles and a no-skill baseline. Promote optional specialists only after useful evidence-support gains at comparable budgets. |
| B34 | Ten contextual checks name criterion IDs | Map the other 33 expectations to current criterion clauses and link methods to execution evidence and cases. Separate classified, implemented, attempted and validated coverage in generated summaries. |
B10's first task and evaluation criteria are specified in the assessment. Relevance ranking and adaptive browser actions follow only if they address measured failures.
Evaluation¶
| ID | Current foundation | Remaining work and acceptance |
|---|---|---|
| B07 | Structured graders and owner report feedback | Independently judge recommendation correctness, usefulness, priority, prerequisites and accessibility regressions. Include unsupported extra findings, not only missed expected findings. |
| B08 | 57 later development cases plus six original tasks | Add WSG/STAR knowledge and legitimate exceptions beyond observed examples; protect release labels and split by site/template/evidence family. |
| B11 | Provider telemetry research | Reconcile complete Harbor campaigns with Neuralwatt usage, including retries and escalation. Missing energy/cost is unknown, not zero. |
| B16 | Adapter setup skip path investigated | Build a clean pinned image with the requested Codex binary and effective Node runtime; record image identity and exclude credentials. Validate startup and unchanged-input retries before another large campaign. |
| B17 | Known strict-label disagreements retained | Separate claim-support questions from criterion outcomes and observed counters from unmeasured quantities. Review references before inference; do not adjust labels to improve published results. |
| B19 | Live distinctions represented in methodology cases | Retain regressions for cold/live preferences, working declared manifests, redirects, localization and code/content licensing; add complete recovery tasks. |
| B20 | Specialist instructions used in live audits | Run matched skill/no-skill comparisons with model, tools, evidence and budgets fixed; use blinded usefulness judgments. Execution alone does not prove skill benefit. |
| B22 | Recommendation-label ambiguities identified | Separate proposal support from action category and use atomic recommendations or explicitly allowed action sets. Review acceptable alternatives before freezing cases. |
| B27 | Frozen-bundle handoffs executed in Harbor | Exercise scope clarification, real browser collection, partial failures, review and final delivery. Run both required high-reasoning arms; then repeat under another harness. |
| B31 | Improved report layout, priorities and evidence appendix | Test whether owners understand actions, priorities, score unknowns and measurements without reading raw JSON; judge source excerpts and screenshot usefulness. |
| B32 | Qualitative batch designed | Implement the proposed 18 review cases and three execution tasks one family at a time, after B17/B22. Cover positive, negative, missing, conflicting, wrong-scope and inapplicable evidence. |
| B38 | Nine live reports across content and institutional sites | Tate, NHS, Google Blog, W3C and Low-tech Magazine add media, consent and public contextual evidence. Obtain independent owner review and add transactional completion, multilingual/mobile tasks and accessibility constraints. Keep related templates and context variants together when splitting evaluation sets. |
Campaign results retain model settings and exceptions. The setup investigation found that the pinned Harbor adapter can skip repeated installation when the expected Codex version exists; it did not establish an end-to-end speedup. The qualitative design owns the proposed cases.
Installation and report delivery¶
| ID | Current foundation | Remaining work and acceptance |
|---|---|---|
| B12 | Chart-ready JSON/CSV and accessible report tables | Add useful charts with accessible equivalents, explicit missing values and no cumulative double counting. Validate enrichment joins by scope and capture. |
| B14 | Shared skills/contracts and sequential fallback | Validate complete tasks under Codex and Claude Code first, then OpenCode/pi. Separate installer support, skill discovery, delegation and proven audit completion. |
| B28 | Plugin manifests, runtime resolver and project installer | Test fresh installation from an unrelated directory, minimal runtime dependencies, browser launch, persistent outputs, updates, uninstall and relocated/replaced plugin caches. Supply a supported update path that respects no-overwrite protection. |
| B29 | Deterministic renderer and editorial skill implemented | Add a portable report/bundle export with valid links, inspected archives, one editorial-receipt contract and optional energy. Eliminate campaign-specific packaging knowledge from ordinary delivery. |
| B36 | Review-task status and reconciliation receipts implemented | Add whole-audit status and completion validation across setup, collection, reviews, final review, supplements and report. Validate the complete guideline-result schema before accepting it, including top findings and next-evidence fields. Bind reconciliation to the final review; distinguish standalone scoring from a fully reviewed audit. Persist failed/partial attempts and test interruption without overwriting accepted work. |
| B37 | Frozen scorer identity prevents silent recomputation | Provide an explicit versioned recomputation operation that retains original evidence and failed receipts while identifying new scorer/rubric/review inputs. Reproduce the latest audits' scorer-fix scenario without manual session copying. |
| B39 | Separate user, reference, project and knowledge sections | Keep one current assessment and roadmap; update counts from catalogs. Avoid calling implemented packaging “proposed” or experimental host support “validated.” Move reusable principles out of campaign histories when updating them. |
B36 should not make the CLI a model runner. The host still owns conversation, delegation and semantic reconciliation; code validates the stage transitions and resulting artifacts. B37 must retain the current strict provenance checks, not bypass them.
Route owner feedback¶
| Feedback | Primary component |
|---|---|
| Wrong or incomplete measurement | Collector/protocol: B01, B15, B23, B35 |
| Unsupported finding or unsuitable fix | Evidence/rubric/skills: B07, B21, B25, B33, B34 |
| Confusing score or priority | Scoring and owner evaluation: B13, B31 |
| Missing public or internal evidence | Discovery and source scope: B06, B24, B26, B30 |
| Too much agent or user intervention | Runtime, completion and delivery: B28, B29, B36, B37 |
| Benchmark result disagrees with sensible advice | Question/reference contract: B17, B22 |
Name the check/case ID and the missing observation or incorrect claim when possible. That gives the next change a testable target without rewriting the historical evidence.