Working approach: start with task-level evidence, then use literature to refine the evaluation. A task may span several sessions. Citation popularity and network position are discovery aids, not quality assessments.
Evaluation question
Evidence to examine
Reading / scope
Did we identify the actual goal?
Manually compare a sample of extracted goals with the user request, subsequent corrections, and the delivered change. Record ambiguous or incorrect extraction before scoring completion.
Proposed local review protocol; not a validated automatic metric.
Did the change satisfy the task?
Acceptance criteria, patch, relevant tests and observed behavior. Record partial and unknown outcomes separately.
SWE-bench · issue-based benchmark. Useful for task formulation; benchmark success does not establish long-term quality.
What human effort did it require?
Review, corrections, retries and total elapsed work, including work outside the agent session.
METR developer study · randomized study in a specific setting. Read with its 2026 design update; results are not timeless estimates for every tool or developer.
Did apparent success survive later use?
Linked regressions, reverts, follow-up fixes, and maintenance effort. Record the observation window; absence of a reported bug is not proof of correctness.
Proposed longitudinal follow-up. Attribution of a later defect to this change requires evidence.
Are we measuring useful work?
Keep correctness, effort, user acceptance and later outcomes separate. Avoid equating activity counts with productivity.
SPACE · multidimensional developer-productivity framework, not an agent scoring formula.
Reading triage is based on the linked abstracts and research summaries, not full-paper appraisal. The Dagstuhl technical-debt report develops terminology and a research agenda; treat it as conceptual background. Here, technical debt is one possible later maintenance consequence, not the umbrella score for every outcome.
Next step: a small task-review pilot. Start with about ten completed task attempts spanning apparent successes, failures and ambiguous outcomes. This is a proposed calibration exercise, not a representative sample for estimating success rates. Compare each extracted goal with the conversation; examine the delivered change, validation evidence, human corrections and any later fixes. Preserve “unknown” where evidence is missing. Group the disagreements before inventing a combined score.
Reading rule: consult a paper when it can change a concrete review decision—for example, how to link evidence to a conclusion or handle conflicting judgments. Use the empirical collection’s screened shortlist to connect those questions to concrete evidence. Case review can reveal failure patterns; it cannot by itself establish whether using an agent caused better outcomes than working without one.
Runeson and Höst’s case-study guidelines are a methods reference, not findings about coding-agent quality. Their age alone does not make the methodology obsolete: the ACM SIGSOFT empirical standards still list the work among case-study readings. For our purpose, skim the evidence, analysis and validity guidance when needed. A citation count alone does not reveal why authors cited it, or whether it will be useful for this question.
Agent quality · six studies and a follow-up
Select a study for versions, dates, outcome definitions and primary sources. On narrow screens, scroll the table sideways for transfer limits.
Foundations · one starting book, two complements
Read to settle evaluation decisions: define the outcome, choose defensible measures, then examine what survives review and later change. These are conceptual foundations, not empirical findings about LLMs or a validated agent scoring system.
1. Start here · Software Metrics: A Rigorous and Practical Approach
Norman Fenton & James Bieman · 3rd edition · CRC Press, 2014 release / 2015 copyright
Read: chapter 3 on goal-based measurement, then chapter 4 on empirical investigation. Consult chapters 10 (external product attributes) and 11 (reliability measurement and prediction) when defining quality and delayed failures.
Contribution: a disciplined bridge from evaluation goals to measurements; separates attributes of code from outcomes in use. This is the most useful starting point for deciding whether a proposed score measures what we care about.
Dated / adaptation: examples and prediction approaches predate coding agents. Revalidate proxies locally; a task can span sessions, retries and human repairs. Record model/tool versions, operational exposure and uncertainty instead of treating generated volume or tests passed as quality.
Use in Atlas: explain disagreements between throughput and time in and ; distinguish static warnings from later corrective work in and .
Peter H. Rossi, Mark W. Lipsey & Gary T. Henry · 8th edition · SAGE, December 2018 / 2019 copyright
Read: chapters 3–5 for intervention theory, implementation and outcomes; chapter 6 for causal impact. Use chapter 10 when accounting for costs. Chapters 7–8 are optional when designing a comparison.
Contribution: separates “was the intended result delivered?” from “did this intervention improve the result?” and from “was it worth the resources?” It gives goal fulfillment, unintended effects and human effort a place beyond output counts.
Dated / adaptation: social-program examples and implementation context predate modern agents; the causal distinction remains useful. Define the intervention as the model, tools, prompts, permissions and human involvement. Task selection, model updates and concurrent sessions complicate comparisons.
Use in Atlas: read through selection and implementation differences. A successful individual session does not establish a causal productivity gain.
Contribution: makes maintainability concrete through future change, review comprehension and tests that support evolution. Productivity goals include tradeoffs; a fast review process can conceal lost quality.
Dated / adaptation: pre-agent practices reflect Google’s scale and tooling, not representative causal evidence. Apply principles without importing its organization. For agents, distinguish author understanding from reviewer understanding, inspect generated tests independently, and retain prior requirements during iterative edits.
Use in Atlas: connect to review effort, and to another maintainer’s subsequent change.
Verified scope: official online contents and selected sections of chapters 1, 7, 9 and 12. Free official book ↗
Minimal route: Fenton chapter 3 → Rossi chapters 5–6 → Google chapters 9 and 12. Return to the other chapters only when a concrete evaluation question requires them. Chapter selection and agent adaptations are reviewer recommendations; verified 7 September 2026.
Study / design
Finding · author-reported
Could apply to your sessions
Cannot generalize
Proposed local measurements · not a validated scoring system
Ten readings for evaluating coding-agent work
Reading order, not a quality ranking. Select a paper for its finding, setting, limitations and reason to read. Filter all 299 records in Documents.
Paper / evidence stage
Question / reason to read
Entire network
No connections match these filters.
Web pagePDFUnresolved referenceArrows point to referenced documents
How the scraper works · coverage and limitations
The agent collects and publishes these networks. Browsing, filtering, and opening evidence do not start scraping. The published snapshot loads on each visit; your selected document is remembered locally.
Topic/category listings and article indexes are excluded from the document list by default using conservative URL patterns. They remain in the network for discovery; choose “Hubs / article lists” to inspect them. Unknown page types remain included. Incoming-source rankings count distinct source nodes, excluding navigation and self-links; they are local to the full published collection and do not change with display filters. OpenAlex counts cover its wider database. Larger nodes reflect the chosen ranking; citations measure attention, not research quality. How OpenAlex builds citations.
Starting from the SEI article, the scraper follows selected research links and parses HTML and PDF text. Each inspected document records its collection time, outgoing links, and extraction warnings. Uninspected neighbors retain only the label observed in their source.
Article hyperlinks, reference-section links, navigation links, embedded PDF links, DOI mentions, and bibliography candidates are separate edge types. Connection evidence preserves the observed label or bibliography text and PDF page where available. A hyperlink does not establish agreement or a semantic relationship.
PDF bibliography splitting is heuristic. Entries without an exact URL or DOI remain unresolved; they are not automatically merged with similarly titled papers. Scanned PDFs require a text layer; OCR is not included.
Fetches honor robots.txt, validate every redirect and public destination, and never execute source JavaScript. Each document is limited to 12 MB, 120 PDF pages, 100 connections, and 45 seconds of processing. Blocked documents and extraction failures are retained as explicit coverage gaps. This is a bounded research sample, not a complete citation index.
