Requirements engineering digest, week 33 of 2026

1. A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents

   Xin Zhou, Chun Yong Chong, Kisub Kim, Yun Peng, Rui Shu, Zihan Wu, Xu Han, Guowen Yuan, Zeyang Zhuang, Jounghoon Kim, Jeongjin Ju, Seongmin Ju, Taein Yoon, David Lo

   arXiv (cs.SE) · preprint

   A repository-level benchmark of 163 tasks that scores a coding agent's whole trajectory rather than only whether the final patch passes, carrying validated ground truth for requirement clarification and for implementation planning as stages of their own. Three agents across six model backends resolve 31.5% on average, and the stage-level diagnosis puts recovery of implicit requirements at the head of the failure list, behind 24.5% to 46.0% of runs.

   Abstract

   Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs. Yet existing repository-level benchmarks typically evaluate only whether the final patch passes tests. Satisfying a user request requires a long chain of interdependent reasoning and decisions: an agent must recover explicit and implicit requirements, formulate a repository-grounded implementation plan, and translate it into correct code. A pass/fail outcome cannot characterize how an unsuccessful trajectory diverges from the requirements and implementation process needed for a correct patch. To address this gap, we introduce SWE-RPG, a repository-level benchmark that combines executable patch evaluation with validated ground-truth references (GTs) for (1) Requirement Clarification and (2) Implementation Planning. These intermediate GTs support retrospective, GT-aligned diagnosis of complete coding-agent trajectories across clarification, planning, code generation, and artifact submission. SWE-RPG comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions. We evaluate 3 coding agents, including Claude Code, Codex, and OpenCode, with 6 large language model backends, including Claude-Sonnet-5 and GPT-5.6-Terra. Results show that the evaluated popular coding agents still struggle to implement user requests in existing repositories, achieving an average resolved rate of only 31.5% on SWE-RPG. Intermediate-GT diagnosis further identifies implicit requirement recovery as the main bottleneck, accounting for 24.5%--46.0% of agent runs. This result suggests implicit-requirement recovery as a key candidate direction for improving coding agents. The benchmark data and evaluation code are available at this https URL.

   Source: arxiv.org/abs/2608.09072

2. SpecPath: Testing Coding Agents Across Contract-Equivalent Specification Histories

   Yangfan Wu, Haozhe Wang, Huanyu Yang, Jianmin Ji, Fangzhen Lin

   arXiv (cs.SE) · preprint

   Names specification-path sensitivity: two requirement histories that mean the same thing by the end lead the same agent to behaviourally different programs. Holding the repository, the final contract, the verifier and the budget fixed and varying only the revision path, 35 of 100 blocks that succeed on the consolidated specification fail on at least one equivalent history.

   Abstract

   Modern coding agents increasingly appear capable of following complex software requirements, yet their success leaves a critical ambiguity: do they resolve the active specification, or merely follow the most salient path by which it was stated? We identify specification-path sensitivity, a failure mode in which requirement histories that are equivalent in their final meaning lead the same agent system to produce behaviorally different programs. This reframes evolving-requirement evaluation as active-contract resolution: before writing code, an agent must determine which requirements still count. Building on this view, we introduce SpecPath, a diagnostic evaluation that holds the repository, final contract, verifier, agent system, and execution budget fixed while changing only the revision path that leads to the contract. Rather than treating each patch as an isolated pass or failure, SpecPath uses paired executable outcomes to reveal whether an agent realizes the same tested behavior across contract-equivalent histories. Across five calibrated software tasks and fourteen coding-agent configurations, aggregate direct and revision-history accuracy is nearly unchanged; nevertheless, 35 of 100 complete blocks that succeed on the direct specification fail on at least one equivalent history. These results show that implementation success on a consolidated request does not guarantee specification-path invariance. Evaluating evolving requirements therefore calls for controlled tests of whether agents are robust to the path by which a specification becomes final.

   Source: arxiv.org/abs/2608.09799

3. Simplifying Requirements Engineering in the Context of the LGPD: An LLM-Based Investigation

   Cinara Gomes de Melo Carneiro, Renato de Freitas Bulcão Neto

   arXiv (cs.SE) · preprint

   Asks whether a large language model can carry Brazil's General Data Protection Law into the start of requirements engineering, generating user stories and acceptance test scenarios from the legislation itself, so that privacy compliance is a requirement from the outset rather than an audit afterwards.

   Abstract

   Compliance with privacy legislation poses a complex challenge to Requirements Engineering (RE): translating legal norms into software requirements. In this context, this study investigates whether Large Language Models (LLMs) can simplify RE within the framework of the Brazilian General Data Protection Law (LGPD). The proposed approach utilizes current legislation to automatically generate User Stories and Acceptance Test Scenarios. The evaluation results demonstrated high performance, confirming the potential of LLMs to ensure regulatory compliance from the software's inception.

   Source: arxiv.org/abs/2608.11454

4. From Safety Documentation to Safety Knowledge Support: An Evidence-Grounded LLM Framework for Medical Devices

   Tuhinangshu Gangopadhyay, Rasmus Adler, Peter Liggesmeyer, Jan Reich

   arXiv (cs.SE), to appear at ISSRE 2026, AISQ workshop · accepted

   Argues that the research problem in medical-device safety work is not generating safety text but supporting safety knowledge with its sources attached. The proposed framework links device artefacts, requirements, design decisions, software changes and verification results, generates candidate safety items per method, checks them for uncertainty, and records the expert review rather than standing in for it.

   Abstract

   Medical devices are becoming more software-intensive, connected, and AI-enabled. Their development requires risk-management evidence aligned with ISO 14971 and, for software, IEC 62304. This evidence must be kept consistent across requirements, design decisions, software changes, verification results, complaints, and post-market data. These tasks are costly and depend on scarce safety and domain experts. Large language models (LLMs) may reduce parts of this effort because medical-device safety work is highly document-based. However, current LLM-based safety-engineering studies often address isolated methods, rely on generic prompting or public examples, and provide limited support for source links, traceability, uncertainty handling, lifecycle updates, and recorded expert review. This limits their use in regulated medical-device development. This paper argues that the central research problem is not safety-text generation, but source-linked safety-knowledge support. We propose an evidence-grounded framework that connects device artifacts, controlled knowledge storage and retrieval, method-specific generation of candidate safety items, critique and uncertainty checks, and recorded expert review. The framework prepares, links, checks, and updates candidate safety artifacts for expert decision-making. It does not decide whether a device is safe and does not provide regulatory approval. We also outline an evaluation strategy using non-public or newly built medical-device case studies and expert reference analyses to assess coverage, correctness, relevance, traceability, duplicate rate, unsupported claims, and review effort.

   Source: arxiv.org/abs/2608.12025

5. SoK: From Generation to Consumption of Privacy Documents in Software Systems

   Shidong Pan, Clark LaChance, Zhen Tao, Sepideh Ghanavati

   arXiv (cs.CR), to appear at NDSS 2027 · accepted

   Reviews 290 papers published between 2010 and 2025 on privacy policies, privacy labels and interface-level transparency, organised by lifecycle: how such documents are defined and scoped, generated, analysed, checked for inconsistency and noncompliance, and evaluated for usability. Reports 15 trends, 21 open opportunities and four broader directions, among them unified policy-code analysis.

   Abstract

   Privacy documents (e.g., privacy policies) are a central mechanism through which digital services disclose data practices and seek user consent. Over the past decades, research on privacy documents has expanded significantly, encompassing not only traditional privacy policies but also short notices (e.g., privacy labels) and interface-level transparency mechanisms. As this research area continues to grow, it has become increasingly difficult to obtain a coherent view of how privacy documents are created, analyzed, evaluated, and maintained across their lifecycle. This SoK provides a unified, lifecycle-oriented view of privacy documents from a software engineering perspective. We systematically review and analyze 290 papers published between 2010 and 2025, organizing them around five research questions that examine how privacy documents are (1) defined and scoped, (2) generated, (3) analyzed and extracted, (4) checked for inconsistencies and noncompliance, and (5) evaluated and improved for usability. Building on our findings, we identify 15 key research trends and 21 open opportunities. We further chart four broader research directions that highlight (i) emerging challenges in AI-centric platforms, (ii) the need for diverse and up-to-date data foundations, (iii) LLM-based unified policy-code analysis, and (iv) dual usability for end-users and developers. We hope this SoK provides a shared foundation for future research on privacy policies and privacy documents.

   Source: arxiv.org/abs/2608.12511

6. Requirements-Augmented Generation for Trustworthy Acceptance Testing of LLM-Based Software

   Fanyu Wang, Chetan Arora, Zhenping Xie, Yonghui Liu, Kla Tantithamthavorn, Aldeida Aleti, Siwei Jiang

   arXiv (cs.SE), to appear at ASE 2026 · accepted

   Acceptance testing for software whose core is a language model, where the same query may call for different answers depending on the user's persona and the context. Test oracles are generated by retrieving the relevant requirements, domain knowledge and personas, and a confidence-calibrated cascade then decides whether to accept a verdict, escalate it or abstain, with reliability guarantees from conformal risk control.

   Abstract

   LLM-based software (LBS) integrates large language models as core components to deliver flexible, personalised responses. Unlike traditional software with deterministic outputs, LBSs exhibit context-dependent, stochastic behaviour that renders classical acceptance testing and test oracles insufficient: the same query may require fundamentally different responses depending on user personas and software context. This gap creates an urgent need for automated acceptance testing frameworks that autonomously interpret user instructions, while reliably inferring user intentions in a changing environment. In this paper, we present an automated acceptance testing framework for LBS with calibrated verdict reliability via two technical contributions. First, we introduce Requirements-Augmented Generation (REAG), which interprets user intentions by retrieving relevant software requirements, domain knowledge, and personas via adaptive RAG and self-reasoning to generate context-aware test oracles. Second, recognising that oracle generation may retrieve irrelevant constraints, misinterpret intent, or hallucinate requirements, we introduce a confidence-calibrated cascade judgment. This method quantifies verdict reliability via simulated expert agreement -- accepting high-confidence verdicts, escalating ambiguous cases, or abstaining when uncertain -- with empirical reliability guarantees backed by conformal risk control. An industrial case study on a production nutrition advisory application demonstrates that REAG achieves a 3.91/5 oracle quality score, reaching qualified or marginal oracle quality in 82% of cases. The confidence-calibrated cascade achieves 98.8% accuracy, improves oracle quality from 3.91 to 4.30 by filtering unqualified outputs, and delivers a 31.7% cost-efficiency improvement over single-judge baselines, validating industrial viability

   Source: arxiv.org/abs/2608.12970

7. Does ISO-Grounded NFR Specification Improve LLM Code Generation? A Comparison of Rich and Structured Interventions against a Natural-Language Baseline

   Joào Pedro Monteiro Pereira, Vinicius Cardoso Garcia

   arXiv (cs.SE), to appear at SBCARS 2026 · accepted

   Asks whether grounding a non-functional requirement in ISO/IEC 25010 rather than stating it as a terse one-line phrase improves the code a language model generates from it. It improves the static quality proxies and makes the result less sensitive to prompt wording, but does not reliably improve functional correctness, and whether the grounded content arrives as prose or as JSON barely matters.

   Abstract

   In LLM-based code generation, Non-Functional Requirements (NFRs) are often specified as terse one-line phrases. We ask whether grounding those specifications in ISO/IEC 25010 Quality Model, either as rich natural-language prose (NL-rich) or as structured JSON (Structured), improves code generated on HumanEval/HumanEval-ET compared to a RobuNFR-style one-line baseline (NL-simple). We evaluate four NFRs (performance, error handling, code smell, readability) with ten prompt variations per condition under a fixed model snapshot and paired non-parametric analysis. Primary finding: ISO-grounded enrichment improves static quality proxies (unreadability density falls across all four NFRs (e.g., Performance 0.88 -> 0.69 for NL-rich)) and reduces sensitivity to prompt wording, but does not reliably improve functional correctness; for error handling, extended-test pass rate decreases, suggesting tension between defensive coding patterns and exact-output benchmarks. Secondary finding: when ISO content is held constant, NL-rich and Structured differ negligibly in correctness (|delta| <= 0.023), indicating that semantic content matters more than JSON-vs-prose format. Practitioners should invest in standard-grounded NFR content rather than serialization form. A fully traceable replication package is provided.

   Source: arxiv.org/abs/2608.13742

8. An Empirical Study on the Impact of Normalized Use-Case Specifications on Traceability

   Luoyuan Shi, Yuanzhao Zhai, Dawei Feng, Jialin Zhao, Zhaoxie Xu, Bo Ding, Huaimin Wang

   arXiv (cs.SE) · preprint

   Attacks the semantic gap in traceability from the requirements side rather than the algorithm side, normalising raw requirements into standardised use-case specifications with a controlled natural language and prompt engineering. Across four datasets and two frameworks, semantically ambiguous requirements trace better afterwards, while requirements already close to the code's own vocabulary trace worse, which marks the boundary of the technique.

   Abstract

   Traceability link recovery between requirements and source code is vital for software quality assurance and evolution analysis. Although automated traceability techniques have advanced greatly, the large semantic gap between vague natural-language requirements and precise source code still hinders accurate link recovery. Most existing approaches optimize traceability algorithms yet ignore the inherent quality of requirement descriptions, which prevents fundamental reduction of the semantic gap. This work proposes a requirement-oriented normalization method. Using controlled natural language and large-language-model-based prompt engineering, raw requirements are decomposed and converted into standardized use-case specifications to strengthen semantic representation and mitigate semantic divergence. Evaluated on four public datasets under two typical traceability frameworks, the normalized specifications improve tracing performance for semantically ambiguous raw requirements. However, over-normalization may degrade results for already high-quality requirements closely aligned with code semantics. The results validate source-side requirement normalization as a promising strategy for traceability improvement and reveal its applicable boundaries for practical usage.

   Source: arxiv.org/abs/2608.15726

9. Can Formal Specifications Be Synthesized from Tests Alone?

   Tianhai Liu, Maximilian Müller, Tobias Hey, Vitus Lüntzel, Muhammad Minhas, Anne Koziolek, Bernhard Beckert

   arXiv (cs.SE) · preprint

   Infers candidate Java Modeling Language specifications without white-box access: the model sees only the program interface, selected inputs and the resulting outputs or state changes, drawn from test code and execution traces, while the implementation stays hidden. Bounded model checking validates each candidate and its feedback drives the refinement. Initial results on SpecGenBench suggest tests can guide a model towards meaningful specifications, with checker compatibility and diagnostic feedback as the obstacles.

   Abstract

   Formal specifications offer strong guarantees, but remain costly to write manually. Recent LLM-based approaches automate this by inferring specifications from source code, yet their reliance on white-box access poses barriers to industrial adoption due to intellectual property risks and deployment costs. Our approach uses LLMs to infer candidate specifications solely from test code and dynamic execution traces: the LLM observes only the program interface, selected inputs, and corresponding outputs or state changes, while the implementation internals remain hidden. Candidate specifications are validated locally using bounded model checking, with feedback guiding iterative refinement. Initial results on the SpecGenBench benchmark suggest that tests can guide LLMs towards meaningful Java Modeling Language specifications, while also highlighting checker compatibility and diagnostic feedback as key challenges for reliable refinement.

   Source: arxiv.org/abs/2608.13240

10. How Powerful are LLMs in Generating Formal Program Specifications?

    Fanpeng Yang, Xing Li, Shuling Wang, Jie An, Zeyu Sun, Shenghua Feng, Wenhan Wang, Weiyi Wang, Naijun Zhan, Fanjiang Xu

    arXiv (cs.SE) · preprint

    Coins, a Rocq-based framework that judges a generated specification by instantiating it on trusted test cases and generating concrete proof obligations, rather than by proving conformance or equivalence, which conflates proof difficulty with specification quality. A large study on HumanEval against curated human-written specifications finds the task still formidable, and argues that how specifications are evaluated, not model scale alone, is what decides what we know about the models.

    Abstract

    Formal verification provides strong guarantees of software correctness, but its adoption is limited by the high cost of writing precise formal specifications. While recent large language models (LLMs) have shown strong capabilities in theorem proving and verified code generation, their true ability to generate program specifications remains unclear. Existing evaluations require either verifying implementation conformance or proving semantic equivalence between specifications, both of which are formidably difficult and may conflate proof difficulty with specification quality. To address this problem, we introduce Coins, a Rocq based evaluation framework that assesses specification quality by instantiating specifications under evaluation on trusted test cases and generating concrete proof obligations. This design aligns with the asymmetric nature of formal reasoning, where successful proofs provide reliable evidence while proof failures are inherently ambiguous. Using Coins, we conduct a large scale study on HumanEval with a curated set of human written Rocq specifications. Our results show that specification generation remains a formidable challenge, and that verification complexity can obscure genuine differences in specification quality. Overall, we find that accurate specification evaluation, rather than model scaling alone, is central to understanding the power of LLMs for specification synthesis, and that test case based formal reasoning offers a more faithful and discriminative measure of progress.

    Source: arxiv.org/abs/2608.13077

11. Specification-first convergence with an AI coding agent: a case study of dismantling a core architectural invariant across 189 files in a 717k-line codebase with no test oracle and no human code review

    Joel Abenhaim

    arXiv (cs.SE) · preprint

    A single instrumented case study of an AI agent dismantling a core lifetime invariant across 189 files of a 717,725-line production codebase, with no test oracle and no human review of the generated code. The protocol is specification first: the agent writes the specification, audits it against the source over 14 cycles, implements atomically, then audits the code against the frozen specification over 17 more. 201 defects were corrected across the 31 passes before anyone ran the program.

    Abstract

    This paper reports a single, fully instrumented case study of a large-scale architectural refactoring by an AI coding agent under a specification-first protocol, with no human review of the generated code and no pre-existing oracle to validate the target behaviour. The task, dismantling a central invariant across a large interdependent codebase, was assessed by the author as effectively infeasible through incremental refactoring, the kind of change that conventionally calls for a rewrite instead. Under the protocol described here, the agent completed it successfully. The system is a 717,725-line production TypeScript application across 3,648 files. The task required dismantling a core lifetime invariant: the guarantee that a UI panel remains open for the duration of an AI request. The target behaviour was that a streaming generation survives the closing of its panel and can be reattached, on reopening, to the same live stream with no loss or duplication. The protocol: formal specification by the agent, 14 refinement cycles auditing that specification against the source code, atomic implementation, a compile/test feedback loop, then 17 verification cycles auditing the code against the frozen specification. Across 31 audit passes, 201 defects were corrected before any human executed the program. The convergence criterion was empirical: two consecutive verification passes returning zero findings. The change touched 189 files (31 new); with the extraction phase, the two commits total 288 files, 34,770 insertions, 16,422 deletions. Across the first and roughly thirty later sessions, the software behaved as specified, no bug observed. Elapsed: three days; cost: USD 2,430. The full specification and raw session logs, 1,500+ pages in French, are published as evidence, allowing inspection of the process and submission to a language model for consistency checking.

    Source: arxiv.org/abs/2608.12440