Requirements engineering digest, week 41 of 2026

1. Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development

   Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, Pinjia He

   arXiv (cs.SE) · preprint

   Introduces ReqClarifyBench, a benchmark of 341 repository-level tasks for how well LLMs turn a vague user intent into a requirements document through interaction. Finds that better clarification does not necessarily mean better document quality.

   Abstract

   As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want. Users often begin with vague intent, leaving important requirements to be clarified through interaction. In repository-level development, misunderstandings can propagate and cause costly rework. However, existing benchmarks typically evaluate implementation from complete specifications or underspecified requests, overlooking pre-coding requirement clarification. Prior clarification studies mainly focus on function-level tasks. It therefore remains unclear how well LLMs can transform vague intent into clear requirements through interaction and how this capability affects subsequent code generation. We introduce ReqClarifyBench, an interactive requirement clarification benchmark for repository-level software development. It contains 341 tasks spanning game, miniapp, and web development, each with a vague user intent, a human-validated requirement document, and code evaluation artifacts. We employ LLM-based user simulators with diverse personas derived from real-world conversations. We evaluate six LLMs in terms of clarification effectiveness and generated requirement document quality. Results show that stronger clarification effectiveness does not necessarily translate into higher document quality, and performance varies substantially across user personas. We also find that coding-agent harnesses do not further improve requirement document quality compared with their base LLMs. These findings offer insights for building LLM-based end-to-end software development systems.

   Source: arxiv.org/abs/2610.13022

2. Agent4RE: A Self-Refining Multi-agent Framework for End-to-End Software Requirements Engineering and Benchmarking

   Yongjian Tang, Linhan Li, Thomas Runkler

   arXiv (cs.SE) · preprint

   Presents Agent4RE, a self-refining multi-agent system covering requirements work from elicitation to generation, and RE-E2E, a dataset built from human-written specifications for end-to-end evaluation.

   Abstract

   Existing LLM-based approaches for software Requirements Engineering (RE) typically rely on basic prompting strategies or rudimentary agent collaboration, under-utilizing the full potential of multi-agent systems. Meanwhile, available datasets focus on isolated subtasks, such as requirements extraction, classification, and completeness detection, leaving the absence of an end-to-end RE benchmark that spans from requirements elicitation to generation. We present Agent4RE - a self-refining multi-agent RE system that orchestrates specialized agents and incorporates two iterative improvement loops. To support evaluation, we construct RE-E2E - a real-world dataset built from human-written requirement specifications, enabling end-to-end assessment of RE workflows. Building on this foundation, we further propose two enhanced Agent4RE versions that incorporate either autonomous self-refinement or structured human feedback, and analyze their strengths and limitations across different scenarios. Evaluation on 8 Large Language Models (LLMs) demonstrates that all three Agent4RE variants consistently outperform a domain-context-augmented prompting baseline by average 8% in text-based metrics. The two enhanced variants achieve the highest LLM-as-a-judge and human ratings, surpassing two RE baselines by approximately 0.8 points on a four-point scale. This consistent performance establishes Agent4RE as a practical end-to-end RE solution for industrial environments.

   Source: arxiv.org/abs/2610.10628 · doi:10.1145/3843779.3844634

3. Beyond Type-checking: Towards Holistic Evaluation of Formal Specification Generation

   Srijith Nair, Aditya Vempaty, Jia Liu, Ashish Jagmohan

   arXiv (cs.SE) · preprint

   Proposes a framework for judging generated formal specifications beyond type-checking, covering validity, similarity to a reference and behavioural adequacy, over 350 Lean tasks.

   Abstract

   When generating verifiable code, natural language requirements are mapped to machine checked code using LLMs and agentic workflows. A crucial component of this pipeline is specification generation (SpecGen), which produces a formal contract against which an agent can prove implementation correctness. Proof generation can obtain deterministic feedback from a theorem prover, but SpecGen lacks a definitive check that a generated specification captures the user's intent. A checked proof can therefore establish correctness against a specification that misrepresents the intended behaviour. We take a step towards holistic SpecGen evaluation with a unified dataset assembled from 350 existing Lean tasks, including 189 from VERINA and 161 from CLEVER, and a framework covering formal validity, reference similarity and equivalence, and behavioural adequacy. We distinguish acceptance of required inputs from acceptance of valid outputs and rejection of invalid outputs, while making each metric's evidence scope explicit. Across four SpecGen configurations, restricting the generalized tree edit distance (GTED) comparison, a reference similarity measure, to 32 jointly measurable VERINA tasks changes the VERINA configuration's position from second to fourth in mean similarity, showing the importance of measurement coverage. In an authored control, a specification achieves 100% positive test recall and negative test rejection while accepting 0% of required inputs. This demonstrates that perfect postcondition scores can miss an unusable input contract, motivating separate feedback on input coverage and output constraints.

   The source publishes this abstract as raw markup. Macros have been replaced by the text the publisher's own rendering shows for them, and the rest of the markup removed. The authors' wording is unchanged.

   Source: arxiv.org/abs/2610.10604

4. Applying Security by Design at the Point of Execution: How Governed Security Requirements Affect the Security of AI-Generated Code

   Pedro Farinha

   arXiv (cs.SE) · preprint

   Measures what happens when governed security requirements are supplied to a code generator before it writes code. Reports fewer exploitable solutions and states where the joint secure-and-functional metric did not improve.

   Abstract

   Security by design asks that security requirements are defined before code is written. Secure-code benchmarks typically measure the opposite situation: the agent receives the task without requirements and is scored with security tests it has not seen. We measured what changes when security requirements, selected from a governed security-by-design knowledge base (SbD-ToE) and delivered through a Model Context Protocol (MCP) server, are given to the generator at the point of execution. On DualGauge, which scores with a language-model judge, we ran 59 Python tasks: the share of tasks passing all security tests rose from 44.1% to 78.0% and the share of security tests passed from 77.4% to 93.0%. On BaxBench, which runs functional tests and real exploits in containers, we ran 28 backend scenarios: among functionally correct solutions, the share with no successful exploit rose from 65% to 86%, comparable to the authors' Oracle Security Reminder (85%), which they call an unrealistic upper bound; the requirement delivery reached it without knowing the tests. The study has limits. The pre-specified joint secure-and-functional metric did not improve significantly on either benchmark. The code written with the governed requirements is stricter, and it failed functional tests that fix values the task never states; we attribute those failures case by case in a post-hoc analysis. Each experiment used one model and one generation per task. The requirements come from the author's open-source manual; we do not assess its completeness. Governed security requirements delivered at the point of execution appear to improve the security of generated code. Current benchmarks cannot tell whether that code meets the security requirements that apply to it, only whether it shows the weaknesses their evaluators were built to detect. We propose measuring conformance with the applicable requirements.

   Source: arxiv.org/abs/2610.10659

5. CYBERFORT: A Compliance-Chain Platform Operationalising the Cyber Resilience Act for SMEs

   Nikolaos Kekatos, Georgios Koutidis, Mihaela Curcă, Angelos Fountoulakis, Marios Ioannou, Michael Ioannou, Elias Iosif, Paul Lacatus, Alexios Lekidis, Avgi Michael, Tom Nianios

   arXiv (cs.SE) · preprint

   Describes an open-source platform that makes EU Cyber Resilience Act obligations traceable from product risk through controls and evidence to the declaration of conformity, with a first deployment and case study.

   Abstract

   The EU Cyber Resilience Act (CRA) turns product cybersecurity into a lifecycle compliance obligation for manufacturers, importers, distributors, and integrators of products with digital elements on the EU market, a load that falls largely on small and medium-sized enterprises (SMEs) that rarely have dedicated governance, risk, and compliance (GRC) capacity. We present CYBERFORT, an open-source CRA-first compliance platform developed under the EU Digital Europe Programme and one of twelve projects in the EU CRA cluster. CYBERFORT operationalises the CRA through a guided scope self-assessment, a question bank tied to Annex I and the vulnerability-handling obligations, and a compliance-checking engine that links every answer to controls, policies, and machine-attested evidence, reusing ISO/IEC 27001, NIS2, and GDPR controls only where they coincide with CRA obligations. Its central contribution is the compliance chain, a traceable structure linking each product risk through its controls and policies to the CRA obligations it satisfies, and onward through evidence to the technical-documentation file and EU declaration of conformity, so that every operational gap is traceable and can be closed before market placement. Deployed at https://access.cyber-fort.eu/ for a first cohort of 43 organisations, the platform is presented with the engineering behind the chain, measured results from a completed end-to-end case study on a SIEM/XDR product with AI-driven remediation spanning the CRA obligation chapters, and the controlled effort study that remains in progress.

   Source: arxiv.org/abs/2610.09918

6. Implementing the Spec Growth Engine: Preventing Spec-Code Divergence, and Growing the Spec with Agents

   Hartwig Grabowski

   arXiv (cs.SE) · preprint

   Describes an implementation that keeps a graph of specifications coupled to the code and checks for spec-code divergence deterministically, with agents extending the spec under human-set delegation.

   Abstract

   The Spec Growth Engine anchors AI-assisted software development in a graph of specifications that the code is coupled to. This paper describes its implementation, which serves two tasks and keeps them apart as two layers. The first layer prevents spec-code divergence: a deterministic engine validates the spec graph, compares it with the code's import graph, earns a node's verified status from recorded test evidence, and classifies every change by what it can break -- without calling a model. The second layer grows the spec with agents: an intent author, a planner and a coder, each played by its own model, extend the graph in rounds, and a deterministic rule decides after each round whether the run goes on. How much of the human's judgement is delegated is set by three independent switches -- a draft gate, a delegation for breaking changes, and the run mode -- which, with two ways of laying a project's floor, give eighteen ways to run a project. We describe each of them, the gates, requests and waivers through which agents and the human communicate, and the spectrum of operation from entirely manual work to an unsupervised run whose decisions the human reviews afterwards. Throughout, one claim holds the design together: an autonomous run is worth only as much as the deterministic instance that measures it.

   Source: arxiv.org/abs/2610.11725

7. The EPIC Framework for Spec-Driven Development

   Bruno Claudino, Ruizhe Xing, Yuanhao Zuo, Tyler Menezes, Anita Sarma, Kostadin Damevski, Zixuan Feng

   arXiv (cs.SE) · preprint

   Scores the artefacts of 114 spec-driven development repositories against ISO/IEC/IEEE 29148 and derives EPIC, 40 practices for writing specifications, plans and tasks for coding agents.

   Abstract

   One practitioner we interviewed said their team writes "must" instead of "should" when instructing a coding agent, because the agent may treat "should" as optional. Small wording choices matter because agents often fill gaps in their instructions with their own assumptions. Spec-driven development (SDD) asks developers to write a specification, plan, and tasks before the agent writes code. SDD frameworks provide templates for these artifacts, but the templates do not help developers judge whether they have written enough or clearly enough. We studied what good SDD specifications contain. We scored the artifacts of 114 open-source SDD repositories against ISO/IEC/IEEE 29148 and derived practices from the highest-scoring ones. The resulting framework, EPIC, has 40 practices in 10 quality dimensions that guide developers in making expectations and decisions explicit in specifications, plans, and tasks for coding agents. The majority of SDD practitioners (N=15) endorsed every practice. Repositories in the top third of specification quality spent 11.8% of their commits on bug fixes, compared with 20.4% in the bottom third, and had a median of 4x as many contributors. Developers can use EPIC to find and fill the gaps in a specification before the agent does.

   Source: arxiv.org/abs/2610.07534

8. Agent4Spec: An LLM-based Agentic System to Optimize API Specifications through Automated Validation and Market Analysis

   Yongjian Tang, Costel Moraru, Doruk Tuncel, Hanya Elhashemy, Steffen Klepke, Thomas Runkler

   arXiv (cs.SE) · preprint

   Treats API specifications as artefacts to validate and evolve rather than generate once, with agents that repair OpenAPI errors and propose missing functionality from market analysis.

   Abstract

   API specifications serve as critical documentation for modern software development, directly impacting integration efficiency and client code generation. While recent advancements have leveraged Large Language Models (LLMs) to generate initial specifications from source code, existing approaches treat this process as a one-time event - overlooking the needs for continuous validation, long-term maintenance, and market-driven evolution of these artifacts. To address this issue, we present Agent4Spec, an autonomous multi-agent framework featuring two distinct pipelines: (1) a validation pipeline that detects and self-corrects OpenAPI syntax errors, and (2) a market analysis pipeline that conducts competitive research, identifies functional gaps against industry leaders, and automatically generates missing API-spec functionalities. We evaluate Agent4Spec on two curated datasets of API specifications spanning both general-purpose and industrial scenarios. Among 9 SOTA LLMs, the top-performer achieves 100% syntax error detection rates and over 95% implementation success for proposed corrections, while securing over 50% adoption rates for suggested endpoint improvements, representing a practical tool for API development.

   Source: arxiv.org/abs/2610.13273

9. Leakage-Resistant Evaluation of AI-Generated Database Specifications in Specification-Driven Migration

   Oleg Grynets, Ivan Fedorchuk, Vasyl Lyashkevych

   arXiv (cs.SE) · preprint

   Provides a benchmark and leakage controls for evaluating AI-generated Oracle PL/SQL specifications, and argues for evidence-linked, typed findings over binary defect detection.

   Abstract

   This paper presents a leakage-resistant framework and benchmark (LRFB) for measuring evaluators of AI-generated Oracle PL/SQL specifications. The benchmark contains 494 labelled source-specification pairs across 19 quality metrics grouped into seven dimensions; each metric contains 26 records divided into development and held-out subsets of 13 records each, with three clean and ten defective specifications per subset. Ground truth is mechanically sealed from inference and checked with canary and label-permutation tests. Three LLM judges were evaluated with three runs per record and compared against two trivial baselines. On 247 held-out records, GPT-4o, DeepSeek V4 Pro, and Claude Sonnet 4.5 reached balanced accuracies of 0.87, 0.89, and 0.81, respectively, while a specification-length baseline reached 0.88. Exact paired tests did not separate any judge from that baseline. The judges nevertheless provided information unavailable to length: the labelled defect category appeared among findings in 84%-92% of violated records, and quoted evidence overlapped the labelled span in 92%-93%. The study therefore reframes specification evaluation from binary issue detection toward evidence-linked, typed findings and benchmark-validity controls.

   Source: arxiv.org/abs/2610.12778

10. SpecGuard: Proving a Task Is Broken Before the Agent Cheats

    Param Biyani, Krishnamurthy Dvijotham

    arXiv (cs.SE) · preprint

    Autoformalises a task description into a Lean 4 specification and checks it against formalised tests, certifying when no implementation could satisfy both, to flag broken tasks before an agent acts.

    Abstract

    As autonomous coding agents get increasingly deployed, the risk that accidental or adversarially injected misspecifications in tasks lead to dangerous agent behavior is critical to address. Prior work has shown that agents given such tasks rarely flag the conflict and instead cheat, editing tests or hard-coding expected outputs, and the actions taken to cheat can cause real damage, such as deleting a security defense to make a corrupted test pass. It remains unclear whether such conflicts can be established with independently verifiable evidence before the agent acts. We present SpecGuard, which detects and formally certifies these conflicts between task intent and tests. Given only the task description and codebase, SpecGuard autoformalizes the intended behaviour into a Lean 4 specification. The tests are formalized independently, and the Lean kernel checks whether any implementation could satisfy both formalizations, producing a machine-checked certificate when none can. On conflicted SWE-bench tasks, SpecGuard detects up to 72.8% of conflicts and formally certifies up to 51.1%, with a nearly five-fold lower conflict miss rate than model-based judgment. SpecGuard provides a pre-execution safety check that identifies reward-hacking opportunities through formal certification of task-level conflicts, before any agent behavior is observed. Our code is available at https://github.com/prmbiy/specguard.

    Source: arxiv.org/abs/2610.09159