A failed repository-analysis run, and what it revealed about evidence, working memory and autonomous tool selection in REAPER-AI.
REAPER-AI technical case study | October 2026 | D01 / S02
A straightforward task that went nowhere
The D01 test was meant to establish whether a local language model could investigate an unfamiliar Python repository without step-by-step instructions. The agent had to locate relevant files, inspect the implementation and produce a report grounded in actual source code. No source modifications were required. The work was deliberately open-ended: a successful run would depend on choosing useful inspection steps and knowing when enough evidence had been gathered.
Devstral Small 2 began by calling index_files(language=“python“). The repository index returned twelve Python files, exactly the information requested. Yet the model invoked the same index operation repeatedly. By the timeout, 117 index calls had been recorded, including 115 results explicitly marked as adding no new information. After 600.2 seconds, 120 model calls and 119 tool calls, the run ended without a technical report.
What the execution record actually showed
The tool was not broken and the requested list had not vanished from context. Decision traces showed that the first index result was available, together with later observations identifying redundant calls. Devstral also issued one general file-listing request and read engine.py lines 1–203, but the bulk of its activity returned to the repository index. The outstanding working-state item, “List of Python files in the repository,” remained unresolved despite the successful initial response.
That is a narrower, more useful diagnosis than describing the behavior as a generic context-window failure. The model had evidence that its intermediate objective was complete but did not translate that evidence into a change of strategy. It repeatedly chose an operation that could no longer advance the investigation under the observed repository state.
Figure 1. Sentinel’s iterative execution path: a model decision is executed through a tool, recorded and returned as an observation.
Why this matters in an autonomous coding agent
REAPER-AI uses Sentinel, an iterative LangGraph-based execution loop, to connect model decisions with filesystem, Git, shell and repository-index tools. Python dispatches each requested operation and records the actual outcome. The language model then selects the next step using a bounded context assembled from the task, recent observations, retained source evidence and working state. A successful tool call is therefore a factual event, but it is not automatically a useful engineering step.
Three separate questions must be kept apart: whether the tool worked, whether the result reached the next decision, and whether the model understood its significance. The Devstral trace supports affirmative answers to the first two questions. It supplies no comparable evidence that the model correctly recognized the completed intermediate task. More tokens or a larger context window would not, by itself, address that decision error.
D01: comparison with two other local models
The same D01 analysis exercise was also run with Qwen3.8-Q5 and Qwen3.6-35B-A3B. Their outcomes put Devstral’s loop in context. Qwen3.8 completed the inspection after 18 model calls and 17 tool calls in 128.9 seconds. It recovered from invalid source-range requests and did not repeat an already successful identical read. Its report, however, contained factual mistakes; execution was rated grade 2, report accuracy grade 4 and the overall result grade 3.
Qwen3.6 finished in 25 seconds with only three tool calls, but examined too little source and presented several incorrect or overstated findings. It received overall grade 4. Devstral timed out without a report and received grade 5. Thus the fastest run was not the soundest analysis, and the greatest number of actions represented almost no useful progress. These are individual measurements within REAPER-AI v0.7.0, not general model rankings.
| D01 metric | Qwen3.8-Q5 | Qwen3.6-35B | Devstral 24B |
| Time | 128.9 s | 25.0 s | 600.2 s |
| Model / tool calls | 18 / 17 | 5 / 3 | 120 / 119 |
| Input tokens | 152,249 | 30,746 | 1,555,309 |
| Result | Report; factual errors | Report; shallow analysis | Timed out; no report |
| Overall grade | 3 | 4 | 5 |
S02: keeping relevant source in view
Not every repeated read arose from the same cause. Earlier D01 investigations had exposed problems in REAPER-AI’s own prompt assembly: a current, narrowly targeted source excerpt could be displaced by older and less relevant material. The application retained the code, but the next model decision did not necessarily receive the best excerpt. S02 addressed this selection problem by ranking source fragments individually, taking the current task and evidence needs into account.
The implementation now distinguishes persisted source coverage, which describes what was captured during the run, from visible source coverage, which records which exact ranges were included in a particular model prompt. The Decision Trace makes that distinction inspectable. S02 was therefore a correction to evidence selection and observability, not a claim that models would always interpret visible code correctly. Its focused checks concerned the constructed decision context and its trace, rather than a new three-model benchmark.
Information gain without artificial tool limits
The dispatcher can identify repeated observations that add no new confirmed information. That signal is useful diagnostically, but blindly prohibiting repetition would create another failure mode. A second read may be justified after a file changes, when the first response was truncated or when a different range is required. The relevant question is whether the action has a concrete purpose given the current evidence, not simply whether a tool has been used before.
The development response was to improve the existing state and source-selection mechanisms rather than introduce a separate supervisory agent, model-specific rules or fixed tool-call quotas. S03 extended the same direction by distinguishing observed facts, proposed model assessments, open questions and the evidence needed to resolve them. These measures make bad decisions easier to diagnose; they cannot guarantee sound reasoning from the chosen LLM.
The remaining limitation
A later S03 comparison confirmed that the underlying model-dependent behavior had not disappeared. Qwen3.8 correctly explained three tested exception paths from visible source, while Qwen3.6 made a control-flow error and Devstral repeatedly requested an already available range. The source was present in the relevant context, yet model responses still differed substantially. This limits how much confidence can be placed in a terminal completed status or a convincing final narrative.
For REAPER-AI, the D01 incident established a practical rule: tool activity should be judged against evidence and the task objective. A model that runs a tool successfully has established only that the operation succeeded. An autonomous engineering task advances when the result changes what can reasonably be concluded or done next. The 117 index calls exposed the gap between those two conditions with unusual clarity.
Sources: REAPER-AI MODELCHOICE.md and ROADMAP.md (D01, S02 and S03 records); https://github.com/robertbackhaus-a11y/REAPER-AI