ToolBottleneckBench explained: where do AI agents stumble?

David

While developing Kurachan and its core technology, Data Brain, we became interested in how to evaluate an AI agent's use of tools. This article examines ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets, explaining its research question, experimental design, and main findings.

The question: what does a success rate measure?

A tool-using AI agent must select a tool, read its result, and decide what to do next. That process involves the model's reasoning, but also the tools it is shown, the format of returned data, token budgets, and limits on calls.

The paper starts from the observation that task success reflects both the model and its execution environment. It calls the surrounding system a harness: the machinery that manages model inputs, tool execution, limits, and stopping conditions.

If success falls when more tools are introduced, did choosing become harder? Or was the response cut off before the model finished choosing? A success/failure total cannot distinguish these possibilities. ToolBottleneckBench studies them through matched comparisons and records of how episodes terminate. Sections 1 and 3

Experimental design: separate four kinds of pressure

The main evaluation covers 36 task families within 12 enterprise intents and 21 model-and-service configurations. A configuration includes the provider route and tool-calling protocol, alongside the model. Important facts and goals are held fixed within each task family while pressure conditions vary.

There are four pressures. Registered catalog size R concerns whether retrieval supplies the required tools. V, W, and D concern the menu, workflow, and data representation presented to the model. Section 3 and Table 1

Figure 1. A simplified view of the design, based on Section 3 and Table 1. R targets retrieval; V, W, and D target conditions the model processes. Matched conditions are compared, with combinations also tested.

The strict success measure, VTS, requires all nine obligations to pass, including the answer, exact evidence set, workflow, permitted state changes, and budgets. There is no partial credit. The registered-catalog experiment uses a separate success measure that includes whether retrieval supplied the required tools. Section 3.3 and Appendices A–B

The main limits are 1,024 completion tokens per model request, 128,000 tokens per episode, and the certified minimum number of calls plus two. Completion is the generation budget for a response; some providers include internal reasoning tokens in it. The same nominal cap therefore does not necessarily behave identically across configurations.

Finding 1: a larger visible menu produces a clear penalty

Increasing the visible menu from 8 to 128 tools reduced VTS by 12.3 percentage points in the main experiment, with a 95% confidence interval of −14.6 to −9.9 points. Success declined in 34 of the 36 task families. Section 5.1 and Figure 2

This result comes from the routed-menu lane. After the first executed tool call, it uses ground-truth operational-tool information to narrow the menu, making it an oracle-assisted reference condition. Keeping the full menu visible throughout produced a −17.5-point contrast.

Increasing the main experiment's workflow pressure from low to high produced a pooled contrast of +0.8 points [−1.2, +2.8], with no detectable overall effect. Individual configurations differed considerably, so this does not establish that longer workflows pose no difficulty.

Result-encoding effects were smaller. Compact scored −3.7 points relative to raw, but a separate intent-level test did not support the effect after multiple-comparison correction. Typed and linked encodings had no detectable effects. The clearest pressure in the main experiment was therefore the larger visible menu.

Finding 2: much of that penalty involves the per-request cap

Failing with a large menu does not necessarily mean selecting the wrong tool. Of 411 episodes truncated at the completion cap with 128 tools, 391 ended before the first tool call.

However, a correlation between truncation and success cannot by itself isolate the cap's effect. The researchers therefore intervened: they increased the completion cap from 1,024 to 16,384 tokens for the same configurations, leaving the episode budget and call limit unchanged and running two repeats per condition. Section 5.2 and Appendix E.1

For six truncation-heavy configurations, the success contrast between 128-tool and 8-tool menus shrank from −43.7 to −2.7 points. Their truncation rates with 128 tools ranged from 30.6% to 69.8% at the lower cap; at the higher cap, all six were at or below 1.4%.

Figure 2. Redrawn from Table 9. The axis is VTS(128 tools) − VTS(8 tools), in percentage points (pp), not absolute success. Lines show 95% task-family-cluster bootstrap confidence intervals. These six configurations are a different population from the full main panel of 21.

This intervention supports the conclusion that the completion cap accounts for most of the menu penalty in these six configurations. A small penalty remains at the higher cap. The paper also includes configurations with substantial menu losses and little truncation, so not every failure can be attributed to the budget.

Finding 3: a failure label is not its cause

The grader checks nine obligations in a fixed order and labels the first unmet one. As a result, selection failure need not mean that the model chose an incorrect tool.

In fact, all 849 selection-failure labels belonged to episodes with zero tool calls. Of those, 838 ended at the completion cap. These episodes include cases that never reached the point of fulfilling the selection requirement. Section 5.3 and Table 3

Figure 3. The first unmet obligation and the termination reason are different observations. Explanatory diagram based on Section 5.3 and Table 3. The label alone does not establish the cause.

Of 20,408 exactly graded episodes, 3,725 failed. 91.5% of those failures ended at harness limits. This includes early stops when too few calls remain to finish; excluding those stops gives 78.7%.

The denominator is failed episodes. Hitting a limit also does not automatically imply a harness defect: a model can spend its budget on unnecessary steps. The paper distinguishes three questions: which obligation was unmet, which limit ended the episode, and why the episode reached that point.

Repair experiments: which changes help?

Alongside measuring pressure, the paper compares repairs that change how tools reach the model or who executes the calls. Section 5.4 and Figure 4

Choose the connector before retrieving tools. With 2,048 registered tools, this increased required-tool availability from 19.4% to 77.8%. Availability is a retrieval-stage measure, distinct from final task success. When a model-independent retriever omits required tools, switching the downstream model cannot change the omission. Appendix G

Use a controller to follow public handoffs. Another repair follows next-record links in tool results with a program rather than repeated model decisions. In the 30-call condition across 24 held-out intents and 17 configurations, success was 96.6%, compared with 0% for the model-run baseline. The program issued 28.7 of the average 29.9 executed calls. This is an effect of delegating execution, not evidence that the model itself acquired better long-workflow ability. Appendix F and Table 10

Add prompt guidance. Two repairs adding workflow or interface explanations produced no detectable improvement. However, they were tested on conditions without detectable baseline pressure losses. These results do not establish that prompt improvements in general are ineffective.

The comparison with model switching also depends on the setting. In a separate development pilot using only public information, switching to a fixed alternative configuration achieved 99.0% success, exceeding each public-information repair. The paper cannot be read as showing that harness repairs always outperform a model switch. Appendix J and Table 13

Limits of the conclusions

The study uses synthetic enterprise tasks, fixed English prompts, and exact all-obligations scoring. Most main capacity cells have one run per configuration. The budget intervention tests one higher cap, rather than broadly varying episode budgets and call limits.

The main routed-menu lane uses ground-truth tool information, so it is not an ordinary deployable router. In a separate 30-call experiment, success collapsed to zero under the fixed budget. That does not establish a general capability limit that AI cannot execute 30 steps. Limitations and Appendix F

In the external ToolSandbox environment, exposing all tools changed full success by −2.9 points [−7.0, +0.5], without a clearly detected decline. The increase in token consumption was clearer. This limits direct generalization of the main menu effect to another environment. Appendix K and Table 14

What the paper contributes

ToolBottleneckBench provides a benchmark for interpreting tool-agent success with explicit pressure conditions and harness budgets. It measures the loss from a larger visible menu, isolates part of that loss through a budget intervention, and demonstrates the distinction between failure labels and termination reasons.

Success measures a model together with its execution environment. The paper's central message is that comparisons need to account for the tools shown, the budgets granted, and the reasons episodes actually ended.

---

Original paper

Zichuan Xu, Yuyang Dong, Makoto Onizuka, Chuan Xiao. ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets. arXiv:2610.21098v1, October 15, 2026. Paper · PDF · Research code. Licensed under CC BY 4.0.

This article explains preprint v1. Numerical results are reported by the paper; we have not independently rerun the experiments. Figures 1 and 3 are explanatory diagrams based on the paper; Figure 2 is redrawn from Table 9.