Researchers compared two ways large language models (LLMs) call external tools. The first way is the common “JSON tool calling” where the model outputs structured JSON objects for each tool call. The alternative, called programmatic tool calling (PTC), has the model write a short Python script that calls typed tool stubs and is executed once by an agent. On a representative subset of the BFCL v4 benchmark, PTC matched or outperformed JSON tool calling for most models tested.

The team tested 14 models released between November 2024 and July 2026 on 309 entries from BFCL v4. In the PTC setup each available tool was provided as a typed Python stub. The model wrote a script that imported those stubs and printed the tool-call results. The script ran in a subprocess and the scorer compared printed outputs to the ground truth. In the JSON setup the models used the usual API-style function-calling interface that returns one JSON tool-call object per model turn. The experiments used the same task descriptions and a deterministic scorer that normalizes arguments before matching.

Across the full set, PTC matched or exceeded JSON tool calling in 11 of 14 models. The newest GPT-5.6 family showed the largest lift, about 10.6% absolute improvement over the JSON baseline. All five Anthropic models and the three newest GPT generations matched or beat the JSON baseline, while three older GPT models did not. PTC also held up better when tasks required many parallel calls: it matched or outperformed baseline in 13 of 14 models under parallel fan-out.

Some concrete stress tests underline where PTC helps. When tasks required long chains of dependent calls, the PTC advantage grew with chain length and reached an 18.8% absolute gap at chain lengths of 12 or more. Under high fan-out, the native JSON method began to drop calls above a model-specific threshold (for example, Claude Sonnet 5 started losing calls around 70–72 parallel items), while PTC maintained full enumeration accuracy even at 100 items. In context-flooding tests, PTC remained stable while the JSON baseline degraded by about 2.3% on average. Against a secondary “filesystem-discovery” condition used as a reference (not an industry practice), PTC improved by about 5.5% whereas that filesystem approach degraded by about 32%.