Brief
Clinical calculator AI: code solver lifts 32B model, not 7B, preprint finds
A preprint tests whether language models do clinical arithmetic better by writing Python for a restricted executor to run instead of calculating directly. It reports a gain for a 32B model but not a 7B one, and says verified formulas remain essential.
A preprint posted to arXiv on 12 September tests whether language models handle clinical arithmetic better when they write case-specific Python for a restricted executor to run rather than calculating directly. The authors evaluated the approach on MedCalc-Bench Verified, a set of 1,100 cases spanning 55 calculators.
With formulas and variables supplied, the solver lifted Qwen2.5-32B-AWQ from 83.47% to 90.53%, a paired gain of 7.05 points with a 95% interval of [0.47, 14.60]. For Qwen2.5-7B the gain was 3.29 points, from 72.02% to 75.31%, with an interval of [-3.49, 10.38] that includes zero.
A hand-written 22-calculator library was exact on its 440 supported cases but abstained elsewhere, scoring 40.0% overall. The authors audited the benchmark's formulas against current clinical guidelines and flagged 16 of 55 for version, use or coefficient concerns.
Our reading
Our reading is that this is preliminary evidence that program-solvers may help larger clinical language models, but with only two models tested and 16 benchmark formulas flagged, the finding is far from settled.
What to do or watch
Watch for tests of the Program-Solve approach beyond the two Qwen2.5 models here, since the 32B gain was clear of zero while the 7B interval included it, and note that the evaluation supplied gold formulas and variables rather than testing extraction from notes. The unresolved question is whether the solver advantage holds once the 16 of 55 benchmark formulas flagged for version, use or coefficient concerns are corrected against current guidelines.
Source details and supporting facts
Each line is stated by the page named above it.
Stated by arXiv
- The study evaluates a Program-Solve interface on MedCalc-Bench Verified, which contains 1,100 cases and 55 calculators.
- At 32B, the solver achieved 90.53% versus 83.47% for direct arithmetic, a paired +7.05 points with a 95% interval of [0.47, 14.60].
- At 7B, the solver achieved 75.31% versus 72.02%, a paired +3.29 points with a 95% calculator-cluster interval of [-3.49, 10.38].
- The hand-written 22-calculator library was exact on its 440 supported cases but abstained elsewhere, yielding 40.0% overall.
- The benchmark's formulas were audited against current clinical guidelines, and 16 of 55 were flagged for version, use or coefficient concerns.
Sources
- arXivText stored 13 September 2026
How this story was checked. Written from the 1 page listed above, stored 13 September 2026; claims checked against that stored text on 14 September 2026.
What that means
- 5 of 5 reported statements were confirmed against the page that carries them; the rest were removed rather than published.
- Figures in the text were required to appear in the stored source text: yes. Identifiers: yes.
- The check reads stored text only: no claim rests on a fresh look that did not happen.
- Where the reporting was silent, the text says so instead of filling the gap.