Trying Small Local AI Models on STEM Problems

This is my 2nd post about local AI, you can read my informal and rather naive benchmark here. My first post covered models that felt smart enough and responsive enough, this meant I focused on the sparse MoE models and barely touched potentially more capable dense models. Around the time I was writing it, Muse Glimmer and Qwen 3.8 came out with much fanfare, with the latter seeming to steal the former’s thunder. This prompted me to revisit the other options. My earlier trials were on llama.cpp b10430 and I’ve since upgraded to b10472, I’d experienced issues with speculative drafting on the older version and they seem to have been resolved in the more recent version. While my earlier benchmark gave me an indication about model feasibility, I’d not explored ways to make it more efficient, nor had I evaluated the quality or speed of answering a real question. I’d subjectively evaluated the models by talking to them but nothing rigorous, I felt they were all similar albeit with certain quirks and strengths.

This time I’ve sought to find a more optimal configuration for this hardware based on the ability of the models to correctly answer a question and measured how fast it can do that.

There were three main things I wanted to answer:

• Can a more optimal configuration make slower models more viable?
• Could more capable models turn out to be a better choice even if their token generation was slower?
• Are some models better at navigating problems? Whether that’s applying a level of thinking to the task that is appropriate to its complexity, or reaching it in a way that’s more reliable due to directness/rigour etc.

I also had some meta questions about how to more robustly test agentic performance around particular tasks. So many LLM applications end up being “does a human think this is good enough?” - both the magic of RLHF during training and the genius idea to push AI as a sidekick with a human (supposedly) in the loop to supervise the process and the output. If you’re looking at this from a user experience and a behavioural perspective, that human supervision starts to look risky - there’s no shortage of disasters in history that started with “My job is to press this button to confirm that everything’s OK, and it always is” or other human factors related causes. Again, I’m not proposing to trust AI with anything here, but I did want to explore and develop more intuition about the ways that models could be stressed - from the perspective of inducing failures and potentially minimising them.

If you read any traces of the MoE models, they can be incredibly indirect - lots of “But wait” and covering the same ground again and again to re-verify. The more capable dense models were arriving at answers with far fewer thinking tokens, and while I didn’t expect that directness to make up for their slower generation, I had a hunch that the more ordered thinking would at least prove more robust.

Three things made the difference: speculative drafting, a lower temperature, and turning the thinking effort down on the models that have that dial.

I’ve enabled speculative drafting - MTP for Qwen 3.6, Qwen 3.8 and Gemma 4, and DFlash (a similar idea) for Muse Glimmer - with --spec-draft-n-max 4, which I found optimal on this hardware. All models benefited, more so the slower ones. I ran each model over the whole suite three times with drafting and three times without, changing nothing else:

Wall times throughout this post are the mean over every run, including the ones that failed. Without drafting, Gemma 4 31B ran out of wall clock on 13 of its 66 runs and Qwen 3.8 on 9, always on the hardest tasks, where with drafting neither hit the limit once. Muse Glimmer is fast enough that it never came close either way. So on this hardware, what drafting mostly bought me was getting the hard tasks to finish at all. That answers the first of my three questions: configuration alone took two models from failing the hardest tasks outright to completing every one of them.

That does mean the speed-ups above are understated: a run that was stopped at 1800 s enters the mean at 1800 s even though it hadn’t finished, so the true no-drafting means are higher than the table and the true ratios larger than 2×. Comparing generation rate instead, on the runs that did complete, it’s more like 2.3×, 2.6× and 3.3×.

Wrong answers barely moved either way - the apparent drop in accuracy without drafting is just the wall cap catching the slower runs. The token counts are the better check, since they measure the model rather than the clock: comparing like for like on the tasks where neither arm was ever truncated, the median moves by -1%, +15% and -7%. No consistent direction, and well inside the run-to-run scatter. Speculative decoding is supposed to be distribution-preserving and on this evidence it is - just as well, since every other number in this post was measured with it switched on.

I’ve reduced the temperature from 1.0 to 0.7 for all models. The official recommendation for all of these (except Nemotron Lightning) is to use 1.0 - at least for thinking use cases. Generally lower temperatures are favoured for agentic workflows for consistency, but we certainly wouldn’t want to go to a temperature of zero because we’d lose much of the benefit of thinking models being able to explore and break out of dead ends - and that matched my experiences when I tried it briefly. At 1.0 I was struggling to reproduce anything, so I settled on 0.7. Reducing the temperature wasn’t purely about performance, it was also about being able to rely on the measurements I was getting.

Having since run the whole suite at four temperatures, there appears to be a small improvement in correctness from lowering the temperature, though how much depends on the model:

These are counts of passing runs rather than percentages. Every cell is three passes through the suite, so 66 runs, and at that size a percentage would imply a precision the numbers don’t have.

1.0 comes out worst or joint-worst on all three. Below that it’s less obvious, so moving off 1.0 looks worth doing and I can’t separate what’s below it - and on Gemma 4 31B the whole difference is a fraction of a task either way. The penalty lands almost entirely on the harder tasks: across all twelve cells there’s exactly one easy-tier failure, Qwen 3.8 at 1.0. And it’s free - Muse Glimmer averages 102, 109, 110 and 106 seconds across the four temperatures, so if lowering it helps even slightly there’s nothing to trade away. I’ve not established the optimal temperature and I haven’t tried below 0.3.

Muse Glimmer and Qwen 3.8 have a configurable level of thinking effort. For this suite of tasks, I’ve found that low or medium is appropriate, which is much faster than the xhigh/high defaults. The two dials aren’t quite the same mechanism, incidentally - Muse Glimmer’s is a free-text variable that ends up in the system prompt as a suggestion, where Qwen 3.8’s is validated against a fixed list and actually changes what the template emits.

Thinking effort turns out to be a much bigger lever on draft acceptance than temperature is, at least on Qwen 3.8 - low and medium both sit around 78% and xhigh drops to 56%. Whatever xhigh does to the output, the draft model stops being able to follow it. I wouldn’t generalise that though, because Muse Glimmer goes the other way, drifting up from 53% at low to 58% at xhigh, so it’s presumably telling us something about MTP or about Qwen 3.8 rather than about thinking effort as such.

The suite is STEM focused, weighted towards practical rather than theoretical problems, and meant to be answered in one shot without tools or user interaction. Where a task needs a specific fact, that fact is in the prompt, but the method is not - so a task needs enough world and domain knowledge to solve without depending on niche recall. That is deliberate - in an agentic workflow I’d far rather a fact came from the prompt or a tool call than a hallucination. Answers are sometimes numeric and sometimes a selection, some tasks can be acceptably solved in more than one way, and I devised each one myself (no “Claude write me a benchmark”). Some are machine-translated into various European languages to see how the models coped.

The tasks aren’t deliberately adversarial, but there is ample room to go wrong: some carry puzzles that don’t need solving, some present information the question doesn’t need, and some present it in a heterogeneous way - all chances to do unnecessary work or magnify a mistake. I confirmed those traps bite by running less capable models and by forcing thinking to end early. I was aiming for something like an A level or university exam paper - you need to understand the problem and remember much of the domain knowledge like formulae, but not commit every niche fact to memory, and the better you know the topic the faster you answer and the more likely you are to catch your own mistakes. The harder tasks are several problems combined, where later parts depend on earlier answers but the earlier reasoning may turn out to be irrelevant.

Here are the results of each local model on that suite. All of these were tested locally over the course of about a fortnight, and the table below comes from 1,012 graded runs. Performing a similar benchmark on more capable hardware and trying to extrapolate to this hardware would be a mistake because of all the potential interactions.

As with the drafting table, the four rows marked ‡ are floors rather than measurements - a truncated run enters the mean at its 1800 s cap - so the true figure sits somewhere above.

Not everything I ran belongs in that table. These are the models that couldn’t keep up, on the same suite at the same settings, ranked and measured the same way - ‡ means the same thing here:

The easy tier is what separates these from the table above. Every configuration up there passes every easy run it attempted. Down here only GLM 4.7 Flash and the 8B and 30B of Granite 4.2 manage that, and the Granite 4.1 family drops easy runs badly - nine of fifteen, seven of fifteen and six of fifteen. That’s the difference between a model that struggles as the work gets harder and one that can’t reliably do the simplest thing in the suite.

Granite 4.1 3B fails in the most unusual way I saw. It averages nine seconds a task and about 190 tokens, and every one of its 66 runs ended normally: it answers quickly and simply gets it wrong. It passes nothing at all above the easy tier.

Granite 4.2 is better than 4.1 at answering and much worse at finishing. 4.2 30B gets 14 of 22 where 4.1 30B gets 20 of 66, but takes about seven times as long per task and was stopped by the clock on 6 of its 22 runs. It’s also the only row in either table using a draft model, so it isn’t quite like for like with the other Granites.

GLM 4.7 Flash is the closest of these to being usable and still isn’t - 35 of 66, at over nine minutes a task.

This is where the second of my three questions gets answered, and the answer surprised me. Wall time is really just two things multiplied together - how fast the model emits tokens, and how many of them it decides it needs - and on this hardware they don’t matter equally. Generation rate across these configurations spans about four times, from 8.8 to 36.2 tokens a second. How much they write spans about eleven times, from a median of 780 tokens a task to 8,700. Gemma 4 26B-A4B is the fastest thing here per token by a comfortable margin and it is nowhere near the fastest at finishing a task, because it spends eleven times as many tokens getting there. Muse Glimmer at low effort generates at about a third of Gemma 4 26B-A4B’s rate and still finishes the average task in well under half the time - 110 seconds against 286.

So the hunch in my introduction was right, but not for the reason I assumed - I’d expected directness to partly offset the slower generation, and in fact it dominates it. Which is a bit awkward for anyone choosing hardware or a model on tokens per second, because that’s the number everyone quotes and it’s the smaller of the two effects.

Failures do not all mean the same thing, and they separate almost perfectly by configuration. Qwen 3.5 9B failed six times and every one was a confident wrong answer. Qwen 3.8 at xhigh failed twelve times without ever getting an answer wrong - it simply never finished. Ornith 1.5 and Nemotron fail a third way, filling the token budget without ever committing to an answer. In the table those are all just a failure, but in practice they’re different problems, and if you’re putting a model into anything with a human meant to be supervising it, worry about the one that hands you a plausible wrong number.

Splitting each configuration by difficulty answers the third of my opening questions, so the figure below gives each one three rows, grouped by family so a model’s effort settings stack together. How much extra a configuration spends as the work gets harder varies substantially. Muse Glimmer at low effort scales the way I’d hope, staying quick on the easy tier and only opening up on the hard one, where Qwen 3.6 27B is already slow on the easy tasks and thinks hard about everything, whether or not it’s needed.

Within a family, the effort dial mostly affects the hard tier. Turning effort up barely changes the easy tasks but piles time onto the hard ones, until they start hitting the wall clock. Qwen 3.8’s settings behave like Muse Glimmer’s.

The whole grid, every configuration against all 22 tasks grouped by difficulty. Each cell is passing runs out of total runs - three for most, one for the single-pass configurations - shaded green for all passed, amber for some, red for none.

A pass meant a correct answer produced within 25,000 tokens and 1800 s, whichever came first. Those limits were somewhat arbitrary, and they do real work in the results table - four of the six worst-ranked configurations are there mostly because they hit a limit I chose rather than because they got anything wrong. I’d still count that as a failure, since a task that doesn’t finish inside half an hour is no use to me either, but it isn’t the same kind of failure as a wrong answer and I’ve tried not to blur the two.

Two things aren’t quite like for like: Nemotron is at temperature 0.6 where everything else is at 0.7, because that’s what its own documentation asks for. And the two Ornith rows are different quantisations - 1.0 is UD-Q4_K_XL and 1.5 is Q4_K_M - so some part of the gap between them may be the quant rather than the model. I’ve not explored quantisation properly, but for what it’s worth, what I did try suggested that whatever a smaller quant buys you in generation speed gets lost again in the time to finish the task. Three runs across the whole set were graded partial and I’ve counted those as failures.

The winner on this suite is Muse Glimmer at low effort, though I should be careful about how much weight that puts on the correctness column. The top four configurations all score exactly 65/66, so I can’t tell them apart on accuracy at all. Time separates them, and not by a little: 110 seconds against 199, 267 and 359. So I’m not claiming Muse Glimmer is more correct than the others. It’s the fastest member of a group I can’t distinguish on correctness. That’s a weaker claim, but it’s the one the data supports.

Qwen 3.8 27B and Gemma 4 31B are also both solid choices, but lose out on speed. Gemma 4 26B-A4B and Qwen 3.6 35B-A3B are still strong models, but there is some flakiness on the hardest tasks and the speed ends up very similar to their denser siblings. A major consideration for me in developing an agentic workflow is how easily the AI model can be pushed outside its envelope - because the real world is messy and I’m sure that I’m not alone in finding myself choosing a slightly more capable model than I think I’ll need for the extra “trust”.

Here are the three, as I actually ran them. They’re the three different models at the top of the table - Muse Glimmer also takes second place at medium effort, but that’s the same weights with one word changed, so it didn’t seem worth a fourth block. GGUFs are Unsloth’s dynamic quants except the Gemma draft head. Paths are relative to wherever you keep your models.

# Muse Glimmer 30B, low effort
llama-server -m Muse-Glimmer-30B-UD-Q4_K_XL.gguf \
  --spec-draft-model muse-glimmer-dflash-kquant.gguf \
  --spec-type draft-dflash --spec-draft-n-max 4 --spec-draft-ngl 99 \
  -c 32768 -ngl 99 -fa on -ctk f16 -ctv f16 -ub 2048 --jinja \
  --chat-template-kwargs '{"reasoning_strength":"low"}' --reasoning-budget -1

# Gemma 4 31B
llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf \
  --spec-draft-model mtp-gemma-4-31B-it-Q4_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-ngl 99 \
  -c 32768 -ngl 99 -fa on -ctk f16 -ctv f16 -ub 2048 --jinja \
  --chat-template-kwargs '{"enable_thinking":true}' --reasoning-budget -1

# Qwen 3.8 27B, medium effort
# no --spec-draft-model here: Qwen ships the MTP head inside the main GGUF,
# where Gemma 4 needs it as a separate file
llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -c 32768 -ngl 99 -fa on -ctk f16 -ctv f16 -ub 2048 --jinja \
  --chat-template-kwargs '{"enable_thinking":true}' --reasoning-budget -1

Sampling goes per request rather than on the server, and it’s where the two effort dials part company - Muse Glimmer’s lives in the template kwargs above, Qwen 3.8’s is the reasoning_effort sampling parameter. A dash means I left the parameter unset:

A word on how hard these tasks actually are. The suite makes no attempt to prove which of the top models is most capable - task difficulty is the floor, can-run-on-this-machine is the ceiling, and what separates the models is how long they take. I sought to set the floor at around the level where AI first became useful to me beyond fancy autocomplete or natural-language glue, which for me was roughly Sonnet 4 in mid-2025 (I’m not an early adopter - I wait for the kinks to be ironed out). The table below shows how some historic frontier models handle the tasks, purely as an indication of difficulty. A like-for-like comparison with the local models wouldn’t be meaningful, given the differences in size, training and feasible context. But it lets you read the floor as “the models you started using around mid-2025 could tackle this well”. Some models of that era have already been retired, which is a shame - it would have made the shift from “can’t” to “can” more obvious. I used what was available to me on OpenRouter.

These were run once each at a temperature of 1.0 and with a 16k reasoning budget through OpenRouter. I made no attempt to make them comparable with the local models.

I have decided not to publish the task prompts in case I want to try this with future models without the risk of contamination. One of the reasons I did this test was because of concerns about benchmaxxing - not that I think anyone will be looking for my benchmark! Here is a brief description of each task without giving the game away:

None of the tasks used vision or audio capabilities, all were text prompts. The rated difficulty was based on constructing a 3-axis difficulty space (share of tokens used per model family, correctness rate and the time to first correct answer by any local model) and then k-means clustering (k=3). All nine tasks that were translations came out in the same difficulty tier as their untranslated counterparts.

One caution, I wouldn’t be trusting any of these problems to an AI model unsupervised - and if I had to, I’d ask it to write a Python script and then audit that. But testing against these tasks provides an indication of whether these models can understand the topic and apply what they know beyond mere parroting.

Same machine and OS as my previous post, which has the full specification - a Minisforum AI X1 Pro-370, so a Ryzen AI 9 HX 370 with a Radeon 890M iGPU and 64 GB of DDR5-5600 across two channels, of which the iGPU can address 30432 MiB.

What’s new since then is the software: llama.cpp build b10472 rather than b10430, the prebuilt Ubuntu Vulkan x64 binary used as shipped on the Vulkan/RADV backend. Everything here ran on that one build, on Linux 7.0.0-29-generic as before - so the numbers are at least internally consistent, whatever else they are.

Given that drafting nearly tripled throughput on the slowest models, and that the terse ones finish far sooner than their token rate suggests, it would be fair to ask whether any of this moves the hardware verdict I reached last time. The best configuration here still averages nearly two minutes a task and over four on the hard tier, and the ones just behind it run to ten or fifteen minutes - on problems that the stronger hosted models further up answered correctly and, in my experience of running them, in a small fraction of that time. It’s an improvement, but it’s still not fast.

So my general conclusion about hardware remains the same - good for narrow and light AI use, but wouldn’t recommend it for a local AI enthusiast or heavy use - unless you’re particularly interested in what can be squeezed out of less capable hardware, which I am at the moment! Like a lot of people I’m concerned about where AI is going, the potential to concentrate power, the risks to communities associated with scaling and the largely uncompensated capture of the knowledge and creative output of humanity - I’d be able to sleep more easily if we can get to a point where capable open models can be run locally on systems that are within reach of more people - as a way to rebalance that somewhat.

I’ve adapted this benchmark as I’ve gone along, not to lean on the scales in favour of any particular model or towards local AI - but to add more harder tasks around the weaker areas to get the floor where it needed to be, and easier ones to expose models that overthink everything. In an attempt to try to keep this honest I have only added tasks, I have not removed tasks or modified them. This isn’t good scientific practice, but my goal wasn’t a rigorous study, it was “what can I squeeze out of the hardware and models available to me”.

There are a great many smart people doing far more sophisticated research than this and I’m sure there are all sorts of consultancies and products in this space, but I’ve always appreciated the accessibility of write ups of DIY experiments like this, by someone who’s not trying to sell you something - and that’s always in short supply.

The data for this article was gathered over the last few weeks, and llama.cpp b10472 was released on 17 August 2026. There have been various developments since then that this post doesn’t cover - for example DFlash2, which may further improve speculative decoding for Muse Glimmer and Qwen 3.8, though it would need a newer llama.cpp build.

Thanks for reading my anecdata!