~/sameer-batra
← research

Harness vs Model

Experiment · 2026

Can a general LLM running locally replace a purpose-built 200M specialist for knowledge-graph extraction? The answer depended on one default setting far more than on the model.

0.33 → 0.51

relation F1, from turning one setting off

11% → 0%

inputs where the model returned no usable answer

139 s → 10 s

time per sentence

288

test sentences, strict scoring, nothing tuned on the test set

PythonOllamaGemma 4 12BGLiNER-RelexCoNLL04

The question

Agents lose context every time they hand work to each other. A knowledge graph is one way to give them shared context, but someone has to build it, which means pulling entities and relations out of plain text.

I run everything locally and did not want to load a second model just for that step. So: can the LLM I already run, Gemma 4 12B, do the extraction, or do I still need a dedicated specialist? I benchmarked it against GLiNER-Relex, a 200M model built only for this task, on CoNLL04.

The first run went badly

Run the default way, Gemma tied the tiny specialist on relations while taking about 200 times longer, and it failed outright on roughly one input in nine. The easy conclusion was that the model was not good enough and I should swap it.

I read the raw outputs and the token-level timing instead.

What was going wrong

Of the 32 failures, three causes accounted for all of them:

  1. Runaway reasoning, 25 of 32. Gemma 4 ships with a step-by-step “thinking” mode on by default. On hard sentences it reasoned in circles until it filled the context window and never wrote an answer. One reproduced case generated 3,886 tokens and returned nothing.
  2. Repetition loops, 4 of 32. At temperature 0, the model would emit a stray digit on a low-confidence word and loop on it until it was cut off.
  3. Stray special tokens, 3 of 32. A valid answer with an internal marker glued to the end, which broke strict JSON parsing.

Decisions

decision 01

Diagnose before replacing

Situation
On the first run Gemma only tied the small specialist, took about 200 times longer, and failed on roughly one input in nine. The easy conclusion was to swap the model.
Task
Find out why it failed before spending another long run on a different model.
Action
I traced each of the 32 failures to a cause in the raw outputs and the token-level timing.
Result
Three causes explained all 32 failures, and 25 of them came from a default "thinking" mode. None had to do with the model's ability.

decision 02

Rule out the plausible causes with data

Situation
Two explanations looked reasonable: a token cap cutting answers short, or the forced JSON output format slowing the model down.
Task
Check both explanations against data before changing anything.
Action
I checked the token cap against the raw outputs. Then I ran all four combinations of two settings: thinking on or off, and forced JSON output on or off.
Result
The token cap was not the cause: the failed runs were the longest ones, not good answers cut short. Forcing JSON adds only about 20% to the run time. The real cost was thinking tokens the model generates but never shows, which Ollama's token counter (eval_count) does not report.

decision 03

Keep the scoring strict

Situation
Lenient or semantic matching would have made both models look better.
Task
Produce a comparison that cannot quietly favour one model.
Action
I used strict micro-F1 with exact matching, the same rule for both models, and tuned nothing on the test set.
Result
"The United States" against "United States" counts as a miss. The numbers are conservative, and there is no leniency setting to argue about.

Results

CoNLL04 test set, 288 sentences, strict micro-F1.

Model Entity F1 Relation F1 Failures Hallucinated entities Time / sentence
GLiNER-Relex (200M specialist) 0.543 0.322 0% 0 0.6 s
Gemma 4 12B, thinking on (default) 0.648 0.328 11.1% 18 139 s
Gemma 4 12B, thinking off 0.725 0.506 0% 4 10 s

The gain is not only from removing the failures. On sentences where thinking-on did return an answer, thinking-off still scored higher, so the reasoning was also making valid answers worse.

What it does not show

The specialist is still far cheaper, at 0.6 seconds per sentence on a CPU, and it cannot produce a broken answer by design, because it points at words already in the sentence instead of generating text. With thinking off, Gemma wins on quality. Which one to use depends on what the system can afford.

Takeaway

The model was capable the whole time. The outcome was decided by the harness around it: the prompt, the output format, the decoding settings and the parsing. How you run a model can matter as much as which model you pick.