Harness vs Model
Experiment · 2026
Can a general LLM running locally replace a purpose-built 200M specialist for knowledge-graph extraction? The answer depended on one default setting far more than on the model.
0.33 → 0.51
relation F1, from turning one setting off
11% → 0%
inputs where the model returned no usable answer
139 s → 10 s
time per sentence
288
test sentences, strict scoring, nothing tuned on the test set
The question
Agents lose context every time they hand work to each other. A knowledge graph is one way to give them shared context, but someone has to build it, which means pulling entities and relations out of plain text.
I run everything locally and did not want to load a second model just for that step. So: can the LLM I already run, Gemma 4 12B, do the extraction, or do I still need a dedicated specialist? I benchmarked it against GLiNER-Relex, a 200M model built only for this task, on CoNLL04.
The first run went badly
Run the default way, Gemma tied the tiny specialist on relations while taking about 200 times longer, and it failed outright on roughly one input in nine. The easy conclusion was that the model was not good enough and I should swap it.
I read the raw outputs and the token-level timing instead.
What was going wrong
Of the 32 failures, three causes accounted for all of them:
- Runaway reasoning, 25 of 32. Gemma 4 ships with a step-by-step “thinking” mode on by default. On hard sentences it reasoned in circles until it filled the context window and never wrote an answer. One reproduced case generated 3,886 tokens and returned nothing.
- Repetition loops, 4 of 32. At temperature 0, the model would emit a stray digit on a low-confidence word and loop on it until it was cut off.
- Stray special tokens, 3 of 32. A valid answer with an internal marker glued to the end, which broke strict JSON parsing.
Decisions
decision 01
Diagnose before replacing
- Situation
- On the first run Gemma only tied the small specialist, took about 200 times longer, and failed on roughly one input in nine. The easy conclusion was to swap the model.
- Task
- Find out why it failed before spending another long run on a different model.
- Action
- I traced each of the 32 failures to a cause in the raw outputs and the token-level timing.
- Result
- Three causes explained all 32 failures, and 25 of them came from a default "thinking" mode. None had to do with the model's ability.
decision 02
Rule out the plausible causes with data
- Situation
- Two explanations looked reasonable: a token cap cutting answers short, or the forced JSON output format slowing the model down.
- Task
- Check both explanations against data before changing anything.
- Action
- I checked the token cap against the raw outputs. Then I ran all four combinations of two settings: thinking on or off, and forced JSON output on or off.
- Result
- The token cap was not the cause: the failed runs were the longest ones, not good answers cut short. Forcing JSON adds only about 20% to the run time. The real cost was thinking tokens the model generates but never shows, which Ollama's token counter (eval_count) does not report.
decision 03
Keep the scoring strict
- Situation
- Lenient or semantic matching would have made both models look better.
- Task
- Produce a comparison that cannot quietly favour one model.
- Action
- I used strict micro-F1 with exact matching, the same rule for both models, and tuned nothing on the test set.
- Result
- "The United States" against "United States" counts as a miss. The numbers are conservative, and there is no leniency setting to argue about.
Results
CoNLL04 test set, 288 sentences, strict micro-F1.
| Model | Entity F1 | Relation F1 | Failures | Hallucinated entities | Time / sentence |
|---|---|---|---|---|---|
| GLiNER-Relex (200M specialist) | 0.543 | 0.322 | 0% | 0 | 0.6 s |
| Gemma 4 12B, thinking on (default) | 0.648 | 0.328 | 11.1% | 18 | 139 s |
| Gemma 4 12B, thinking off | 0.725 | 0.506 | 0% | 4 | 10 s |
The gain is not only from removing the failures. On sentences where thinking-on did return an answer, thinking-off still scored higher, so the reasoning was also making valid answers worse.
What it does not show
The specialist is still far cheaper, at 0.6 seconds per sentence on a CPU, and it cannot produce a broken answer by design, because it points at words already in the sentence instead of generating text. With thinking off, Gemma wins on quality. Which one to use depends on what the system can afford.
Takeaway
The model was capable the whole time. The outcome was decided by the harness around it: the prompt, the output format, the decoding settings and the parsing. How you run a model can matter as much as which model you pick.