~/sameer-batra
← research

Context-length extension

Research experiment · 2025 – 2026

Small language models can beat large ones on a specific task, but they cannot read long inputs. I tested whether two context-extension methods, Linear Position Interpolation and YaRN, let Phi-2 and TinyLlama read inputs longer than the limit they were built for.

286 → 15

Phi-2 perplexity on 8K-token inputs, base model against fine-tuned with Linear PI (lower is better)

100 / 91 / 36%

retrieval at 2K, 4K and 8K tokens for Phi-2 with Linear PI

56 / 44 / 32%

reading comprehension at 2K, 4K and 8K tokens, against 25% for random guessing

2 × 2 × 3

models, extension methods and benchmarks

PythonPyTorchHugging FacePEFT4-bit quantizationPhi-2TinyLlama

The question

A small language model can beat a large one on a specific task. Its weak point is context length: it can only read a short input.

Methods for extending context were designed and tested on larger models. I wanted to see what happens when they are applied to small ones.

What I tested

  • Models. Phi-2 (2.7B parameters) and TinyLlama (1.1B).
  • Methods. Linear Position Interpolation (Linear PI), the simpler one, and YaRN, the more sophisticated one, implemented from the ICLR 2024 paper. Both were followed by fine-tuning with 4-bit quantization.
  • Benchmarks. Needle-in-a-Haystack for retrieval, perplexity for how well the model predicts long text (lower is better), and QuALITY for reading comprehension.

Each experiment is defined in a YAML config, so any run can be repeated.

Results

Phi-2, by input length.

Configuration Retrieval at 2K / 4K / 8K Perplexity at 8K Comprehension at 2K / 4K / 8K
Phi-2, base model — 286 —
Phi-2 + Linear PI 100% / 91% / 36% 15 56% / 44% / 32%
Phi-2 + YaRN 0% at 4K and 8K — 20% at 8K

Comprehension is multiple choice, so random guessing scores 25%. A dash means the project’s results summary does not give that figure.

  • Linear PI was the only setup that held up on all three tests. Retrieval was still 91% at 4K and perplexity stayed at 15 at 8K, but at 8K retrieval and comprehension fell to 36% and 32%.
  • YaRN did not carry over. Retrieval fell to 0% at 4K and 8K, past the input length Phi-2 was built for, and comprehension at 8K was below random guessing.
  • TinyLlama mostly failed. Fine-tuning kept its perplexity between 27 and 31 at 8K, where the base model reached 986. But retrieval at 2K dropped from 73% to 45% after fine-tuning with Linear PI, and comprehension fell below random guessing.

Takeaway

On these two small models, the more sophisticated method did not give the better result. The simpler one, Linear PI, was the only one that extended what Phi-2 could do, and even then quality fell steadily as inputs grew: retrieval at 8K was 36%.

What it does not show

These results cover two models and inputs up to 8K tokens. They do not show how either method behaves on other small models or on longer inputs.