Bigger but Compressed, or Smaller but Precise?

COMP516 Group30 · Qwen2.5-Instruct on the GSM8K test set ({{ nq }} questions per configuration) · llama.cpp b5001 on an RTX 4060 Laptop GPU (8 GB).

{{ t.sub }}

Each line is one model size; points move left as precision drops from FP16 to Q8_0 to Q4_K_M. At any file size the higher line, the bigger model, is more accurate. Whiskers show 95% bootstrap confidence intervals.

{{ s }}

Two models with similar file sizes, answering the same questions. The bigger compressed model wins every pair.

Bigger, compressedSmaller, precise
Metric
{{ p }}

Accuracy change against FP16 for the same model size. Paired McNemar test; Holm correction across the six tests. Statistics were run after data collection.
Model size {{ s }} Precision {{ p }}
GSM8K test split, all {{ nq }} questions; same questions and order for every configuration Qwen2.5-Instruct 0.5B, 1.5B, 3B; official post-training quantized GGUF files (FP16, Q8_0, Q4_K_M); no retraining NVIDIA RTX 4060 Laptop GPU (8 GB VRAM), Intel Core i9-14900HX, 16 GB RAM, Windows 11 llama.cpp b5001 (CUDA 12.4), llama-server; Python 3.12 evaluation scripts System: "You are a helpful assistant." User: {question} + "Solve the problem step by step. At the end, write the final answer as: The answer is <number>." Greedy, seed 42, context 2,048 tokens, max 1,024 output tokens, one request at a time Paired McNemar test, 10,000-sample bootstrap 95% CIs, Holm correction (post hoc)

File size is GGUF bytes divided by 10⁹, not runtime memory. Energy is the GPU's sampled power minus its idle baseline, integrated over the run; it excludes the rest of the laptop. The scorer misread 44 of 11,871 boxed answers; a corrected scorer changes accuracy by under one point and no conclusion. Sources: experiment/results/formal-v3/main_summary.csv, results/analysis-2026-10-19/paired_statistics.json, configs/model_manifest.csv. Generated 2026-10-30.