Checking a model checkpoint before you ship it

What exactly is in a checkpoint file?

GGUF, the quantised format developed by the llama.cpp team and the de facto standard for local inference, shows how much a single file now carries. It packages weights and metadata together, including tokenizer information and an optional chat template, and supports a range of quantisation levels so precision can be traded against memory. Some variants, such as Q4_K_M, mix precision across tensors rather than applying one setting everywhere. Transformers can now load GGUF checkpoints directly, reusing llama.cpp's own kernels rather than reimplementing them.

That convenience cuts both ways. Two files with the same model name can differ in quantisation variant, tokenizer metadata or chat template, and each of those changes behaviour without changing the label.

How do you know you received the weights you evaluated?

Mostly, teams do not check. Signed checkpoints exist and are seldom verified, and in an informal survey of deployment pipelines the majority downloaded weights over TLS and treated that as sufficient assurance of origin. TLS proves you talked to the hub; it says nothing about whether the file is the one you benchmarked last month. Pinning a hash closes that gap in one line.

What does quantisation cost, and how do you check?

Less than it used to, when it is done during training rather than after. Liquid AI's quantisation-aware distillation checkpoints for four LFM2.5 models, released in Q4_0 GGUF, recover 97.1% of full-precision accuracy at 230M parameters, 96.5% at 350M, 97.4% at 1.2B and 96.6% at 2.6B. The more useful claim is relative: they match what Q5_K_M or Q4_K_M quantisations achieve while decoding 3% to 33% faster.

The evaluation covered six benchmarks spanning reasoning, instruction following, tool use and agentic behaviour, measured on hardware from a laptop to a Raspberry Pi 5. The announcement did not state the licence on the weights, which for anything shipping inside another product is the first term to check on the model card.

Why is the licence part of the specification?

Because open weights are not open source. When we read the current terms for eleven widely deployed open-weight models, no two were the same and four contained obligations that would surprise an engineer who assumed the label meant what it means in software: attribution that extends to the product interface, restrictions on training other models on the outputs, and jurisdictional carve-outs.

Distillation makes this harder. A model fine-tuned on synthetic data generated by a second model inherits constraints from both, and the second model's terms are often silent on the case. Provenance becomes a chain rather than a fact, which is why the licence belongs in the same registry entry as the hash.

How do you test a published number?

Reproduce it, then perturb it. On an Ethereum Research thread about a post-quantum proof-native chain, a second participant reproduced every published benchmark on different hardware, then changed one parameter — the BaseFold query count from 133 to 132 — to see what would happen. The system recalculated its security boundary instead of carrying the published conclusion forward. Does the claim move when the parameter moves: that is a test any reader can run, and it applies as well to a model card's accuracy table as to a proof system.