Research Datasets Models Blog About Contact

{{ post.title }}

{{ post.excerpt }}

{{ post.date }}

February 2026

Why Aggregate Accuracy Lies

A model that scores 95% overall accuracy sounds finished. It usually isn't. Aggregate accuracy averages over every example equally, which means it treats a model that fails randomly the same as a model that fails on the same ten patients, or the same rare class, every single time. Those two models are not equally safe to deploy, but a single number can't tell them apart.

In our own audits, we've found that errors are rarely random. They cluster, in minority classes, in specific subpopulations, in inputs that share some structural quirk the model never learned to handle. A benchmark score hides exactly the information a practitioner needs most: not "how often does this fail," but "who does it fail on, and how confidently."

This is why we report class-level and instance-level error breakdowns wherever we can, even when the aggregate number looks good. Especially when it looks good.

April 2026

The Hidden Tax in How We Write

Every time you choose a curly quote over a straight one, a dash over a hyphen, or a contraction over the full phrase, you are quietly changing how many tokens a language model needs to read the same sentence. Nobody decided this on purpose. Tokenizer vocabularies are learned from whatever text happened to be common during training, so some perfectly ordinary formatting choices are cheap and others are not, for reasons that have nothing to do with meaning.

We think this matters more than it sounds. Token-based billing is now a real cost that shapes what gets written, and a tax nobody can see is a tax nobody can budget for. Measuring it is the first step toward tokenizers, and pricing, that don't quietly penalize ordinary writing.

March 2026

Small Models, Small Budgets

A model that runs entirely on a phone or a laptop, with no data center behind it, has to be judged differently than a frontier model with unlimited compute. The question isn't "how smart is it," it's "exactly where does it stop working", after how many reasoning steps, how many simultaneous instructions, how long a response before it runs out of budget and produces something unparseable.

We've found that a surprising share of small-model failures aren't reasoning failures at all, they're formatting failures. The model knew the answer and ran out of room to say it cleanly. That distinction matters enormously if you're deciding which small model to ship, and most single-score benchmarks erase it entirely.

June 2026

Confidence Is Not Correctness

Two models can reach the same accuracy and behave completely differently when they're wrong. One hedges, its wrong answers come with low confidence, giving a human or a downstream system a chance to catch the mistake. The other is confidently wrong, which is far more dangerous in any setting where a wrong answer has real consequences, from a diagnosis to a financial decision.

Calibration is not a footnote to accuracy, it's a separate property that deserves separate reporting. An uncertain wrong answer can be caught. A confident one usually can't.

© 2026 Invariance Labs.