What was open-sourced
Discussions about "which model is better" almost always run into the same problem: nobody has verified the numbers. Measurements are taken on a different machine, with different quantization, in a different runtime build — and comparing them to each other is meaningless. Liquid AI proposed to fix this with openness and reproducibility: it released Pipette, a platform for benchmarking foundation models on edge devices. The methodology was independently validated by Artificial Analysis.
The core idea the whole construct rests on: on-device behavior is a property of the deployed system, not of the model in isolation. That means the unit of measurement shouldn't be "the model" either, but the full configuration: weights plus quantization scheme plus runtime plus specific hardware. From this thesis grows both the structure of the dataset and the way all the comparisons below should be read.
What made it into the starting dataset
- five on-device performance metrics;
- over a thousand configurations across the combination model × quantization × runtime × device × context length;
- over 30 models and several quantization formats;
- llama.cpp builds for macOS, iOS, Windows, and Android;
- context lengths from 256 to 8,192 tokens.
The first published measurements were taken on a MacBook Pro with M5 Max, iPhone 17 Pro, and Galaxy S26 Ultra. For AMD Ryzen AI Max+ 395 and Radeon 8060S, the results are still marked "coming soon" — that is, the platform was announced as cross-platform from the start, but hardware coverage is still growing.
Four takeaways from the first runs
Same parameters — different behavior on long context
A good illustration of why such measurement is needed at all. Two 350M-parameter models with the same Q4_K_M on the same phone behave differently as input tokens grow: Granite-4.0-H-350M retains 78.4% of decoding throughput when going from 256 to 4,096 input tokens, while Granite-4.0-350M retains only 33.8%. The parameter count tells you nothing here: the difference lies in the architecture and in how it maps onto a specific runtime and chip.
Sparse activation saves compute, but not memory
LFM2.5-8B-A1B on the same phone at 2,048 input tokens decodes 2.4 times faster than Qwen3.5-4B and 2.6 times faster than Ministral-3-3B-Instruct-2512. The secret is sparse activation: roughly 1.5B of 8.5B parameters are engaged per token. But peak memory consumption is 5.29 GiB, because all expert weights still have to reside in memory in full. The practical takeaway: MoE-like architectures win on time but don't save your RAM budget — and on a phone the limit is often exactly memory.
Faster doesn't mean better
On iPhone 17 Pro with Q4_K_M, MiniCPM5-1B completes a workload of 2,048 input / 256 output tokens in 3.47 s, while LFM2.5-1.2B-Instruct takes 4.12 s — that is, the former is 15.8% faster. However, on the same artifacts LFM scores 9.0 points higher on MATH-500. Throughput and quality are different axes, and choosing a model by a single number in a table is meaningless.
Nearly identical system profiles can hide a reversal on tasks
On M5 Max with Q4_K_M and 2,048 input tokens, Granite-4.1-8B and Ministral-3-8B-Instruct-2512 differ by just 2.4% in decoding throughput and 1.2% in peak RAM. But on tasks the picture changes: Granite wins by 7.3 points on IFBench, while Ministral beats it by 14.0 points on GPQA Diamond. The difference in the "hardware" profile is within noise, the difference in behavior is fundamental.
How the measurements are structured
Performance runs are built on fixed token shapes, greedy decoding, discarded warmup, and five measured repetitions. Before each repetition, readiness gating kicks in: a platform-specific check verifies that thermal conditions and background load are within normal range. Failed runs don't make it into the publication — this is exactly what distinguishes a reproducible benchmark from a one-off script.
A separate storyline is quality. It isn't measured by the same run: IFBench, GPQA Diamond, and MATH-500 are used for that, and the scores are taken from llama.cpp evaluation runs on reference systems with NVIDIA H100 80GB. They are then matched against on-device runs for the same model and the same quantization. An important consequence: the quality number sitting next to a phone's throughput was not obtained on the phone. It's a convenient metric for comparing models to each other, but not a measurement of what happens on a specific smartphone.
What exactly is being open-sourced
Pipette ships in full, with no waitlist:
- infrastructure under Apache 2.0 — the pipette-mgmt, pipette-clients, and pipette-scores repositories;
- a public results dataset;
- a hosted dashboard;
- native benchmarking apps for iOS and Android.
The only part not yet ready for general access is the publication of community-submitted results: it's in beta. Everything else can be run on your own, including deploying the pipeline inside your own perimeter.
Who needs it and why
The easiest way to describe the target audience: it's any team that ships a model onto hardware it doesn't own. Beyond that, the options diverge by scale.
- A solo developer and a seed-stage startup can get by with the dashboard and mobile apps — no infrastructure of their own needed.
- A mid-sized product team can deploy clients across its internal fleet of devices and get measurements on its own configuration.
- Large OEMs, chipmakers, and enterprises can keep the entire pipeline behind a firewall — which removes questions about where measurement data goes.
Typical tasks are also clear without much explanation: choosing a model and quantization format before sprint tasks are locked in; justifying a SoC or hardware purchase; catching a regression when updating the runtime, OS, or driver; planning capacity by context length; independently verifying vendors' marketing claims. Industries — consumer electronics and smartphone OEMs, automotive, manufacturing and robotics, medical devices, financial services, defense: anywhere latency, privacy, or lack of connectivity forces you to run the model directly on the device.
What to look at before trusting the numbers
Three things that are easy to forget when reading any table of measurements.
First, quality numbers and performance numbers come from different places. Throughput is measured on-device, quality — on a reference system with an H100 and then matched. For comparing models this is valid, for predicting behavior in a specific application — it isn't.
Second, quantization can't be factored out. The same format on different models and different runtimes yields different results, and the memory limit often becomes decisive — as in the sparse activation example, where speed went up but the 5.29 GiB didn't go anywhere.
Third, identical performance says nothing about how models will handle specific tasks. The reversal of Granite and Ministral on IFBench and GPQA Diamond with nearly identical system profiles is exactly the case where you first need to decide on the task and only then look at the tables.
