Almost all the attention in inference optimization goes to things you do at runtime. Quantization schemes, batching strategy, KV cache management, speculative decoding, serving frameworks. That is where the interesting research is and it is where the conference talks are.
Meanwhile, there's a dimension most teams never touch at all, and on Arm hardware it's worth a multiple rather than a percentage.
I built a tuner for llama.cpp that sweeps the compile dimension for a target Arm chip: the -march and -mcpu settings, the instruction-set features the build is allowed to assume, and the quantization format, then benchmarks each resulting binary on real hardware. Running it on AWS Graviton, compile-flag discovery alone produced a 2 to 3x prefill speedup, reproduced across runs.
Nothing about the model changed. Same weights, same runtime code, same machine. Only the instructions the compiler was permitted to emit.
Why the gap exists
The default build targets a conservative baseline. It has to, because a binary that assumes instructions your chip doesn't have does not run slowly, it crashes. So the safe default is the lowest common denominator across a family, and on Arm that family is unusually wide.
The Arm ecosystem isn't one target. There is a real spread across generations and vendors in which extensions are present: the dot-product instructions, the matrix-multiply extensions, the various vector features. A generic aarch64 build cannot assume any of the good ones. A build that names your actual core can use all of them, and the kernels that matter for transformer inference are exactly the kernels those extensions were designed for.
The result is a large, boring, reproducible gap between "it compiled and runs" and "it compiled for this chip." Most deployments sit on the wrong side of it, because the default worked and nobody had a reason to look.
Why nobody checks
Three reasons, and they're all understandable.
The feedback loop is invisible. If you pick a bad flag your build fails or the binary faults immediately. If you pick a merely conservative flag everything works, and there's no signal at all that you left performance on the table. Silent suboptimality is the hardest kind to notice.
The search space is unpleasant. Architecture flags, feature flags, quantization formats, and their interactions don't decompose cleanly. You can't reason your way to the answer from the documentation, and the interactions are real, meaning the best quantization format isn't independent of the instruction set you compiled for.
And the whole area is culturally unfashionable. Build configuration reads as sysadmin work. It's not a model improvement, it doesn't make a good paper, and there is no framework to adopt.
Which is exactly why it should be automated
The problem has a shape that suits a search rather than an expert. Candidates are enumerable, each one is cheap to build, and the evaluation is an unambiguous number from a benchmark on the actual target hardware. You do not need theory. You need a loop and the patience to run it.
That's what I built. Enumerate plausible configurations for a named target, build each, benchmark each on that hardware, keep what wins. The result is a build recipe for one chip, which is the correct granularity, because the answer genuinely differs between Graviton generations and there's no reason to expect one recipe to transfer.
Two practical notes from doing it.
Benchmark on the target, never on a proxy. The entire premise is that chip-specific instruction availability is what moves the number, so measuring anywhere else answers a different question.
Separate prefill from decode. They stress different things: prefill is compute-bound matrix work and responds strongly to instruction-set improvements, while decode is far more memory-bound. Reporting one blended number will hide most of what you found. My headline result is a prefill number, and I state it that way deliberately.
What this means if you deploy on Arm
The Arm inference story is usually told as a cost story, and the cost argument is real. But part of the price-performance advantage people attribute to the hardware is sitting unclaimed in build configuration, and a lot of published Arm-versus-x86 comparisons are quietly measuring a tuned build against an untuned one, in whichever direction the author was hoping.
If you serve models on Arm instances, the check is small. Build once for your specific target rather than the generic baseline, benchmark prefill and decode separately, and see what you get. It's an afternoon. If the answer is nothing, you have learned something for cheap. If the answer looks like mine, you found a multiple in your inference throughput without touching a line of model code, and it will hold for as long as you keep running on that chip.
The broader point I keep coming back to: as models commoditize, the differentiated performance work moves down the stack, into compilation, memory layout, and the specific silicon. That's old-fashioned systems engineering, and it's where a surprising amount of the remaining headroom lives.
About Nick Sawinyh
Nick Sawinyh is a product and engineering leader with over a decade of experience across DeFi, AI tooling, and government technology. He works on on-device inference and performance benchmarking with llama.cpp and GGML on aarch64, and writes at sawinyh.com.