Abstract

Efficient on-device neural network (NN) inference offers predictable latency, improved privacy and reliability, and lower operating costs for vendors than cloud-based inference. This has sparked recent development of microcontroller-scale NN accelerators, also known as neural processing units (μNPUs), designed specifically for ultra-low-power applications. We present the first comparative evaluation of a number of commercially-available μNPUs, including the first independent benchmarks for multiple platforms. To ensure fairness, we develop and open-source a model compilation pipeline supporting consistent benchmarking of quantized models across diverse microcontroller hardware. Our resulting analysis uncovers both expected performance trends as well as surprising disparities between hardware specifications and actual performance, including certain μNPUs exhibiting unexpected scaling behaviors with model complexity. This work provides a foundation for ongoing evaluation of μNPU platforms, alongside offering practical insights for both hardware and software developers in this rapidly evolving space.

What problem does this paper address?

Commercial microcontroller-scale NPUs are difficult to compare because vendor measurements use different models, quantization schemes, toolchains, and definitions of inference cost. This paper asks how these platforms compare under consistent workloads when end-to-end latency, power, memory transfers, initialization, and CPU post-processing are measured together.

Main contributions

• A side-by-side evaluation of commercially available μNPU platforms under consistent workloads and measurement conditions.
• Independent, fine-grained benchmarks for platforms that previously lacked third-party evaluation.
• An open-source compilation toolchain for deploying equivalent quantized neural networks across diverse μNPU hardware.
• Practical platform-selection and model-optimization recommendations derived from stage-by-stage measurements.

Key results

• Dedicated neural accelerators achieved up to two orders of magnitude higher energy efficiency than general-purpose MCUs in the evaluated workloads.
• The HX-WE2 averaged about 1.93× lower end-to-end latency than the MAX78000, with about 3.13× higher power consumption.
• The MAX78000 spent up to 90% of execution time on memory I/O, showing why advertised GOPS alone did not predict end-to-end performance.

Relevant research topics

• TinyML
• Neural Processing Units
• Microcontrollers
• Embedded AI
• Hardware Benchmarking
• Energy-Efficient Inference
• Model Compilation

Preferred citation

@inproceedings{millar2025benchmarking,
  author    = {Josh Millar and Yushan Huang and Sarab Sethi and Hamed Haddadi and Anil Madhavapeddy},
  title     = {Benchmarking Ultra-Low-Power {$\mu$NPUs}},
  booktitle = {Proceedings of the 31st Annual International Conference on Mobile Computing and Networking},
  year      = {2025},
  pages     = {1060--1074},
  doi       = {10.1145/3680207.3765264}
}