Edge AI Fundamentals

What edge AI is, and why teams do it

Cloud inference is simple to build: send the input to a server with big GPUs, get the answer back. Edge inference runs the same kind of model locally, on the device's own CPU, GPU or dedicated neural accelerator, or on a nearby box rather than a distant data centre. "On-device AI" is the most extreme point of edge AI: the model runs on the very device that owns the sensor and the user.

The device-edge-cloud continuum

"Edge" is not one place; it is a spectrum of locations between the sensor and the data centre. Each step toward the sensor lowers latency and data exposure but shrinks the compute and power budget.

  Sensor / MCU        Device             Near edge            Far edge / regional      Cloud
  (TinyML)            (phone, watch,     (gateway, camera     (telco MEC, on-prem      (data centre
                      car ECU, laptop)   NVR, Jetson box)     server, store server)     GPU clusters)
  ------------------------------------------------------------------------------------------------
  mW, KB-MB           1-10 W, 4-24 GB    10-60 W, 8-64 GB     100s of W, GPUs          MW, TB of HBM
  <1 ms to sensor     ~1-50 ms           ~1-10 ms LAN         ~5-30 ms                 ~50-300+ ms mobile
  keyword spotting    camera effects,    multi-camera video   city-scale video,        frontier LLMs,
  gesture, anomaly    on-device LLM,     analytics, robot     AR offload, factory      training, batch
  detection           photo search       perception           quality inspection       analytics
  ------------------------------------------------------------------------------------------------
  <-- more private, lower latency, works offline        more capable, easier to update, costs per call -->

Why run AI at the edge?

The price you pay

Hybrid is the common answer

Most real products are hybrid. A small on-device model handles the fast, private or frequent cases, and hard cases fall back to the cloud. Examples: on-device wake-word detection that then streams to a cloud assistant; an on-device small LLM that answers simple prompts and escalates long or complex requests; on-device OCR followed by cloud translation for rare languages. A good design states explicitly which requests stay local, what triggers fallback, and what happens offline. The hybrid architectures section below goes deeper.

A decision checklist: should this model run at the edge?

1. Latency budget If the end-to-end budget is below the network round trip plus server time (for example under 100 ms at the 99th percentile), the edge is almost mandatory.
2. Data sensitivity Would users or regulators object to raw data leaving the device? If yes, process locally and send only results, or nothing.
3. Connectivity Must it work offline or on poor networks? Then at least a degraded local path is needed.
4. Volume and cost Multiply requests per user per day by users by cloud cost per request; high-frequency features are expensive in the cloud.
5. Feasibility Can a model that meets the quality bar fit the device's memory, latency and energy budgets on the lowest tier you must support?
6. Update cadence Does the model need to change weekly (cloud-friendly) or is it stable for months (edge-friendly)?

The constraints: compute, memory, bandwidth, power, heat, storage

Every edge design is a negotiation between six budgets. Beginners focus on compute (TOPS); experienced engineers look first at memory capacity and memory bandwidth, then at sustained power and heat, because those are what usually break a feature in the field.

Memory bandwidth: the constraint people underestimate

Bandwidth is how many bytes per second can move between DRAM and the compute units. It is fixed by the memory technology and bus width: a phone with LPDDR5X at 8533 MT/s on a 64-bit bus delivers at most 8533e6 × 8 bytes ≈ 68 GB/s, and effective bandwidth for one workload is usually 60-80% of that, shared with the display, camera and other units. A laptop may reach 100-250 GB/s; a data-centre GPU 3-8 TB/s. Because every weight must be read at least once per inference at batch size 1, bandwidth alone sets a hard floor on latency:

Energy is dominated by data movement

At modern process nodes, fetching a value from DRAM costs roughly two to three orders of magnitude more energy than an 8-bit multiply-accumulate on it. Rough, widely quoted orders of magnitude: an INT8 add is well under a picojoule, a 32-bit SRAM read from a small local buffer is a few picojoules, and a 32-bit DRAM read is hundreds of picojoules. That is why accelerators have large on-chip SRAM, why operator fusion saves power as well as time, and why smaller data types help energy even when compute is not the bottleneck.

Fragmentation: the hidden seventh budget

A cloud team targets one GPU type. An Android team targets thousands of device models, several chipset vendors, multiple NPU generations, different driver versions and OS versions, and devices with 3 GB to 24 GB of RAM. A model that is fully accelerated on one phone may fall back to the CPU on another. This is why device tiering, capability checks and a reliable CPU fallback path are part of every production design.

Worked example: does this feature fit?

Feature: live background blur in video calls, 30 fps, mid-range phone
Budget per frame        : 33 ms total; leave ~10 ms for the model (camera, render, encode need the rest)
Model                   : segmentation net, 4 GFLOPs per frame (2 GMACs), 3 M parameters
Weights                 : 3 M x 1 byte (INT8) = 3 MB  -> fits easily in memory and cache
Compute                 : 4 GOPs x 30 fps = 120 GOPS sustained
NPU (say 10 TOPS INT8 at ~20% real utilization = 2 TOPS effective)
Time per frame          : 4e9 / 2e12 = 2 ms   -> OK
Energy per frame        : ~1.5 W while active x 2 ms = 3 mJ -> 90 mW at 30 fps (plus pre/post-processing)
CPU fallback (100 GOPS effective): 40 ms per frame -> misses 30 fps -> needs a smaller model or lower resolution
Conclusion              : ship on NPU devices; for CPU-only tiers use a half-resolution variant at 15 fps

The hardware landscape

A modern phone system-on-chip (SoC) has several compute units that can run neural networks, trading flexibility for efficiency. Beyond phones, edge AI runs on embedded GPU modules, USB and M.2 accelerators, PC NPUs and microcontrollers. Picking the right unit, and keeping the model on it without falling back, is a core edge skill.

CPU: SIMD and matrix extensions

Arm CPUs process many values per instruction using SIMD (single instruction, multiple data). NEON registers are 128 bits wide, so one instruction can operate on 16 INT8 or 8 FP16 values. The dot-product extension (SDOT/UDOT) multiplies four INT8 pairs and accumulates into an INT32 lane in one instruction; the INT8 matrix-multiply extension (SMMLA, "i8mm") computes a small 2x8 by 8x2 INT8 matrix product per instruction. SVE/SVE2 make vector length hardware-dependent, and the newest cores add SME/SME2 (Scalable Matrix Extension) with outer-product engines. Libraries such as XNNPACK, KleidiAI (Arm's low-bit matmul micro-kernels), and llama.cpp's hand-written kernels exploit these; enabling the right kernels can change CPU LLM prefill speed by 20% or more. x86 edge devices use AVX2, AVX-512 and VNNI/AMX for the same purpose.

Mobile GPU

Mobile GPUs (Adreno, Mali, Apple GPU, Immortalis, Xclipse) are wide SIMD machines designed for graphics. They excel at FP16 convolutions and element-wise work on large tensors. Their costs: kernels must be compiled at first use (shader compilation can take hundreds of milliseconds to seconds unless cached), the GPU is shared with UI rendering and the camera preview, and on many chips INT8 throughput is not much higher than FP16. They are a strong middle ground when an NPU is unavailable or the model uses operators the NPU lacks.

Inside a typical mobile NPU

Vendors design NPUs differently, but many combine three kinds of units. Qualcomm's Hexagon NPU is a well-known example: a scalar unit for control flow, a wide vector unit (HVX, Hexagon Vector eXtensions) for SIMD maths, and a tensor unit (HMX, Hexagon Matrix eXtensions) for dense matrix multiplication, together often called the HTP (Hexagon Tensor Processor). Other vendors (MediaTek APU, Samsung NPU, Google Tensor TPU, Apple Neural Engine) follow similar ideas: a large on-chip SRAM (tightly coupled memory) so data does not keep going out to DRAM, and hardware for low-precision multiply-accumulate (MAC) operations arranged as systolic arrays or dot-product engines.

                +---------------------------- SoC ----------------------------+
                |                                                             |
 App / Runtime  |   CPU cluster        GPU            NPU (e.g. Hexagon)       |
 (LiteRT, ORT,  |   big + LITTLE      shader       scalar | vector | tensor    |
  ExecuTorch)   |   NEON/SVE/SME      cores          ctrl   | HVX    | HMX      |
      |         |      |               |               \       |      /       |
      |         |      +-------+-------+----------------+  on-chip SRAM (TCM)  |
      |         |              |                                              |
      +-------->|        System cache / interconnect                          |
                |              |                                              |
                +--------------+----------------------------------------------+
                               |
                        LPDDR DRAM (shared by everything, ~50-100 GB/s)

Why an NPU is efficient

• Low-precision MAC arrays: an INT8 multiplier is several times smaller and cheaper in energy than an FP32 one, so far more fit in the same area and power.
• Data reuse in SRAM: a systolic array passes each loaded weight and activation through many MACs before discarding it, so each DRAM byte feeds many operations.
• Less control overhead: a CPU spends most of its energy fetching, decoding and scheduling instructions; an NPU runs a fixed dataflow with coarse-grained commands.
• Ahead-of-time compilation: static shapes and static quantization let the compiler plan tiling, memory and scheduling offline, which is also why NPUs dislike dynamic shapes and runtime-computed scales.

Platform-specific accelerators

TOPS versus reality

Vendors quote peak TOPS (trillions of operations per second), usually for INT8 (sometimes INT4 or with sparsity, which doubles the number) with ideal utilization, counting a multiply-accumulate as two operations. Real models rarely reach that because of:

• Memory bandwidth. If weights or activations must stream from DRAM, the compute units wait. LPDDR5/5X bandwidth on phones is roughly 50-100 GB/s, far below desktop and data-centre GPUs (1-8 TB/s).
• Unsupported operators. An op the NPU cannot run forces a round trip to the CPU, which adds copy and synchronization time.
• Small batch sizes. On device, batch size is usually 1, so there is little data reuse to hide memory latency.
• Poor mapping. Depthwise convolutions, small channel counts, softmax, normalization and reshapes do not fill a large MAC array.
• Thermal throttling. Peak clocks last seconds to minutes; sustained performance is what users feel.
• Marketing arithmetic. Check whether the quoted figure is INT4 or INT8, dense or sparse, and whether it sums CPU, GPU and NPU together.

The roofline model: compute-bound versus memory-bound

The roofline model is the single most useful mental tool in edge AI. It tells you, before writing any code, whether a layer will be limited by arithmetic or by memory traffic, and therefore which optimization will help.

The two numbers you need

Arithmetic intensity (AI) is the number of operations performed per byte moved to or from DRAM. The ridge point of a machine is its peak compute divided by its bandwidth: the intensity at which a workload stops being memory-bound.

attainable
performance |                 ______________  peak compute (TOPS)
            |               /
            |             /   compute-bound region
            |           /
            |         /  memory-bound region
            |       /    (slope = memory bandwidth)
            |_____/_______________________________
                    arithmetic intensity (ops per byte)
   LLM decode: far left (memory-bound)   big conv / prefill: right (compute-bound)

Worked example 1: a phone NPU

Machine: NPU with 40 TOPS INT8 peak, 60 GB/s effective DRAM bandwidth
Ridge point = 40e12 / 60e9 = ~667 ops per byte

Layer A: matrix-VECTOR multiply, W is 4096 x 4096 INT8 (LLM decode, batch 1)
  ops   = 2 x 4096 x 4096        = 33.6 M ops
  bytes = 4096 x 4096 x 1 byte   = 16.8 MB (weights dominate; the vector is tiny)
  AI    = 33.6 M / 16.8 M        = 2 ops/byte   (far below 667 -> memory-bound)
  attainable = 60 GB/s x 2       = 120 GOPS = 0.3% of peak
  time  = 16.8 MB / 60 GB/s      = 0.28 ms

Layer B: same weights, but N = 512 tokens at once (LLM prefill)
  ops   = 2 x 4096 x 4096 x 512  = 17.2 G ops
  bytes ~ 16.8 MB weights + 2 x (512 x 4096) activations ~ 21 MB
  AI    ~ 17.2 G / 21 M          = ~820 ops/byte (above the ridge -> compute-bound)
  time  ~ 17.2e9 / (40e12 x 0.5 utilization) = 0.86 ms for 512 tokens (1.7 us/token)

Same weights, same chip: per token, prefill is ~165x cheaper than decode.

Worked example 2: what quantization does on the roofline

Halving bytes per weight doubles arithmetic intensity. For a memory-bound layer, that moves the point up the sloped line and nearly doubles speed. For a compute-bound layer, lower precision helps only if the hardware has faster low-precision units (for example INT8 MACs that run at twice the FP16 rate).

Arithmetic intensity of common layers

Model basics for systems engineers

You do not need to train models to deploy them, but you must understand their shapes, costs and sensitive parts. A few families dominate edge work, and a handful of formulas let you estimate size, compute and memory in your head. For a full treatment of the architectures themselves, see Deep Learning and Transformers and LLMs.

Parameters, size and FLOPs

• Parameters are the learned weights. Model file size is roughly parameters × bytes per parameter. A 3B-parameter model is about 12 GB in FP32, 6 GB in FP16, 3 GB in INT8 and about 1.5-1.8 GB in INT4 (group scales add a little overhead).
• FLOPs (floating-point operations) measure compute. A dense layer with N inputs and M outputs costs about 2 × N × M FLOPs per sample (one multiply plus one add per weight).
• A convolution costs about 2 × K × K × C_in × C_out × H_out × W_out FLOPs. Depthwise-separable convolution splits this into a cheap per-channel filter plus a 1x1 convolution, often cutting FLOPs by 8-9x for 3x3 kernels.
• A transformer forward pass costs about 2 × parameters FLOPs per token, plus attention cost that grows with context length.
• MACs (multiply-accumulates) are often quoted instead of FLOPs; 1 MAC is roughly 2 FLOPs. Always check which one a paper or vendor uses.
• Activations are intermediate tensors. Peak activation memory matters on small devices, sometimes more than weight size; on MCUs it is often the binding constraint.

Worked examples

1) Convolution cost
   3x3 conv, 64 -> 128 channels, 56x56 output
   MACs  = 3 x 3 x 64 x 128 x 56 x 56 = 231 M MACs  (~462 M FLOPs)
   Depthwise-separable version:
     depthwise 3x3 x 64 x 56 x 56   =   1.8 M MACs
     pointwise 64 x 128 x 56 x 56   =  25.7 M MACs
     total                          =  27.5 M MACs  -> ~8.4x cheaper

2) Small transformer size
   d = 2048, 16 layers, vocabulary 128k
   layers     = 12 x 2048^2 x 16  = ~805 M parameters
   embeddings = 128,000 x 2048    = ~262 M parameters (often tied with the LM head)
   total      = ~1.07 B -> ~2.1 GB FP16, ~0.6 GB INT4
   Note: in a 1B model the embedding table is ~25% of all weights, which is why
   embedding quantization matters so much for small LLMs.

3) Per-token compute
   1B model: ~2 GFLOPs per token. A 500-token prompt = ~1 TFLOP of prefill work.

Training versus inference

Training adjusts weights using large datasets, gradients and a lot of memory (weights, gradients, optimizer state and saved activations, often 16+ bytes per parameter); it happens in data centres. Inference only runs the forward pass with fixed weights. Edge work is almost entirely inference, with occasional light on-device adaptation (for example small adapters or federated learning updates).

Transformer building blocks worth knowing

• Tokens and embeddings: text is split into tokens; each token maps to a vector (embedding). The vocabulary embedding table can be a large part of a small model.
• Attention: each token creates Query, Key and Value vectors. Scores are softmax(Q × K^T / sqrt(d)), then multiplied by V. During generation only K and V of past tokens are needed again, which is why the cache holds K and V but not Q.
• Multi-head, multi-query and grouped-query attention (MHA, MQA, GQA): GQA shares Key/Value heads across several Query heads, shrinking the KV cache, which is why most modern small LLMs use it.
• Positional encoding (RoPE): rotary embeddings rotate Q and K by position-dependent angles; context-extension tricks rescale those angles. RoPE is applied before K is cached.
• Feed-forward (MLP) blocks: hold most of the parameters; they are big matrix multiplies and quantize well.
• Normalization and activation functions (LayerNorm, RMSNorm, GELU, SiLU): cheap in FLOPs but numerically sensitive; often kept at higher precision.
• Mixture of experts (MoE): only a few expert MLPs run per token, so compute per token is small, but all experts must still be stored, which makes MoE memory-heavy for edge devices.

Number formats: FP32 to INT4, FP8 and block formats

Every tensor on an accelerator is stored in some number format, and the choice decides size, bandwidth, energy, which hardware units can be used and how much accuracy is lost. Floating-point formats split their bits between an exponent (range) and a mantissa (precision); integer formats have uniform steps and need an external scale.

Floating-point anatomy

Worked example: the same number in different formats

Store 0.1:
  FP32  -> 0.100000001490116...   (error ~1.5e-9)
  FP16  -> 0.0999755859375        (error ~2.4e-5)
  BF16  -> 0.10009765625          (error ~9.8e-5)  less precise than FP16...
Store 70,000:
  FP16  -> inf  (overflow, max is 65,504)
  BF16  -> 70,144 (fine, just coarse) ...but BF16 never overflows where FP32 would not.
Store 1e-8:
  FP16  -> 0 (underflow; smallest subnormal is ~6e-8)
  BF16  -> ~1e-8 (fine)

Lesson: FP16 has better precision, BF16 has better range.
Activations with large values (attention logits, some LLM hidden states) can overflow FP16.

import numpy as np
print(np.float16(70000))         # inf  -> overflow
print(np.float16(300) * np.float16(300))   # inf  (90,000 > 65,504)
print(np.float16(0.1))           # 0.1 displayed, stored as 0.09998
print(np.float32(np.float16(0.1)))         # 0.099975586

Block (microscaling) formats

A small group of values shares one scale, so each element can use very few bits while the group still covers a wide range. This is the same idea as per-group integer quantization, standardized for hardware.

Accumulators and mixed formats

Products are accumulated in a wider format than the inputs: INT8 × INT8 products accumulate in INT32 (a sum of thousands of 16-bit products would overflow INT16), FP16 products usually accumulate in FP32 or FP16 depending on hardware, and FP8 products accumulate in FP32 or FP16. Notation such as W4A16 means 4-bit weights and 16-bit activations; W8A8 means both 8-bit; 8da4w means 8-bit dynamically quantized activations with 4-bit weights; KV8 or KV4 describes the KV-cache precision.

Quantization in depth

Quantization stores and computes numbers with fewer bits. It is the single most important optimization in edge AI: it shrinks the model, cuts memory bandwidth (the real bottleneck for many layers), saves energy, and unlocks integer engines on NPUs and DSPs. This section builds it up from the maths; the next section covers the special techniques for LLMs.

The affine (asymmetric) mapping

A real value x is mapped to an integer q using a scale (the step size between integer levels) and a zero-point (the integer that represents real zero exactly):

q = clamp(round(x / scale) + zero_point, q_min, q_max)
x_approx = scale * (q - zero_point)

// Example, asymmetric INT8 (q in 0..255) for a tensor ranging -1.0 .. 3.0
scale      = (3.0 - (-1.0)) / 255  = 0.01569
zero_point = round(0 - (-1.0) / scale) = 64
x = 1.0  ->  q = round(1.0 / 0.01569) + 64 = 128  ->  x_approx = 0.01569 * (128 - 64) = 1.004

Symmetric mapping and a worked example

Symmetric quantization forces the zero-point to 0 and uses a range centred on zero: s = max|x| / 127 for INT8 (many toolchains use the restricted range -127..127 so the grid is symmetric). It removes the zero-point arithmetic from the inner loop, which is why it is the default for weights.

Weights: [-0.80, 0.30, 1.20, -0.05]      symmetric INT8, range -127..127
scale = max|w| / 127 = 1.20 / 127 = 0.009449

w      w / scale   q (rounded)   dequantized    error
-0.80   -84.67       -85          -0.8031       0.0031
 0.30    31.75        32           0.3024       0.0024
 1.20   127.00       127           1.2000       0.0000
-0.05    -5.29        -5          -0.0472       0.0028
Max possible rounding error = scale / 2 = 0.0047

Now add one outlier weight, 12.0, to the same tensor:
scale = 12.0 / 127 = 0.0945  ->  -0.05 becomes q = -1 -> -0.0945 (error 0.045, ~90%)
                                  0.30 becomes q =  3 ->  0.2835
One outlier made every small weight 10x less precise. Per-channel or per-group
scales isolate the outlier to its own channel or group.

How much error does quantization add?

With round-to-nearest, the error for values inside the range is roughly uniform between -s/2 and +s/2, so its variance is s²/12. Values outside the range are clipped, which can create much larger errors. Every extra bit halves the step and adds about 6 dB of signal-to-quantization-noise ratio (SQNR).

Design choices

Per-tensor, per-channel, per-group

Weight matrix W  (rows = output channels, columns = input features)

per-tensor       per-channel (per row)     per-group (row split into groups of g)
+----------+     +----------+ s0           +----+----+----+ s00 s01 s02
|          |     |----------| s1           |----+----+----| s10 s11 s12
|  one s   |     |----------| s2           |----+----+----| s20 s21 s22
|          |     |----------| s3           |----+----+----| s30 s31 s32
+----------+     +----------+              +----+----+----+
1 scale          one scale per row         one scale per g weights (g = 32..128)
cheapest         standard for INT8         standard for INT4 LLM weights

Per-channel scales for weights cost nothing at runtime: because each output channel's dot product uses only its own weights, the channel scale can be applied once to the accumulator. Per-channel scales for activations along the reduction dimension are a different story: they cannot be factored out of the dot product, which is why activation outliers are hard and why SmoothQuant moves them into the weights.

Calibration: choosing the range

For static quantization, a calibration set (typically 100-1000 representative samples) is run through the model and statistics are collected per tensor. The range-setting method matters as much as the data.

Static versus dynamic quantization

Integer matrix multiplication with scales

For y = W x with W ≈ sw(qw - zw) and x ≈ sx(qx - zx), the expensive inner loop is pure integer arithmetic, and all the floating-point scale work happens once per output value.

Weight-only schemes (W4A16) work differently: the kernel loads 4-bit weights, dequantizes them to FP16 in registers using the group scale, and does FP16 maths. The win is bandwidth (fewer bytes loaded), not integer compute, which is exactly what memory-bound decode needs.

PTQ versus QAT

QAT needs gradients through the rounding function, whose true derivative is zero almost everywhere. The straight-through estimator (STE) pretends rounding is the identity in the backward pass (with zero gradient outside the clipping range). Methods such as LSQ (learned step size quantization) also learn the scales themselves.

Code: quantization from scratch

import numpy as np

def quantize(x, bits=8, symmetric=True, axis=None, group=None):
    """Return integer tensor q, scale s, zero-point z (fake-quant helpers)."""
    if group is not None:                        # per-group along the last axis
        x = x.reshape(*x.shape[:-1], -1, group)
        axis = -1
    reduce = dict(axis=axis, keepdims=True) if axis is not None else {}
    if symmetric:
        qmax = 2 ** (bits - 1) - 1               # 127 for INT8, 7 for INT4
        s = np.abs(x).max(**reduce) / qmax
        z = 0
        q = np.clip(np.round(x / s), -qmax, qmax)
    else:
        qmin, qmax = 0, 2 ** bits - 1            # 0..255 for UINT8
        lo, hi = x.min(**reduce), x.max(**reduce)
        lo, hi = np.minimum(lo, 0), np.maximum(hi, 0)   # zero must be representable
        s = (hi - lo) / (qmax - qmin)
        z = np.round(qmin - lo / s)
        q = np.clip(np.round(x / s) + z, qmin, qmax)
    return q, s, z

def sqnr_db(x, x_hat):
    return 10 * np.log10((x ** 2).sum() / ((x - x_hat) ** 2).sum())

np.random.seed(0)
W = np.random.randn(256, 1024).astype(np.float32)
W[3, 7] = 40.0                                   # one outlier
for bits in (4, 8):
    for name, kw in [("per-tensor", {}), ("per-channel", {"axis": 1}),
                     ("per-group-32", {"group": 32})]:
        q, s, z = quantize(W, bits=bits, **kw)
        W_hat = (s * (q - z)).reshape(W.shape)
        print(f"INT{bits} {name:13s} SQNR = {sqnr_db(W, W_hat):5.1f} dB")

# Output:
# INT4 per-tensor    SQNR =   0.1 dB   (scale 40/7 = 5.7: almost every weight rounds to 0)
# INT4 per-channel   SQNR =  16.2 dB
# INT4 per-group-32  SQNR =  20.2 dB
# INT8 per-tensor    SQNR =  20.8 dB
# INT8 per-channel   SQNR =  40.3 dB   (~6 dB per extra bit: 16.2 + 4 x 6 = ~40)
# INT8 per-group-32  SQNR =  45.3 dB

Code: PyTorch 2 export-based PTQ (sketch)

import torch
from torch.export import export
from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
# Module paths move between releases; in recent versions the quantizer lives in
# executorch.backends.xnnpack.quantizer.xnnpack_quantizer
from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import (
    XNNPACKQuantizer, get_symmetric_quantization_config)

model = MyModel().eval()
example = (torch.randn(1, 3, 224, 224),)

graph = export(model, example).module()          # capture the graph
quantizer = XNNPACKQuantizer().set_global(
    get_symmetric_quantization_config(is_per_channel=True))
prepared = prepare_pt2e(graph, quantizer)        # inserts observers

with torch.no_grad():
    for batch in calibration_loader:             # 100-1000 production-like samples
        prepared(batch)

quantized = convert_pt2e(prepared)               # observers -> quantize/dequantize ops
# then lower `quantized` to a backend (see the runtimes section)

Why quantization hurts accuracy, and how to fix it

• Outliers: a few very large activation values stretch the range, so normal values lose resolution. Fixes: per-channel scales, clipping with percentile or MSE calibration, SmoothQuant (moves difficulty from activations to weights), rotations (QuaRot/SpinQuant), keeping outlier layers in higher precision.
• Sensitive layers: first and last layers, attention softmax, layer norms and the embedding/LM head are often more sensitive. Use mixed precision: keep them at INT16/FP16.
• Bad calibration data: if calibration images are all daytime, night images will clip. Calibrate on data that matches production.
• Error accumulation: errors add up through deep networks. Measure per-layer error (for example SQNR, cosine similarity, mean and max absolute error) between an FP32 reference run and the quantized run to find the layer that breaks.
• Uneven channel ranges: cross-layer equalization rescales consecutive layers (using the fact that ReLU is scale-equivariant) so per-channel ranges become similar without changing the FP32 output; bias correction removes the systematic mean shift that quantization error introduces.
• Rounding choices: AdaRound learns, per weight, whether to round up or down to minimize each layer's output error, using a small calibration set.

Layer-wise error analysis: the debugging workflow

1. Build a reference Run the FP32 model on the host and dump the output of every layer for a fixed set of inputs.
2. Capture the quantized run Run the quantized model (on host simulation and, ideally, on the device) and dump the same tensors, dequantized.
3. Diff per layer Compute SQNR, cosine similarity, mean and max absolute error for each tensor.
4. Rank and localize Plot error against depth; look for the layer where error jumps rather than grows smoothly. Separate "this layer is sensitive" (large local error) from "error accumulated" (small local error on a damaged input).
5. Fix with evidence Keep the worst layers at higher precision, change granularity, or apply an outlier technique; re-measure accuracy and latency, since each higher-precision island may create a new partition on the NPU.

Quantizing LLMs: GPTQ, AWQ, SmoothQuant, rotations and GGUF

Plain round-to-nearest (RTN) INT8 works well for most CNNs. LLMs are harder at low bit-widths for two reasons: we want to go to 4 bits or below to fit memory and bandwidth budgets, and transformer activations contain a few channels with values tens to hundreds of times larger than the rest. A family of post-training methods addresses this with only a small calibration set and no full retraining.

Why LLM activations are hard, weights are easy

• Weights are roughly bell-shaped per channel with few outliers; with small groups (32-128) even 4 bits keep good accuracy.
• Activations have a handful of fixed channels with huge magnitudes that appear for almost every token. Per-tensor INT8 activations then waste nearly all of their range on those channels. Because the outliers lie along the reduction dimension, per-channel activation scales cannot be factored out of an integer matmul.
• Decode is memory-bound, so weight-only quantization (W4A16, or W4 with 8-bit dynamic activations) captures most of the speed benefit without touching activations. Full W8A8 or W4A8 matters mostly for compute-bound prefill and for integer-only NPUs.
• The embedding table and LM head are large in small models and unusually sensitive; they are often kept at 6-8 bits even when the body is at 4 bits.

GPTQ: error-compensating rounding

GPTQ quantizes a layer's weight matrix one column (input feature) at a time. After rounding a column, it updates the not-yet-quantized columns to cancel the error that rounding introduced in the layer output, using second-order information from calibration activations.

Practical notes: ~128 calibration sequences are typical; it runs layer by layer so memory stays manageable; a 7B model takes minutes to an hour on a single data-centre GPU. It works best combined with group-wise scales (for example INT4, group size 128).

AWQ: protect the salient channels

Activation-aware weight quantization observes that a small fraction (around 1%) of weight input channels matter most, namely those multiplied by large activations. Instead of keeping them in FP16 (which would complicate kernels), AWQ multiplies those weight channels by a factor s > 1 before quantization and divides the matching activations by s (folded into the previous layer or normalization). The product is unchanged, but the important weights now use more of the integer grid, so their relative rounding error shrinks.

SmoothQuant: move the difficulty into the weights

SmoothQuant targets W8A8 by dividing each activation channel by a smoothing factor and multiplying the corresponding weight row by the same factor. This is mathematically exact in full precision and makes both sides easy to quantize.

Rotation methods: QuaRot and SpinQuant

If you multiply activations by an orthogonal matrix R and the next weights by R^T, the output is unchanged (R R^T = I), but the outlier energy gets spread across all channels, so no single channel dominates the range. QuaRot uses random Hadamard rotations (cheap to apply with a fast transform); SpinQuant learns the rotations on calibration data. Rotations make 4-bit weights, 4- or 8-bit activations and 4-bit KV caches workable, and they were used, together with QAT plus LoRA, for official quantized mobile releases of small open LLMs, which reached roughly 2.5x faster decode and 50-60% smaller models than BF16 on phone CPUs.

GGUF and k-quants (llama.cpp)

GGUF is llama.cpp's single-file format holding tensors, tokenizer, chat template and metadata. Its quantization types are block formats, named by approximate bits and scheme.

K-quants organize 256 weights into a super-block of 8 sub-blocks of 32, with small quantized per-sub-block scales (and minimums) plus one FP16 super-block scale: two-level scaling keeps metadata overhead low. An importance matrix (imatrix), computed from calibration text, weights the rounding error by how much each weight matters, similar in spirit to AWQ.

KV-cache quantization

At long context the KV cache can outgrow the weights, so quantizing it pays off directly in memory and decode bandwidth. INT8 KV is close to lossless; 4-bit needs care; research shows usable results near 2-3 bits with the right grouping. Keys and values behave differently: keys have outlier channels (so they quantize better per channel), while values are better quantized per token. Recent tokens are often kept in higher precision in a small residual buffer.

Precision by target: what each runtime typically expects

INT8 versus INT4: what interviewers expect

Treat "INT4" as a family of recipes, not one format. Group size, which tensors stayed at 6-8 bits, RTN versus GPTQ/AWQ, and whether activations are dynamic 8-bit or static 16-bit can move quality more than the nominal bit-width. Always state the full recipe and re-measure the product task.

Evaluating a quantized LLM

• Perplexity on held-out text is cheap and sensitive, but small perplexity changes do not always predict task quality.
• Task benchmarks relevant to the feature (summarization quality, extraction accuracy, instruction following) catch behavioural regressions.
• Agreement with the reference: KL divergence of next-token distributions or top-1 agreement versus the FP16 model on the same prompts.
• Long-context and edge cases: quantization damage often shows first on long inputs, rare languages, maths and formatting.

# llama.cpp: build an importance matrix, quantize with it, and compare perplexity
./llama-imatrix   -m model-f16.gguf -f calibration.txt -o imatrix.dat
./llama-quantize  --imatrix imatrix.dat model-f16.gguf model-IQ4_XS.gguf IQ4_XS
./llama-quantize  model-f16.gguf model-Q4_K_M.gguf Q4_K_M
./llama-perplexity -m model-Q4_K_M.gguf -f wiki.test.raw     # compare against the F16 run

Beyond quantization: pruning, sparsity, distillation, low-rank and NAS

A model trained in FP32 on a data-centre GPU is almost never shipped as-is. Quantization reduces bits per weight; the techniques here reduce the number of weights or operations, or produce a better small model in the first place. They stack: a distilled, pruned, quantized model is common.

Pruning

Pruning removes weights that contribute little.

• Unstructured pruning zeroes individual weights (usually those with the smallest magnitude). It can remove 50-90% of weights with small accuracy loss, but most mobile hardware cannot skip scattered zeros, so it mainly helps compression (the zeros compress well on disk), not speed.
• Structured pruning removes whole channels, filters, attention heads or layers. The model genuinely becomes smaller and faster on any hardware, but accuracy drops more and usually needs fine-tuning. For LLMs, width pruning (hidden size, MLP size, heads) and depth pruning (whole layers) followed by distillation is a proven recipe for producing small models from larger ones.
• Semi-structured (N:M) sparsity, for example 2:4 (two zeros in every four weights), is supported by some accelerators and gives real speed-ups (up to about 2x for the matrix maths) because the pattern is regular enough for hardware to exploit.
• Iterative pruning (prune a little, fine-tune, repeat) keeps more accuracy than one-shot pruning. One-shot LLM methods such as SparseGPT and Wanda use calibration data to choose which weights to drop without retraining.
• Activation sparsity: after ReLU-like activations many values are zero; some runtimes and research systems skip the corresponding weight rows, which can reduce bytes read during decode.

import torch.nn.utils.prune as prune
# Unstructured: zero the 50% smallest-magnitude weights of a layer
prune.l1_unstructured(model.fc1, name="weight", amount=0.5)
# Structured: remove 25% of output channels (rows) by L2 norm
prune.ln_structured(model.conv2, name="weight", amount=0.25, n=2, dim=0)
prune.remove(model.fc1, "weight")   # make the pruning permanent
# Note: masks only zero values; to get real speed from structured pruning you must
# rebuild the layer with fewer channels (and adjust the next layer's inputs).

Knowledge distillation

A large "teacher" model trains a small "student" model to match its outputs (soft probabilities), not just the hard labels. Soft outputs carry extra information ("this looks 70% like a cat, 25% like a fox"), so the student learns more from the same data. Most good small on-device LLMs are distilled from larger models, often after pruning.

Low-rank factorization

A weight matrix W of size m × n is approximated by the product of two thin matrices U (m × r) and V (r × n), usually from a truncated SVD followed by fine-tuning.

Adapters: LoRA and QLoRA

LoRA (low-rank adaptation) fine-tunes a model by learning small low-rank update matrices while freezing the base weights: W' = W + B A, with rank r of 4-64. On device, one base model can serve several features by swapping small LoRA adapters (a few MB to tens of MB each) instead of shipping several full models; OS-level AI services use exactly this to specialize one shared foundation model per feature. QLoRA fine-tunes LoRA adapters on top of a 4-bit (NF4) quantized base, cutting fine-tuning memory. Adapters can be merged into the weights for zero overhead, or kept separate for hot-swapping at a small compute cost. See Fine-Tuning for training details.

Neural architecture search and efficient architecture design

The biggest wins often come before any compression: pick a mobile-first architecture (MobileNet-style, efficient ViTs, small LLMs with GQA), lower input resolution, shorter context, fewer layers, or split into a small always-on model that triggers a larger one only when needed (cascade). Neural architecture search (NAS) automates this: it searches over block types, widths, kernel sizes and depths, scoring candidates by accuracy and by measured or predicted latency on the target hardware ("hardware-aware NAS"). MobileNetV3 and EfficientNet came from NAS; "once-for-all" approaches train one super-network and extract sub-networks sized for each device tier without retraining.

• Scaling knobs: width multiplier (channels), depth, input resolution; compute roughly scales with width squared and resolution squared.
• Early exit: add intermediate classifiers so easy inputs stop early.
• Weight sharing and clustering: store a small codebook of distinct values and an index per weight (palettization), popular on Apple devices.
• Matryoshka / nested models: one set of weights contains smaller sub-models that can be selected at runtime to fit the device or the moment's power budget.

Graph and kernel optimizations

Between the model and the silicon sits a compiler (inside the converter or the runtime) that rewrites the computation graph and chooses kernels. These optimizations change no weights (or change them in exactly equivalent ways), so they are free accuracy-wise, yet they often deliver 10-50% speed-ups and large memory savings.

The main rewrites

• Operator fusion: merge sequences like Conv + BatchNorm + ReLU into one kernel, or MatMul + bias + activation, or a whole attention block into a fused attention kernel. This avoids writing intermediate tensors to memory and launching several kernels.
• BatchNorm folding: at inference, BatchNorm is a fixed per-channel affine transform, so it can be folded into the preceding convolution's weights and bias. Essentially free accuracy-wise.
• Constant folding: precompute anything that does not depend on input (shape arithmetic, fixed masks, positional tables).
• Dead-code elimination and common-subexpression elimination: remove unused outputs and compute repeated expressions once.
• Layout transforms: choose the memory layout (NHWC vs NCHW, or vendor-specific tiled layouts) that the accelerator prefers, and avoid repeated transposes. Mobile CPUs and many NPUs prefer channels-last (NHWC).
• Static shapes: fixed input shapes let compilers plan memory and pick kernels ahead of time; dynamic shapes often force CPU fallback. For variable inputs, compile a few fixed buckets and pad.
• Memory planning: reuse activation buffers whose lifetimes do not overlap to cut peak RAM (liveness analysis plus a packing algorithm). On MCUs this decides whether a model fits at all.
• Tiling: split big tensors into tiles that fit in on-chip SRAM so each byte fetched from DRAM is reused as much as possible.
• Kernel selection and weight prepacking: pick algorithm variants (direct, im2col+GEMM, Winograd for 3x3 convolutions) per shape, and reorder weights once at load time into the layout the micro-kernel wants.
• Model splitting: NPUs have per-graph memory and size limits; large LLMs are split into several subgraphs (for example several compiled context binaries executed in sequence) that share buffers.

Worked example: folding BatchNorm into a convolution

Fold before quantizing: otherwise the conv weights are quantized with one range and then rescaled by BN, wasting precision.

Worked example: what fusion saves

Activation tensor: 1 x 112 x 112 x 64 in FP16 = 1.6 MB
Unfused Conv -> BN -> ReLU:
  conv writes 1.6 MB, BN reads 1.6 + writes 1.6, ReLU reads 1.6 + writes 1.6
  = 8.0 MB of DRAM traffic (+3 kernel launches)
Fused ConvBNReLU:
  conv computes, applies BN and ReLU in registers, writes 1.6 MB once
  = 1.6 MB of DRAM traffic (1 launch)
At 50 GB/s: 0.16 ms vs 0.03 ms for this block's memory traffic, and ~5x less DRAM energy.

Compilers and graph tools

Runtimes and SDKs

A runtime loads a converted model and executes it on the device, deciding which operators run on which hardware. Most runtimes use a plug-in called a delegate, execution provider or backend to hand parts of the graph to a GPU or NPU. Parts the accelerator cannot run fall back to the CPU.

What happened to NNAPI

The Android Neural Networks API (NNAPI) was introduced in Android 8.1 as a vendor-neutral interface: apps (or runtimes) described a graph, and each vendor's NNAPI driver ran it on their accelerator. In practice, driver quality and operator coverage varied a lot across devices, updates were tied to OS releases, and performance was unpredictable: a lowest-common-denominator API implemented inconsistently. Google deprecated NNAPI starting with Android 15. Existing apps keep working, but new work should use LiteRT with its GPU delegate or vendor-specific NPU delegates/accelerators, or other runtimes (ONNX Runtime QNN execution provider, ExecuTorch vendor backends) that talk to vendor SDKs directly.

How a delegate partitions a graph

Original graph:   [Conv]-[Conv]-[CustomOp]-[Conv]-[Softmax]-[TopK]

Delegate check:    NPU    NPU     CPU only   NPU    NPU       CPU

Partitioned:      { NPU subgraph 1 } -> CPU -> { NPU subgraph 2 } -> CPU
                                  ^ copy/sync      ^ copy/sync
Each boundary costs memory copies, layout conversion and synchronization.
Goal: as few partitions as possible, ideally one fully delegated graph.

Code: choosing execution providers and engines

# ONNX Runtime with the Qualcomm NPU (QNN execution provider) and CPU fallback
import onnxruntime as ort
sess = ort.InferenceSession(
    "model.qdq.onnx",                                   # QDQ-quantized ONNX model
    providers=["QNNExecutionProvider", "CPUExecutionProvider"],
    provider_options=[{"backend_path": "QnnHtp.dll"},   # libQnnHtp.so on Android/Linux
                      {}])
print(sess.get_providers())                             # confirm what actually loaded
out = sess.run(None, {"input": x})

# Jetson: build a TensorRT engine from ONNX, INT8 + FP16, on the DLA with GPU fallback
trtexec --onnx=detector.onnx --int8 --fp16 \
        --useDLACore=0 --allowGPUFallback \
        --saveEngine=detector.plan
# Engines are specific to the GPU/DLA generation and TensorRT version: build on (or for) the target.

Choosing a runtime

1. Start from the source framework PyTorch models fit ExecuTorch or ONNX Runtime naturally; TensorFlow/Keras/JAX models fit LiteRT. PyTorch can also reach LiteRT via AI Edge Torch.
2. Check operator coverage on your target accelerator Convert, then inspect the partition report: how many ops are delegated?
3. Check device coverage One chipset family (use its vendor SDK for peak performance) or thousands of Android models (use a cross-platform runtime with a reliable CPU/GPU fallback).
4. Check binary size and startup time Runtime library size (from tens of KB to several MB) and model initialization/compilation time matter for app install size and cold start.
5. For LLMs look at tokenizer support, KV-cache handling, quantization formats and streaming APIs: llama.cpp, ExecuTorch, MediaPipe/LiteRT LLM APIs, MLC-LLM or vendor LLM stacks.
6. For embedded Linux boxes TensorRT on Jetson, OpenVINO on Intel, ONNX Runtime as a portable layer; for MCUs, LiteRT for Microcontrollers or a vendor compiler.

The deployment pipeline at a glance

Going from a research checkpoint to a shipped feature follows a repeatable loop. Expect to go around it many times. This section gives the concepts and representative commands; the step-by-step hands-on pipeline, benchmark harness, NPU deployment walkthrough and project ideas live on the companion page Edge AI Deployment.

Pick model (e.g. from Hugging Face / model zoo)
   -> Export graph      (torch.export, ONNX export, TF SavedModel)
   -> Optimize          (fold BN, fuse ops, fix static shapes)
   -> Quantize          (PTQ with calibration set, or QAT; INT8 / INT4 / mixed)
   -> Convert           (LiteRT .tflite | ONNX/.ort | ExecuTorch .pte | QNN context binary | TensorRT engine)
   -> Validate on host  (compare outputs vs FP32 reference; accuracy on eval set)
   -> Deploy to device  (bundle in APK/AAB, download on demand, or system model)
   -> Run on NPU/GPU/CPU via delegate / backend
   -> PROFILE           latency p50/p95, init time, peak memory, tokens/s,
                        power (mW / mJ per inference), thermal over 10+ minutes
   -> Iterate           (fix fallback ops, change precision, change model)

Export and convert: two representative paths

Post-training full-integer quantization to LiteRT from a TensorFlow SavedModel:

import tensorflow as tf

converter = tf.lite.TFLiteConverter.from_saved_model("saved_model_dir")
converter.optimizations = [tf.lite.Optimize.DEFAULT]

def representative_dataset():
    for sample in calibration_samples[:300]:      # real, production-like inputs
        yield [sample[None, ...].astype("float32")]

converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8          # integer-only I/O for NPUs
converter.inference_output_type = tf.int8

with open("model_int8.tflite", "wb") as f:
    f.write(converter.convert())

Exporting a PyTorch model to ExecuTorch with the XNNPACK CPU backend:

import torch
from torch.export import export
from executorch.exir import to_edge_transform_and_lower
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner

model = MyModel().eval()
example_inputs = (torch.randn(1, 3, 224, 224),)

exported = export(model, example_inputs)           # captures a static graph
program = to_edge_transform_and_lower(
    exported, partitioner=[XnnpackPartitioner()]   # swap for a vendor NPU partitioner
).to_executorch()

with open("model.pte", "wb") as f:
    f.write(program.buffer)

Run it in the app with a delegate

// Kotlin, LiteRT interpreter API with GPU delegate and CPU fallback
val options = Interpreter.Options()
val compat = CompatibilityList()
if (compat.isDelegateSupportedOnThisDevice) {
    options.addDelegate(GpuDelegate(compat.bestOptionsForThisDevice))
} else {
    options.setNumThreads(4)          // XNNPACK CPU path
}
val interpreter = Interpreter(loadModelBuffer(context, "model.tflite"), options)

// Create once, reuse for every frame: initialization is expensive
interpreter.run(inputBuffer, outputBuffer)

Benchmark on the device

# LiteRT benchmark tool (prebuilt binary pushed to the device)
adb push benchmark_model /data/local/tmp/
adb push model_int8.tflite /data/local/tmp/
adb shell chmod +x /data/local/tmp/benchmark_model
adb shell /data/local/tmp/benchmark_model \
    --graph=/data/local/tmp/model_int8.tflite \
    --num_threads=4 --warmup_runs=10 --num_runs=200 \
    --use_gpu=true --enable_op_profiling=true

# System-level view while the model runs
adb shell dumpsys thermalservice          # thermal status and throttling
adb shell dumpsys meminfo <package>       # app memory (PSS, native heap)
adb shell "for z in /sys/class/thermal/thermal_zone*; do echo \$(cat \$z/type) \$(cat \$z/temp); done"
adb shell "cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq"
# Record a Perfetto trace with CPU frequency, scheduling and power rails
# (on devices that expose On-Device Power Monitor rails)

Model delivery options

Production concerns

• Device tiering: ship different model variants (or disable the feature) based on chipset, RAM and accelerator support.
• Warm-up and caching: GPU delegates compile shaders and NPU runtimes may compile graphs on first load; cache compiled artifacts where the runtime supports it, or ship precompiled binaries per SoC.
• Threading: never run inference on the UI thread; reuse interpreter instances; watch for contention with rendering.
• Pre/post-processing: image resize, color conversion, tokenization and non-max suppression can take longer than the model; move them to GPU or optimized native code.
• Monitoring: log latency, fallback rate and crashes per device model; keep a kill switch or remote config to disable a model variant.
• Security: models in APKs can be extracted; if the model is valuable, consider encryption at rest, integrity checks, or server-side parts.
• Validation on real silicon: emulators and host simulators are fine for functional checks but useless for performance, and accelerator numerics can differ from host simulation.

LLMs on device

Running a language model locally is the hottest and hardest edge workload. It stresses memory capacity, memory bandwidth and thermals all at once, and it has two phases with opposite performance characteristics.

The two phases of generation

Back-of-envelope: decode speed

Example: 3B-parameter model, INT4 weights (~1.7 GB including scales)
Phone LPDDR5X peak ~ 60-75 GB/s, effective maybe ~ 40-50 GB/s
Short context (KV still small): ceiling ~ 45 / 1.7  ~= 26 tokens/s
Same model in FP16 (~6 GB weights): ceiling ~ 7 tokens/s
=> quantization speeds up decode almost in proportion to the bytes saved.

Do not omit the KV term. This 3B GQA model is ~112 KB/token in FP16 KV
(~56 KB INT8, ~28 KB INT4). At 4k context, INT8 KV is ~224 MB:
  45 / (1.7 + 0.22) ~= 23 tok/s. At 16k, INT8 KV is ~0.90 GB:
  45 / (1.7 + 0.90) ~= 17 tok/s. Long chats get slower even with fixed weights.

Sanity check against published phone numbers: small (1B-class) INT4 models reach
roughly 40-50 decode tokens/s on recent flagship CPUs, and 3B-class models ~20 tokens/s
at short context, consistent with this bandwidth model.

Back-of-envelope: time to first token

3B model, 1,000-token prompt: 2 x 3e9 x 1000 = 6 TFLOP (6e12 ops)
  NPU at ~10 effective TOPS      -> ~0.6 s TTFT
  CPU at ~0.5 effective TFLOPS   -> ~12 s TTFT   (this is why prefill belongs on the NPU)
1B model, same prompt: 2 TFLOP  -> ~0.2 s on the NPU, ~4 s on the CPU

The KV cache

During attention, each new token needs the Keys and Values of all previous tokens. Recomputing them every step would be wasteful, so they are cached. The cache grows linearly with context length (and with batch size, which is usually 1 on device):

Example (a ~3B model with GQA): 28 layers, 8 KV heads, head_dim 128, 4096 tokens, FP16
  = 2 x 28 x 8 x 128 x 4096 x 2 bytes  ~= 470 MB   (~112 KB per token)
With full multi-head attention (24 KV heads instead of 8) it would be ~1.4 GB.
INT8 KV cache halves it; a 2048-token limit halves it again.

A ~1B model with GQA: 16 layers, 8 KV heads, head_dim 64 -> 32 KB per token
  4K context = 128 MB; 32K context = 1 GB (more than its ~0.7 GB of INT4 weights).
An older 7B model with full MHA: 32 layers, 32 heads, head_dim 128 -> 512 KB per token
  4K context = 2 GB; it passes its ~3.5 GB of INT4 weights at ~7K tokens.

Crossover point (cache = weights) = weight bytes / KV bytes per token.
At long context the KV cache, not the weights, becomes the memory problem.

• GQA/MQA reduce KV heads, directly shrinking the cache.
• KV-cache quantization (INT8, sometimes INT4 or lower) saves memory with small quality impact.
• Sliding-window attention caps how many past tokens each layer keeps; some models alternate local (windowed) and global layers.
• Attention sinks and eviction: keeping the first few tokens plus a recent window (and dropping the middle) keeps generation stable in streaming settings; smarter policies evict tokens that receive little attention.
• Paged / block-allocated caches allocate the cache in fixed blocks (typically 16-64 tokens) mapped by a block table. Contiguous reservation wastes (T_max − T_actual) × KV per token; a paged cache wastes at most one partially filled block per sequence and avoids reallocating a giant buffer as the conversation grows.
• Context limits: on device, 2K-8K tokens is typical; long documents need summarization, chunking or retrieval.
• Prompt/prefix caching: reuse the KV of a byte-identical prefix (system prompt, tools, a shared document). Prefill savings are about P / (P + U) when prefix P is cached and only suffix U is new. Put static tokens first; a timestamp or user id at the top of the prompt misses the cache every request.

Speculative decoding and friends

Decode leaves compute idle because it is memory-bound. Speculative decoding spends that spare compute: a tiny draft model proposes k tokens; the large target model checks all of them in one parallel forward pass (which costs about the same as one decode step, since the weights are read once) and accepts the longest prefix that matches what it would have produced. With the right acceptance rule, the output distribution is exactly the target model's.

• Self-speculative variants avoid a second model: extra prediction heads (Medusa-style), a light draft head reusing the target's features (EAGLE-style), or skipping layers to draft.
• Prompt-lookup / n-gram drafting copies candidate continuations from the prompt itself, which works very well for summarization, rewriting and code editing where output repeats input.
• On device, the draft model's memory and the verification cost of rejected tokens must fit the budget; gains are largest when the task is predictable.

Other speed-up techniques

• Operator fusion for attention (fused attention kernels, and runtime options that fuse scaled-dot-product attention with KV-cache updates) reduces memory traffic.
• Heterogeneous execution: prefill on the NPU, decode on whichever unit gives the best tokens per watt; tokenization and sampling on CPU.
• Adapter swapping (LoRA): one shared base model with small task-specific adapters instead of several full models.
• Model cascades: answer simple requests with a tiny model; escalate to a larger local or cloud model when needed.
• Constrained decoding: grammars or JSON schemas restrict sampling to valid outputs, so small models produce reliable structured results without retries.

Small model families (landscape at the time of writing)

Memory reality check

Android also shares RAM between apps; a large resident model can push other apps out of memory. Memory-mapping weights (mmap) lets the OS page them, but paging from storage during decode destroys performance. Watch peak resident memory (RSS): it is the number that decides whether the low-memory killer terminates your process.

Code: running a quantized LLM

# llama.cpp: convert, quantize, benchmark, run
python convert_hf_to_gguf.py ./my-hf-model --outfile model-f16.gguf --outtype f16
./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M
./llama-bench -m model-Q4_K_M.gguf -p 512 -n 128 -t 4     # prefill and decode tokens/s
./llama-cli   -m model-Q4_K_M.gguf -p "Explain KV cache in one line" -n 64

# ExecuTorch LLM export (shape of the command; flags change between releases)
python -m extension.llm.export.export_llm \
    base.model_class="llama3_2" base.checkpoint="$CKPT" base.params="$PARAMS" \
    model.use_kv_cache=True model.use_sdpa_with_kv_cache=True \
    quantization.qmode="torchao:8da4w" quantization.group_size=128 \
    quantization.embedding_quantize="torchao:4,32" \
    export.output_name="llm_int4.pte"
# 8da4w = 8-bit dynamic activations, 4-bit weights. Leaving the KV-cache options off
# produces a model that recomputes attention for the whole context every token.

Quality and safety on device

• Small models hallucinate more; constrain tasks (summarize this text, extract fields) rather than open-ended knowledge questions, or ground them with on-device retrieval.
• Evaluate on task-specific sets plus perplexity; compare quantized against unquantized outputs.
• Safety filters and prompt-injection defences still apply, even when everything is local.
• Licences of open-weight models vary; check them before shipping.

Vision, audio, sensor models and TinyML

LLMs get the headlines, but most edge AI running today is perception: camera pipelines, speech and sound, and sensor streams from motion, heart-rate or vibration sensors. These workloads are continuous, latency-critical and often always-on, so they are shaped by duty cycle and energy far more than by peak speed. At the extreme end, TinyML runs them on microcontrollers with kilobytes of memory.

On-device vision

• Tasks: classification, detection (YOLO-style, SSD), segmentation (portrait blur, background replacement), pose and landmark estimation, OCR, super-resolution and denoising, depth estimation, image embeddings for search.
• Resolution is the biggest knob: compute scales with pixel count, so going from 640x640 to 320x320 cuts compute about 4x, at the cost of small-object accuracy.
• Pipeline cost: cameras deliver YUV frames; conversion to RGB, resizing, rotation and normalization can cost more than the model if done naively on the CPU. Use the GPU, the ISP or fused native code; fold normalization into the first layer.
• Post-processing: non-max suppression, decoding anchor boxes and upsampling masks are often left on the CPU and can dominate small models.
• Temporal tricks: run the detector every few frames and a cheap tracker in between; skip inference when the scene is static; process regions of interest instead of full frames.

On-device audio and speech

• Keyword spotting (wake words): tiny always-on models on a DSP, typically a few tens of thousands of parameters, operating on log-mel or MFCC features.
• Speech recognition: streaming architectures (RNN-transducer, conformer-transducer) for live captions and dictation; encoder-decoder models such as Whisper-tiny/base-class for offline transcription.
• Real-time factor (RTF) = processing time / audio duration; streaming needs RTF well below 1 with bounded latency per chunk.
• Other audio: noise suppression in calls, sound event detection (alarms, glass breaking), speaker verification, on-device text-to-speech.

Sensor models

• Motion (IMU): activity recognition, fall detection, gesture recognition from accelerometer and gyroscope windows, often on a sensor hub.
• Health (PPG, ECG): heart-rate estimation, arrhythmia screening, sleep staging on wearables; see Wear OS for platform constraints.
• Industrial: vibration anomaly detection with small autoencoders (flag inputs the model reconstructs badly), predictive maintenance on motors and pumps.
• Radar and ultrasound: presence detection and gesture sensing at very low power.

TinyML: machine learning on microcontrollers

A microcontroller has no DRAM, often no OS, and memory measured in kilobytes. Weights and code live in flash (read-only, hundreds of KB to a few MB); activations live in SRAM (tens to hundreds of KB). The binding constraint is usually peak activation memory, not weight size.

Worked example: does a keyword spotter fit?

Input: 1 s of 16 kHz audio -> 49 frames x 10 MFCC features (INT8) = 490 bytes
Model: small depthwise-separable CNN, ~25 K parameters (INT8) = ~25 KB flash
       ~3-6 M MACs per inference
Largest two live activations: 49x10x64 input feature map + same-size output
       = 31 KB + 31 KB = ~63 KB peak arena (after in-place / memory planning tricks, less)
Target MCU: 256 KB SRAM, 1 MB flash, Cortex-M4 at 80 MHz
Latency: a few million MACs with SIMD kernels -> tens of ms; run ~every 100-250 ms
Energy: e.g. 10 mW while active x 30 ms = 0.3 mJ per inference; 4 inferences/s = ~1.2 mW average
Verdict: fits comfortably; spend remaining SRAM on the audio ring buffer.

Code: LiteRT for Microcontrollers

#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "model_data.h"                    // model compiled into flash as a C array

constexpr int kArenaSize = 64 * 1024;      // all activations live here: no malloc
alignas(16) static uint8_t tensor_arena[kArenaSize];

void setup_and_run(const int8_t* features) {
  const tflite::Model* model = tflite::GetModel(g_model_data);
  static tflite::MicroMutableOpResolver<5> resolver;   // register only what you use
  resolver.AddConv2D();
  resolver.AddDepthwiseConv2D();
  resolver.AddFullyConnected();
  resolver.AddReshape();
  resolver.AddSoftmax();

  static tflite::MicroInterpreter interpreter(model, resolver, tensor_arena, kArenaSize);
  if (interpreter.AllocateTensors() != kTfLiteOk) { /* arena too small */ return; }

  TfLiteTensor* input = interpreter.input(0);
  memcpy(input->data.int8, features, input->bytes);
  interpreter.Invoke();
  int8_t* scores = interpreter.output(0)->data.int8;   // dequantize with output scale/zero-point
}

Always-on design patterns

1. Stage 0: hardware trigger A sensor interrupt, voice activity detector or motion threshold in the sensor hub, costing microwatts.
2. Stage 1: tiny model Keyword spotter or activity classifier on a DSP or MCU, tuned for high recall.
3. Stage 2: verifier A larger model on the NPU or CPU, woken only when stage 1 fires, tuned for precision.
4. Stage 3: heavy lifting Full ASR, a local LLM or a cloud request, only after verification.

The false-trigger rate of each stage directly drives energy: every false wake costs a stage-2 run. Tune thresholds on real-world audio or sensor data, not clean benchmarks.

Power and thermal budgets

Phones, watches and glasses are passively cooled and battery-powered. A sustained workload of a few watts heats the device, and the thermal governor lowers CPU/GPU/NPU clocks. Design for sustained performance and energy per result, not a single warm run. The platform-level details of power management are covered in Power and Thermal; this section focuses on what matters for inference.

Energy, not power, is the metric

Compare two paths for the same model:
  CPU : 1.0 W above idle x 60 ms = 60 mJ per inference
  NPU : 2.0 W above idle x 10 ms = 20 mJ per inference   (2x the power, 1/3 the energy)

A continuous camera feature at 30 fps on the NPU:
  20 mJ x 30 = 0.6 W for the model alone (+ camera, ISP, display, pre/post-processing)
  0.6 W / 19.25 Wh = ~3.1% battery per hour from inference alone

The same feature at 15 fps with a model half the size (~10 mJ):
  10 mJ x 15 = 0.15 W  -> ~0.8% per hour. Duty cycle and model size multiply.

Race to idle versus DVFS

Dynamic power scales roughly as P ≈ C × V2 × f. Lower frequency allows lower voltage, so energy per operation falls at lower clocks; that argues for running slowly. But the chip also burns static (leakage) power and keeps memory, rails and other blocks awake while working; that argues for finishing fast and powering down ("race to idle"). The best operating point depends on the chip and workload: accelerators usually win on both counts because they finish faster at lower energy per operation, and for CPUs the most efficient point is typically a mid frequency on the big or mid cores, not the maximum. Measure rather than assume.

Thermal behaviour

• Thermal mass buys time: a phone can absorb heat for seconds to a few minutes before the skin temperature limit triggers throttling, which is why short benchmarks look great.
• Sustained-to-peak ratio: throughput after 10-30 minutes divided by the first-minute throughput. Ratios of 0.5-0.7 are common for heavy workloads on phones; the ratio is what a shipped feature actually gets.
• Environment matters: ambient temperature, phone cases, charging (which adds heat), and simultaneous camera and display use all change the result.
• Shared budget: the SoC's thermal budget is shared; a hot GPU from gaming or a camera pipeline reduces what the NPU can sustain.

throughput
 (tok/s)   |****
           |    ****
           |        ******                     <- throttling begins (skin temp limit)
           |              ***********************   sustained plateau
           |
           +------------------------------------------------ time
             0    1    2    3    5         10          20 min

Designing for the budget

• Duty cycle: a camera feature at 30 fps runs continuously; a photo-edit feature runs once. Budgets differ by orders of magnitude. Lower frame rate, skip frames or run on events.
• Race to idle: finishing quickly and letting hardware sleep is often more efficient than running slowly.
• Android APIs: PowerManager.getThermalHeadroom() and thermal status listeners let you degrade gracefully (lower frame rate, smaller model, lower resolution) before the OS throttles hard. The Android Dynamic Performance Framework (ADPF) helps give the scheduler hints for steady workloads.
• Always-on features (wake words, activity detection) belong on a low-power DSP or sensor hub, waking the main processor only on a trigger.
• Batch background work: index photos or fine-tune adapters only when charging and idle.

// Kotlin: degrade gracefully before the OS throttles
val pm = context.getSystemService(PowerManager::class.java)

fun chooseModelTier(): Tier {
    val headroom = pm.getThermalHeadroom(10)   // forecast 10 s ahead; 1.0 = severe throttling
    return when {
        headroom.isNaN() -> Tier.DEFAULT         // not supported / called too often
        headroom > 0.9f  -> Tier.SMALL_15FPS
        headroom > 0.7f  -> Tier.MEDIUM_30FPS
        else             -> Tier.LARGE_30FPS
    }
}

pm.addThermalStatusListener { status ->
    if (status >= PowerManager.THERMAL_STATUS_SEVERE) pauseNonEssentialInference()
}

How to measure power

• On-device power rails (On-Device Power Monitor) captured in Perfetto traces on supported devices: per-subsystem energy counters.
• External power monitor in place of the battery: the most precise lab method.
• Fuel gauge and batterystats: coarse, but available on any device and in the field.
• Always subtract an idle baseline measured with the same screen, radio and camera state, and report energy per inference plus battery percent per hour for continuous features.

Privacy, federated learning and security

Keeping inference on the device keeps raw data local, but products also need to improve models from user data and to protect the model itself. Federated learning, differential privacy and secure aggregation let a fleet learn without collecting raw data; device security measures protect the model and its outputs.

Federated learning (FL)

1. Select clients The server picks a sample of eligible devices (typically charging, idle, on unmetered Wi-Fi, with enough local data).
2. Broadcast Devices download the current global model (or just an adapter).
3. Train locally Each device runs a few steps of training on its own data.
4. Report updates Devices send weight updates (not data), usually clipped, compressed and protected by secure aggregation.
5. Aggregate The server averages the updates, weighted by the amount of local data, and produces a new global model. Repeat for many rounds.

• Challenges: non-IID data (each user's data looks different), unreliable and slow clients, communication cost (updates can be larger than the data), stragglers and dropouts, and debugging a model you cannot inspect the data for.
• Good fits: keyboard next-word prediction and emoji suggestions, wake-word and speaker models, on-device ranking and recommendation.
• Lighter alternative: on-device personalization that never leaves the device, such as fine-tuning a small adapter or a last layer locally, or keeping a local embedding index.

Differential privacy (DP)

Differential privacy is a mathematical guarantee that the output of a computation changes very little whether or not any single individual's data is included.

• DP-SGD / DP-FedAvg: clip each example's (or each client's) update to a maximum norm C, then add Gaussian noise proportional to C before aggregation. A privacy accountant tracks the cumulative ε across rounds.
• Central versus local DP: in central DP a trusted aggregator adds noise to the combined result; in local DP each device adds noise before sending, which needs no trust but costs far more accuracy.
• Trade-off: more noise gives stronger privacy but slower convergence and lower accuracy; large client populations make the noise affordable.

Secure aggregation and trusted hardware

Secure aggregation is a cryptographic protocol in which devices mask their updates so that the server can compute only the sum across many devices, never an individual update. Trusted execution environments (TEEs) and hardware-backed keystores protect keys, model decryption and sensitive processing on the device itself.

Threats to on-device models

Hybrid edge-cloud architectures

Pure on-device and pure cloud are the two ends; most successful products sit in between, using each tier for what it does best. The design question is not "edge or cloud?" but "which requests, which stages and which data go where, and what happens when the network or the device cannot cope?"

Common patterns

Designing the router

1. Classify the request By privacy sensitivity, expected difficulty (length, task type), latency requirement and cost.
2. Try local first where allowed Start local immediately for responsiveness; use a confidence signal (model self-score, a small classifier, or task rules) to decide whether to escalate.
3. Escalate with consent and minimal data Send only what is needed (a summary, extracted fields, embeddings) and only if the user and policy allow.
4. Keep a consistent contract The UI receives the same response format whichever path answered, with a flag for provenance.
5. Handle failure Offline or timeout means a local answer with honest messaging, never a hang.
6. Log and tune Record routing decisions and outcomes (privacy-safely) to tune thresholds for quality, cost and latency.

Worked example: routing economics

Assistant feature: 50 M daily users x 20 requests/day = 1 B requests/day
Cloud cost: ~0.0005 per request (illustrative)            -> 500,000 per day if all cloud
Local model handles 70% of requests acceptably             -> 150,000 per day
Local energy per request: ~2 J (short generation on NPU)   -> 20 requests = 40 J/day per user,
                                                              ~0.06% of a 69 kJ battery
Latency: local first token ~0.3 s vs cloud ~1 s on a good network, several s on a poor one
Trade-off: 70% cost saving and better latency, paid for with device engineering effort,
a download of ~1-2 GB of weights (or an OS-provided model), and a quality gap on hard requests.

A platform AI service: what an OS-level design looks like

Operating systems increasingly host one shared foundation model as a system service rather than letting every app ship its own. The design is a good reference architecture for any multi-client edge AI system:

• One resident model shared across apps instead of N copies, with a memory budget and eviction under memory pressure.
• Device gating: enabled only on devices with enough RAM and a capable NPU.
• Per-feature LoRA adapters (tens of MB each) hot-swapped on the shared base, stored per app or in a shared location.
• Model updates as binary deltas rather than full multi-GB downloads.
• Permission isolation and safety filtering on inputs and outputs, enforced centrally.
• Request queueing and fairness across clients, with thermal-aware rate limiting.
• Graceful degradation policy: NPU, then GPU, then CPU, then refuse or defer, documented and observable.

Benchmarking and evaluation metrics

Edge success is multi-dimensional. A model that is accurate but drains 10% battery per hour will be disabled by users; a fast model that drops accuracy by 8% will fail product review; a model that is fast in the lab but throttles in a user's pocket will get one-star reviews. Benchmarking concepts are summarized here; the companion page Edge AI Deployment walks through building a full measurement harness.

Why percentiles matter

Averages hide the stalls users notice. If a camera feature has a mean latency of 12 ms but a p99 of 45 ms, one frame in a hundred misses a 33 ms frame deadline and the preview visibly stutters. Report p50 (typical), p95/p99 (tail) and the maximum, and look at latency over time to separate random jitter (scheduling, garbage collection) from trends (throttling, growing KV cache).

A fair benchmarking protocol

1. Fix the environment Same device, OS build, charge level (or on external power supply for power tests), airplane mode, screen brightness, room temperature.
2. Warm up Run 10-50 inferences before measuring and report initialization separately.
3. Measure distributions Hundreds of runs; report p50/p95/p99 and standard deviation.
4. Test sustained load Run continuously for 10-30 minutes and plot latency and temperature over time.
5. Compare against a baseline FP32 on CPU, then each optimization step, in one table.
6. Test across device tiers Flagship, mid-range and low-end chipsets behave very differently.
7. Measure end to end Include pre/post-processing and data copies, and measure inside the real app as well as in the benchmark tool.

Useful tools

• LiteRT benchmark_model and its op profiler; ONNX Runtime onnxruntime_perf_test; ExecuTorch developer tools (ETDump profiling); llama-bench for GGUF models; trtexec and Nsight Systems on Jetson.
• Perfetto / Android System Trace for CPU frequency, scheduling, GPU activity and power rails.
• Vendor profilers (for example Snapdragon Profiler, Qualcomm AI Hub profile jobs) for per-layer NPU timing.
• Netron to visualise a model graph and spot odd ops or layouts.
• Android GPU Inspector for GPU delegate behaviour.
• Industry benchmark suites (MLPerf Mobile, MLPerf Tiny, MLPerf Inference edge categories) for standardized cross-device comparisons.

Industry use cases

Edge AI is already everywhere; what differs by industry is the dominant constraint. Knowing a few concrete deployments per sector, and the constraint that shaped each one, makes design interviews much easier.

Constraint by sector

Common failure modes and how to avoid them

Most edge AI projects that fail do not fail because the model is wrong; they fail because of measurement mistakes, silent fallbacks, unplanned memory pressure or scope creep. This section collects the traps, organized by where they appear.

Skills map, roles and further study

Edge AI rewards people who combine two halves: enough ML understanding to know what a model is doing, and deep systems skill to make it run well on silicon. Engineers from mobile, platform, embedded, driver, DSP or power-optimization backgrounds already own the scarce systems half. Many edge roles are explicitly inference-engineering roles (C++ runtimes, SDKs, kernels, deployment), not model-training roles. Detailed learning schedules, benchmark projects and target numbers are on Edge AI Deployment; this section maps the territory.

How platform skills transfer

Skills map

A generic learning path

1. AI literacy Be able to explain transformers, attention, tokens, embeddings, training vs inference and fine-tuning vs RAG in plain words; get comfortable with Python, PyTorch basics and Hugging Face.
2. ML through a systems lens Quantization in depth, compression, model formats and graph optimization, and inference maths (compute vs memory bound, KV cache, why NPUs are fast).
3. Edge runtimes and toolchains Deploy the same model with two runtimes, use one vendor NPU SDK, run a quantized LLM with llama.cpp or ExecuTorch, and profile latency, memory, tokens/s, power and thermal on a real device.
4. Evidence Build two or three benchmarked projects with a clear results table and write them up (see the companion page for project designs).

Roles

Portfolio project ideas (summary)

What a good project write-up contains

1. Goal and constraints Target device, latency/power budget, accuracy floor.
2. Baseline FP32 on CPU numbers with the measurement method.
3. Changes Each optimization step with before/after numbers.
4. Findings What surprised you (for example an op that fell back, or throttling after 8 minutes).
5. Reproducibility Scripts, versions, device build, commands, and raw data.

Curated resources (official sources first)

APIs in this area change quickly, so prefer each project's current official documentation.

FAQ

Do I need to be good at maths or model training?

Not for most edge roles. You need to understand models (architectures, where compute and memory go, how quantization affects them) well enough to optimize and deploy them. Basic linear algebra and probability intuition are enough to start; the job is inference efficiency, not research.

Is Python a problem if I come from C, C++ or Java?

Python is a quick add for an experienced engineer, usually a few weeks of practice. Much of edge AI (runtimes, kernels, llama.cpp, delegates) is C/C++, where systems engineers are already strong, and many inference-SDK roles require strong C++ specifically.

Is a GenAI application course wasted if I want edge AI?

No. It builds the vocabulary you need to talk to model teams. Just avoid making application-layer GenAI your whole identity if your strength is systems; stack edge-specific skills on top.

How do I prove skill without AI job experience?

Benchmarked projects. A concrete, reproducible result such as "ran model X on device Y at N tokens/s with M% less energy than CPU, sustained over 10 minutes" is verifiable and rare.

Which runtime should I learn first?

LiteRT or ExecuTorch, depending on whether your models come from TensorFlow/JAX or PyTorch, plus llama.cpp for LLMs. Then learn one vendor NPU SDK to understand what happens below the runtime.

Will I lose seniority by switching?

If you position yourself as an AI application engineer, possibly. If you position yourself as an AI systems or on-device engineer, deep platform experience counts fully, because that half is what is scarce. Building skills while in a current role and preferring an internal move reduces the risk.

Quick revision

• Edge AI runs models near the data source, on a continuum from MCUs to devices, edge boxes, edge servers and the cloud; on-device AI is the extreme end.
• Drivers: latency, privacy, cost, offline use, bandwidth, data sovereignty and personalization; costs: compute, memory, bandwidth, power, thermal, storage and fragmentation.
• Most products are hybrid: a small local model for common cases with a clear rule for cloud fallback and a defined offline behaviour.
• Phone DRAM bandwidth is ~50-100 GB/s versus 2-8 TB/s on data-centre GPUs; bandwidth sets a hard latency floor of bytes moved divided by bandwidth.
• Moving data from DRAM costs orders of magnitude more energy than computing on it, so reuse, fusion and smaller data types save power.
• CPU is flexible, GPU is parallel and FP16-friendly, NPU gives the best performance per watt for low-precision tensor maths, DSP suits always-on low-power tasks, MCUs run TinyML at milliwatts.
• NPUs are efficient because of low-precision MAC arrays, on-chip SRAM reuse, fixed dataflow and ahead-of-time compilation, which is also why they want static shapes and static quantization.
• Peak TOPS = MACs x 2 x clock; real utilization is often 20-50% because of bandwidth, operator coverage, batch size 1 and throttling.
• Roofline: attainable performance = min(peak compute, bandwidth x arithmetic intensity); the ridge point is peak / bandwidth.
• Batch-1 matrix-vector work (LLM decode) has ~2 ops/byte at INT8 and is deeply memory-bound; prefill with hundreds of tokens is compute-bound.
• Model size is roughly parameters times bytes per parameter: 3B params is ~6 GB FP16, ~3 GB INT8, ~1.5-1.8 GB INT4.
• A dense layer costs about 2 x inputs x outputs FLOPs; a transformer costs about 2 x parameters FLOPs per token plus attention; depthwise-separable convs cut 3x3 conv cost ~8-9x.
• FP16 has more precision but overflows above 65,504; BF16 has FP32's range with less precision; FP8 comes as E4M3 (precision) and E5M2 (range).
• Block formats (MXFP4, NVFP4, GGUF blocks) share one scale per small group; effective bits = element bits + scale bits / group size.
• Quantization maps reals to integers with a scale and zero-point: x is approximately scale x (q - zero_point); symmetric sets zero_point to 0.
• Rounding error is about scale squared / 12 in variance; each extra bit adds ~6 dB SQNR; calibration balances rounding error against clipping error.
• Per-channel weight scales are standard for INT8; per-group scales (32-128) are standard for INT4 LLM weights; one outlier ruins a per-tensor scale.
• Calibration methods: min-max, percentile, MSE and KL/entropy; calibration data must match production.
• Static quantization fixes activation scales at compile time (NPU-friendly); dynamic computes them at runtime (CPU-friendly, no calibration).
• Integer matmul accumulates in INT32 and requantizes with a fixed-point multiplier M = s_w x s_x / s_y; weight-only W4A16 dequantizes on the fly and wins on bandwidth.
• PTQ needs only a calibration set; QAT uses fake quantization with the straight-through estimator and recovers accuracy at low bits. Try PTQ first.
• Accuracy loss usually comes from outliers, sensitive layers or bad calibration; fix with per-channel scales, clipping, equalization, mixed precision, advanced PTQ or QAT, guided by layer-wise error analysis.
• GPTQ rounds column by column with Hessian-based error compensation; AWQ scales up salient channels; SmoothQuant moves activation outliers into weights; rotations spread outliers.
• GGUF is llama.cpp's single-file format; Q4_K_M (~4.8 bits) is a popular balance; imatrix improves low-bit quants.
• CPU paths commonly use 4-bit group-wise weights with 8-bit dynamic activations; NPUs commonly use 4- or 8-bit weights with 16-bit static activations.
• Unstructured pruning compresses but rarely speeds up mobile hardware; structured and N:M sparsity give real speed-ups.
• Distillation trains a small student on a big teacher's soft outputs (temperature-scaled KL); most good small LLMs are distilled, often after pruning.
• Low-rank factorization turns m x n parameters into r(m + n); LoRA adapters let one base model serve many features.
• Hardware-aware NAS searches architectures using measured latency on the target device.
• Operator fusion and BatchNorm folding remove memory round trips at no accuracy cost; static shapes, layout choice and memory planning matter as much.
• Delegates/execution providers hand subgraphs to accelerators; every CPU fallback boundary adds copies and sync; check partition logs.
• LiteRT is the new name of TensorFlow Lite; ExecuTorch is PyTorch's on-device runtime; ONNX Runtime is framework-agnostic; QNN/QAIRT is Qualcomm's NPU SDK; TensorRT serves Jetson; llama.cpp and MLC-LLM serve LLMs.
• NNAPI is deprecated from Android 15 because of inconsistent drivers and fragmentation; use LiteRT delegates or vendor backends instead.
• The pipeline is: export, optimize, quantize, convert, validate on host, deploy, run on accelerator, profile, iterate.
• LLM prefill is compute-bound and sets time to first token (~2 x params x prompt tokens FLOPs); decode is memory-bound and sets tokens/s.
• Decode tokens/s ceiling is roughly effective memory bandwidth divided by (weight bytes + KV bytes at the current context). Long chats slow down even with fixed weights.
• KV cache = 2 x layers x KV heads x head_dim x context x bytes; at long context it can exceed the weights; GQA, KV quantization, windows and shorter context shrink it.
• Paged KV: contiguous waste is (T_max - T_actual) x KV per token; a block table wastes at most one block per sequence and can share prefix blocks.
• Prefix caching reuses KV for a byte-identical prefix; savings about P / (P + U). Put static tokens first.
• Speculative decoding drafts k tokens and verifies them in one pass: expected tokens per pass = (1 - alpha^(k+1)) / (1 - alpha); speed-up ≈ that / (1 + k c). Unchanged output distribution; helps most at low batch when acceptance is high.
• INT8 weights are ~2x smaller than FP16 and usually near-lossless. INT4 is ~4x smaller and the usual on-device decode lever, but quality loss is larger on reasoning and maths; keep embeddings and the LM head higher and always re-evaluate the task.
• Realistic phone LLMs are about 0.5-4B parameters in INT4; 7-8B needs 12-16 GB+ RAM.
• TinyML's binding constraint is usually peak activation memory in SRAM (the tensor arena), not weight size in flash.
• Always-on features use cascades: hardware trigger, tiny model on DSP/MCU, verifier on NPU, heavy work last; false triggers cost energy.
• Energy per inference (power x time) matters more than peak power; racing to idle often wins; plan for the sustained, throttled clock.
• Use thermal headroom APIs to degrade gracefully (smaller model, lower fps) before the OS throttles hard.
• Federated learning shares updates, not data (FedAvg); secure aggregation hides individual updates; differential privacy bounds what any individual's data can reveal.
• Benchmark with warm-up, hundreds of runs, p50/p95/p99, sustained 10-30 minute runs, baseline rows, device tiers and real silicon only.
• Pre- and post-processing can cost more than the model; profile the whole pipeline inside the real app.
• Ship per-device-tier model variants, monitor latency and fallback in production, and keep a remote kill switch.

Glossary

ADPF

Android Dynamic Performance Framework: APIs for performance hints and thermal awareness so apps can sustain steady workloads.

Arithmetic intensity

Operations performed per byte of memory traffic; decides compute-bound versus memory-bound behaviour.

AWQ

Activation-aware Weight Quantization: a PTQ method that scales up the most important weight channels (found from activation magnitudes) before low-bit quantization.

BF16

bfloat16: a 16-bit float with FP32's 8-bit exponent and a 7-bit mantissa; wide range, low precision.

Block (microscaling) format

A format in which a small block of values shares one scale, such as MXFP4, NVFP4 or GGUF blocks.

Calibration set

A small set of representative inputs used during PTQ to measure activation ranges.

Cascade

A chain of models of increasing cost where each stage runs only if the previous one fires or is unsure.

Core ML

Apple's on-device inference framework, running on CPU, GPU and the Neural Engine.

Cross-layer equalization

Rescaling consecutive layers so per-channel weight ranges are even, improving quantization without changing FP32 outputs.

Decode

The token-by-token generation phase of an LLM; memory-bandwidth-bound.

Delegate

A runtime plug-in that executes supported parts of a model graph on an accelerator such as a GPU or NPU.

Depthwise-separable convolution

A per-channel spatial convolution followed by a 1x1 convolution; far cheaper than a standard convolution.

Differential privacy

A mathematical guarantee, parameterized by epsilon and delta, that one individual's data barely changes a computation's output.

Distillation

Training a small student model to imitate a larger teacher model's outputs.

DSP

Digital signal processor: a power-efficient processor for fixed-point vector maths, often used for audio and always-on models.

Dynamic quantization

Quantization where activation scales are computed at runtime for each input.

Edge server (MEC)

A server near users, on premises or at a telecom site (multi-access edge computing), offering low-latency compute.

Energy per inference

Average power above idle multiplied by latency; the right metric for comparing backends on battery devices.

Execution provider

ONNX Runtime's term for a hardware backend (CPU, QNN, Core ML, XNNPACK, TensorRT and so on).

ExecuTorch

PyTorch's runtime for on-device inference, using models exported with torch.export into .pte files.

Fake quantization

Quantize-then-dequantize operations inserted during QAT to simulate rounding in floating point.

Fallback

Running an operator on the CPU because the accelerator does not support it.

Federated learning

Training across many devices by sharing model updates instead of raw data, then aggregating them on a server.

FLOPs

Floating-point operations; a measure of compute work (one multiply-accumulate is about two FLOPs).

FP16

Half-precision float with a 5-bit exponent and 10-bit mantissa; maximum value 65,504.

FP8

8-bit floating point, in E4M3 (more precision, max 448) and E5M2 (more range) variants.

GGUF

llama.cpp's single-file model format containing quantized weights, tokenizer and metadata.

GPTQ

A post-training quantization algorithm that quantizes weights column by column while compensating the error using second-order information.

GQA

Grouped-query attention: several query heads share one key/value head, shrinking the KV cache.

Hexagon NPU

Qualcomm's neural processor, combining scalar, vector (HVX) and tensor (HMX) units.

Importance matrix

Per-weight importance statistics from calibration text, used by llama.cpp to reduce low-bit quantization error.

INT4

4-bit integer format with 16 levels; used mainly for LLM weights with per-group scales.

Jetson

NVIDIA's family of embedded modules combining Arm CPUs, a CUDA GPU with tensor cores and deep-learning accelerators.

K-quants

llama.cpp quantization types using super-blocks with quantized sub-block scales, such as Q4_K and Q6_K.

KV cache

Stored keys and values of previous tokens so an LLM does not recompute them at each step.

Paged attention

Storing the KV cache in fixed-size blocks mapped by a block table, so waste is at most one block per sequence and prefixes can be shared.

Prefix caching

Reusing computed KV for a byte-identical prompt prefix (system prompt, tools, shared document) to cut prefill and TTFT.

LiteRT

Google's on-device runtime, formerly TensorFlow Lite, using .tflite models.

LiteRT for Microcontrollers

The MCU variant of LiteRT (formerly TFLite Micro), running models from a fixed memory arena without an OS.

llama.cpp

An open-source C/C++ engine for running quantized LLMs on CPUs and GPUs of laptops and phones.

LoRA

Low-rank adaptation: fine-tuning with small low-rank update matrices while freezing the base model.

MAC

Multiply-accumulate operation, the basic building block of neural network maths.

MediaPipe

Google's framework of ready-made on-device ML pipelines built on LiteRT.

Memory bandwidth

Bytes per second that can move between DRAM and compute units; the main limit for memory-bound layers.

Mixed precision

Using different numeric formats for different layers, for example INT8 for most and FP16 for sensitive ones.

MLC-LLM

A compiler-based LLM deployment stack built on TVM, targeting mobile GPUs and other backends.

N:M sparsity

A structured sparsity pattern with N zeros in every M consecutive weights (for example 2:4) that hardware can accelerate.

NAS

Neural architecture search: automated search for architectures, often scored by measured latency on the target device.

NF4

4-bit NormalFloat: a non-uniform 4-bit format whose levels follow a normal distribution; used by QLoRA.

NNAPI

Android Neural Networks API, a vendor-neutral accelerator interface, deprecated from Android 15.

NPU

Neural processing unit: a dedicated accelerator for low-precision tensor maths with high performance per watt.

ONNX

Open Neural Network Exchange: an open model format used to move models between frameworks and runtimes.

ONNX Runtime

A cross-platform inference engine for ONNX models with pluggable execution providers.

Operator fusion

Merging several consecutive operations into one kernel to cut memory traffic and launch overhead.

Per-channel quantization

Using a separate scale for each output channel of a weight tensor for better accuracy.

Per-group quantization

Using a separate scale for each small group (for example 32-128) of consecutive weights.

Prefill

The phase where an LLM processes the whole prompt in parallel; compute-bound; sets time to first token.

Pruning

Removing weights, channels, heads or layers that contribute little to the output.

PTQ

Post-training quantization: quantizing an already-trained model using calibration data.

QAT

Quantization-aware training: training with simulated quantization so the model tolerates low precision.

QNN / QAIRT

Qualcomm AI Engine Direct, packaged as the Qualcomm AI Runtime SDK, for running models on the Hexagon NPU, Adreno GPU and CPU.

Race to idle

Finishing work quickly so hardware can return to low-power sleep states, often saving energy overall.

Real-time factor

Processing time divided by input audio duration; below 1 means faster than real time.

Ridge point

Peak compute divided by memory bandwidth; the arithmetic intensity where a workload becomes compute-bound.

Roofline model

A chart relating attainable performance to arithmetic intensity, bounded by memory bandwidth and peak compute.

Rotation methods

Quantization techniques (QuaRot, SpinQuant) that apply orthogonal transforms to spread outliers across channels.

Scale and zero-point

The two parameters that map real values to integers in quantization.

Secure aggregation

A cryptographic protocol that lets a server see only the sum of many device updates, never an individual one.

SmoothQuant

A technique that divides activation channels and multiplies weight rows by the same factor to move outliers into weights, enabling W8A8.

Speculative decoding

A draft model proposes tokens that the main model verifies in parallel, speeding decode without changing outputs.

SQNR

Signal-to-quantization-noise ratio in decibels; a standard per-tensor measure of quantization damage.

Static quantization

Quantization where activation scales are fixed ahead of time from calibration data.

Straight-through estimator

A QAT trick that treats rounding as the identity in the backward pass so gradients can flow.

Tensor arena

The fixed, pre-allocated SRAM buffer holding all activations in LiteRT for Microcontrollers.

TensorRT

NVIDIA's inference optimizer and runtime that builds device-specific engines with fusion and low precision.

Thermal headroom

An OS forecast of how close the device is to thermal throttling, used to degrade workloads gracefully.

Thermal throttling

The OS lowering clock speeds to limit heat, reducing sustained performance.

TinyML

Machine learning on microcontrollers with kilobytes of memory and milliwatt power budgets.

TOPS

Trillions of operations per second; a peak compute figure, usually for INT8.

TTFT

Time to first token: delay from submitting a prompt to receiving the first generated token.

W4A16

Notation for 4-bit weights with 16-bit activations; similar forms describe other weight/activation precisions.

XNNPACK

A highly optimized CPU inference library used as the default CPU backend by LiteRT, ExecuTorch and ONNX Runtime.

Interview questions