Edge AI Deployment & Projects

The end-to-end deployment pipeline

Every edge AI deployment, whether it is a 5 MB keyword spotter or a 1-billion-parameter language model, walks the same eight stages. The names of the tools change; the stages do not. If you can explain each stage, its artifact, and what goes wrong there, you can reason about any stack an interviewer throws at you. The theory behind each stage (what an NPU is, how quantization maths works, how runtimes differ) lives on the concepts page, Edge AI; this page is about doing it.

  +-------------+   +-----------+   +-----------+   +------------+
  | 1. CHOOSE / |--▶| 2. EXPORT |--▶| 3. CONVERT|--▶| 4. QUANTIZE|
  |    TRAIN    |   | static    |   | to runtime|   | PTQ / QAT  |
  +-------------+   | graph     |   | format    |   | mixed prec.|
                    +-----------+   +-----------+   +------------+
                                                          |
  +-------------+   +-----------+   +-----------+   +------------+
  | 8. MONITOR  |◀--| 7. BENCH- |◀--| 6. INTE-  |◀--| 5. COMPILE |
  | field data, |   |  MARK on  |   |  GRATE in |   | for target |
  | A/B, rollbk |   |  device   |   |  the app  |   | CPU/GPU/NPU|
  +-------------+   +-----------+   +-----------+   +------------+
         |                                               ▲
         +------------- iterate: fix ops, precision, model ----+

The eight stages and their artifacts

How to read this page

1. Set up once Read the stack decisions, buy or borrow the right hardware, and complete the environment checklist before writing any model code.
2. Learn the paths Walk through the export paths section and run at least two of them end to end (ExecuTorch and llama.cpp are the easiest starting pair).
3. Integrate Put one model into a real Android app so you understand packaging, threading and memory.
4. Build the five projects in order Each project reuses the previous project's harness and understanding; do not reorder them.
5. Harden for production Profiling, accuracy debugging, shipping and the failure-mode checklist turn a demo into a product.
6. Revise Use the quick revision list, glossary and interview questions at the end.

Stack decisions: what to learn and what to skip

The on-device stack churned heavily in 2024 and 2025 and has now largely consolidated. Choosing the right tools up front saves months of learning APIs that are being retired. The rule of thumb: learn one general-purpose runtime deeply, one fast LLM prototyping tool, and one vendor NPU toolchain. Everything else you should recognise and be able to discuss.

Do not invest in these

• NNAPI (Android Neural Networks API). Introduced in Android 8.1 and marked deprecated / not recommended for new work from Android 15. Vendors implemented it inconsistently, so behaviour and performance varied wildly between devices. The replacement direction is vendor delegates and backends that plug directly into frameworks (LiteRT accelerators, ExecuTorch backends, ONNX Runtime execution providers). If a tutorial is built on NNAPI, treat it as historical.
• Training from scratch, distributed training, cloud MLOps pipelines, data-centre serving. Useful elsewhere, but not what edge deployment roles screen for. Fine-tuning small adapters is the one training skill worth having (see Fine-tuning).
• Chasing 7B-plus models on phones early. Quantizing them needs 80 GB-class GPUs, iteration is slow, and a 1B-class model teaches every lesson.

The stack that matters

Why a closed system AI service is worth studying

A platform AI service such as Android's AICore shows what a production-grade on-device design looks like. It is updated independently of the OS (as an updatable system module), it only enables itself on devices with enough RAM and an NPU, it ships model updates as binary deltas instead of re-downloading gigabytes, it runs safety filtering around the model, it isolates the model from callers behind a permission boundary, and it supports small LoRA adapters (tens of MB) that specialise one shared base model per feature. Every one of those decisions is an answer to a real constraint: storage, bandwidth, memory, privacy, safety and multi-tenancy. It is the blueprint for the shared-inference-service option in Project 5.

Choosing a runtime for a given job

Hardware, software and budget

Performance work on edge AI is only meaningful on physical silicon. Emulators and simulators do not model memory bandwidth, cache sizes, DVFS, thermal throttling, or the NPU at all; a number measured on an emulator is not wrong by a small factor, it is meaningless.

Devices: match the NPU generation

Vendor NPU libraries are versioned per hardware generation. For Snapdragon, the Hexagon library folder must match the SoC; using the wrong one fails to load or silently falls back to CPU.

For other vendors the same principle applies: MediaTek Dimensity APU generations map to specific NeuroPilot versions; Google Tensor chips expose their TPU through LiteRT accelerators; Apple's Neural Engine is reached only through Core ML.

Minimum viable kit

• One Android phone with a recent flagship SoC (for example Snapdragon 8 Gen 2 or newer) and 12 GB+ RAM. A second-hand device is perfectly fine.
• A Linux workstation (Ubuntu 22.04+ native, or WSL2 on Windows) with 16-32 GB RAM and 100 GB+ free disk. Most vendor toolchains are Linux-first.
• Android SDK platform-tools (adb) and NDK r26 or newer.
• A reliable USB-C data cable. Flaky cables cause intermittent adb disconnects that look like software bugs.

Ideal kit

• Two phones from different SoC generations or vendors: cross-silicon comparisons are the most informative measurements you can produce.
• A Pixel device to explore the platform AI service path and Tensor TPU accelerators.
• A MediaTek device for second-vendor work later.
• A USB power meter or, better, a bench power supply / power monitor for repeatable energy numbers; otherwise use on-device power rails and the fuel gauge.
• Optional: a single-board computer or microcontroller dev kit (Arm Cortex-M with an Ethos-U NPU, or a Linux SBC with an NPU) for the wearable/sensor project.

GPU access for quantization

Running a model is cheap; quantizing a large one is not. Algorithms such as GPTQ, SpinQuant, AdaRound or QAT need the full-precision model in GPU memory plus calibration activations. Rough requirements for LLM quantization workflows:

Strategy: use 1B-class models with post-training quantization for most learning; rent a cloud GPU by the hour for the one or two runs that genuinely need it (a QAT comparison, a vendor LLM export). Shut instances down immediately after; idle GPUs are the main cost leak.

Budget (approximate, varies by region)

Environment setup

Toolchain setup is where most people lose several weekends and give up. Do it once, carefully, in a reproducible way, and timebox it. If the heavy toolchain fights you for more than a day, build llama.cpp first to get a model running on the phone, then come back.

Checklist

1. Operating system Ubuntu 22.04+ (native or WSL2) with 100 GB+ free disk. Model artifacts, SDKs and build trees add up quickly.
2. Python Python 3.10-3.12 in a dedicated virtual environment or conda environment per toolchain. Never install into system Python; vendor SDKs often pin conflicting versions.
3. Android tools SDK platform-tools with adb devices showing your phone as device (not unauthorized).
4. NDK Android NDK r26+ installed; ANDROID_NDK exported.
5. Build tools CMake 3.24+, Ninja, a recent Clang, git with LFS.
6. Model access A model hub account with accepted licences for gated models (Llama-family weights require accepting the licence).
7. Vendor SDK A vendor developer account, AI Hub token, and the QAIRT SDK downloaded and unpacked; QNN_SDK_ROOT exported.
8. Phone settings Developer options on, USB debugging on, "stay awake while charging" on, battery optimisation disabled for your test app, adaptive brightness off.
9. Version control One git repository for the whole effort. Commit measurement CSVs and configs, not just code, so every number is reproducible.

Host environment

# Base tools (Ubuntu)
sudo apt update && sudo apt install -y build-essential cmake ninja-build git git-lfs \
     python3-venv python3-dev clang unzip wget

# Android NDK (adjust version/path to what you installed)
export ANDROID_HOME=$HOME/Android/Sdk
export ANDROID_NDK=$ANDROID_HOME/ndk/26.3.11579264
export PATH=$ANDROID_HOME/platform-tools:$PATH

# Sanity checks
adb devices                     # should list your phone as "device"
adb shell getprop ro.soc.model  # e.g. SM8650 for Snapdragon 8 Gen 3
adb shell getprop ro.product.cpu.abi   # must be arm64-v8a
adb shell cat /proc/meminfo | head -3  # total RAM

ExecuTorch host build

git clone --recursive https://github.com/pytorch/executorch.git
cd executorch
python -m venv .venv && source .venv/bin/activate
./install_executorch.sh          # installs torch, torchao and the executorch pip package

# Verify before touching Android
python -c "import executorch, torch; print(executorch.__file__, torch.__version__)"

ExecuTorch Android runner build (the flags that matter)

cmake -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
      -DANDROID_ABI=arm64-v8a \
      -DANDROID_PLATFORM=android-26 \
      -DCMAKE_BUILD_TYPE=Release \
      -DEXECUTORCH_BUILD_XNNPACK=ON \
      -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
      -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
      -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
      -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
      -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
      -DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
      -DEXECUTORCH_XNNPACK_ENABLE_KLEIDI=ON \
      -Bcmake-out-android .
cmake --build cmake-out-android -j$(nproc) --target install

# Then build the LLM example runner against that install
# (see examples/models/llama in the repo for the exact current target)

llama.cpp for Android (fast baseline)

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build-android \
  -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-28 \
  -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DGGML_OPENMP=OFF
cmake --build build-android -j$(nproc) --target llama-cli llama-bench

adb shell mkdir -p /data/local/tmp/lcpp
adb push build-android/bin/llama-cli build-android/bin/llama-bench /data/local/tmp/lcpp/

Vendor SDK environment (Qualcomm example)

# After unpacking QAIRT
export QNN_SDK_ROOT=/opt/qcom/aistack/qairt/<version>
source $QNN_SDK_ROOT/bin/envsetup.sh        # sets PATH, PYTHONPATH, LD_LIBRARY_PATH
python $QNN_SDK_ROOT/bin/check-python-dependency   # installs pinned Python deps

# AI Hub client (separate venv recommended)
python -m venv ~/venvs/aihub && source ~/venvs/aihub/bin/activate
pip install qai-hub qai-hub-models
qai-hub configure --api_token <YOUR_TOKEN>
qai-hub list-devices | head

Phone preparation for stable measurements

# Keep the screen on while plugged in, and set a fixed brightness
adb shell svc power stayon usb
adb shell settings put system screen_brightness_mode 0
adb shell settings put system screen_brightness 100

# Record the software build you measured on
adb shell getprop ro.build.fingerprint

# Create a scratch area on the device
adb shell mkdir -p /data/local/tmp/bench

Export and conversion paths

"Export" captures a model as a static, framework-independent graph; "conversion" turns that graph into the format a specific runtime executes. Most modern paths start from PyTorch. Below are the six paths you will meet most often, with working command and code patterns. Treat exact flag names as version-dependent and check the current documentation.

                         PyTorch nn.Module (FP32/BF16)
                                    |
      +-------------+-------------+-+-----------+-------------+--------------+
      |             |             |             |             |              |
 torch.onnx    torch.export  ai-edge-torch  HF safetensors  coremltools   AI Hub /
  .export           |        (torch.export  convert_hf_to_   ct.convert    QAIRT
      |             |          inside)       gguf.py            |        converters
   .onnx        ExecuTorch        |             |            .mlpackage      |
      |          to_edge +     .tflite       .gguf (f16)        |       QNN model
 ORT quantize    partitioner   (LiteRT)    llama-quantize    Core ML     + context
  / QNN EP         |              |             |            runtime     binary .bin
      |          .pte         LiteRT +      Q4_K_M .gguf       (ANE)          |
 ONNX Runtime   ExecuTorch    delegates     llama.cpp                   QNN HTP /
   Mobile        runtime                                                  Genie

Path 1: PyTorch to ONNX (and ONNX Runtime quantization)

import torch, torchvision

model = torchvision.models.mobilenet_v3_small(weights="DEFAULT").eval()
dummy = torch.randn(1, 3, 224, 224)

torch.onnx.export(
    model, (dummy,), "mobilenet_v3.onnx",
    input_names=["image"], output_names=["logits"],
    opset_version=17,
    dynamic_axes=None,          # keep static for NPUs; use {"image": {0: "batch"}} only if needed
    do_constant_folding=True,
)

# Validate numerically against PyTorch before doing anything else
import onnxruntime as ort, numpy as np
sess = ort.InferenceSession("mobilenet_v3.onnx", providers=["CPUExecutionProvider"])
ref = model(dummy).detach().numpy()
out = sess.run(None, {"image": dummy.numpy()})[0]
print("max abs diff:", np.abs(ref - out).max())   # expect ~1e-5 for FP32

Static (QDQ) INT8 quantization with a calibration reader:

from onnxruntime.quantization import (quantize_static, CalibrationDataReader,
                                      QuantFormat, QuantType, CalibrationMethod)
from onnxruntime.quantization.shape_inference import quant_pre_process

quant_pre_process("mobilenet_v3.onnx", "mobilenet_v3.pre.onnx")   # shape inference + folding

class Reader(CalibrationDataReader):
    def __init__(self, samples):          # samples: list of float32 arrays, shape (1,3,224,224)
        self.it = iter([{"image": s} for s in samples])
    def get_next(self):
        return next(self.it, None)

quantize_static(
    "mobilenet_v3.pre.onnx", "mobilenet_v3.int8.onnx",
    calibration_data_reader=Reader(calibration_samples[:300]),
    quant_format=QuantFormat.QDQ,             # QuantizeLinear/DequantizeLinear pairs; EPs fuse them
    activation_type=QuantType.QUInt8, weight_type=QuantType.QInt8,
    per_channel=True, calibrate_method=CalibrationMethod.MinMax,
)
# For the QNN EP, ORT provides helpers (qnn_preprocess_model, get_qnn_qdq_config)
# that produce a QDQ model with the uint8/uint16 activation types the HTP expects.

Optionally convert to the ORT format and build a reduced-operator runtime to shrink the mobile binary: python -m onnxruntime.tools.convert_onnx_models_to_ort model.int8.onnx produces a .ort file and a list of required operators you can feed into a custom minimal build.

Path 2: PyTorch to ExecuTorch (.pte)

Generic model with the XNNPACK CPU backend and PT2E INT8 quantization:

import torch
from torch.export import export
from executorch.exir import to_edge_transform_and_lower
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import (
    XNNPACKQuantizer, get_symmetric_quantization_config)
from torchao.quantization.pt2e.quantize_pt2e import prepare_pt2e, convert_pt2e

model = MyModel().eval()
example = (torch.randn(1, 3, 224, 224),)

# 1. Capture a graph suitable for quantization
graph = export(model, example).module()

# 2. Insert observers, calibrate, convert (post-training static quantization)
quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config(is_per_channel=True))
prepared = prepare_pt2e(graph, quantizer)
for x in calibration_batches:          # ~100-500 representative inputs
    prepared(x)
quantized = convert_pt2e(prepared)

# 3. Re-export, lower supported subgraphs to XNNPACK, serialize
program = to_edge_transform_and_lower(
    export(quantized, example), partitioner=[XnnpackPartitioner()]
).to_executorch()
with open("model_xnnpack_int8.pte", "wb") as f:
    f.write(program.buffer)

LLM export using the built-in LLM exporter (running example: Llama-3.2-1B-Instruct; you need consolidated.00.pth, params.json and tokenizer.model from the model repository):

python -m extension.llm.export.export_llm \
  base.model_class="llama3_2" \
  base.checkpoint="${LLAMA_DIR}/consolidated.00.pth" \
  base.params="${LLAMA_DIR}/params.json" \
  model.use_kv_cache=True \
  model.use_sdpa_with_kv_cache=True \
  model.dtype_override="fp32" \
  base.metadata='"{\"get_bos_id\":128000, \"get_eos_ids\":[128009, 128001]}"' \
  quantization.qmode="torchao:8da4w" \
  quantization.group_size=128 \
  quantization.embedding_quantize="torchao:4,32" \
  backend.xnnpack.enabled=True \
  export.max_seq_length=2048 \
  export.output_name="llama3_2_1b_8da4w.pte"

• 8da4w means 8-bit dynamic activations and 4-bit weights; group_size=128 gives one scale per 128 weights.
• embedding_quantize="4,32" quantizes the large embedding table (128256 x 2048 for this model, about a fifth of all parameters) to 4-bit with groups of 32.
• use_kv_cache and use_sdpa_with_kv_cache are essential for speed. Without them every decode step recomputes attention over the whole sequence and numbers are badly wrong.
• The BOS/EOS ids in metadata must match the tokenizer, or generation will not stop (or will stop immediately).
• Older ExecuTorch versions used python -m examples.models.llama.export_llama with flags like -kv --use_sdpa_with_kv_cache -X -qmode 8da4w; recognise both.

Swap XnnpackPartitioner for a vendor partitioner (Qualcomm QNN, MediaTek, Core ML, Vulkan) to target accelerators; the rest of the flow stays the same. That uniformity is ExecuTorch's main selling point.

Path 3: PyTorch to LiteRT via ai-edge-torch

import torch, torchvision
import ai_edge_torch        # package name may be litert-torch in newer releases

model = torchvision.models.resnet18(weights="DEFAULT").eval()
sample = (torch.randn(1, 3, 224, 224),)

edge_model = ai_edge_torch.convert(model, sample)      # uses torch.export under the hood
edge_model.export("resnet18_fp32.tflite")

# Check parity on host
import numpy as np
ref = model(*sample).detach().numpy()
out = edge_model(*sample)
print("max abs diff:", np.abs(ref - out).max())

# Optional: NHWC inputs to avoid transposes on GPU/NPU delegates
nhwc_model = ai_edge_torch.to_channel_last_io(model, args=[0])
edge_nhwc = ai_edge_torch.convert(nhwc_model, (torch.randn(1, 224, 224, 3),))

Quantization uses the PT2E flow with a PT2E quantizer for LiteRT (ai_edge_torch.quantize) before conversion, or post-conversion quantization for weight-only compression. For LLMs, ai-edge-torch's generative API includes re-authored model definitions and conversion scripts that emit prefill and decode signatures with an external KV cache, then bundle them into a .task or .litertlm file for MediaPipe / LiteRT-LM.

From TensorFlow/Keras, the classic converter still applies: tf.lite.TFLiteConverter.from_saved_model() with optimizations=[tf.lite.Optimize.DEFAULT], a representative_dataset generator for full-integer quantization, and TFLITE_BUILTINS_INT8 ops for integer-only NPU targets.

Path 4: Hugging Face checkpoint to GGUF (llama.cpp)

cd llama.cpp
pip install -r requirements.txt

# 1. Convert HF safetensors to a 16-bit GGUF (weights + tokenizer + metadata in one file)
python convert_hf_to_gguf.py /models/Llama-3.2-1B-Instruct \
       --outfile llama-3.2-1b-f16.gguf --outtype f16

# 2. Optional: importance matrix from calibration text improves low-bit quality
./build/bin/llama-imatrix -m llama-3.2-1b-f16.gguf -f calib.txt -o imatrix.dat

# 3. Quantize
./build/bin/llama-quantize --imatrix imatrix.dat llama-3.2-1b-f16.gguf llama-3.2-1b-Q4_K_M.gguf Q4_K_M
./build/bin/llama-quantize llama-3.2-1b-f16.gguf llama-3.2-1b-Q4_0.gguf Q4_0   # Arm repack-friendly

# 4. Quality check on host: perplexity on held-out text
./build/bin/llama-perplexity -m llama-3.2-1b-Q4_K_M.gguf -f wiki.test.raw

# 5. On device
adb push llama-3.2-1b-Q4_0.gguf /data/local/tmp/lcpp/
adb shell "cd /data/local/tmp/lcpp && ./llama-bench -m llama-3.2-1b-Q4_0.gguf -p 128,512 -n 128 -t 4"
adb shell "cd /data/local/tmp/lcpp && ./llama-cli -m llama-3.2-1b-Q4_0.gguf -p 'Explain KV caching in one paragraph.' -n 128 -t 4 -no-cnv"

• Q4_0 is a simple 4-bit format with a scale per 32 weights; on recent Arm CPUs llama.cpp repacks it at load time into layouts that use i8mm/dotprod instructions, which is often the fastest CPU option on phones.
• Q4_K_M uses super-blocks with mixed 4/6-bit precision for sensitive tensors; better quality per byte, a popular default.
• Thread count matters: on big.LITTLE phones, use the number of performance cores (often 4-6); using all cores can be slower because little cores become stragglers.

Path 5: To a QNN context binary (Snapdragon NPU)

Option A, via AI Hub (hosted compile, quantize and profile on real devices):

import torch, qai_hub as hub

model = MyModel().eval()
example = torch.randn(1, 3, 224, 224)
traced = torch.jit.trace(model, example)

device = hub.Device("Samsung Galaxy S24 (Family)")

# Compile straight to a QNN context binary for this SoC
compile_job = hub.submit_compile_job(
    model=traced, device=device,
    input_specs={"image": (1, 3, 224, 224)},
    options="--target_runtime qnn_context_binary",
)
target_model = compile_job.get_target_model()

# Profile on a real device in the cloud: per-layer timing, compute unit per op, memory
profile_job = hub.submit_profile_job(model=target_model, device=device)

# Run inference on-device to check numerics against the PyTorch reference
infer_job = hub.submit_inference_job(model=target_model, device=device,
                                     inputs={"image": [example.numpy()]})
out = infer_job.download_output_data()
target_model.download("model_ctx.bin")

For quantized models, submit a quantize job (INT8 or W4A16/W8A16 style activations) with calibration data before compiling, or export a pre-quantized ONNX (QDQ) model. For supported LLMs, the model zoo has ready export scripts that quantize and split the model into several context binaries:

python -m qai_hub_models.models.llama_v3_2_3b_instruct.export \
    --device "Snapdragon 8 Elite QRD" \
    --skip-inferencing --skip-profiling \
    --output-dir ./genie_bundle_src

Option B, locally with the QAIRT tools. Binary names and flags are release-specific and are the most common source of copy-paste failures. Older QNN SDK releases ship qnn-onnx-converter, qnn-model-lib-generator and qnn-context-binary-generator. Newer QAIRT releases document qairt-converter / qairt-quantizer (exact names: check the SDK bin/ of the version you installed). The sketch below is the older converter shape so you can recognise it; copy flags from that release's HTML docs or --help, do not invent them.

# 1. Convert (and quantize with a calibration input list of raw tensor files)
# Flag names (--act_bitwidth, --weights_bitwidth, --input_list) are from older
# qnn-onnx-converter help text. Confirm on your SDK before running.
qnn-onnx-converter --input_network model.onnx \
    --input_list calib_list.txt \
    --act_bitwidth 16 --weights_bitwidth 8 \
    --output_path model_qnn.cpp

# 2. Build a model library for the target
qnn-model-lib-generator -c model_qnn.cpp -b model_qnn.bin \
    -t x86_64-linux-clang -o model_libs

# 3. Generate an HTP context binary (graph finalized for a specific SoC arch)
qnn-context-binary-generator \
    --backend $QNN_SDK_ROOT/lib/x86_64-linux-clang/libQnnHtp.so \
    --model model_libs/x86_64-linux-clang/libmodel_qnn.so \
    --config_file htp_config.json \
    --binary_file model_ctx

# 4. Run on device from the context binary (no on-device graph compile)
adb push model_ctx.bin $QNN_SDK_ROOT/lib/aarch64-android/libQnnHtp*.so \
         $QNN_SDK_ROOT/lib/hexagon-v75/unsigned/libQnnHtpV75Skel.so \
         $QNN_SDK_ROOT/bin/aarch64-android/qnn-net-run /data/local/tmp/qnn/
adb shell "cd /data/local/tmp/qnn && export LD_LIBRARY_PATH=. ADSP_LIBRARY_PATH=. && \
  ./qnn-net-run --backend libQnnHtp.so --retrieve_context model_ctx.bin \
                --input_list inputs.txt --profiling_level basic"

Option C, via frameworks: the ExecuTorch Qualcomm backend (a QnnPartitioner plus QnnQuantizer, producing a .pte with embedded context binaries) or the ONNX Runtime QNN execution provider with EP context caching (covered in the Android integration section).

Path 6: To Core ML (Apple)

import torch, coremltools as ct

model = MyModel().eval()
example = torch.randn(1, 3, 224, 224)
traced = torch.jit.trace(model, example)          # or torch.export program in newer coremltools

mlmodel = ct.convert(
    traced,
    inputs=[ct.TensorType(name="image", shape=example.shape)],
    convert_to="mlprogram",
    compute_units=ct.ComputeUnit.ALL,              # CPU + GPU + Neural Engine
    minimum_deployment_target=ct.target.iOS17,
    compute_precision=ct.precision.FLOAT16,
)

# Weight compression: 8-bit linear or 4-bit palettization / per-block quantization
from coremltools.optimize.coreml import (OpLinearQuantizerConfig, OptimizationConfig,
                                         linear_quantize_weights)
cfg = OptimizationConfig(global_config=OpLinearQuantizerConfig(mode="linear_symmetric"))
mlmodel = linear_quantize_weights(mlmodel, config=cfg)
mlmodel.save("MyModel.mlpackage")

Use Xcode's Core ML performance report to see which operations actually ran on the Neural Engine; ops that the ANE cannot run (certain shapes, dynamic dimensions, some activation types) fall back to GPU or CPU without any error.

Making a model exportable

1. Remove Python-level dynamism Replace data-dependent if branches with torch.cond or tensor operations; make loops fixed-length.
2. Fix shapes Pick concrete input sizes (and for LLMs, fixed prefill chunk sizes such as 128 plus a decode length of 1). Use torch.export.Dim only where the target runtime truly supports dynamic dimensions.
3. Replace unsupported ops Rewrite custom or exotic operators with standard ones (for example GELU-tanh approximation, explicit RMSNorm, attention written as matmul + softmax).
4. Keep the KV cache explicit For LLMs, make the cache an input/output or registered buffer with a fixed maximum length rather than a growing Python list.
5. Validate parity After every conversion, run the same inputs through the original and converted model; FP32 differences should be around 1e-5 to 1e-4.

Integrating into an Android app

A model file is not a feature. Integration means choosing where inference runs (Kotlin via a runtime's Java API, or native C++ via JNI), how the model gets onto the device and into memory, which accelerator executes it, and how the app behaves when things go wrong. For Android framework background (services, lifecycles, threading) see Android frameworks.

Architecture of an on-device inference feature

+---------------------------- App process ------------------------------+
|  UI (Compose/View)  --▶  ViewModel  --▶  InferenceRepository          |
|                                   |  (single background dispatcher)   |
|                                   ▼                                    |
|              Kotlin runtime API  or  JNI bridge ──▶ C++ engine        |
|              (LiteRT / ORT /        (ExecuTorch, llama.cpp, QNN,      |
|               ExecuTorch / MediaPipe)  custom pre/post-processing)    |
|                                   |                                    |
|        pre-processing ──▶ model ──▶ post-processing ──▶ result         |
+-----------------------------------|------------------------------------+
                                    ▼
      Delegate / backend:  NPU (HTP, APU, TPU)  |  GPU (OpenCL/Vulkan)  |  CPU (XNNPACK/KleidiAI)
                                    ▼
      Model file: mmap from app storage (downloaded) or uncompressed asset

Option 1: LiteRT Interpreter with GPU delegate and CPU fallback (Kotlin)

// build.gradle.kts
// Artifact IDs moved from org.tensorflow:tensorflow-lite* to com.google.ai.edge.litert:*.
// Confirm the current Maven coordinates and the Java package they export in the
// official LiteRT Android docs for the version you pin.
dependencies {
    implementation("com.google.ai.edge.litert:litert:<version>")
    implementation("com.google.ai.edge.litert:litert-gpu:<version>")
}
android {
    androidResources { noCompress += listOf("tflite") }   // required to mmap from assets
}

// Interpreter is the long-lived, widely documented path. Some LiteRT artifacts
// still export org.tensorflow.lite.* for compatibility; newer packages may not.
import org.tensorflow.lite.Interpreter
import org.tensorflow.lite.gpu.CompatibilityList
import org.tensorflow.lite.gpu.GpuDelegate
import java.io.FileInputStream
import java.nio.MappedByteBuffer
import java.nio.channels.FileChannel

class Classifier(context: Context) : AutoCloseable {
    private var gpu: GpuDelegate? = null
    private val interpreter: Interpreter

    init {
        val options = Interpreter.Options()
        val compat = CompatibilityList()
        if (compat.isDelegateSupportedOnThisDevice) {
            gpu = GpuDelegate(compat.bestOptionsForThisDevice)
            options.addDelegate(gpu)
        } else {
            options.setNumThreads(4)          // XNNPACK CPU path
        }
        interpreter = Interpreter(mapAsset(context, "model_int8.tflite"), options)
    }

    private fun mapAsset(ctx: Context, name: String): MappedByteBuffer {
        ctx.assets.openFd(name).use { fd ->
            FileInputStream(fd.fileDescriptor).channel.use { ch ->
                return ch.map(FileChannel.MapMode.READ_ONLY, fd.startOffset, fd.declaredLength)
            }
        }
    }

    // Reuse buffers: allocate once, fill per call
    private val input = ByteBuffer.allocateDirect(1 * 224 * 224 * 3).order(ByteOrder.nativeOrder())
    private val output = Array(1) { ByteArray(1000) }

    fun classify(pixels: ByteArray): ByteArray {
        input.rewind(); input.put(pixels)
        interpreter.run(input, output)       // call from a background thread only
        return output[0]
    }

    override fun close() { interpreter.close(); gpu?.close() }
}

LiteRT Next also documents a CompiledModel-style API that binds a model to an accelerator (CPU, GPU, or an NPU compiler plugin) and can consume SoC-specific compiled artifacts. Class names, packages, accelerator enums and compile options change between releases. Check the current official LiteRT docs before writing production code; do not assume Interpreter.Options maps one-to-one onto CompiledModel, and do not copy guessed method names from older blog posts. The concepts stay the same: create once, choose an accelerator, reuse buffers, fall back gracefully.

Option 2: ONNX Runtime Mobile with the QNN execution provider

// build.gradle.kts: use the onnxruntime-android-qnn package for the QNN EP
dependencies { implementation("com.microsoft.onnxruntime:onnxruntime-android-qnn:<version>") }

import ai.onnxruntime.*

val env = OrtEnvironment.getEnvironment()
val opts = OrtSession.SessionOptions().apply {
    // Java helper name (addQnn vs addConfigEntry) and option keys are
    // version-specific. Confirm against the current ONNX Runtime QNN EP docs.
    // Common documented keys include backend_path and an HTP performance mode.
    addQnn(mapOf(
        "backend_path" to "libQnnHtp.so",
        "htp_performance_mode" to "burst"
        // Other documented modes include sustained_high_performance; do not
        // invent extra flags. Graph-finalization and context-cache keys also
        // move between ORT releases — look them up for the AAR you ship.
    ))
    addConfigEntry("ep.context_enable", "1")
    addConfigEntry("ep.context_file_path", File(context.filesDir, "model_ctx.onnx").path)
    addConfigEntry("session.disable_cpu_ep_fallback", "1")
}
val session = env.createSession(modelFile.path, opts)

val input = OnnxTensor.createTensor(env, floatBuffer, longArrayOf(1, 3, 224, 224))
session.run(mapOf("image" to input)).use { result ->
    val logits = (result[0].value as Array<FloatArray>)[0]
}

• The QNN HTP backend requires a quantized (QDQ) model for NPU execution; FP32 models either use the HTP's FP16 path on newer chips or fall back.
• Turn off CPU fallback in development so partitioning problems surface as errors; turn it back on in production for robustness.
• EP context caching avoids re-finalizing the graph on every launch, which can take seconds for large models.

Option 3: MediaPipe LLM Inference API (fastest LLM integration)

// dependencies { implementation("com.google.mediapipe:tasks-genai:<version>") }
// Engine options vs session options (temperature / top-k / LoRA) were split
// across releases. Check the current MediaPipe Tasks GenAI docs and the
// LiteRT-LM notes; method names below are the long-lived engine-level ones.
import com.google.mediapipe.tasks.genai.llminference.LlmInference

val options = LlmInference.LlmInferenceOptions.builder()
    .setModelPath(File(context.filesDir, "models/gemma.task").path)
    .setMaxTokens(1024)                 // prompt + output budget: sizes the KV cache
    .build()
val llm = LlmInference.createFromOptions(context, options)   // expensive: do once, off main thread

val answer = llm.generateResponse("Summarise this note in two lines: ...")

llm.generateResponseAsync(prompt) { partial, done ->
    uiFlow.tryEmit(partial)
    if (done) onFinished()
}

This path only loads models converted into its bundle format (with tokenizer and metadata). It is ideal when a supported model family fits the product; for arbitrary architectures, use ExecuTorch or llama.cpp.

Option 4: ExecuTorch from Kotlin

// Packages and constructors move between ExecuTorch Android releases
// (Module vs ExecutorchModule, LlmModule constructor args). Confirm against
// the current ExecuTorch Android / LLM runner docs for the AAR you ship.
import org.pytorch.executorch.EValue
import org.pytorch.executorch.Module
import org.pytorch.executorch.Tensor

val module = Module.load(File(filesDir, "model_xnnpack_int8.pte").path)   // mmap-based loader
val input = Tensor.fromBlob(floatArray, longArrayOf(1, 3, 224, 224))
val out = module.forward(EValue.from(input))[0].toTensor().dataAsFloatArray

import org.pytorch.executorch.extension.llm.LlmCallback
import org.pytorch.executorch.extension.llm.LlmModule

val llm = LlmModule(ptePath, tokenizerPath, /* temperature = */ 0.7f)
llm.load()
llm.generate(prompt, /* seqLen = */ 512, object : LlmCallback {
    override fun onResult(token: String) { appendToUi(token) }
    override fun onStats(stats: String) { Log.i("LLM", stats) }
})

Option 5: your own C++ engine through JNI

Native integration gives you full control (custom pre/post-processing, direct QNN or Genie calls, llama.cpp, zero-copy buffers). Keep the JNI surface small and coarse-grained: one call per request, not one per tensor.

// Kotlin side
object NativeEngine {
    init { System.loadLibrary("edgeengine") }
    external fun create(modelPath: String, backend: Int): Long      // returns a handle
    external fun generate(handle: Long, prompt: String, maxTokens: Int, cb: TokenCallback): String
    external fun destroy(handle: Long)
}
fun interface TokenCallback { fun onToken(piece: String) }

// C++ side: engine_jni.cpp
#include <jni.h>
#include <memory>
#include <string>
#include "engine.h"          // your wrapper around ExecuTorch / llama.cpp / QNN

extern "C" JNIEXPORT jlong JNICALL
Java_com_example_edge_NativeEngine_create(JNIEnv* env, jobject, jstring jpath, jint backend) {
    const char* path = env->GetStringUTFChars(jpath, nullptr);
    auto* engine = new Engine(path, static_cast<Backend>(backend));   // loads + mmaps weights
    env->ReleaseStringUTFChars(jpath, path);
    return reinterpret_cast<jlong>(engine);
}

extern "C" JNIEXPORT jstring JNICALL
Java_com_example_edge_NativeEngine_generate(JNIEnv* env, jobject, jlong h, jstring jprompt,
                                            jint maxTokens, jobject cb) {
    auto* engine = reinterpret_cast<Engine*>(h);
    const char* p = env->GetStringUTFChars(jprompt, nullptr);
    std::string prompt(p);
    env->ReleaseStringUTFChars(jprompt, p);

    jclass cls = env->GetObjectClass(cb);
    jmethodID onToken = env->GetMethodID(cls, "onToken", "(Ljava/lang/String;)V");

    std::string full;
    engine->generate(prompt, maxTokens, [&](const std::string& piece) {
        full += piece;
        jstring js = env->NewStringUTF(piece.c_str());
        env->CallVoidMethod(cb, onToken, js);
        env->DeleteLocalRef(js);            // avoid local-ref table overflow in long loops
    });
    return env->NewStringUTF(full.c_str());
}

extern "C" JNIEXPORT void JNICALL
Java_com_example_edge_NativeEngine_destroy(JNIEnv*, jobject, jlong h) {
    delete reinterpret_cast<Engine*>(h);
}

# CMakeLists.txt (app/src/main/cpp)
cmake_minimum_required(VERSION 3.22)
project(edgeengine CXX)
set(CMAKE_CXX_STANDARD 17)
add_library(edgeengine SHARED engine_jni.cpp engine.cpp)
target_compile_options(edgeengine PRIVATE -O3 -march=armv8.2-a+dotprod+fp16)
# Link prebuilt runtime libs (ExecuTorch, llama, or QNN) imported as IMPORTED targets
target_link_libraries(edgeengine executorch_llm_runner log android)

• Token-by-token callbacks into Java cost microseconds each; fine for tens of tokens per second, but batch pieces if you stream very fast.
• Call JNI from a thread that was attached to the JVM; callbacks from a native worker thread need AttachCurrentThread.
• Vendor NPU libraries must be packaged in jniLibs/arm64-v8a (or extracted to app storage) and the DSP library search path (ADSP_LIBRARY_PATH) must point to where the Hexagon skeleton libraries live; set it before initializing the backend.
• Set useLegacyPackaging or extractNativeLibs according to whether your runtime can load libraries directly from the APK.

Threading and lifecycle rules

1. Never on the main thread Use a single dedicated background dispatcher or thread for inference; most interpreters are not thread-safe for concurrent calls.
2. Create once, reuse Loading, delegate initialization and graph compilation can take hundreds of milliseconds to seconds. Keep the session in an application-scoped object.
3. Warm up Run one dummy inference after load, while the user is not waiting, to trigger shader compilation and cache fills.
4. Release on memory pressure Respond to onTrimMemory by freeing caches and, for large models, unloading entirely when backgrounded.
5. Cancel LLM generation must be cancellable (a flag checked every token) so leaving the screen stops work and power draw.

Model packaging and delivery

A robust self-managed download flow:

// Pseudocode for a WorkManager download worker
1. Fetch manifest: {modelId, version, url, sha256, sizeBytes, minRamMb, socAllowList, runtimeVersion}
2. Check eligibility: RAM tier, SoC, free storage (size x 2 for safety), unmetered network, charging
3. Download to  files/models/.tmp/<id>-<version>.part  with HTTP Range resume
4. Verify SHA-256 (and signature if the model is sensitive)
5. Atomically rename to files/models/<id>/<version>/model.pte
6. Update "active version" pointer only after a smoke-test inference succeeds
7. Keep the previous version until the new one has run successfully N times; then delete

Memory-mapped weights

Memory mapping (mmap) makes the model file appear as memory without copying it into the heap. Pages are read from storage on first touch and can be dropped by the kernel under pressure because they are clean and file-backed.

Two subtleties: first, if the runtime repacks weights into a kernel-friendly layout at load time (common for CPU kernels and GPU delegates), the repacked copy is anonymous memory and the mmap benefit shrinks; second, if pages are evicted during generation, decode stalls on storage reads, which appears as sudden latency spikes. For latency-critical paths, touch all pages after load (prefault) or use mlock where permitted, and budget memory honestly.

Project 1: baseline and benchmark harness

Duration: 2-3 weeks. Goal: an LLM running on your phone plus a measurement harness that every later project reuses. Skills: latency, memory and power profiling; token-rate measurement; PyTorch export tooling; accelerator basics. Output: a repository containing the harness and a sustained-load thermal analysis write-up.

What you build

Run Llama-3.2-1B-Instruct (or an even smaller model such as a 0.5-0.6B one for a quicker first win) on Android through ExecuTorch with the XNNPACK backend, and in parallel through llama.cpp as a cross-check. Then build a harness that records, on every run:

Step 1: export, push and run

# Export (see the ExecuTorch path above), then:
adb shell mkdir -p /data/local/tmp/llama
adb push llama3_2_1b_8da4w.pte  /data/local/tmp/llama/
adb push tokenizer.model        /data/local/tmp/llama/
adb push cmake-out-android/examples/models/llama/llama_main /data/local/tmp/llama/

adb shell "cd /data/local/tmp/llama && ./llama_main \
    --model_path=llama3_2_1b_8da4w.pte \
    --tokenizer_path=tokenizer.model \
    --prompt='Explain KV caching in one paragraph.' \
    --seq_len=256 --cpu_threads=4 --warmup=1"
# The runner prints prompt/generated token counts and timing stats
# (model load, first token latency, prefill and generation rates).

For an instruct model, wrap the prompt in the model's chat template (for Llama 3.x: <|begin_of_text|><|start_header_id|>user<|end_header_id|> ... <|eot_id|><|start_header_id|>assistant<|end_header_id|>). A missing template is the most common cause of rambling or empty output.

Step 2: define a benchmark protocol

1. Fix the environment Same device, OS build, airplane mode, fixed screen brightness (or screen off via a CLI run), battery 50-90% or external supply, device at room temperature with the case removed, no other apps, same prompt set.
2. Cool down between configurations Wait until the SoC thermal zone returns near its idle temperature (for example within 2 degrees of baseline) before the next configuration.
3. Warm-up Discard 1-3 full generations (or 10-50 runs for small models) so caches, page faults and kernel selection settle. Record load time and first-run latency separately.
4. Repeat At least 10 generations for LLMs, 100-1000 inferences for small models; report median and p90/p99, plus standard deviation.
5. Vary prompt length Measure prefill at several lengths (for example 64, 256, 1024 tokens) because attention cost and cache behaviour change with length.
6. Record everything Device, SoC, build fingerprint, runtime version, model hash, quantization config, thread count, ambient temperature, start/end temperatures.

Step 3: the harness (host-side driver)

#!/usr/bin/env python3
"""bench.py - drive an on-device LLM runner over adb and collect metrics into CSV."""
import csv, json, re, statistics, subprocess, time, datetime

DEV_DIR = "/data/local/tmp/llama"
RUNNER = f"cd {DEV_DIR} && ./llama_main --model_path={{pte}} --tokenizer_path=tokenizer.model " \
         "--prompt=\"{prompt}\" --seq_len={seq} --cpu_threads={threads}"

def adb(cmd: str) -> str:
    return subprocess.run(["adb", "shell", cmd], capture_output=True, text=True, timeout=900).stdout

def soc_temp_c() -> float:
    # pick the zone(s) that represent CPU/SoC on your device; names vary by vendor
    out = adb("for z in /sys/class/thermal/thermal_zone*; do echo $(cat $z/type) $(cat $z/temp); done")
    temps = [int(t) / 1000 for name, t in (l.split() for l in out.splitlines() if l.strip())
             if re.search(r"cpu|soc|tsens", name, re.I) and t.lstrip("-").isdigit()]
    return max(temps) if temps else float("nan")

def wait_cool(target_c: float, timeout_s=600):
    t0 = time.time()
    while soc_temp_c() > target_c and time.time() - t0 < timeout_s:
        time.sleep(10)

def parse_stats(text: str) -> dict:
    # Adapt these regexes to your runner's output format
    pats = {
        "load_ms":    r"Model load time:\s*([\d.]+)",
        "ttft_ms":    r"(?:Time to first generated token|first token).*?([\d.]+)",
        "prefill_tps": r"Prompt evaluation:.*?([\d.]+)\s*tokens/s",
        "decode_tps":  r"Generated \d+ tokens:.*?([\d.]+)\s*tokens/s",
    }
    return {k: float(m.group(1)) if (m := re.search(p, text, re.S)) else None for k, p in pats.items()}

def peak_rss_kb(proc_name="llama_main") -> int:
    out = adb(f"pid=$(pidof {proc_name}); [ -n \"$pid\" ] && grep VmHWM /proc/$pid/status")
    m = re.search(r"(\d+)", out)
    return int(m.group(1)) if m else -1

def run_config(pte, prompt, seq=256, threads=4, warmup=2, reps=10, idle_c=None):
    if idle_c: wait_cool(idle_c + 2)
    cmd = RUNNER.format(pte=pte, prompt=prompt, seq=seq, threads=threads)
    for _ in range(warmup):
        adb(cmd)
    rows = []
    for i in range(reps):
        t_start = soc_temp_c()
        text = adb(cmd)
        s = parse_stats(text)
        s.update(rep=i, temp_start=t_start, temp_end=soc_temp_c(), pte=pte, threads=threads)
        rows.append(s)
    return rows

def summarize(rows, key):
    xs = sorted(r[key] for r in rows if r.get(key) is not None)
    if not xs: return {}
    pct = lambda p: xs[min(len(xs) - 1, int(round(p / 100 * (len(xs) - 1))))]
    return {"p50": pct(50), "p90": pct(90), "p99": pct(99),
            "mean": statistics.mean(xs), "stdev": statistics.pstdev(xs)}

if __name__ == "__main__":
    meta = {"fingerprint": adb("getprop ro.build.fingerprint").strip(),
            "soc": adb("getprop ro.soc.model").strip(),
            "time": datetime.datetime.now().isoformat()}
    idle = soc_temp_c()
    rows = run_config("llama3_2_1b_8da4w.pte", "Explain KV caching in one paragraph.", idle_c=idle)
    with open("results.csv", "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=rows[0].keys()); w.writeheader(); w.writerows(rows)
    print(json.dumps({"meta": meta, **{k: summarize(rows, k)
                      for k in ("ttft_ms", "prefill_tps", "decode_tps")}}, indent=2))

Note: VmHWM in /proc/<pid>/status is the peak resident set ("high water mark"); poll it while the process runs or print it from the runner itself at exit, because it disappears with the process. For app-based runs, dumpsys meminfo <package> gives PSS broken down into native heap, graphics, and file mappings.

Step 4: in-app timing (C++ side)

#include <chrono>
#include <vector>
#include <algorithm>

using Clock = std::chrono::steady_clock;            // monotonic; never use wall clock

struct GenStats { double load_ms, ttft_ms, prefill_tps, decode_tps; std::vector<double> itl_ms; };

GenStats timed_generate(Engine& e, const std::vector<int>& prompt, int max_new) {
    GenStats s{};
    auto t0 = Clock::now();
    e.prefill(prompt);                                // fills KV cache, returns logits
    int tok = e.sample();
    auto t1 = Clock::now();
    s.ttft_ms = std::chrono::duration<double, std::milli>(t1 - t0).count();
    s.prefill_tps = prompt.size() / (s.ttft_ms / 1000.0);

    auto prev = t1;
    for (int i = 1; i < max_new && tok != e.eos(); ++i) {
        e.decode(tok);
        tok = e.sample();
        auto now = Clock::now();
        s.itl_ms.push_back(std::chrono::duration<double, std::milli>(now - prev).count());
        prev = now;
    }
    double decode_s = std::chrono::duration<double>(prev - t1).count();
    s.decode_tps = s.itl_ms.size() / decode_s;
    return s;
}

double percentile(std::vector<double> v, double p) {
    std::sort(v.begin(), v.end());
    size_t idx = static_cast<size_t>(p / 100.0 * (v.size() - 1) + 0.5);
    return v[idx];
}

Step 5: power and energy

Energy is what users feel as battery drain. Three levels of rigour:

# Fuel gauge sampling (units are usually microamps / microvolts; sign convention varies by device)
adb shell "while true; do echo $(date +%s%N) $(cat /sys/class/power_supply/battery/current_now) \
  $(cat /sys/class/power_supply/battery/voltage_now); sleep 0.2; done" > power.log

# batterystats
adb shell dumpsys batterystats --reset
adb shell dumpsys battery unplug          # simulate unplugged so stats accumulate over USB
# ... run workload ...
adb shell dumpsys batterystats > batterystats.txt
adb shell dumpsys battery reset

Step 6: the thermal soak (the differentiating measurement)

Most published numbers are a single cold run. Instead, generate continuously for 10-20 minutes and plot throughput, temperature, CPU frequency and power over time. This reveals when throttling begins, how far throughput falls, and the sustained-to-peak ratio, which is the number that actually matters for a shipped feature. For the thermal framework itself see Power and thermal.

# thermal_soak.sh - run generations back-to-back and sample system state every 2 s
DUR=${1:-600}
adb shell "cd /data/local/tmp/llama; end=\$((\$(date +%s)+$DUR)); \
  while [ \$(date +%s) -lt \$end ]; do ./llama_main --model_path=llama3_2_1b_8da4w.pte \
  --tokenizer_path=tokenizer.model --prompt='Write a long story about a lighthouse.' \
  --seq_len=512 --cpu_threads=4 2>&1 | grep -i 'tokens/s'; done" > soak_tps.log &

while kill -0 $! 2>/dev/null; do
  ts=$(date +%s)
  temps=$(adb shell "for z in /sys/class/thermal/thermal_zone*; do echo -n \$(cat \$z/type)=\$(cat \$z/temp),; done")
  freqs=$(adb shell "cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq | tr '\n' ,")
  status=$(adb shell "dumpsys thermalservice | grep -m1 'Thermal Status'")
  echo "$ts;$temps;$freqs;$status" >> soak_sys.log
  sleep 2
done

decode tok/s
  45 |■■■■■■■■■■■
  40 |           ■■■■■                          peak  (first 60-120 s)
  35 |                ■■■■■
  30 |                     ■■■■■■■■■■■■■■■■■■■■■  sustained plateau
  25 |
     +-----+-----+-----+-----+-----+-----+-----+--▶ time (min)
     0     1     2     3     4     5     6     10
  SoC temp rises ~15-25 °C; big-core frequency steps down; ratio sustained/peak ≈ 0.6-0.8 (typical, device-dependent)

Step 7: compare BF16 and INT4

Export the same model twice (BF16/FP32 and 8da4w INT4) and fill a comparison table on all metrics: load time, TTFT, prefill tok/s, decode tok/s, peak RSS, file size, energy per 100 tokens and sustained ratio. Expect roughly 2-3x faster decode, 3-4x faster prefill (with KleidiAI), about 50-60% smaller file and 30-40% lower peak memory for the quantized model. Deviations beyond that mean something is misconfigured (see the target numbers section).

Deliverables checklist

• Model exported and running on device with coherent output.
• Harness scripted so a full measurement run is one command, producing a CSV plus a summary.
• BF16 vs INT4 comparison across all metrics.
• Ten-minute sustained-load curve with temperature and frequency overlay.
• Numbers sanity-checked against reference values.
• Short write-up of what surprised you, with the raw data committed.

Project 2: quantization sweep and layer-wise error analysis

Duration: 3-4 weeks. Goal: understand quantization numerically and build the tool that finds where it breaks. Skills: PTQ vs QAT, weight and activation quantization, mixed precision, layer-wise mismatch, host reference vs on-device validation, fixed-point representation. Output: a layer-wise activation diff tool plus a sweep results table. For the theory of scales, zero-points and granularity, see Edge AI.

Part A: the configuration sweep

Take one model (Llama-3.2-1B is ideal) and walk the configuration space, measuring both quality and cost every time. Record every result in one CSV; the sweep table is itself a useful artifact.

Measure quality two ways: perplexity on held-out text (statistical degradation) and at least one task-level score (for example a small multiple-choice benchmark, or exact-match on your product's prompts) for behavioural degradation. Perplexity can move very little while a task collapses, and vice versa.

INT8 versus INT4, what to expect before you sweep. INT8 (or FP8) weights are about 2× smaller than FP16 and usually near-lossless after PTQ; that is the control row. INT4 weight-only is about 4× smaller and is the decode-speed lever, but quality loss is larger and shows first on reasoning, maths, rare languages and long context. Keep embeddings and the LM head at 6-8 bits in the first INT4 recipe. Two "4-bit" rows that differ in group size or algorithm are not comparable; write the full recipe on every CSV line.

# Sweep driver sketch (ExecuTorch exporter; one row per config)
for bits in 4 8; do for gs in 32 128 256; do for emb in "none" "torchao:8,0" "torchao:4,32"; do
  name="l32_1b_w${bits}_g${gs}_emb${emb//[:,]/_}"
  python -m extension.llm.export.export_llm base.model_class=llama3_2 \
    base.checkpoint=$CKPT base.params=$PARAMS model.use_kv_cache=True \
    model.use_sdpa_with_kv_cache=True quantization.qmode="torchao:8da${bits}w" \
    quantization.group_size=$gs quantization.embedding_quantize="$emb" \
    backend.xnnpack.enabled=True export.output_name="$name.pte"
  python eval_ppl.py --pte $name.pte --data heldout.txt >> sweep.csv   # host-side quality
  python bench.py --pte $name.pte >> sweep.csv                         # device-side cost
done; done; done

Part B: layer-wise error analysis (the core skill)

Analysing quantization error and layer-wise mismatches is a named responsibility in many edge AI roles, and very few candidates can demonstrate it. The method:

1. Reference run Run the FP32 model on the host (x86 or Arm workstation) and capture the output of every layer boundary (attention projections, attention output, MLP gate/up/down, norms, residual stream after each block).
2. Quantized run Run the quantized model on the same inputs and capture the same tensors: first as a simulated (fake-quantized) model on the host, then from the device.
3. Diff per tensor Compute cosine similarity, mean and max absolute error, and SQNR for each captured tensor.
4. Rank and plot Rank layers by divergence; plot SQNR across depth to see whether error compounds or recovers.
5. Isolate Quantize one layer at a time (everything else FP32) to measure each layer's individual sensitivity; this separates "this layer is fragile" from "this layer receives bad inputs".
6. Derive a recipe Keep the most sensitive layers at higher precision, re-run the sweep and the Project 1 harness, and quantify quality recovered versus latency and memory cost.

Host-side tool: capture and diff with forward hooks

import torch, math, csv
from collections import OrderedDict

def capture(model, inputs, names_filter=lambda n, m: isinstance(m, torch.nn.Linear)
                                              or "norm" in n.lower() or n.endswith("layers")):
    acts, hooks = OrderedDict(), []
    for name, mod in model.named_modules():
        if names_filter(name, mod):
            hooks.append(mod.register_forward_hook(
                lambda m, i, o, name=name: acts.__setitem__(name, (o[0] if isinstance(o, tuple) else o)
                                                             .detach().float().cpu())))
    with torch.no_grad():
        model(*inputs)
    for h in hooks: h.remove()
    return acts

def metrics(ref, q):
    ref, q = ref.flatten(), q.flatten()
    err = ref - q
    sqnr = 10 * math.log10((ref.pow(2).sum() / err.pow(2).sum().clamp_min(1e-20)).item())
    cos = torch.nn.functional.cosine_similarity(ref, q, dim=0).item()
    return {"sqnr_db": sqnr, "cosine": cos,
            "mae": err.abs().mean().item(), "max_abs": err.abs().max().item(),
            "ref_absmax": ref.abs().max().item()}        # large absmax hints at outliers

ref_acts = capture(fp32_model, (tokens,))
q_acts   = capture(fake_quant_model, (tokens,))          # e.g. torchao-quantized copy

rows = [{"layer": n, **metrics(ref_acts[n], q_acts[n])} for n in ref_acts if n in q_acts]
rows.sort(key=lambda r: r["sqnr_db"])                    # worst first
with open("layerwise.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=rows[0].keys()); w.writeheader(); w.writerows(rows)
for r in rows[:10]:
    print(f'{r["layer"]:50s} SQNR {r["sqnr_db"]:6.1f} dB  cos {r["cosine"]:.5f}')

Single-layer sensitivity sweep (the "isolate" step):

def per_layer_sensitivity(build_model, layer_names, quantize_only, eval_fn):
    base = eval_fn(build_model())                      # FP32 score (e.g. perplexity)
    out = []
    for name in layer_names:
        m = quantize_only(build_model(), {name})       # quantize a single layer
        out.append((name, eval_fn(m) - base))          # degradation attributable to this layer
    return sorted(out, key=lambda t: -t[1])

Capturing intermediate tensors on the device

Comparing host-simulated quantization with the device run is itself informative. If fake-quant on the host matches FP32 well but the device diverges, the problem is not the quantization scheme but the device implementation: different rounding, accumulator width, requantization, fused operator behaviour, or an FP16 overflow in a GPU/NPU kernel.

Typical findings to look for

Mixed-precision fallback recipe

1. Start uniform Everything at the target precision (for example W4 group 32/128, A8 dynamic).
2. Rank Use the single-layer sensitivity list, not just the full-model diff.
3. Promote greedily Move the top-k layers to W8 (or FP16) one at a time until the quality target is met.
4. Price it Each promoted layer costs size and decode time in proportion to its parameter share; report the Pareto curve of quality vs size vs latency.
5. Validate on device Re-run the Project 1 harness and the device-side diff; mixed precision can split graphs on NPUs if the backend cannot run both precisions in one partition.

Also explore

• SpinQuant and QAT + LoRA: the two quantization-aware recipes with published Llama 3.2 numbers. Compare their quality against your PTQ results at equal bit-width.
• GPTQ / AWQ: weight-only PTQ that uses calibration data to minimise output error or protect salient channels.
• AIMET techniques: cross-layer equalisation, bias correction and AdaRound for CNN-style models on NPUs.
• Operator fusion: inspect the exported graph before and after lowering (for example with a graph visualiser or the printed edge program) and understand which ops fused and why fused quantized kernels change numerics slightly.
• Fixed-point representation: verify by hand how an INT8 x INT8 multiply accumulates into INT32 and requantizes with a multiplier and shift; this is exactly what NPU kernels do.

Deliverables checklist

• Full sweep table: quality, latency, size and memory for every configuration.
• Layer-wise diff tool working for host reference vs host fake-quant vs device.
• Ranked list of the most sensitive layers with a hypothesis for each.
• Evidence-derived mixed-precision recipe validated on the Project 1 harness.
• PTQ vs rotation-based vs QAT comparison at equal bit-width.

Project 3: NPU deployment on vendor silicon

Duration: 4-5 weeks. Goal: a model running on the NPU (Snapdragon Hexagon HTP as the worked example) integrated into an Android app through the native C API. Skills: vendor SDK (QAIRT/QNN), HTP deployment, static quantization, model splitting, operator support gaps. Output: working NPU inference plus an honest CPU vs GPU vs NPU comparison including power. This is the hardest project and the most valuable, because few engineers outside chipset vendors have done a full NPU deployment. Expect SDK version mismatches, unsupported operators and cryptic errors; debugging them is the curriculum.

The constraint that changes everything: static quantization

The Hexagon NPU is built around fixed-point arithmetic with quantization parameters known at compile time. For LLMs the typical scheme is 4-bit (or 8-bit) weights with 16-bit static activations (W4A16 / W8A16); for CNNs, full INT8. On the CPU path in Projects 1-2, activations were quantized dynamically: scales computed at runtime from each tensor's actual range. The NPU cannot afford a data-dependent range computation per tensor per step, so ranges must come from calibration data. If production inputs exceed the calibrated range, values clip. Understanding why the NPU needs this, and measuring what it costs in accuracy, is the core lesson of the project.

Step 1: prepare the model (AI Hub path)

pip install qai-hub-models
qai-hub configure --api_token <YOUR_TOKEN>

# Export a supported LLM to QNN context binaries, quantized and split for the NPU
python -m qai_hub_models.models.llama_v3_2_3b_instruct.export \
    --device "Snapdragon 8 Elite QRD" \
    --skip-inferencing --skip-profiling \
    --output-dir ./export_8_elite
# If you produced a custom quantized checkpoint in Project 2, pass it through
# with the export script's checkpoint option instead of the default weights.

For the 1B model, use the corresponding 1B export if available in the zoo, or go through the ExecuTorch Qualcomm backend, which has its own Llama export scripts that produce a .pte with embedded HTP context binaries.

Why the model is split into several binaries

• The NPU process has limits on how much memory a single graph and its mapped weights can use; large models are therefore partitioned by layers into several context binaries executed in sequence.
• Prefill and decode are compiled as separate graphs with different static shapes (for example 128-token chunks for prefill, 1 token for decode) that share weights ("weight sharing") so memory is not doubled.
• The KV cache is managed outside the graphs as input/output tensors of fixed maximum context length; the runtime updates it between calls.

Step 2: assemble the Genie bundle

mkdir -p genie_bundle

# 1. Model config for Genie (start from the sample config for your model in the vendor tutorials)
cp <tutorial-configs>/llama_v3_2_3b.json genie_bundle/genie_config.json
cp <tutorial-configs>/htp_backend_ext_config.json genie_bundle/

# 2. Hexagon DSP-side libraries: MUST match the SoC architecture
cp "$QNN_SDK_ROOT"/lib/hexagon-v73/unsigned/* genie_bundle/   # Snapdragon 8 Gen 2
# cp "$QNN_SDK_ROOT"/lib/hexagon-v75/unsigned/* genie_bundle/ # 8 Gen 3
# cp "$QNN_SDK_ROOT"/lib/hexagon-v79/unsigned/* genie_bundle/ # 8 Elite

# 3. Android (ARM) side libraries and the CLI runner
cp "$QNN_SDK_ROOT"/lib/aarch64-android/* genie_bundle/
cp "$QNN_SDK_ROOT"/bin/aarch64-android/genie-t2t-run genie_bundle/

# 4. Context binaries and tokenizer from the export step
cp export_8_elite/*.bin export_8_elite/tokenizer.json genie_bundle/

# 5. Verify: every file listed in ctx-bins[] of genie_config.json exists in the bundle

An abridged Genie dialog config. Field names (ctx-bins, kv-dim, sampler keys) vary between SDK versions. Always start from the sample JSON shipped for your model in that QAIRT / AI Hub export, then change only paths and context length.

{
  "dialog": {
    "type": "basic",
    "context":  { "size": 4096, "n-vocab": 128256, "bos-token": 128000, "eos-token": [128001, 128009] },
    "sampler":  { "seed": 42, "temp": 0.7, "top-k": 40, "top-p": 0.95 },
    "tokenizer": { "path": "tokenizer.json" },
    "engine": {
      "n-threads": 3,
      "backend": {
        "type": "QnnHtp",
        "QnnHtp": { "use-mmap": true, "poll": true, "kv-dim": 128 },
        "extensions": "htp_backend_ext_config.json"
      },
      "model": {
        "type": "binary",
        "binary": { "ctx-bins": ["model_part_1_of_3.bin", "model_part_2_of_3.bin", "model_part_3_of_3.bin"] }
      }
    }
  }
}

Step 3: run on device

adb push genie_bundle /data/local/tmp/
adb shell
  cd /data/local/tmp/genie_bundle
  export LD_LIBRARY_PATH=$PWD          # ARM-side libraries
  export ADSP_LIBRARY_PATH=$PWD        # DSP-side (Hexagon skeleton) libraries
  ./genie-t2t-run -c genie_config.json \
    -p "<|begin_of_text|><|start_header_id|>user<|end_header_id|>

Explain NPU static quantization.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

"

Step 4: integrate the C API into an app

Genie wraps the tokenizer, the QNN backend, KV-cache management, decoding and sampling behind a dialog API; the engine executes forward passes on the HTP (with CPU for the rest). Wrapping it in JNI gives you the complete path: PyTorch checkpoint to quantized model to context binaries to NPU to Android UI. The C symbols below match the public dialog-API shape (JSON config, create, query with a callback, free). Header names, status codes, callback signatures and JSON keys are release-specific — copy them from the QAIRT samples for the version you link. Do not invent extra flags or config fields.

#include "GenieDialog.h"
#include <fstream>
#include <sstream>
#include <string>

struct Ctx { std::string out; void (*on_piece)(const char*, void*); void* user; };

static void on_response(const char* piece, const GenieDialog_SentenceCode_t code, const void* ud) {
    auto* c = static_cast<Ctx*>(const_cast<void*>(ud));
    if (piece) { c->out += piece; if (c->on_piece) c->on_piece(piece, c->user); }
    (void)code;   // BEGIN / CONTINUE / END / COMPLETE / ABORT
}

class NpuLlm {
    GenieDialogConfig_Handle_t cfg_ = nullptr;
    GenieDialog_Handle_t dlg_ = nullptr;
public:
    bool init(const std::string& config_path) {
        std::ifstream f(config_path); std::stringstream ss; ss << f.rdbuf();
        if (GenieDialogConfig_createFromJson(ss.str().c_str(), &cfg_) != GENIE_STATUS_SUCCESS) return false;
        return GenieDialog_create(cfg_, &dlg_) == GENIE_STATUS_SUCCESS;   // loads context binaries
    }
    std::string ask(const std::string& prompt, void (*cb)(const char*, void*), void* user) {
        Ctx c{{}, cb, user};
        GenieDialog_query(dlg_, prompt.c_str(), GENIE_DIALOG_SENTENCE_COMPLETE, on_response, &c);
        return c.out;
    }
    ~NpuLlm() { if (dlg_) GenieDialog_free(dlg_); if (cfg_) GenieDialogConfig_free(cfg_); }
};

Before creating the dialog inside an app, set ADSP_LIBRARY_PATH (with setenv) to the directory containing the Hexagon skeleton libraries (for example the app's native library directory or an extracted folder), and ship the ARM-side QNN libraries in jniLibs/arm64-v8a.

Step 5: the honest three-way comparison

Report energy per 100 tokens and the ten-minute curve, not just peak tok/s. Vendors publish peak numbers; independent, power-inclusive, sustained comparisons are rare and far more useful.

NPU deployment gotchas (and fixes)

Graph partitioning, visualised

Ideal (1 partition):     [ NPU: conv..attn..mlp..conv ................ ]      1 dispatch

Bad (5 partitions):      [NPU][CPU:op X][NPU][CPU:op Y][NPU]
                             ▲copy+sync  ▲copy+sync ▲copy+sync ▲copy+sync
Each boundary costs data transfer, format conversion (quantize/dequantize, layout)
and a synchronisation round trip; with many small partitions the NPU sits idle.

HTP context caching and performance settings

• Context binary: the serialized, finalized graph for one SoC architecture. Loading it skips graph optimisation and is typically 10-100x faster than compiling from the model at startup. It is not portable across Hexagon versions, and often not across SDK versions.
• On-device caching: if you must compile on device (for example via ORT QNN EP), write the compiled context to app storage on first run and reuse it; invalidate the cache when the model, SDK or OS/driver version changes.
• Performance modes: burst gives maximum clocks for short tasks; sustained high performance gives a stable profile for long workloads; power saver trades latency for energy. Match to the duty cycle.
• VTCM and spill-fill: the HTP has fast on-chip memory (VTCM); graphs that exceed it spill to DDR. Tiling and model splitting keep working sets on-chip.

Deliverables checklist

• Model exported to context binaries (via AI Hub, QAIRT or ExecuTorch QNN backend).
• CLI runner generating coherent output on the NPU.
• C API integrated into an Android app via JNI.
• Three-way CPU/GPU/NPU comparison with power and sustained curves.
• Documented accuracy cost of static vs dynamic activation quantization.
• A written log of every operator gap and SDK issue encountered, with the workaround. This log is gold in interviews.

Project 4: KV-cache engineering

Duration: 3-4 weeks. Goal: own the memory bottleneck that limits on-device long context. Skills: KV caching, KV-cache quantization, footprint profiling, attention internals. Output: a KV-cache profiler plus a quantization study across bit-widths and context lengths. The KV cache grows linearly with sequence length and at long contexts it exceeds the model weights; for long-context edge deployment, compressing it is often more impactful than squeezing weights further. Yet most tutorials focus only on weight quantization. For attention mechanics, see Transformers and LLMs.

The memory math

Worked example with Llama-3.2-1B (from its params.json: dim 2048, 16 layers, 32 query heads, 8 KV heads, so head dim = 2048 / 32 = 64):

per token (FP16) = 2 x 16 layers x 8 kv_heads x 64 head_dim x 2 bytes = 32,768 B = 32 KiB
per token (INT8)  = 16 KiB (+ small scale overhead)
per token (INT4)  =  8 KiB (+ scales)

Llama-3.2-3B: 28 layers, 24 query heads, 8 KV heads, head_dim 128
per token (FP16) = 2 x 28 x 8 x 128 x 2 = 114,688 B = 112 KiB

A 7B model WITHOUT GQA (32 layers, 32 KV heads, head_dim 128)
per token (FP16) = 2 x 32 x 32 x 128 x 2 = 524,288 B = 512 KiB   (16x the 1B model)

INT4 weights are about 0.7-1.1 GiB for the 1B model and roughly 1.7-2.3 GiB for the 3B (depending on how embeddings and the output layer are quantized). Crossover (FP16 cache = INT4 weights) is weight bytes / 32 KiB per token for the 1B: about 22k tokens at 0.7 GiB and about 35k at 1.1 GiB. For the 3B (112 KiB/token) it is about 16-20k tokens. A 7B without GQA (512 KiB/token, ~3.5 GiB INT4 weights) crosses below 8k. Those are exactly the context lengths long-document and chat-history features want. Also remember that NPU and many CPU runtimes allocate the cache statically for the maximum context length, so you pay the full cost even for short prompts.

Part A: profile it

1. Footprint table Measure actual RSS (not just the formula) at 512, 1k, 2k, 4k, 8k and 16k context for the 1B and 3B models; the difference from the formula reveals allocator overhead and duplicated buffers.
2. Crossover point Find the context length where cache bytes exceed weight bytes.
3. Speed impact Plot decode tok/s against current context length: each decode step reads the whole cache, so decode slows as the conversation grows.
4. Allocation pattern Trace allocations over a long generation (heapprofd in Perfetto, or malloc hooks): growth, fragmentation and reallocation stalls when a contiguous cache is resized.
5. Kill threshold Find the context length at which the low-memory killer starts terminating your process on an 8 GB device with a real foreground workload.

def kv_bytes(layers, kv_heads, head_dim, tokens, bytes_per=2, batch=1, group=None, scale_bytes=2):
    base = 2 * layers * kv_heads * head_dim * tokens * batch * bytes_per
    if group:                                   # per-group scales for quantized KV
        base += 2 * layers * kv_heads * (head_dim // group) * tokens * batch * scale_bytes
    return base

for T in (512, 1024, 2048, 4096, 8192, 16384):
    fp16 = kv_bytes(16, 8, 64, T) / 2**20
    int4 = kv_bytes(16, 8, 64, T, bytes_per=0.5, group=32) / 2**20
    print(f"{T:6d}  fp16 {fp16:7.1f} MiB   int4(g32) {int4:7.1f} MiB")

Why decode slows as context grows

Part B: compress and manage it

Code: per-token INT8 value cache and ring-buffer window

import torch

def quant_per_token(x, bits=8):                     # x: [heads, T, head_dim]
    qmax = 2 ** (bits - 1) - 1
    scale = x.abs().amax(dim=-1, keepdim=True).clamp_min(1e-8) / qmax
    q = torch.clamp(torch.round(x / scale), -qmax - 1, qmax).to(torch.int8)
    return q, scale.to(torch.float16)

def quant_per_channel(x, bits=8):                   # keys: scale per head_dim channel
    qmax = 2 ** (bits - 1) - 1
    scale = x.abs().amax(dim=-2, keepdim=True).clamp_min(1e-8) / qmax
    q = torch.clamp(torch.round(x / scale), -qmax - 1, qmax).to(torch.int8)
    return q, scale.to(torch.float16)

class RingKV:
    """Sliding window + attention sinks, fixed memory."""
    def __init__(self, layers, heads, head_dim, window=2048, sinks=4, dtype=torch.float16):
        self.W, self.S = window, sinks
        shape = (layers, heads, sinks + window, head_dim)
        self.k = torch.zeros(shape, dtype=dtype); self.v = torch.zeros(shape, dtype=dtype)
        self.n = 0                                   # total tokens seen
    def slot(self):
        if self.n < self.S: return self.n            # sinks are never overwritten
        return self.S + (self.n - self.S) % self.W   # ring over the window
    def append(self, layer, k_t, v_t):               # k_t, v_t: [heads, head_dim]
        s = self.slot()
        self.k[layer, :, s] = k_t; self.v[layer, :, s] = v_t
    def advance(self): self.n += 1
    def valid(self): return min(self.n, self.S + self.W)
# Note: with RoPE, positions of cached keys are already baked in; implementations that
# roll a window either keep original positions or re-assign positions within the cache.

Code sketch: paged KV allocator in C++

struct BlockPool {
    size_t block_tokens, bytes_per_block;
    std::vector<void*> free_list;
    explicit BlockPool(size_t tokens, size_t bpt, size_t n_blocks)
        : block_tokens(tokens), bytes_per_block(tokens * bpt) {
        for (size_t i = 0; i < n_blocks; ++i) free_list.push_back(aligned_alloc(64, bytes_per_block));
    }
    void* get() { if (free_list.empty()) return nullptr; auto* b = free_list.back(); free_list.pop_back(); return b; }
    void put(void* b) { free_list.push_back(b); }
};

struct SequenceKV {
    std::vector<void*> blocks;      // block table: logical block i -> physical block
    size_t tokens = 0;
    bool append_token(BlockPool& pool) {
        if (tokens % pool.block_tokens == 0) {
            void* b = pool.get();
            if (!b) return false;       // caller evicts, compresses, or stops generation
            blocks.push_back(b);
        }
        ++tokens; return true;
    }
};
// Pre-allocating the pool once at startup avoids reallocation stalls and makes the
// memory budget explicit; the attention kernel walks the block table.

Part C: the Android systems finding

Tie the work to device reality. On a real 8 GB phone with a foreground app and background services, what is the maximum usable context length before the system kills your process, and how does each compression technique move that number? This requires understanding Android's memory management (see Linux kernel and BSP and Power and thermal for related platform detail):

• lmkd (the userspace low-memory killer daemon) monitors memory pressure (PSI, pressure stall information) and kills processes in order of oom_score_adj: cached apps first, then services, then perceptible and finally foreground.
• A foreground app is killed last, but a large model can push the system into killing everything else (including the launcher or music playback), which users experience as "the phone got weird".
• Anonymous memory (heap-allocated KV cache, repacked weights) cannot be reclaimed without swap/zRAM compression; file-backed mmapped weights can be dropped and re-read.
• zRAM compresses anonymous pages; low-entropy tensors compress well, quantized ones do not.

# Observe memory pressure and kills while increasing context length
adb shell cat /proc/pressure/memory             # PSI: some/full stall percentages
adb shell dumpsys meminfo -s <package>         # PSS summary for the app
adb logcat -b events | grep -i am_kill           # framework kill events
adb logcat | grep -i lowmemorykiller             # lmkd decisions and reasons
adb shell cat /proc/<pid>/oom_score_adj

Deliverables checklist

• KV footprint curve vs context length for two model sizes, formula vs measured.
• Documented crossover point where the cache exceeds the weights.
• Quantization study at 8/4/3 bits with quality cost quantified, including key vs value grouping.
• At least one alternative strategy implemented (sliding window with sinks, or paged).
• Maximum usable context under real Android memory pressure, with lmkd behaviour.

Project 5: the system project

Duration: 4-6 weeks. Goal: a complete system where the model is one component among many, built to survive memory pressure, heat and bad inputs. Skills: LoRA adapters, system integration, memory- and thermal-aware inference, platform architecture. Output: a working system, a short demo, and an architecture write-up of the trade-offs. Pick one of the four designs below; each reuses the harness (P1), quantization recipe (P2), NPU path (P3) and cache management (P4). For wider design framing, see GenAI system design.

Option A: a shared on-device inference service

Build a small version of what a platform AI service is: one Android service hosting one model, shared by several client apps.

  Client app A ─┐                                      ┌─ LoRA: summarise (≈20-40 MB)
  Client app B ─┼─ AIDL (bound service, permission) ─▶ │  Scheduler / queue (fairness, priorities)
  Client app C ─┘                                      │  Policy: memory + thermal governor
                                                       │  Engine: one resident base model (mmap)
                                                       │     NPU ──▶ GPU ──▶ CPU ──▶ refuse
                                                       └─ LoRA: rewrite / classify (hot-swap)

• Stable IPC surface: an AIDL interface with request id, prompt, options and a streaming callback; enforce a custom permission; validate inputs; cap sizes (see Binder transaction limits: stream results rather than returning large payloads).
• One model resident, not one copy per client: memory is the scarcest resource.
• Memory-pressure awareness: listen to onTrimMemory and PSI; shrink the KV budget, unload adapters, or unload the model when idle and under pressure.
• Thermal-aware throttling: poll PowerManager.getThermalHeadroom() and thermal status listeners; lower the generation rate or max tokens proactively instead of being throttled arbitrarily by the OS.
• Hot-swappable LoRA adapters: fine-tune two small adapters for different tasks (see Fine-tuning), swap at runtime, and measure the switch cost in latency and memory. Merged weights are fastest but not swappable; unmerged adapters add a small matmul per adapted layer.
• Queueing and fairness: per-client quotas, priority for foreground callers, cancellation when the client dies (linkToDeath).
• Graceful degradation: NPU, then GPU, then CPU, then refuse with a clear error, following a documented policy.

// IInferenceService.aidl
package com.example.edge;
import com.example.edge.IInferenceCallback;

interface IInferenceService {
    int  getApiVersion();
    long submit(String task, String prompt, in Bundle options, IInferenceCallback cb); // returns request id
    void cancel(long requestId);
    Bundle getStatus();     // backend in use, thermal state, loaded adapters, queue depth
}

// IInferenceCallback.aidl
oneway interface IInferenceCallback {
    void onToken(long requestId, String piece);
    void onComplete(long requestId, in Bundle stats);
    void onError(long requestId, int code, String message);
}

// Thermal- and memory-aware policy (Kotlin, inside the service)
class Governor(private val pm: PowerManager) {
    fun decide(req: Request): Plan {
        val headroom = pm.getThermalHeadroom(10)          // forecast 10 s ahead; 1.0 = throttling
        val status = pm.currentThermalStatus
        return when {
            status >= PowerManager.THERMAL_STATUS_SEVERE -> Plan.Refuse("device too hot")
            headroom > 0.85f -> Plan.Run(backend = Backend.NPU, maxTokens = req.maxTokens / 2,
                                          tokenDelayMs = 20)   // pace generation
            memoryTight()     -> Plan.Run(backend = Backend.NPU, maxContext = 1024)
            else              -> Plan.Run(backend = Backend.NPU, maxTokens = req.maxTokens)
        }
    }
}

Option B: an on-device assistant with local RAG (documents or device logs)

A fully local assistant that ingests documents (or device logs such as a bug report or logcat capture) and answers questions with citations. Nothing leaves the device.

Ingest ──▶ Parse/clean ──▶ Chunk (200-500 tokens, overlap) ──▶ Embed (small encoder, INT8)
                                                                   │
                                                          Vector index (flat / HNSW) + keyword index (BM25)
Query ──▶ Embed ──▶ Hybrid retrieve top-k ──▶ Rerank (optional) ──▶ Pack context within budget
                                                                   │
                         Local LLM (NPU, KV budget from P4) ──▶ Answer with cited chunk ids ──▶ UI

• Chunking: for logs, chunk on natural boundaries (timestamps, process ids, crash blocks) rather than fixed characters, and keep metadata (time, tag, pid) for filtering.
• Embeddings: a small sentence encoder (tens of millions of parameters) quantized to INT8 runs in milliseconds per chunk; batch ingestion in the background while charging.
• Index: a flat index is fine up to tens of thousands of chunks; beyond that use an approximate index (HNSW) in SQLite or a small native library. Combine with keyword search, because identifiers and error codes are poorly captured by embeddings.
• Context budget: a 40 MB log does not fit a 4-8k context; retrieval plus filtering is mandatory. Budget tokens explicitly: system prompt (prefix-cached) + retrieved chunks + question + answer.
• Grounding and refusal: instruct the model to answer only from retrieved text with chunk citations, and to say it cannot find evidence otherwise; verify citations exist before displaying.
• Measure the whole pipeline: ingestion time per MB, retrieval recall@k on a labelled question set, TTFT and total answer latency, peak memory, energy per question.

Option C: a wearable sensor model

An always-on activity, gesture or anomaly classifier on accelerometer/gyroscope/PPG data for a watch or band, where the budget is milliwatts, not watts.

• Model: a tiny 1-D CNN or small recurrent model (tens of KB to a few hundred KB), INT8, with fixed windows (for example 2-second windows at 50 Hz, 50% overlap).
• Placement: run on the sensor hub / low-power DSP or microcontroller (for example a Cortex-M with a micro NPU via a micro runtime), and wake the application processor only on detected events. Batch sensor data with hardware FIFOs to avoid waking the CPU every sample.
• Budget: energy per inference in microjoules, average power in low milliwatts, RAM in tens to hundreds of KB, flash for weights.
• Evaluation: per-user variation matters; evaluate with leave-one-subject-out splits, and measure false wake-ups per hour, which dominate battery cost.
• Updates: over-the-air model updates via the companion phone, with version checks and rollback.

Option D: a real-time camera pipeline

Detection, segmentation or super-resolution on live camera frames at 30 fps, where the budget is a 33 ms frame period and the model shares the device with the camera stack and display.

Camera (YUV_420_888) ──▶ GPU/ISP resize + colour convert ──▶ NPU model (INT8, static 256-640 px)
          │                                                        │
          └──── preview surface (no copy) ◀── overlay render ◀── post-process (NMS, masks) on GPU/CPU

• Zero-copy: use hardware buffers (AHardwareBuffer) shared between camera, GPU and NPU where the runtime supports it; each CPU copy of a 1080p frame costs milliseconds.
• Pipelining: overlap capture, pre-processing, inference and rendering across frames; drop frames rather than queueing them (latency matters more than throughput).
• Pre/post-processing: often slower than the model itself; move resize/colour conversion to GPU or ISP and NMS to optimized native code.
• Thermal: camera plus display plus NPU is a heavy sustained load; measure 20-30 minutes, and adapt resolution or frame rate to thermal headroom.
• Metrics: end-to-end frame latency (sensor to overlay), p99 frame time, dropped-frame rate, accuracy on device-captured test footage (not only curated datasets).

Deliverables for any option

• System runs end to end on device, reproducibly.
• Behaviour under memory pressure documented (not just the happy path).
• Thermal behaviour under sustained use documented.
• A short demo recording.
• Architecture write-up explaining the design trade-offs and why.

Profiling tools and how to use them

Benchmark numbers tell you what is slow; profilers tell you why. Profile at three levels: the system (CPU scheduling, frequencies, thermal, power), the runtime (per-operator timings and placement), and the native code (hot functions and cache behaviour).

Recording a Perfetto trace with CPU frequency, scheduling and power rails

# config.pbtxt
buffers { size_kb: 131072 fill_policy: RING_BUFFER }
data_sources { config { name: "linux.ftrace" ftrace_config {
    ftrace_events: "sched/sched_switch"
    ftrace_events: "power/cpu_frequency"
    ftrace_events: "power/cpu_idle"
    ftrace_events: "thermal/thermal_temperature"
    atrace_categories: "gfx"
    atrace_categories: "view"
    atrace_apps: "com.example.edge"
} } }
data_sources { config { name: "android.power" android_power_config {
    battery_poll_ms: 250
    collect_power_rails: true
    battery_counters: BATTERY_COUNTER_CURRENT
    battery_counters: BATTERY_COUNTER_VOLTAGE
} } }
data_sources { config { name: "linux.process_stats" target_buffer: 0 } }
duration_ms: 60000

# record (the config is streamed through stdin)
adb push config.pbtxt /data/local/tmp/
adb shell "cat /data/local/tmp/config.pbtxt | perfetto --txt -c - -o /data/misc/perfetto-traces/edge.pftrace"
adb pull /data/misc/perfetto-traces/edge.pftrace
# open the file in the Perfetto UI (runs locally in a browser; the trace stays on your machine)

Add custom slices so the trace shows model phases alongside system activity:

// Kotlin
Trace.beginSection("llm_prefill"); engine.prefill(tokens); Trace.endSection()

// C++ (NDK)
#include <android/trace.h>
ATrace_beginSection("llm_decode_step");
engine.decode(tok);
ATrace_endSection();

simpleperf on the native engine

adb shell simpleperf record -p $(adb shell pidof llama_main) -g --duration 15 \
    -o /data/local/tmp/perf.data
adb shell simpleperf report -i /data/local/tmp/perf.data --sort symbol | head -40

# Hardware counters (availability varies by device)
adb shell simpleperf stat -e cpu-cycles,instructions,cache-misses,raw-l2d-cache-refill \
    -p $(adb shell pidof llama_main) --duration 10
# For app processes, the NDK's simpleperf scripts (app_profiler.py) handle symbols and flame graphs.

QNN per-layer profiling

adb shell "cd /data/local/tmp/qnn && ./qnn-net-run --backend libQnnHtp.so \
   --retrieve_context model_ctx.bin --input_list inputs.txt \
   --profiling_level detailed --output_dir out"
adb pull /data/local/tmp/qnn/out/qnn-profiling-data_0.log
qnn-profile-viewer --input_log qnn-profiling-data_0.log   # per-op cycles/time, init vs execute

A profiling workflow that finds real problems

1. Start at the system trace Is the work actually on the expected unit? Are threads on big cores? Are frequencies capped (thermal) or low (governor idle)? Is anything else competing?
2. Check placement In the runtime profile, confirm what fraction of ops or time runs on the accelerator and how many partitions exist.
3. Find the top ops Typically matmuls and attention dominate; if a reshape, transpose, gather or quantize/dequantize is near the top, it is a conversion artifact to fix.
4. Roofline sanity check For decode, compare achieved bytes/s (weights + KV per token x tok/s) with the device's memory bandwidth; if you are far below, the kernels or threading are the problem, not the hardware.
5. Drill into native code Only then use simpleperf to see whether the expected SIMD kernels are running.

Debugging accuracy drift

Accuracy drift is when the deployed model's outputs differ from the reference by more than expected. It can come from conversion, quantization, the device implementation, or the pipeline around the model. The key is to bisect systematically instead of guessing.

The bisection ladder

  Stage                               Compare against          Expect
  ─────────────────────────────────────────────────────────────────────────
  1 Original framework model (FP32)    ground truth labels      baseline accuracy
  2 Exported / converted FP32 model    stage 1 outputs          max abs diff ~1e-5..1e-4
  3 Converted FP16 model               stage 2                  small diff, no inf/NaN
  4 Quantized model, run on host       stage 2                  small accuracy delta
  5 Quantized model, run on device     stage 4                  near-identical outputs
  6 Full app pipeline on device        stage 5 with same inputs identical if pre/post match
  The first rung where the diff jumps is where the bug lives.

Common causes by rung

LLM-specific drift checks

• Greedy decode comparison: run with temperature 0 on host and device with identical token ids; the first differing token position shows how quickly outputs diverge. Small logit differences can flip an argmax late in a sequence, which is expected; early divergence is not.
• Logit metrics: compare top-1 agreement and KL divergence of next-token distributions over a fixed prompt set, not just generated text.
• Tokenizer parity: the device tokenizer must produce the same ids as the reference (special tokens, BOS handling, whitespace normalisation).
• Chat template: a missing or different template changes behaviour more than any quantization.
• KV-cache bugs: outputs fine for short prompts but degrade after a certain length point to cache indexing, position (RoPE) offsets, or window wrap-around errors.

import torch.nn.functional as F
def logit_agreement(ref_logits, dev_logits):          # [T, vocab] for the same token sequence
    top1 = (ref_logits.argmax(-1) == dev_logits.argmax(-1)).float().mean().item()
    kl = F.kl_div(F.log_softmax(dev_logits, -1), F.log_softmax(ref_logits, -1),
                  log_target=True, reduction="batchmean").item()
    return {"top1_agreement": top1, "kl": kl}

Shipping and updating models

Shipping turns a working build into a feature that runs on thousands of device models, improves over time, and can be switched off safely. Treat models like code: versioned, tested, staged, observable and reversible.

Device tiering

Decide the tier with a combination of static facts (RAM from ActivityManager.MemoryInfo.totalMem, SoC model from build properties) and a quick on-device capability probe at first run (load a tiny test graph on the NPU and check it runs), then cache the result.

Versioning and compatibility

• Model manifest: model id, semantic version, runtime and SDK version required, SoC/NPU architecture, input/output schema, tokenizer version, quantization recipe, checksum, minimum app version.
• Schema contracts: pre/post-processing code in the app must match the model's expected inputs; version them together or embed metadata in the model file and assert on load.
• Runtime compatibility: context binaries and compiled caches are tied to NPU generation and SDK version; ship per-SoC variants and invalidate caches on OS or driver updates.
• Adapters: LoRA adapters are tied to a specific base model version; a base model update invalidates all adapters.

Rollout, A/B testing and remote config

1. Offline gates Quality evaluation on a fixed set, on-device benchmark on a device matrix (at least one device per tier), and memory/thermal soak all pass.
2. Internal and beta cohort Enable via remote config for internal users; monitor crashes and ANRs per device.
3. Staged rollout 1%, 5%, 20%, 50%, 100% with automatic halt thresholds (crash rate, p90 latency, fallback rate, user-visible errors).
4. A/B test Compare new vs old model on product metrics (task success, acceptance of suggestions, retention) and device metrics (latency, battery), segmented by tier.
5. Kill switch A remote flag that disables the model or reverts to the previous version without an app update.

Fallback chain

request ──▶ NPU path ok? ──yes──▶ run
               │ no (unsupported / init failed / too hot)
               ▼
            GPU path ok? ──yes──▶ run (maybe smaller model or shorter context)
               │ no
               ▼
            CPU path within budget? ──yes──▶ run (reduced max tokens)
               │ no
               ▼
            cloud allowed (network + user consent + privacy policy)? ──yes──▶ cloud model
               │ no
               ▼
            refuse gracefully with a clear message

What to monitor in the field

• Load success rate and load time per device model and OS version.
• Backend actually used (NPU/GPU/CPU/cloud) and fallback rate.
• Latency percentiles (TTFT, tok/s, end-to-end) by tier.
• Crashes, ANRs, native crashes inside runtime libraries, low-memory kills during inference.
• Thermal status at request start, and share of requests degraded by the governor.
• Quality proxies: user edits or rejections, thumbs up/down, retries; never upload raw private inputs without explicit consent.
• Model version distribution, to know when old versions can be retired.

Security and integrity

• Verify checksums and signatures of downloaded models before loading; a malicious model file can be a crash or code-path attack surface on native parsers.
• Store models in app-private storage; encrypt at rest if the weights are valuable, knowing that a determined attacker with root can still extract them at runtime.
• Apply safety filters and prompt-injection defences for LLM features even though inference is local.
• Check model licences before shipping open weights.

Target numbers to validate against

Use these as a smoke test. If your measurements are wildly off (for example half the expected decode speed), something is misconfigured: wrong build type, missing KV-cache flags, kernels not enabled, wrong thread count, or the device is already hot. All numbers are approximate, depend heavily on SoC, runtime version, prompt length and thermal state, and will improve with newer releases; re-verify before quoting.

Llama 3.2 on Android with ExecuTorch (CPU, flagship 2024 phone, prompt length 64, KleidiAI enabled)

Another reference point: quantized Llama 3.2 1B (INT4 weights, 8-bit dynamic activations) on a flagship 2024 phone

• Prefill above ~350 tok/s and decode above ~40 tok/s on CPU with KleidiAI kernels.
• About 2 seconds to respond to a ~600-token input.
• .pte size about 1.1 GiB quantized vs about 2.3 GiB in BF16 (not a clean 4x because embeddings and some layers stay at 8-bit and scales add overhead).
• Peak RSS about 1.9 GiB vs about 3.1 GiB for BF16 at a 2048-token maximum sequence length (roughly 40% lower).
• KleidiAI contributes more than 20% of prefill speed at identical accuracy.

Rule-of-thumb ranges for phone-class flagships (approximate)

Classic models (INT8, flagship phone, approximate per-inference latency)

Sanity rules

• Decode ceiling: tok/s cannot exceed effective bandwidth / bytes read per token. With ~50 GB/s effective and ~1 GB of INT4 weights, about 50 tok/s is the ceiling for a 1B model.
• Prefill vs decode: prefill tok/s should be several times (5-20x) decode tok/s; if they are similar, prefill is not batching tokens (for example the KV-cache or SDPA path is disabled).
• Quantization gain: moving BF16 to INT4 should give about 2.4-3x decode (bandwidth ratio minus overheads); much less suggests dequantization overhead or fallback kernels.
• Threads: decode usually peaks at the number of performance cores; more threads can reduce speed.
• Sustained: expect 60-85% of peak after 10 minutes on a phone in free air; much lower means aggressive throttling or a hot environment.

A 24-week learning plan

At 10-15 hours per week, this sequence takes a systems or application engineer to the point of having shipped-quality, measured deployments across CPU, GPU and NPU. Each project depends on the previous project's harness and understanding, so keep the order.

Theory to study alongside (about 20-25 hours total)

Slip discipline

• If you fall behind, cut scope inside a project, never skip a project. A shallow NPU project still teaches the toolchain; skipping it leaves the biggest gap open.
• If the NPU project stalls completely on hardware or SDK access, switch vendor or move to Project 4 and return later.
• Timebox setup and toolchain debugging; when stuck for more than a day, get a baseline working with a simpler tool and come back.

Failure modes checklist

These are the ways edge AI work most often goes wrong, both while learning and in production. Read them before you start and again before you ship.

While learning

Before shipping

• Converted FP32 model matches the original within tolerance.
• Quantized model evaluated on a fixed set including hard and edge cases, not only a few prompts.
• Per-op placement confirmed: no unexpected CPU fallback partitions on the target accelerator.
• Static shapes and max context chosen deliberately; memory budget written down (weights + KV + activations + runtime + app).
• Release build, correct flags (optimized kernels, KV cache, SDPA), correct thread count.
• Load time measured cold and warm; compiled graph or context cached.
• Inference off the main thread; cancellable; session reused; warm-up done off the critical path.
• Model assets uncompressed and memory-mapped, or downloaded with checksum and atomic install.
• Behaviour tested under memory pressure (other apps open, onTrimMemory), with no low-memory kills of the foreground app.
• Ten- to thirty-minute thermal soak done; thermal headroom policy in place.
• Energy per request measured against an idle baseline.
• Tokenizer and chat template parity verified; greedy outputs compared host vs device.
• Hexagon/NPU libraries match the SoC; SDK versions of build and runtime match.
• Fallback chain implemented and tested by forcing each failure.
• Telemetry segmented by SoC, RAM tier and OS build; staged rollout thresholds and kill switch ready.
• Model licence checked; safety filtering in place for generative features; downloaded models verified.

Quick revision

• The deployment loop is choose, export, convert, quantize, compile for target, integrate, benchmark on device, monitor; stages 3-7 iterate many times.
• Export captures a static graph (torch.export, ONNX); conversion produces a runtime format (.pte, .tflite, .gguf, .onnx/.ort, .mlpackage, QNN context binary).
• NNAPI is deprecated from Android 15; new work uses LiteRT delegates/accelerators, ExecuTorch backends or ONNX Runtime execution providers. Confirm current LiteRT CompiledModel / Interpreter package names in official docs; do not invent flags.
• ExecuTorch swaps backends by changing the partitioner (XNNPACK, Vulkan, QNN, MediaTek, Core ML) while keeping one export flow.
• llama.cpp with GGUF is the fastest way to a working on-device LLM baseline; Q4_0 is repacked for fast Arm kernels, Q4_K_M is a quality-per-byte default.
• For ExecuTorch LLM export, use_kv_cache and use_sdpa_with_kv_cache are essential; 8da4w means 8-bit dynamic activations, 4-bit weights.
• KleidiAI kernels in XNNPACK add more than 20% prefill speed on Arm CPUs; always build Release.
• Validate numerics after every conversion: FP32 differences should be around 1e-5 to 1e-4.
• Never benchmark on an emulator: no realistic memory system, DVFS, thermal model or NPU.
• Report load time (cold and warm), TTFT, prefill tok/s, decode tok/s, per-token p50/p90/p99, peak RSS, file size, energy per request and sustained/peak ratio.
• Always state prompt length and generated length with LLM numbers; prefill and decode differ by 5-20x.
• A fair protocol fixes the environment, cools down between runs, discards warm-up runs, repeats, and records device, build and runtime versions.
• Peak RSS comes from VmHWM in /proc/<pid>/status; app memory breakdown from dumpsys meminfo.
• Energy = integral of (power minus idle power) over time; use power rails via Perfetto, fuel gauge sampling, batterystats, or an external monitor.
• A 10-30 minute thermal soak reveals throttling; phones typically sustain 60-85% of peak.
• Quantization quality needs both perplexity and task-level evaluation on a fixed set including hard cases.
• Layer-wise error analysis compares each intermediate tensor to an FP32 reference using cosine similarity, MAE, max error and SQNR.
• SQNR = 10 log10(signal power / error power); each bit adds about 6 dB; low SQNR on the residual stream predicts quality loss.
• Single-layer sensitivity (quantize one layer at a time) separates fragile layers from layers receiving bad inputs.
• Typical sensitive parts: outlier activation channels, MLP down projection, LM head, first/last blocks, norms and softmax.
• Mixed precision promotes only the most sensitive layers to 8-bit or FP16 and prices each promotion in size and latency.
• Host fake-quant matching FP32 but device diverging means a device implementation issue (rounding, accumulators, FP16 overflow), not the scheme.
• NPUs need static quantization: ranges fixed at compile time from calibration; LLMs on Hexagon commonly use W4A16 or W8A16.
• NPUs need static shapes; export separate prefill (chunked) and decode (one token) graphs that share weights.
• One unsupported op mid-graph creates partitions and CPU round trips that can make the NPU slower than the CPU.
• Context binaries are finalized graphs for one Hexagon architecture and SDK version; loading them avoids on-device compilation.
• Hexagon library folders (v73, v75, v79) must match the SoC; set ADSP_LIBRARY_PATH for DSP-side libraries.
• Large models are split into several context binaries because of NPU session memory limits.
• KV bytes = 2 x layers x KV heads x head dim x tokens x bytes; Llama-3.2-1B uses 32 KiB per token in FP16 (verified: 2 × 16 × 8 × 64 × 2).
• At long context the KV cache exceeds the weights (about 22-35k tokens for 1B INT4, about 16-20k for 3B, under 8k for models without GQA).
• Decode speed falls as context grows because each step reads the weights plus the whole cache: tok/s ≤ BW / (W + KV(t)).
• INT8 weights are usually near-lossless; INT4 is the decode lever but needs a task eval, plus higher precision on embeddings and the LM head.
• Quantize keys per channel (outlier channels) and values per token; INT8 KV is near-lossless.
• Sliding-window attention bounds memory; keep attention-sink tokens to avoid collapse on long streams.
• Paged KV: contiguous waste is (T_max - T_actual) x KV per token; paged waste is at most one block. Prefix caching saves about P / (P + U) of prefill if the prefix is byte-identical.
• Speculative decoding: E[tokens per pass] = (1 - alpha^(k+1)) / (1 - alpha); speed-up ≈ E / (1 + k c). Helps on-device when decode is bandwidth-bound and the draft is accurate.
• lmkd kills by oom_score_adj under memory pressure (PSI); anonymous memory is not reclaimable, mmapped clean file pages are.
• Store model assets uncompressed (noCompress) so they can be memory-mapped; downloaded models need resume, checksum, atomic install and a smoke test.
• Create sessions once, off the main thread, warm them up, reuse buffers, make generation cancellable and respond to onTrimMemory.
• ONNX Runtime's QNN EP needs a QDQ model; disable CPU fallback during development and enable EP context caching. Provider option keys (backend_path, HTP performance mode) are version-specific: check the current ORT QNN EP docs.
• QAIRT converter and Genie binary/API names move between SDK releases. Copy flags, JSON keys and C symbols from the samples for the version you link; do not invent them.
• Profile at three levels: system (Perfetto), runtime (per-op profiles, AI Hub, QNN profiler, ETDump), native (simpleperf).
• Accuracy drift is bisected: original, converted FP32, FP16, quantized host, quantized device, full app pipeline; pre-processing bugs are the most common cause.
• Shipping needs device tiering, versioned manifests, staged rollout with halt thresholds, A/B tests, a kill switch and a fallback chain (NPU, GPU, CPU, cloud, refuse).
• Monitor load success, backend used, fallback rate, latency by tier, low-memory kills and thermal state, segmented by SoC, RAM and OS build.

Glossary

Activation quantization

Representing the intermediate outputs of layers with low-bit integers; harder than weight quantization because ranges depend on inputs and contain outliers.

AI Hub

Qualcomm's hosted service to compile, quantize, profile and run models on real Snapdragon devices, with a model zoo of export scripts.

AIMET

Qualcomm's open-source toolkit for quantization and compression, including cross-layer equalisation, AdaRound, QuantAnalyzer and QAT.

Attention sink

The first few tokens of a sequence, which attract a large share of attention; keeping them in a sliding-window cache keeps long generations stable.

Block table

The mapping from a sequence's logical KV-token ranges to physical KV blocks in a paged cache, analogous to a page table.

CompiledModel

LiteRT Next's compiled-model API that binds a .tflite (or a vendor-compiled artifact) to an accelerator. Class names and options are release-specific; confirm in current LiteRT docs.

AWQ

Activation-aware weight quantization: a PTQ method that protects the weight channels most important to activations by scaling before quantizing.

batterystats

Android's battery accounting service; dumpsys batterystats reports estimated energy use per app over a measurement window.

BF16

Brain float 16: 16-bit floating point with the same exponent range as FP32 but fewer mantissa bits; robust to overflow.

Calibration dataset

A small, representative set of inputs run through a model to record activation ranges for static quantization.

Context binary

A serialized, fully compiled QNN graph for a specific Hexagon architecture that loads without on-device graph compilation.

CPU fallback

Execution of operators the accelerator cannot run on the CPU instead, creating partitions and data transfers.

Decode phase

The token-by-token generation phase of an LLM; memory-bandwidth-bound because all weights and the KV cache are read for each token.

Draft acceptance rate

The per-token probability that a speculative draft token is accepted by the target model; it sets expected tokens per verification pass.

Delegate

A LiteRT plug-in that takes over supported parts of a graph and runs them on a GPU, NPU or DSP.

Dynamic quantization

Quantizing activations at runtime with scales computed from each tensor's actual values; common on CPUs, not possible on most NPUs.

EP context cache

An ONNX Runtime feature that saves a compiled execution-provider graph (for example a QNN context) so later sessions skip compilation.

ETDump and ETRecord

ExecuTorch developer-tool artifacts: runtime profiling and debug data (ETDump) and export-time graph metadata (ETRecord) used together by the Inspector.

Execution provider

An ONNX Runtime backend (CPU, QNN, Core ML, XNNPACK and others) that executes the subgraphs assigned to it.

ExecuTorch

PyTorch's on-device runtime that executes .pte programs produced from torch.export, with pluggable hardware backends.

Fake quantization

Simulating quantization in floating point (quantize then dequantize) to measure or train for its effect on a host.

Genie

Qualcomm's generative AI runtime in QAIRT that wraps tokenizer, QNN execution, KV-cache management and sampling behind a dialog API. Confirm headers, JSON keys and CLI names against the SDK samples for your release.

GGUF

llama.cpp's single-file model format containing tensors, quantization types, tokenizer and metadata.

GPTQ

A post-training weight quantization method that quantizes weights column by column while compensating the error using second-order information from calibration data.

GQA (grouped-query attention)

Attention where several query heads share each key/value head, shrinking the KV cache in proportion.

Graph partitioning

Splitting a model graph into subgraphs for different processors; each boundary adds transfer and synchronisation cost.

Group-wise quantization

Using one scale (and zero-point) per small block of weights, for example 32 or 128, to follow local ranges more closely.

heapprofd

Perfetto's native heap profiler that attributes allocations to call stacks in a running process.

Hexagon HTP

The Hexagon Tensor Processor, the matrix/tensor accelerator in Snapdragon NPUs, reached through QNN/QAIRT.

KleidiAI

Arm's library of optimized low-bit matrix-multiply micro-kernels, integrated into XNNPACK and other runtimes.

KV cache

Stored key and value vectors for all previous tokens in every layer, so attention does not recompute them each step.

LiteRT

Google's on-device runtime, formerly TensorFlow Lite, executing .tflite models with CPU, GPU and NPU accelerators.

llama.cpp

A C/C++ LLM inference engine with optimized CPU kernels and GPU backends, using the GGUF format.

lmkd

Android's userspace low-memory killer daemon, which kills processes by priority when memory pressure is high.

LoRA adapter

A small set of low-rank weight updates that specialise a shared base model for a task and can be swapped at runtime.

Memory mapping (mmap)

Mapping a file into the address space so pages load lazily and can be reclaimed by the kernel, avoiding a full copy into the heap.

Mixed precision

Using different numeric precisions for different layers, typically keeping sensitive layers at higher precision.

NNAPI

Android Neural Networks API, the older accelerator abstraction, deprecated from Android 15 in favour of vendor delegates and backends.

ODPM

On-Device Power Monitor: hardware energy counters for power rails, readable via Perfetto on supported devices.

Paged KV cache

A KV cache stored in fixed-size blocks mapped by a block table, avoiding contiguous reallocations and fragmentation.

Perfetto

Android's system-wide tracing tool for scheduling, frequencies, thermal, power rails, memory and custom trace slices.

Prefill phase

Processing all prompt tokens in parallel to fill the KV cache; compute-bound and the main contributor to TTFT.

Prefix caching

Reusing a precomputed KV cache for a shared prompt prefix, such as a system prompt, to reduce TTFT.

PSI

Pressure stall information: kernel metrics showing how much time tasks stall waiting for memory, CPU or IO.

PTQ

Post-training quantization: quantizing a trained model using calibration data, without retraining.

QAIRT

Qualcomm AI Runtime SDK, which includes the QNN libraries, converters, quantizers, HTP backend, profiling and Genie.

QAT

Quantization-aware training: fine-tuning with simulated quantization so the model learns to tolerate rounding.

QDQ format

An ONNX representation of quantization using explicit QuantizeLinear/DequantizeLinear nodes that backends fuse into integer kernels.

RSS and PSS

Resident set size (all resident pages of a process) and proportional set size (shared pages divided among sharers).

simpleperf

Android's sampling CPU profiler for native code, with call graphs and hardware counters.

Sliding-window attention

Attention restricted to the most recent W tokens, bounding KV-cache memory.

SpinQuant

A quantization method that applies learned rotations to weights and activations to spread outliers before low-bit quantization.

Speculative decoding

A draft model (or extra heads / n-grams) proposes tokens that the target verifies in one pass. Expected tokens per pass = (1 − α^k+1) / (1 − α). Output distribution is unchanged.

SQNR

Signal-to-quantization-noise ratio in decibels, comparing reference tensor energy with quantization error energy.

Static quantization

Quantization with activation scales fixed ahead of time from calibration; required by most NPUs.

Static shapes

Tensor dimensions fixed at export/compile time, allowing NPU compilers to plan memory and tiling.

Sustained-to-peak ratio

Throughput after a long run divided by initial throughput; captures the effect of thermal throttling.

Thermal headroom

Android's forecast of how close the device is to severe throttling, from PowerManager.getThermalHeadroom().

Thermal soak

A long continuous workload (10-30 minutes) used to measure throttling and sustained performance.

torch.export

PyTorch's ahead-of-time graph capture producing an ExportedProgram with no Python control flow, used by ExecuTorch and ai-edge-torch.

TTFT

Time to first token: from prompt submission to the first generated token appearing.

VTCM

Vector tightly coupled memory: fast on-chip memory in the Hexagon NPU; working sets that exceed it spill to DRAM.

W4A16

A scheme with 4-bit weights and 16-bit activations, common for LLMs on NPUs.

XNNPACK

A highly optimized CPU inference library for Arm and x86 used as the CPU backend by LiteRT, ExecuTorch and ONNX Runtime.

Interview questions