# Choosing a model file and settings for on-device inference

Which quant to download, CPU or GPU, repacking and mapping: what each choice costs in memory, speed and quality, measured on two Pixels and a laptop.

2026-11-03

Running a model on your own device means making four choices that the model's name doesn't make
for you: **which file** to download (its quantization, or "quant"), **which backend** runs it (CPU
or GPU), whether the engine **repacks** the weights, and whether it **maps** the file or reads it
into memory. Each one trades memory, speed and answer quality against the others, and the right
trade depends on your device.

This article explains what each choice does, shows what it cost on the devices we tested, and ends
with a short procedure for finding the best setup on yours. Bring Your LM's embedded engine is
llama.cpp, so most of this applies to any llama.cpp-based app.

## The short version

1. **Download a 4-bit K-quant made with an importance matrix**: `Q4_K_M` or `Q4_K_XL` from a
   repository marked "imatrix", "UD" or similar, which answers closer to the original than the
   same type made without one. Pick `Q4_0` only if the model was trained for 4 bits (QAT or QAD
   in its name): it reads prompts fastest, but on a model not trained for it, it lost the most.
2. **Leave repacking and mapping on Auto.** Repacking makes the CPU much faster, and Auto turns it
   off only when the model would not fit otherwise.
3. **Try both the CPU and the GPU (Vulkan), and keep the faster.** Whether the GPU is faster depends
   on the GPU and its driver, not on anything a list can tell you: on the Pixel 6 the CPU won on
   every file. **Auto** picks the GPU whenever the phone has one, so pin each backend in turn.
4. **If the app warns about memory, follow its suggestions in order**, and only then reach for a
   smaller file. Turning repacking off saves the most memory and costs the most speed.

The rest of the article explains why.

## What we measured

Two phones and one laptop, all with llama.cpp build b11946:

| Device | Processor | GPU | RAM |
|---|---|---|---|
| Pixel 6 | Google Tensor G1 (Arm, dot product) | Mali-G78 | 8 GB |
| Pixel 9 Pro | Google Tensor G4 (Arm, i8mm, SVE2) | Mali-G715 | 16 GB |
| Laptop | AMD Ryzen AI 9 HX 370 (Zen 5, AVX-512) | Radeon 890M (integrated) | 32 GB |

Two model families, each in every common quant: **LFM3 1.7B** (about 1 GB at 4 bits) and **Gemma
4 E2B** (about 3 GB at 4 bits). Speed is `llama-bench`'s prompt reading (512 tokens) and reply
writing (128 tokens), in tokens per second, with each run started on a cool phone. Memory is what
Android counts against the app. Quality is measured on the laptop as the divergence of each quant's
predictions from the 8-bit file's (KL divergence over a Wikipedia sample; lower is closer).

These are three devices. Your numbers will differ, sometimes a lot; the point of the tables is the
shape of each trade, not a ranking.

## How Android counts an app's memory

Android, and Google Play's quality checks, count an app's _anonymous_ memory: what it allocated
itself. They don't count file pages the app maps, because the system can drop them and read them
back from the file. Play holds apps to limits by device RAM: on an 8 GB phone, about 2.25 GB while
you use the app and 1.5 GB once you leave it.

That makes **where the weights live** the biggest factor in an on-device model's memory:

| Where the weights are | Counted? |
|---|---|
| Mapped from the file | no |
| Read into memory | yes, in full |
| Repacked copies | yes |
| On a Mali GPU (Vulkan) | no, but it is real RAM the phone has to find |

## Repacking: the biggest memory and speed trade

Arm and x86 processors read some weight types much faster when the weights are rearranged into a
layout that fits their vector instructions. llama.cpp does this at load time, and the rearranged
copy lives in memory even when the file is mapped. So repacking costs about as much memory as the
weights it copies, and in return speeds up both reading your prompt and writing the reply.

On the Pixel 6, with the CPU and the file mapped (prompt / reply tokens/s, memory counted):

| File | Repacked | Not repacked |
|---|---|---|
| LFM3 QAD Q4_0 | 95 / 18.3, 1.1 GiB | 33 / 13.1, 0.13 GiB |
| LFM3 Q4_K_M | 68 / 16.2, 1.1 GiB | 26 / 13.3, 0.13 GiB |
| Gemma 4 E2B QAT UD-Q4_K_XL | 64 / 13.0, 1.4 GiB | 23 / 9.6, 0.22 GiB |
| Gemma 4 E2B Q4_K_M | 39 / 10.4, 1.5 GiB | 18 / 8.7, 0.23 GiB |

Without repacking, prompts read 2 to 3.5 times slower and replies 20 to 50% slower, but almost
nothing is counted. That is why Bring Your LM's **Repack the weights** setting defaults to
**Auto**: it repacks unless the repacked model would not fit in what the phone allows the app, and
then loads without.

Not every type has a repacked layout. On these phones `Q4_0`, `Q4_K`, `IQ4_NL` and `Q8_0` repack;
`IQ4_XS` and most of `Q3_K_M` don't. Which types repack depends on the processor, so the app reads
it from the engine rather than from a list, and names them when it warns about the copy. On the
laptop's AVX-512 processor, repacking helps `Q4_0` but makes `Q4_K` prompts about a quarter
slower; there, **Off** is the faster setting for `Q4_K` files.

## Choosing the file

A quant is the same model with its weights stored in fewer bits. Fewer bits mean a smaller file
and less memory, and the answers drift further from the original. Different quant types also run
at different speeds, because only some of them have fast kernels on a given processor or GPU.

Quality, as divergence from the 8-bit file (lower is closer):

| Quant | LFM3 1.7B | Gemma 4 E2B |
|---|---|---|
| UD-Q4_K_XL | — | 0.12 |
| Q4_K_M | 0.12 | 0.17 |
| IQ4_XS | 0.12 | 0.19 |
| IQ4_NL | 0.09 | 0.19 |
| Q4_0 | — | 0.35 |
| Q3_K_M | 0.25 | — |

The QAT and QAD files are not in the table. Each is a retrained model, and we compared it against
the regular model's 8-bit file, so its number mixes the retraining in with the rounding.

What the measurements say about each type:

- **Q4_K_M and the imatrix Q4_K_XL ("UD") files** are the safe default: good quality on both
  models, repacked to full speed on Arm phones.
- **Q4_0** read prompts fastest of the 4-bit files on the phones' CPUs, but on Gemma, which was not
  trained for it, it drifted twice as far as Q4_K_M. Prefer it when the model was trained for it (QAT
  or QAD in the name): that training targets exactly Q4_0's rounding. Bring Your LM notes a Q4_0 file
  whose name says no such training.
- **IQ4_XS** keeps about Q4_K_M's quality in a slightly smaller file and repacks only a quarter of
  a GiB, so it counts the least memory of the 4-bit files with repacking on. The price is speed:
  on the phones' CPUs it reads prompts at a third of Q4_K_M's speed.
- **IQ4_NL** repacks and runs well on the CPU, but was the slowest file on both phones' GPUs.
- **Q8_0** is close to the original, about twice the size of a 4-bit file, and on a phone writes
  replies slower than a 4-bit file even though it reads prompts nearly as fast.
- **Q3_K_M** is a fifth smaller than Q4_K_M, clearly worse, and on the phones' CPUs not faster,
  since it barely repacks.

### Files made with an importance matrix

The same quant type comes in better and worse versions. Whoever made the file can first run the
model over sample text and record which weights matter most, an _importance matrix_ (imatrix), and
let the quantizer spend its precision there. The result is an ordinary file of the same type and
size: Bring Your LM, like any llama.cpp app, loads it without knowing an imatrix was involved.

It is worth looking for. We quantized LFM3 1.7B twice from the same 8-bit file, with and without
an imatrix its publisher made from their own text, and measured both on Wikipedia text:

| Type | Without imatrix | With imatrix |
|---|---|---|
| Q4_K_M | 0.123 | 0.102 |
| Q4_0 | 0.190 | 0.168 |

An imatrix took a sixth off Q4_K_M's divergence and an eighth off Q4_0's, for free in size and
speed. Unsloth's "UD" files, AtomicChat's "AD" files and most repositories with "imatrix" or
"i1" in the name are made this way; the model card usually says. Not every file says so inside:
one Q4_0 we downloaded carried no record of an imatrix and measured like one made with it.

A larger model at 4 bits is usually better than a smaller one at 8 bits, if it fits.

## CPU or GPU

A phone's GPU shares the phone's RAM, so offloading the model to it doesn't free memory for other
apps, though on Mali it moves it out of what Android counts against Bring Your LM. Whether it is
_faster_ depends on the GPU, its driver and the quant:

- **Pixel 6 (Mali-G78):** the CPU repacked was faster on every file, prompts 2 to 8 times, replies
  about twice. Vulkan also took 1.5 to 4.5 GB of graphics memory, more than repacking did.
- **Pixel 9 Pro (Mali-G715):** it depends on the phase. The CPU repacked wrote every reply faster,
  by 10 to 90%: LFM3 Q4_0 at 23 tokens/s against Vulkan's 14. Vulkan read prompts faster for every
  Gemma file, 2 to 3.5 times, and for the files that don't repack (IQ4_XS, Q3_K_M); for the other
  LFM3 files the CPU was as fast or faster. Vulkan kept what Android counts near the unrepacked
  CPU's, and used 2 to 5 GB of graphics memory instead.
- **Laptop (Radeon 890M):** Vulkan read prompts 3 to 6 times faster than the CPU and wrote replies
  no faster; on Gemma, a quarter slower. ROCm, AMD's own backend, was slower than Vulkan for most
  files on this integrated GPU.
- **Snapdragon (Adreno 830):** llama.cpp's Vulkan backend did not work on the one we tried.

So the only reliable way to choose is to try both. Pick **Vulkan** under **Compute backend** in the
model's settings, send the same message as before, and compare **Show performance** under the two
replies: the first row is how fast your prompt was read, the second how fast the reply was written.
For chat, the reply speed matters most; for long documents or images, the prompt speed.

## Mapping the file

With mapping on, the weights stay in the file and the system pages them in as needed, so they don't
count against the app and the model loads in moments the second time. Leave **Map the model file**
on **Auto**: it maps when the file and storage allow and reads the file otherwise. Turning it off
makes every byte of the weights count.

## Threads

On a phone, the app uses half of the cores and never fewer than four: the fast cores. Using all
eight adds the efficiency cores, which then hold the fast ones back. On the Pixel 6, all eight
threads wrote replies at 1.5 to 8 tokens/s where four wrote at 18; on the Pixel 9 Pro, eight
wrote at 3 to 8 where four wrote at 9 to 23. Leave **Decode threads** on
automatic.

## On a desktop

The same choices apply, without Play's limits. On an x86 processor the engine picks the build of its
CPU code that matches your processor's vector instructions: AVX2 reads prompts 6 to 25 times faster
than the plain x86-64 build, and AVX-512 up to 1.7 times as fast again. The provider's **Supported
backends** row shows which ones the engine uses, for example "CPU (AVX2, AVX-512)". With an
integrated GPU, Vulkan is the faster choice for long prompts; for chat, try the CPU too.

## How Bring Your LM helps

The model's settings page sizes the load before it happens, with llama.cpp's own dry run:

- **The memory bar** shows how much of the model the app holds, how much is mapped and how much is
  on the GPU, against your phone's limits.
- **Findings** name what costs memory a different choice wouldn't: a repacked copy (and which
  weight types), weights read into memory, a large context.
- **Suggestions** list the changes that would help, each with what the dry run says it saves and
  what it costs, and **Apply** sets it.
- **Auto repacking** keeps the default load within what the phone allows.

## A procedure for your device

1. Download a 4-bit K-quant made with an imatrix (or the QAT/QAD Q4_0, if there is one) that is well
   under half your phone's RAM.
2. Leave repacking and mapping on Auto. In the model's settings, set **Compute backend** to
   **CPU** and send two or three messages that get replies of a few sentences.
3. Set **Compute backend** to **Vulkan** and do the same. The model's settings page now lists both
   under **Speed on this device**, and if one replied clearly faster, **Apply** switches to it.
   (**Auto** picks Vulkan whenever the phone has it, which is not always the faster choice.)
4. If the settings page warns about memory, apply its suggestions from the top. If the model still
   doesn't fit with repacking on, try the same model as IQ4_XS: less memory, slower prompts, the
   same quality.
5. If replies are too slow on both backends, try a smaller model rather than a smaller quant of the
   same one.

## What we didn't test

Long conversations (we measured a 640-token context; a long context grows the memory and slows
both phases), Snapdragon and Exynos phones beyond the one Adreno we tried, models above 3 GB, and
answer quality on real tasks rather than divergence from the 8-bit file. If your device behaves
differently, the app's **Show performance** numbers are the ones that count.

---

[Bring Your LM](/) · [Docs](/docs/features) · [Marketplace](/marketplace)

[Articles](/articles) · [Releases](/releases) · [Bundle Format](/marketplace/bundle-format) · [Marketplace Terms](/terms) · [App Privacy](/privacy) · [Website Privacy](/website-privacy) · [Open Source](/open-source) · [llms.txt](/llms.txt)