# On-device models — run fully offline The embedded llama.cpp provider runs your own GGUF model files in-process — no server, no URL, no API key — with a compute backend you pick per device, and vision support. Status: Shipped Alongside the remote and local-server providers, Bring Your LM has a built-in llama.cpp inference engine. Point it at a folder of GGUF model files you supply and it runs them in-process, on the device, with nothing leaving it — no server to stand up, no URL, no API key. ## Why it matters It's the most private and the most self-contained way to run a model: fully offline, on hardware you already carry. You still bring the model — a GGUF file you downloaded or converted — but there's no service in the loop. ## How it works Add a **Llama.cpp on device** provider and give it a **models directory**. On desktop that's a path you type; on Android you grant a folder once and the app reads it through the system file-access framework. The directory is scanned recursively and each model file becomes selectable. The provider's settings are saved with **Save**, like any other form. ![The on-device provider's form — models directory, compute backend, and decode threads, no API key](/images/provider_embedded.png) You assemble each model in the app rather than hand-editing anything. A model is built from: - a **base** GGUF (the model itself), - an optional **multimodal projector** (an mmproj file) that gives a vision-capable model the ability to actually read attached images, and - an optional **speculative-decoding draft** for faster generation on supported models. The file picker only offers files that fit each slot and shows each file's size, so you don't pick a projector where a base model belongs. A model's name is derived from its components (with markers like "+ vision"), and you can override it. Pick the **compute backend** per device — **Auto** lets llama.cpp choose, or pin **CPU**, **Vulkan**, or (on Linux) **ROCm** for an AMD GPU. This choice is saved on the device only, since the right hardware for a phone means nothing on a desktop. You can also set the number of decode threads (or leave it automatic, with a recommended value shown). Each reply's metrics show which backend it actually ran on. **Supported backends** lists what this device can run, and its CPU entry names the vector instruction sets the engine uses there, such as "CPU (AVX2, AVX-512)" on a desktop or "CPU (Neon, DotProd)" on a phone. A model running on the CPU names them too, in the conversation's usage strip and in each reply's metrics. A model can run on a backend of its own on this device: pick it under **Compute backend** in the model's settings, which the button beside the model's name in a conversation's expanded usage strip opens. **Inherit** follows the provider's pick. The next message loads the model on the new backend, and the strip's **Backend** row says what it runs on. ## Mapping the model file By default the model file is memory-mapped where the backend allows it: its weights are read from the file as they are needed instead of being copied into memory first. Android can reclaim a mapped file's memory without stopping the app, and a reload while the file is still cached is quick. The provider's **Map the model file** setting chooses, per device: - **Auto** — the default. The file is mapped where the backend supports it, and read into memory when it cannot be mapped. - **On** — always map; a file that cannot be mapped fails to load, with the reason. - **Off** — read the base model's weights into memory. They take as much memory as the file, and Android cannot drop them to re-read later. A draft head is still mapped, and so are the parts a few models (Gemma 4) read only when they need them, so those can still wait on Android re-reading a page. A model can pick its own in its settings, or **Inherit** the provider's. Each reply's metrics say whether its model's weights were **Mapped** or **Read into memory**, so a file Auto could not map shows there. ## Repacking the weights On the CPU, many quantized models run faster once their weights are copied into layouts the processor handles better. The copy lives in memory even when the file is mapped, so it costs about as much memory as the weights it repacks. The provider's **Repack the weights** setting chooses, per device: **On** repacks, **Off** keeps the weights as they are in the file — on a phone, prompts up to three times slower and replies slower too, but with the file mapped, far less memory Android counts against the app. **Auto**, the default, repacks unless, on Android, the repacked model would not fit in the memory the phone allows the app while you use it. On a desktop Off drops the layouts for x86 processors with AVX2, AVX-512 or AMX, which slows Q4_0 prompts; with AVX-512, Q4_K files read prompts faster without them. A model can pick its own in its settings, or **Inherit** the provider's. ## Keeping past thinking A reasoning model's thinking goes back to it with the conversation. Within a reply that calls tools, the model always reads its earlier thinking again. Across your messages, **Keep past thinking**, on the provider's page, decides, per device: - **On** — the default. The model reads its earlier thinking too, so each new message continues from where the last reply ended and only the new part of the conversation is read. - **Off** — earlier thinking is left out, as many reasoning models' own templates do. The conversation takes less of the context, but a model such as LFM3, which cannot pick up part way through, reads the whole conversation again for every message. A model can pick its own in its settings, or **Inherit** the provider's. A model whose template has no way to keep past thinking leaves it out either way. ## How much memory a model takes Each reply's metrics draw the memory the model took on the device that wrote it as a bar, with a colour for each part: the **base model**, a separate **draft head**, the vision **projector** while it is loaded, the **context** (its cache and working buffers), and grey for the rest of the app. Every figure is what was _allocated_. Hover or long-press a part for its size. A context is allocated for its full size when the model loads, however little of it a conversation fills, so a smaller context size is the quickest way to bring that part down. On a reply this phone wrote, the bar is measured against this phone's limits and notes what to change, as below; a reply from another device shows its memory without them. ## Fitting in the phone's memory Android limits how much memory an app may hold, by how much the phone has. The usage strip at the top of a conversation, once expanded, draws the loaded model's memory as the same bar, split three ways: - **App memory**, solid — what Android holds against the app. - **Reclaimable**, hatched — weights read from the model file, which the phone can drop and read again when it needs the memory. - **GPU**, dotted — what the GPU holds outside the app's memory. The bar marks the **Recommended** size once the app's memory comes near it — over it, a model left loaded after you leave the app may get the app stopped, so the note points at **Release the model in the background** — and the **Limit**, over which Android may stop the app. A model well inside both just fills the bar. When a setting makes the model take more memory than it needs to, the part it is behind is outlined and a note under the bar says what happened and how to fix it: weights copied out of the mapped file by repacking (naming each weight type copied and how much of it), weights read into memory instead of mapped (repacked ones included, since repacking off alone leaves them in memory), or, near a limit, a large context or a loaded vision projector. Weights read in with mapping on Auto mean the file could not be mapped on this phone, so the note says that rather than to turn mapping on, which would fail the load. For a context it says how much is the cache, which a smaller context or cache type shrinks, and how much is working buffers, which neither does. A model's settings page draws the same bar for the settings you are editing, before anything loads, counting its draft head, and its vision projector when it loads with the model; otherwise a line under the bar says how much a turn with an image adds. Under each note it lists the changes that help — repacking off, running the model on Vulkan, mapping the file (Auto, so a file that cannot be mapped still loads), a smaller context, an 8-bit or 4-bit cache, loading the projector at the first image — each with what it saves and what it gives up, and **Apply** puts one into the form. Vulkan is offered only where the phone has it, and says how much it puts in GPU memory, which is still the phone's RAM; whether it is faster depends on the GPU, so compare under **Speed on this device**. A cache already 4-bit is said to be the smallest. Over the recommended size, the changes that answer no note are listed under the bar as well. A model whose load would go over the limit for the app in front is not loaded: the reply says how much it would take and what the phone allows. To load it anyway on this phone, turn on **Load even when over the limit** in its settings; Android may then stop the app while the model is loaded. When you leave the app, a loaded model stays for a minute, then is let go; the next message loads it again. A reply still being written in the background finishes first. **Release the model in the background**, on the provider's page, sets how long it stays on this device — 30 seconds to 15 minutes, or **Never**, at the risk of Android stopping the app instead. ## Which setup is fastest on this phone Whether the GPU or the CPU runs a model faster depends on the phone's GPU, its driver and the model's weight types, so the app measures it instead of guessing. Each reply's speed is kept on this device for the setup it ran with — the backend, and on the CPU whether the weights were repacked — and the model's settings page lists them under **Speed on this device**, from two replies each. When another setup replied clearly faster than the one used last, it says how much faster, and **Apply** switches the model to it. A backend the phone has but the model never ran on is named as not tried: pick it under **Compute backend** and send a message to compare. Before any reply, the same section says when the phone's processor has no faster layout for the type most of the model's weights are in (IQ4_XS on an Arm phone, for example), so the model reads prompts at the unrepacked speed whatever the repacking setting, and names the types it does repack: a file of the same model in one of those reads prompts faster, at the cost of the memory the copy takes. Under the model's file, the page notes a Q4_0 file whose name does not say the model was trained for 4 bits (QAT or QAD). Q4_0 is quick to read but keeps less of a model than Q4_K_M unless the model was trained for it, so a Q4_K_M file of an untrained model answers closer to the original. ## When the vision projector loads A projector takes memory of its own, beside the model. By default it loads at the first image, so a vision model holds none of that memory in a conversation that never shows it one. The provider's **Vision projector** setting chooses, per device: - **With the model** — the projector loads with the model, so the first image is read without a wait, but the memory is held even in conversations without images. - **At the first image, then kept** — the default. The projector loads on the first turn that carries an image and stays until the model itself is unloaded. - **Only while a turn has images** — the projector is also released again on the next turn without one. Generating a conversation's title or summary does not count as such a turn. Saving the provider's settings saves this choice too, so a device whose settings were saved while **With the model** was the default keeps it until you pick another. A model with a projector can pick its own policy in its settings, or **Inherit** the provider's — useful when one small vision model should keep its projector loaded while a large one on a phone loads it late. Like the provider's setting, a model's pick stays on this device. When the projector loads late, the model itself stays loaded: only the projector comes and goes. The cost is time, and the app says so under the setting — a turn with an image waits while the projector loads. If the memory it needs has been taken by then (a larger context, say), images go unseen until you ask for the projector again from the note under the reply. A conversation that had an image anywhere in it keeps needing the projector for as long as the image is part of what the model reads. Whenever a reply was written without the conversation's images, a note under it says how many and why: the model reads text only, its projector file is not in this device's models folder, or the projector failed to load (built for another model, or no memory left for it). The note is saved with the reply, so it is still there after a restart and on your other devices. A reply's metrics name the **Projector** when it was loaded for that turn, and the **Draft head** the model speculated with, each by its file's name and quantization, so a reply says what it ran with even after the settings change. A projector file you add to the models folder is picked up by the next turn that needs it. One that failed to load is not tried again on its own, since every try reads the whole file: tap **Try loading it again** on the note, and the next turn that needs the projector loads it. ## Each model's settings belong to the model How a model is loaded — the context size, the KV cache types, whether its file is mapped and its weights repacked, when its vision projector loads, and the draft head it speculates with here — is set on the model itself, not on an agent. Every agent that uses that model loads it the same way, and two quantizations of the same weights are two models with their own settings. Reach them by opening a model's row in the provider's model list — **Model settings** sits under the weights and projector slots, so it opens the settings of the pairing those slots are set to — or from the tune icon beside an agent's bound model, or from a reply's metrics. Opening the settings of a pairing is what sets it up, so it appears under **Assembled models** on the provider page afterwards. Context size and cache types carry a scope: **All devices** stores what the model is _meant_ to run at and travels with it, while **Only this one** is this machine's veto — the cap a phone needs without retuning the desktop. A value neither layer has set is saved for this device; sharing it is one tap. The draft head has no shared option at all, because whether it speeds a model up or slows it down depends on the hardware in front of you. How many tokens the draft head proposes per step is **Auto** unless you set a number. Auto times each step on this device and drafts what turns out fastest, from nothing up to what the head was trained for, and goes on without the head for the rest of a reply when plain generation is faster. A number pins every step to it. ## What it doesn't do yet - **Speculative decoding is experimental.** The draft-model slot and its controls are in the form, but the speed-up currently applies only to text turns on specific paired models — a turn with an attached image runs without it — and depends on runtime pieces that aren't fully in place yet. - **ROCm is Linux-only.** On Android the offered backends are CPU and Vulkan. On Linux it takes the ROCm [download](/#linux-rocm) and AMD's ROCm runtime installed on the machine; the plain download runs on Vulkan or the CPU. The download entry lists the AMD architectures (`gfx…`) its ROCm backend was compiled for. - **No 32-bit Android.** On a phone running a 32-bit Android the provider type is listed but disabled, and one synced in from another device says it can't run there. Everything else in the app works. - **You still supply the model file.** Nothing is bundled or downloaded for you; the provider only runs GGUF files already in the folder you grant it. More in [Features](/docs/features): [Providers — bring your own LLM](/docs/features/providers) · [Agents — the thing you build](/docs/features/agents) · [Tools — giving an agent more than the model knows](/docs/features/tools) · [Workspaces — keep your work in scopes](/docs/features/workspaces) · [Conversations — how a turn runs](/docs/features/conversations) · [Attachments — images and files in a message](/docs/features/attachments) · [Context management — what the model still sees](/docs/features/context-management) · [Background turns — replies that keep running](/docs/features/background-turns) · [Local-first & private by design](/docs/features/local-first) · [Sync — your data across your devices](/docs/features/sync) · [Sharing — move agents, tools and providers as a file](/docs/features/sharing) · [Marketplace — find and publish shared agents, tools and providers](/docs/features/marketplace) · [Running a Couchbase Edge Server](/docs/features/couchbase-edge-server) · [Spell checking — catch typos while you write](/docs/features/spell-check) --- [Bring Your LM](/) · [Docs](/docs/features) · [Marketplace](/marketplace) [Articles](/articles) · [Releases](/releases) · [Bundle Format](/marketplace/bundle-format) · [Marketplace Terms](/terms) · [App Privacy](/privacy) · [Website Privacy](/website-privacy) · [Open Source](/open-source) · [llms.txt](/llms.txt)