2026Author
Device-Aware RAG Assistant
Retrieval assistants usually assume one fixed backend and a live network, so they either burn money on every query or fall over the moment they run somewhere constrained or offline.
The result
The same code detects its runtime and picks its own backend, running fully offline at zero per-query cost when it has to, with no user-facing toggle.
Device-Aware RAG is a retrieval-augmented assistant whose defining idea is a capability-aware provider chain: it reads network, CPU, and RAM at runtime and routes between a cloud model, a quantized on-device model (GGUF via llama-cpp-python), and a fully offline mode. Retrieval is hybrid, narrowing by date before semantic search, and a pinned, model-aware embedding cache keeps vectors from different models from ever mixing. It was validated on an Android emulator with no network at all.
- Backends
- Cloud / on-device
- Demo runtime
- 100% in-browser
- Per-query cost
- $0
- Retrieval
- Date + semantic
Architecture
Capability router
reads network, CPU, RAM; no user toggle
Cloud model
when resources are free
Quantized on-device
GGUF via llama-cpp-python
Fully offline mode
no network path
A query first goes through hybrid retrieval, which narrows the candidate set by date and then ranks the survivors by semantic similarity, reading vectors from a cache keyed to the exact embedding model that produced them. The retrieved context and the query then reach the capability router, which has already inspected the device and the network. If resources are free it calls a cloud model; if not, it runs a quantized GGUF model on the device; if there is no network at all, it stays fully offline.
Key decisions and trade-offs
A capability-aware provider chain with no user-facing toggle
Exposing a cloud-or-local switch pushes a decision onto the user that depends on runtime facts they cannot see (current bandwidth, free memory, whether the device is even online). Detecting those conditions in code and routing automatically means the same build works on a fast laptop and an offline phone without a settings screen or a second code path to maintain.
On-device GGUF over a smaller cloud model
When resources are tight, the reflex is to call a cheaper hosted model, but that still needs a network and still bills per query. Running a quantized GGUF model on the device instead removes the round trip entirely: it costs nothing per query, keeps working with no connection, and never sends the query text off the machine. The trade is a heavier model to load locally, which the router only pays when it actually chooses that path.
Hybrid retrieval that narrows by date before semantic search
Pure vector search treats a stale document and a current one as equally relevant. Filtering by date first shrinks the candidate set to what is actually in scope, so semantic ranking spends its budget on plausible answers and recency is respected instead of accidental.
A pinned, model-aware embedding cache
Embeddings from two different models live in incompatible vector spaces, so comparing them by cosine is meaningless and silently wrong. Keying the cache to the model that produced each vector, and pinning that model, guarantees a query is only ever compared against vectors from the same space, and it makes cache reuse safe when the backend changes.
Results
- 108+ passing tests.
- Validated end to end on an Android emulator with no network.
- Zero per-query cost in offline and on-device modes.
Repository
The on-site search is a live, browser-only slice of the same idea: it embeds every post on your device with a quantized MiniLM model and ranks by cosine locally, so search costs nothing per query and no query text leaves the page.