Our in-product RAG chatbot has two halves. The first is retrieval — hybrid BM25 + bge-m3 + a fine-tuned reranker, running locally as native .NET/ONNX, with zero API surface to anyone. The second is generation — turning the retrieved chunks into an answer, or into a proposed metadata change. That second half talked to a cloud API with a per-token bill.
This post is about replacing that second half with a local model, and about a side effect that turned out to be more useful than the original goal: the same WUIC knowledge, exposed to Cline and Continue in VS Code, gives you a free agentic coding assistant that actually knows the framework — no API key, no per-token cost.
It is also an honest post. A local LLM is not "free" in the sense people usually mean. You trade money and privacy gains for quality and latency losses. The last section is the bill.
What actually moved
The retrieval half did not move — it was already local. Only the brain changed.
BEFORE retrieval (local ONNX) → Claude API ($ per token, cloud round-trip)
AFTER retrieval (local ONNX) → Ollama / qwen2.5 ($0, on a GPU box in the LAN)
The two are not the same model class, and pretending otherwise would be dishonest — more on that below. But the wiring is clean, because the engine already spoke an OpenAI-compatible dialect. Pointing it at a local server is configuration, not surgery:
rag-llm-provider = ollama # OpenAI-compatible wire format
rag-llm-base-url = http://<gpu-host>:11434/v1 # Ollama's OpenAI endpoint
rag-llm-api-key = ollama # dummy, non-empty
rag-llm-default-chat-model = qwen2.5-coder:32b
The topology
Three boxes. Your machine runs the WUIC backend and the local retrieval. A second machine with a GPU runs Ollama (it can be the same machine if it has the VRAM). VS Code talks to both through a small bridge.
Your machine GPU box (same LAN)
┌──────────────────────────────┐ ┌──────────────────────────────┐
│ WUIC backend :5000 │ │ Ollama :11434 │
│ ├─ RAG retrieval (ONNX) │ │ qwen2.5-coder:32b │
│ │ BM25 + bge-m3 + rerank │ ───► │ generation + tool-calling │
│ └─ /api/Rag/Query /Rag/Chat│ └──────────────────────────────┘
└──────────────────────────────┘
▲
│ stdio MCP (server "wuic-rag")
│
┌──────────────────────────────┐
│ VS Code │
│ Cline / Continue ───────► │ agentic chat: edit files, run commands
│ (brain = same Ollama) │ knowledge = WUIC RAG via the MCP server
└──────────────────────────────┘
The arrow from the backend to Ollama is the RAG chatbot's new generation step. The arrow from VS Code back to the backend is the new part: an MCP server that lets any agentic editor query the same WUIC retrieval.
The bridge: a tiny MCP server
MCP (Model Context Protocol) is the open standard for giving an LLM client tools. Cline, Continue, Cursor and Claude Code all speak it. So instead of building a bespoke VS Code extension, we ship a tiny stdio MCP server that wraps the REST API the backend already exposes:
You (in Cline): "where is the cookie login handled?"
│
▼
Cline (local model) decides to call a tool
│ tools/call wuic_codebase_search
▼
wuic-rag-mcp.mjs ──POST /api/Rag/Query──► RAG engine ──► top-8 chunks (file:line + snippet)
│
▼
Cline reads the chunks, answers with real citations — or starts editing
It launched with two tools: wuic_codebase_search (retrieval — file paths and snippets) and wuic_ask (a full RAG answer generated by the local model). It is zero-dependency — Node's built-in http, no npm install, runs on any Node ≥ 14.18 — and stateless: every call just proxies to the backend. That matters for distribution: it ships inside every WUIC app and is wired up automatically at first run, so an end developer gets a .mcp.json already pointing at it.
The crucial design point: the MCP server is not the agent. It adds knowledge, not autonomy. The editing, the command execution, the plan/act loop — that is Cline or Continue. We did not rebuild that; reusing a mature open-source harness was the entire point.
The one hard problem: tool-calling
Retrieval-only worked on day one. The interesting failure was the action path — the chatbot's ability to emit a structured tool call (e.g. "add a row-style rule with this JS condition") rather than prose.
Cloud models return tool calls in a structured field. Our local serving stack, under the hood, often did not: it put the tool call into the message content as raw text, in inconsistent shapes — sometimes valid JSON without the envelope, sometimes a JavaScript object literal with single quotes and key=value instead of key: value. The model knew what to do; the plumbing dropped it on the floor.
The fix was a tolerant parser on our side: detect a known tool name in the content, capture the following {...} block, and normalize that JS-ish literal into JSON — quote-aware, so embedded callback bodies with their own braces and quotes survive. One contained change took routing from unusable to good. The lesson is general: with local models, budget for the serving quirks, not just the model. The weights are capable; the adapter around them is where the work is.
Measured results — no rounding up
We benchmarked against our real tool-routing suite (89 prompt variants, 3 repetitions each, on a 24 GB GPU):
| Model | Routing pass-rate | Latency / call | Notes |
|---|---|---|---|
| qwen2.5-coder:32b | ~91% | ~8 s | best schema fidelity + code/JS generation |
| gpt-oss:20b | lower | ~2.6 s | much faster, weaker on complex tool schemas |
The residual failures, when we dug in, were mostly not the model: a couple were genuinely correct behavior the single-turn test could not credit (the model asked a clarifying question instead of guessing — which the real multi-turn product handles fine), and one was a genuinely hard nested-JSON case. The honest count of real model-quality misses was ~2 of 89.
Context window matters too. A cloud model gives you 200k tokens. A 32B model on 24 GB of VRAM, after the weights, leaves room for ~24k tokens of KV cache (with q8 cache + flash attention). Plenty for a chat turn — but it changes how aggressively you summarize a long conversation.
The bill: pros and cons of going local
This is the part most "run an LLM locally!" posts skip.
What you gain
- Cost. Zero per-token. The whole reason we started. A team hammering an assistant all day costs nothing after the hardware.
- Privacy. The prompt — your code, your schema, your question — never leaves the LAN. For a closed-source product this is not a nice-to-have.
- No rate limits, no outages. Your throughput is your GPU, not someone's quota or status page.
- Availability determinism. The model you tested is the model you run, forever. No silent upgrades changing behavior under you.
What you pay
- Quality. A 20–32B local model is not a frontier model. On scoped tasks (one file, a clear instruction, a single tool call) it is genuinely good. On multi-file refactors, long-horizon planning, and recovering from its own mistakes, it is visibly weaker — and that gap is largest exactly where agentic coding lives.
- Latency. ~8 s per call for the 32B vs ~1–3 s for a small hosted model. Smaller local models are faster but trade away the quality you came for.
- Hardware + ops. You need a GPU with enough VRAM (24 GB for a 32B at 4-bit), and you have to run it: expose it on the LAN, keep it warm, restart it on boot, tune the context window. It is a service you now operate.
- Tooling friction. As above — the serving layer's tool-calling quirks are yours to absorb.
The honest recommendation is hybrid. Use the local model for the bulk — Q&A over the codebase, scoped edits, the always-on assistant that costs nothing. Keep a frontier model in your pocket for the genuinely hard multi-file work where its reasoning earns the per-token price. Replacing the API key everywhere is a slogan; replacing it where the quality gap does not bite is an architecture.
Setting it up
Because it ships in the workspace template, a WUIC app is agentic-ready out of the box:
.mcp.jsonis generated at first run, already pointing at thewuic-ragserver — Claude Code and Cursor pick it up automatically..clinerulesand.cursorrulesare generated from the same single source as the agent rules (AGENTS.md), so Cline and Cursor get the framework conventions and a nudge to search the codebase before inventing — no copy-paste.scripts/mcp/carries the server, a runbook, and anOLLAMA-SETUP.mdthat walks you through standing up your own Ollama box (LAN exposure,num_ctxtuning, model pull, persistent startup) on Windows or Linux.
You install the agentic extension of your choice, point it at your Ollama host, and you have a WUIC-aware agentic assistant in your editor for the price of the electricity.
Local LLMs are not magic and they are not free. But for "an assistant that knows our framework, runs all day, and never sends our code anywhere" — they are exactly the right tool, and the trade-offs are ones you can plan around once you see them written down.
Update — July 2026: a faster brain, a longer memory, and a second serving quirk
A month of running this in anger changed most of the numbers above — for the better. Same architecture, same honest accounting.
The model is now a Mixture-of-Experts, fine-tuned on our own mistakes. The dense qwen2.5-coder:32b gave way to qwen3-coder:30b — a MoE with ~3B active parameters that generates at ~197 tokens/s on the same 24 GB GPU (the dense 32B did ~30). Then we DPO-trained it on the assistant's own failure traces, and that fine-tune is now the default brain for the in-product chatbot too. On the routing suite — grown from 89 to 102 variants — the base MoE scores 88/102; the DPO fine-tune scores 96/102 at identical speed. The ~8 s/call in the table above is now ~2–7 s.
The context window doubled, and here is the honest math. We wrote "~24k tokens of KV cache" above. With grouped-query attention (4 KV heads) and a q8 cache, this model's KV costs ~52 KB/token — so 49k of context adds only ~2.6 GB on top of ~19 GB of weights. 49152 tokens, still 100% on-GPU, generation speed unchanged. 64k also fits, but leaves ~1.7 GB of headroom on a box that also drives a desktop — we passed. Context is never "free at the same VRAM"; it was simply cheaper than we assumed.
A second serving quirk, same lesson as the first. The MoE sometimes serializes array-typed tool arguments as a stringified JS literal — "layout": "[{'tool_name': ...}]" instead of an actual array. The schema validator rightly drops it, and our designer-tool routing quietly fell to 9/21. One more tolerant-parser pass (coerce string arguments that parse as arrays/objects back into structure, code fields excluded) took it back to 20/21. If you keep one sentence from this post, keep this one: the weights are capable; the adapter around them is where the work is.
The MCP server grew from two tools to nine. Alongside retrieval and Q&A it now exposes metadata-aware tools — the real column names of a route, lookup resolution (displayed value → ID), sample records, the exact schema of each metadata action, and the framework's skill playbooks — because the single biggest failure mode of a local agent is inventing field names. The chatbot's own tool catalog, meanwhile, is up to 22 tools.
The toolbox grew a dimension. The chatbot now drives the 3D scene designer the same way it drives the dashboard designer: "add a red cube", "kanban wall of the orders by status", "build a warehouse scene bound to the stock table" — one tool call, applied client-side to the live three.js scene, data-bound through the 3D mesh repeater. Same host-registry pattern, new canvas.
And the frontier-vs-local check, re-run. We benchmarked a newer dense 27B against the MoE: it is genuinely better at routing (perfect on the kinds the MoE fumbles) — and 4.5× slower (44 vs 197 tok/s), with no MTP head in the quantized build to claw the speed back. For an interactive assistant we kept the MoE + DPO: 96/102 at full speed beats ~99/102 at thirty seconds a reply. The hybrid recommendation above stands — now the local half is just much closer to the frontier than it was in June.
Discussion welcome — if you've run a local model behind an agentic editor and hit different serving quirks than these two, I'd genuinely like to hear about them.