AI, Local LLMs, and Agents

I run large language models on my own GPUs, and I route the work that is safe to send out to whatever cloud model is cheapest for the job. That means a private LLM router that decides what leaves my network, a couple of RTX cards in a workstation under the desk, and a benchmark harness that scores models on the actual coding tasks I do rather than on somebody else’s leaderboard. Most of what I write about AI here is the unglamorous half: getting llama.cpp to compile, working out why a model that looked fast is not, catching an agent when it confidently reports a fix that never happened. This hub collects those write-ups, roughly grouped by what I was trying to get working at the time.

Running local models on your own hardware

The starting point was a used Dell Precision T5820, a Quadro I pulled out of it, and an RTX 3090 Ti I dropped in, which is how I ended up building llama.cpp from source on that box after seven power cycles. Once it was running, most of the published throughput numbers I compared against turned out to be measuring something other than what I cared about, which is the subject of why your llama.cpp benchmarks are wrong. I then spent three months trying to make the same card go faster, testing autoregressive decoding against DFlash and MTP on Qwen3.6-27B, where the answer turned out to depend entirely on how many tokens you are generating. On the Apple side, getting a 30B coder model stable on a Mac Studio meant working through LM Studio’s prompt truncation and Jinja failures, and running a whole local agent team on Apple silicon is covered in the OpenClaw build, 13 errors and $1.50 a month.

Routing, cost, and what leaves the network

Every prompt is a decision about money and about privacy, and I did not want to make that decision by hand 200 times a day. So I built a private LLM router where one boring lookup table decides which task types are allowed to reach a cloud API at all and which stay on the GPUs at home. The cost side of the same idea, sending each task to the cheapest model that can actually handle it, is written up as inference arbitrage across five models. Free tiers are part of that math until they are not, which is why I tested what Groq, Cerebras, Mistral, Gemini and Cohere actually give you the week one provider quietly deleted the model my code depended on, and why I track Claude, Codex and Gemini quotas from a single script.

Which model should you actually use

Public benchmarks measure raw intelligence. Deployment decisions also turn on latency, format reliability, and whether the data is allowed to leave the building, so I ran my own: 15 models against 38 real coding tasks, scored on work I would have done anyway, for $2.29 in API spend. The short version, updated with what the results mean if you just want a pick per use case, is the best LLM for coding in 2026.

Building with agents

Coding agents forget things, and they forget them in ways that quietly wreck a long session. Auto-compact was doing exactly that to me, so I wrote up how to stop Claude Code from lobotomizing itself mid-task, and then the fuller answer: a persistent memory system of markdown files, two commands, and five cron jobs, plus the design rules I derived from breaking it repeatedly. The other half of agent work is pointing one at a system that was never designed for it, like driving WordPress from the terminal through REST, Playwright and wp-cli, or turning a local model loose on 2,300 unsorted files in my Obsidian vault to find out what a decade of digital hoarding actually contains.

When the model lies to you

The failure mode that costs the most time is not a model that refuses. It is a model that says “fixed” and moves on, which is what happened for three months on a script that had never once run correctly while every status report said otherwise. Agents also believe whatever they read, so I built a honeypot canary that screens web content with a deliberately gullible LLM before my real agent sees it, and wrote down what it caught. The same credulity applies to instruments: a $15 thermal sensor on an open-air GPU reads load backwards, and the LLM I asked about it fell for the reading exactly the way I did.

Watching the stack it all runs on

A local inference stack fails in ways a hosted API never will, and you only find out if you are watching. Three green lies is about the week I stood up a Grafana dashboard for the router and found three panels reporting healthy data that was not real, none of it a Grafana bug. Loading a big model is also the fastest way to discover how a Linux box behaves under memory pressure, which is how I ended up hardening SSH to survive the OOM killer after locking myself out of my own server.

---

All AI posts

Every AI write-up, newest first. This list updates itself as new posts go live.