It's not about the model

There is a particular kind of satisfaction in watching a model load. The layers march across the terminal one at a time as llama.cpp offloads them to the GPU, the numbers climb, and then comes the line that tells you how much VRAM is left over. I remember sitting in front of that line the first evening the second card was in the machine. Two Radeon AI PRO R9700, sixty four gigabytes of VRAM between them, and llama.cpp reporting [XX.X] GB for the weights of a thirty billion parameter coder model at four bit quantization. Room to spare, I thought. The hard part is over. Everything from here is just usage.

That turned out to be wrong in a way that took a few months to understand properly, and the understanding has been more interesting than the setup ever was. Everything below comes from one particular arrangement, llama.cpp and ROCm on two consumer class cards, so read it as field notes rather than as general truths about how inference works.

The model is the finished part

Almost every conversation about running language models locally circles the same drain. Which model, which quantization, how it scores, whether the new release beats the one from six weeks ago. It is a fun conversation and I have had it many times, usually with a browser tab of benchmark tables open that I am not really reading.

The odd thing is that past a certain threshold the answer stops mattering very much. For the work I actually do at home, reading logs, writing small tools, reasoning over structured data, drafting things, a good open weights model in the twenty to thirty billion range is not the constraint. It is competent. It is occasionally better than competent. When something goes wrong in my setup, and things go wrong constantly, the model is almost never the reason.

Which leads to a slightly deflating realization. The model is the one part of the stack that somebody else already finished. Someone spent an enormous amount of money and compute, packaged the result, and handed it over for free. It arrives done. Everything around it is where the work lives, and nobody hands you that.

So the renaissance of the local model, if we want to call it that, is not really about models at all. It is about what you have to build to make one useful.

The VRAM you did not count

The first thing that surprised me was memory, and specifically the fact that I was counting the wrong memory.

When people size a machine for local inference, they size it for weights. Eighteen or twenty gigabytes for the model, sixty four available, room to spare. You start thinking about whether you could fit something larger, or run two models side by side, or keep an embedding model resident.

Weights are the easy part. They are static, they are predictable, and you can calculate them on a napkin. The memory that actually runs out is the KV cache, and it behaves nothing like weights.

The cache holds key and value tensors for the tokens in the active context, across the model's layers and its KV heads. It grows with how much context you are holding and it grows again with how many requests you are holding at the same time. On the build I run, the context you set with --ctx-size is shared across the slots you ask for with --parallel rather than granted to each of them, which is worth verifying against your own version because this behaviour has moved around over the years. The practical effect is that the generous context you configured for one conversation is not the context an agent gets when it is running four things at once.

Agentic workloads are unusually cruel here. A chat session with a person is short and mostly idle. An agent loop is long, and every iteration carries the entire growing transcript back in, plus tool output, plus the file it just read, plus the error from the thing that failed. Context does not merely fill up. It fills up on the last iteration of a long run, after twenty minutes of work, which is a very specific kind of annoying.

There are things you can do about it, and they all cost something. Flash attention made a visible difference to the memory profile on my setup, enough that it is the first flag I reach for, though how much it buys you clearly depends on backend and model. Quantizing the cache itself from f16 to q8_0 with --cache-type-k and --cache-type-v roughly halves the storage required by the cache tensors themselves, and the quality hit was small enough that I stopped noticing, which I say as someone running no rigorous evaluation at all, only a feeling. Grouped query attention matters more than any of my flags, because it decides how many KV heads there are to cache in the first place, and that is a property of the model rather than something I get a vote on. And the prefill batch settings, -b and -ub, quietly reserve working memory of their own that nobody mentions in the guides.

Then there is the second bill, which is time. Even when a long context fits, processing it is not free. Prefill on a large prompt takes real seconds, and an agent that resends a growing transcript pays that cost again and again unless prefix caching lines up, which it often does not once a tool result changes something in the middle of the conversation.

So the honest version of the hardware question is not how many parameters fit. It is how much simultaneous context you want, at what latency, and the answer is usually less than you hoped.

Eighty percent, five times over

The second surprise was the tooling, and this is the part I did not expect to find interesting.

Open source has every piece. llama.cpp for inference, with a server that speaks something close enough to the OpenAI API. Ollama if you want model management to be pleasant rather than a directory of GGUF files with names you invented at midnight. OpenWebUI as a front end, which is genuinely good and does far more than a front end has any business doing. LiteLLM as a shim when something insists on a slightly different dialect of the same API. SearXNG when you want web search that does not report back to anyone. Embeddings, a reranker, a vector store, all available, all free.

Every single one of these does eighty percent of what you need. The problem is that it is never the same eighty percent.

A small example that cost me an evening. OpenWebUI can use SearXNG for web search, which is exactly the combination you want if the whole point is keeping things local. To make it work, SearXNG has to return JSON, and it does not do that by default, so you add json to the formats list in settings.yml. Fine. Then the rate limiter starts refusing your own requests, because it is designed to protect a public instance from exactly the kind of automated traffic your setup now generates, so you go and adjust that too. Fine. And then the whole thing works, and the retrieval pipeline dutifully pulls full page contents into the prompt, and you watch the context budget you spent the previous week carefully calculating evaporate on three pages of navigation menus and cookie notices.

None of these steps is hard. Not one of them is documented in the same place as the others, because they are five separate projects with five separate maintainers who owe each other nothing. Tool calling is the same story in a different key, split between the chat template, the flags that decide whether the server applies it, and whatever convention the client was written against. When it fails it rarely fails loudly. It fails by the model politely describing the function call it would like to make, in prose, as though narrating.

These projects are extraordinary and they are given away, so none of this is a complaint. But somewhere in the middle of the third integration I understood something that had been sitting in front of me the whole time.

The commercial providers are not selling models. They are selling the assembly.

It is worth being concrete about what that means, because the word model does a lot of quiet work in these conversations. When you send a message to a hosted assistant, the weights are one component among many. There is a system prompt that someone has been iterating on for months and that is longer than most of the code I write in a week. There is a tool harness that knows how to run a search, read a file, execute something in a sandbox and hand the result back in a format the model was trained to expect. There is retrieval that trims a page down before it reaches the context, memory of some kind, a router deciding which size of model gets your particular request, caching underneath all of it, a safety layer inspecting both ends, telemetry, and an evaluation suite that tells the people who built it whether last week's change made anything worse. The model is the part that gets the name on the box. It is arguably the most replaceable component in the whole arrangement.

That is the product. Once several models are good enough for the workload, the differentiation moves up the stack, and everyone in that market seems to know it even while the marketing insists otherwise. What you pay for is that you did not have to be the integrator.

The part where I admit I enjoy this

I should say plainly that a good deal of what I have described is not necessary. I could have paid for a subscription and had a better experience by dinner time.

Some of this is curiosity, some of it is professional, and some of it is that tinkering with infrastructure is one of the most satisfying ways to avoid doing the thing the infrastructure was supposed to enable. I have caught myself optimizing a cache setting for a workload I had not yet written. I have rebuilt a stack that was working. There is a reason the machine has a name and I talk about it as though it has moods.

But the experimenting does teach you something that reading cannot, which is where the actual edges are. Once you have watched your own context budget collapse under a retrieval pipeline you did not tune, you read a provider's context window number differently. Once you have measured prefill time on a long prompt, you understand what a per token price is really charging you for. The laboratory work is not efficient, but it recalibrates you.

What you are actually renting

And this is where the two halves meet.

Once you understand that the assembly is the product, you also see what comes attached to it. Rate limits. Context caps that are a business decision as much as a technical one. Models deprecated on a schedule you did not choose, taking your carefully tuned prompts with them. Regional availability. Data residency questions that matter enormously in some industries and not at all in others. A price per run set by somebody else, revised by somebody else.

There is also the safety layer, and here I have to tread carefully, because I do not think the people who build these systems are wrong.

I work in security, and a fair amount of that work is offensive. Which means that a normal Tuesday involves asking questions that, stripped of all context, are indistinguishable from the questions somebody with entirely different intentions would ask. Explain what this scanner finding actually means in practice. Help me reason about why this control did not stop what it was supposed to stop. Write up the exploitation chapter of a report for a system I have signed authorization to attack, with a scope document, a contact person and a date range.

A hosted model has to make a judgment about that with almost none of the context that makes it legitimate. It cannot see the engagement letter. It cannot see that there is a customer on the other end who asked for exactly this. Given how badly the wrong call goes in one direction, the systems are tuned to err in the other, and the false positives of that tuning fall disproportionately on people doing legitimate work that happens to look dangerous out of context. That is a defensible tradeoff and I would probably make the same one if it were my company. It is still a tax, and sanctioned security work is one of the places where that tax becomes unusually visible.

The version that wears on me is not the clear refusal. A clear refusal is at least information. It is the soft one, where the answer arrives shaped normally but with the useful specificity sanded off, and you cannot tell whether the model does not know, or knows and has decided you should not. For an expert user, that makes capability and policy failure difficult to distinguish. You start second guessing the output on every question, including the ones where nothing was withheld at all.

A local model, in my experience, mostly just answers. I want to resist the urge to frame that as wisdom, because it is not. It is the absence of a component. The responsibility that a provider was carrying for you has simply moved onto your side of the line, where it sits alongside the authorization letter and your own judgment about what you are doing and why. Running locally does not remove the provider. It makes you the provider. For someone with a scope document, that is the correct place for it. For someone without one, nothing about running the model locally makes the work any more legitimate, and it is worth saying so out loud.

Rate limits are the more mundane cousin of the same thing. It is not really about the quota. It is that the quota is invisible until you reach it, that it is usually measured per minute while you plan your work per task, and that an agentic run consumes tokens at a rate that has almost nothing to do with how large the task felt when you started. Twenty minutes into something that was going well, the thing stops. Or it does not stop, and everything simply gets slower, because capacity is shared and somebody else is having a busy afternoon.

I ran into all of this recently in a very ordinary way. There was a managed path available for a piece of work, technically better than what I could build at home, clearly the right answer on the merits. It is currently parked. Not because of anything wrong with it, but because it needs a tenant owner to weigh in, a role assignment somebody has to approve, a cost sign off for a workload measured in tens of dollars per run, and a verification that the data stays in the region it is supposed to stay in. Every one of those is reasonable. Together they mean the work waits.

The local stack exists in the gap that opens up while all of that gets sorted out. Not because it is cheaper, because once you count the cards and the electricity and the evenings it certainly is not. Not because of any purity about sovereignty either, though that word gets used a lot and occasionally means something. It exists because it is the version where I get to decide, and I pay for that in integration work instead of in permission.

That is a real trade and it is not obviously a good one. The hosted stack still does several things better than anything I have assembled, and pretending otherwise would be silly. The point was never to win the comparison.

The point, I think, is that until you have built the thing yourself, badly, on a machine in a room you can walk into, you do not really know what you are renting when you rent it.