The Local AI Box, three budgets

What this actually replaces

Same rule as the photo server: the money first, and honestly.

A chatbot subscription is $20/month. The cheapest box here is around $400, so call it a twenty-month payback. That sounds fine right up until you look at what you're comparing against. A 7-billion-parameter model running in your basement is not as smart as the model behind a $20 subscription. It isn't close, and anyone telling you otherwise has something to sell you.

So don't buy this to save $20. Buy it for the three things the subscription can't do:

• Nothing leaves the house. Your notes, your finances, your medical letters, your kids' names, your employer's source code. The reason most people can't use a cloud model at work isn't the price.
• There is no meter. Point it at ten thousand documents and it costs the same as pointing it at one. This is where local models genuinely win — bulk, boring, repetitive work you'd never authorise a per-call bill for.
• It can't be deprecated, rate-limited, price-hiked or discontinued. The weights are a file on your disk. It works with the internet unplugged.

If none of those three matter to you, keep the $20 subscription and keep your weekend. That's a real answer and for a lot of people it's the right one.

How to read a tokens-per-second number

A token is roughly three-quarters of a word. The number everyone quotes is generation speed — how fast the model writes its answer once it has started.

You read at about 250 words a minute, which is somewhere around 5 tokens per second. That's the line everything else is measured against:

There is a second number nobody quotes and everybody feels: prompt processing speed — how fast it reads what you gave it, before it writes a single word. Paste in a document and that's the silence you sit through. It separates these budgets far more brutally than generation speed does, and it gets its own table further down.

What I measured it on

Two machines, both mine, stated plainly so you can discount the numbers appropriately.

Software: Ollama 0.20.7, models at their default q4_K_M quantisation — qwen2.5:3b (1.9GB), qwen2.5:7b (4.7GB) and llama3.2:3b (2.0GB). Every figure is the median of three warm runs: the model was already loaded, so weight-loading time isn't hiding inside the result. CPU-only numbers were produced on the same machines by forcing every layer off the graphics card rather than by unplugging it.

The whole harness is this, against Ollama's HTTP API:

# evict whatever is resident first, so placement is deterministic
curl -s http://localhost:11434/api/generate -d '{"model":"qwen2.5:7b","keep_alive":0}'

# one warm-up call (thrown away), then the runs that count.
# "num_gpu": 0 forces CPU only. Remove it to let Ollama decide.
curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen2.5:7b",
  "prompt": "Explain in 200 words why a local model differs from a cloud API.",
  "stream": false,
  "options": {"num_gpu": 0, "num_predict": 256,
              "temperature": 0, "seed": 42, "num_ctx": 2048}
}'

The reply carries eval_count and eval_duration (nanoseconds). Generation speed is one divided by the other; prompt speed is prompt_eval_count over prompt_eval_duration. That's the entire trick. There is nothing proprietary about any of it, which is why it bothers me how many published tokens-per-second numbers have no command printed next to them.

If you'd rather not write the loop: ollama run qwen2.5:7b --verbose "your prompt" prints the same statistics after every answer. Run it twice and use the second one — the first includes loading the model off disk.

The numbers

Same machine, same models, same questions. The only thing that changes between the two columns is whether the graphics card is allowed to do the work.

Box B. Median of three warm runs generating 256 tokens each; both GPU rows confirmed at 100% GPU placement, both CPU rows at 0%.

And the same exercise on Box A's desktop card, with a 3B model:

Two things fall out of putting those tables next to each other, and both are more useful than any individual number:

• Generation speed tracks memory bandwidth almost exactly. Box A's desktop RTX 3070 moves memory about 1.75× faster than Box B's laptop RTX 4070 on paper, and it produced a near-identically-sized 3B model about 1.7× faster in practice — 165 against 97. Not cores, not the generation on the box. Bandwidth and VRAM. Buy on those two numbers.
• The same logic explains the CPU column. Box B's newer chip with LPDDR5-7467 memory ran a 3B model at 20 tok/s; Box A's older i9 on DDR4 managed 9.2 on a model the same size — roughly twice the memory speed, roughly twice the tokens. If you're building the no-graphics-card tier, the memory generation matters more than the badge on the CPU.

And now the number nobody quotes

Generation speed is the flattering metric. Here is the same 7B model on the same laptop, handed a ~2,900-token document — about four pages — and timed from send to the first word of the answer:

Sixty-two times. That is the real gap between these tiers, and it's the one you feel every single time you use the thing for work rather than chitchat. Summarise an email thread, ask a question about a manual, hand it a page of code — on the CPU box you are waiting a full minute, staring at nothing, before the first word appears. The generation speed you were shopping on barely matters next to that.

The three budgets

One rule runs through all three: you are buying video memory. Not cores, not the brand, not the generation. The model has to fit in VRAM or it doesn't run at full speed, and that single constraint decides everything else.

~$400 — the no-graphics-card box

What it does: 3B models comfortably above reading speed, 7B models around it. Fine for short questions, classifying things, tidying text, and anything you can set running and walk away from. Painful the moment you paste in something long — that's the 59-second table above, and on a slower CPU than mine it will be worse.

~$900 — the one that actually works

What it does: a 7B model about ten times faster than you read, and a four-page document absorbed in about a second. This is where a local model stops being a demo and starts being a tool. If you take one thing from this page: this is the tier to buy.

Honest caveat on that $289: I measured an 8GB laptop card, not this one. A used desktop 3060 12GB has both more VRAM and more memory bandwidth on paper than the card I tested, so I'd expect the numbers above to be the floor rather than the ceiling — but I haven't measured one, so that's a reasoned expectation and not a benchmark, and I'd rather label it than quietly let you assume otherwise. Used price checked 24 July 2026.

~$1,800 — the 24GB tier, with no benchmark attached

The next real step is a 24GB card — a used RTX 3090 was averaging about $1,255 when I checked on 24 July 2026 — in a similar host, which lands the build near $1,800. 24GB holds a 14B model comfortably and a 32B model at a squeeze.

Buy this tier only once you know the 7B models on the $900 box aren't smart enough for your specific job. That's something you learn by owning the $900 box for a month — not by reading a table.

Four things that broke when I did this

All four happened on the machines above, this week, while producing the tables you just read.

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 4168.09 MiB
  on device 0: cudaMalloc failed: out of memory
llama_model_load: error loading model: unable to allocate CUDA0 buffer
Error: llama runner process has terminated

nvidia-smi --query-compute-apps=pid,used_memory,name --format=csv

$ ollama ps
NAME          ID              SIZE      PROCESSOR          CONTEXT
qwen2.5:7b    845dbda0ea48    5.4 GB    36%/64% CPU/GPU    4096

# who else is talking to your model server?
journalctl -u ollama --since "-1 hour" | grep -c "api/"

Should you actually do this?

One more piece of honesty about scope: these are the models I had pulled and could benchmark this week, not the survivors of a search for the best model available. Model rankings turn over every few months. The hardware conclusions — bandwidth, VRAM, and the reading wall — age far more slowly than the model names do.

Or I'll set it up for you — $199