What decides whether a self-hosted model runs well, and what to run on the machine you already have. Every dial below computes from the same formula the prose teaches.
The best comfortable pick from this page's shortlist, with its four task verdicts (S short text, L long documents, C coding, A agent loops; section 09 explains them). Changing the type refills a typical bandwidth for that classest; your real number comes from the bench in section 13. One more affordance: select any text on this page and a button appears that copies it as a ready-made question for whatever AI assistant you use.
August 2026. A small-enterprise Windows box: Ryzen CPU, 128 GB of RAM, an RTX 2070 Super with 8 GB of VRAM. The job was an agent harness driving a network analytics platform over MCP, all of it local. I sized the machine on paper first.
The formula held. The ranking did not. On a pure CPU box the winner is the model with the fewest active parameters, so a 30B MoE beats a dense 8B. Put an 8 GB card in the same box and the winner becomes whatever fits on the card, even a 4B. Fit first, then speed, then size.
So: the write-up, made runnable. Seven instruments, four figures, one formula. Drag anything. Numbers watched on the real box carry obs; everything else carries est, including every number a slider hands you. No decode rate was ever benchmarked on that box, so this page quotes none.
Hardware talk around local models drowns in specs. Almost all of it is noise. Three numbers decide the experience. Hold them and you can size any machine in your head.
Before the specs, the feeling. The passage below is written token by token at whatever speed you set. Find the rate where the machine stops being the thing you notice. Every spec on this page exists to buy that rate.
Reading runs about 5 tok/s, skimming about 15est. That is where the bands on this page come from: under 10, generation is slower than you read and agent turns feel broken; 15+ is workable; 25+ reads as instant. Every preset is a real configuration from a later section, at the expected 65% of the formula ceiling.
Weights plus KV cache must fit in RAM (CPU) or VRAM (GPU). Decides what you can load at all.
How fast weights stream from memory to compute. Sets generation speed. The spec nobody quotes and everybody feels.
Dense models touch everything every token. MoE models touch a small slice. This is why architecture beats size.
They combine into one line worth memorizing:
Do not read worked examples. Work them. The calculator below is that one line, live. Pick a machine, drag the active-parameter slider, watch the tok/s move. That motion is the whole hardware story of this page.
Expected = 50-80% of the formula ceiling; every output here is est until benched (section 13). Verdict bands from the field doctrine: under 10 tok/s agent turns feel broken, 15+ is workable, 25+ is comfortable.
Three settings worth trying. Desktop DDR5, about 65 GB/s real, with a dense 8B at q4: a ceiling near 13 tok/s. Same box, a 3B-active MoE: about 30. Same box, a dense 27B: about 4. 128 GB of RAM solves N1 completely and does nothing for N2. Capacity buys shelf space, not speed, which is why the whole game on a CPU box is picking models with a small N3. Hold that thought. The field test flips the ranking of these same three numbers the moment a small GPU appears.
Models train in 16-bit weights. Quantization stores them in fewer bits, which shrinks the footprint (N1) and the bytes touched per token (N3) at the same time. After picking the model, it is the biggest lever you hold.
Guide lines: the fit thresholds that matter in this document. Sizes from the rule of thumb q4 ≈ params × 0.6 GB and its siblings; all est, weights only, KV cache not included.
Keep one rule of thumb: q4 size ≈ params × 0.6 GB. It sizes any model you meet on Hugging Face without opening the file list. Quantization applies to the KV cache too, and on the field-test box that turned out to matter more than anything else. Section 11 has that story.
Two sharpenings from the inference-engineering literature. First, quantization moves both walls at once. Fewer bytes per weight is more effective bandwidth for decode, and on GPUs with low-precision tensor cores it doubles raw compute, so one precision step is worth roughly 30-50% in practiceest. Second, not everything tolerates it equally. Sensitivity runs weights, then activations, then KV cache, then the attention math itself. Weights shrug off q4. The cache wants the care section 11 gives it. Attention stays in full precision in every sane recipe.
An inference server does two different jobs. Prefill reads your prompt and builds the model's working state. Compute-bound, parallel. Decode generates tokens one at a time. Bandwidth-bound. GPUs chew through prefill so fast nobody thinks about it. CPUs do not.
For an agent harness this decides the feel, because a harness turn is not a chat message. System prompt plus tool schemas plus history runs 10-20k tokens before your actual question. On the field-test box the harness front-loaded roughly 20k tokens of MCP tool schemas before the first user wordobs. Drag the prompt slider to harness scale and watch what turn one does on CPU.
Fixed worked example: dense 7B at q4. Prefill speeds are order-of-magnitude class estimatesest (CPU ~100 tok/s, Apple unified ~500-900, consumer NVIDIA ~2,500-5,000, H100 ~15,000); decode from the formula at 65% realization. Worth noticing: Apple unified memory decodes fast and prefills modestly, which is why long agent prompts feel different there than on NVIDIA. Turn 2 assumes about 600 new tokens on top of the cached prefix. In the standard vocabulary: the first-token wait is TTFT (time to first token), the per-token gap during decode is ITL (inter-token latency), and tok/s = 1000 / ITL in milliseconds.
The mitigation is prefix caching. The server keeps the computed state of the prompt, so the next turn only processes what changed. llama.cpp's server does this per slot. With it, turn one is slow and every later turn is cheap. Without it, every turn is turn one, which the widget above makes painfully visible. When an agent setup on CPU feels broken, check this before anything else.
One more way to lose that cache. Serving stacks unload an idle model after a few minutes, Ollama's default being about five, and the reload throws the prompt cache away with it. The next turn then pays the full reload plus the full prefill again. On an agent box, pin the model: OLLAMA_KEEP_ALIVE=24h. Cheapest lever on perceived speed there is.
Every machine has two ceilings: how fast it computes (FLOPS) and how fast memory feeds it (bandwidth). Which one you hit depends on a single property of the work, how many operations you perform per byte fetched, its arithmetic intensity. Below a machine's ridge point you are memory-bound and the compute idles. Above it you are compute-bound and the bandwidth idles. The book's worked example: an H100 offers 989 teraFLOPS against 3.35 TB/s, so its ridge sits near 295 operations per byteest. Every machine has its own ridge. The shape is universal.
Now place the two jobs on that chart. Prefill fetches each weight once and applies it to the whole prompt: thousands of operations per byte, compute side. Decode refetches every weight for every single token, roughly two operations per byte, pinned deep in the memory-bound slope. The formula in section 01 is that left slope. This is the mechanism under everything above.
Watch it run. Each sweep below is one token being made: every active byte crossing the bus once, past a compute unit that spends most of its life waiting.
The whole argument in one moving picture. To produce a single token, every active byte must cross the bus once, so the period of the sweep is bytes divided by bandwidth, and nothing about compute appears in the loop. Widen the bus or shrink the block and the rhythm changes on the spot. Rates at the expected 65% of ceilingest.
The book's H100 spec numbers (989 TF dense fp16, 3.35 TB/s, ridge ~295 ops/byte); decode intensity approximated as 2 x batch ops/byte. All est. The shape is the lesson, not the exact figures.
The batch slider is the punchline. A server answering two hundred people at once reuses every fetched weight two hundred times, so decode slides right, crosses the ridge, and the GPU finally earns its FLOPS. That is why cloud tokens are cheap and why datacenters batch hard. Your agent box runs at batch one, the leftmost and worst point on this chart. Which is why memory bandwidth is the entire story for a local box, and compute specs barely matter.
The chart also predicts the one near-free lever a batch-one box has: speculative decoding. Decode leaves most of the compute idle, so a small draft model proposes several tokens and the big model verifies them in a single pass. Accepted drafts arrive in bulk. It buys tokens per second and never buys back the first-token wait, and the payoff lives or dies on the acceptance rate, so it gets measured, not assumed. llama.cpp and friends support it with a small draft model from the same family. Purpose-built drafts, the EAGLE style, are the current default in server engines.
The KV cache is the model's working memory for the conversation. It grows linearly with context length and lives in the same RAM as the weights. The field-test box gave this section its anchor: a Qwen3-4B-class model stores roughly 144 KB of cache per token of context at fp16. Slide the context window and watch the cache dwarf the weights.
Basis: 144 KB per token at fp16 for the 4B-class field-test modelobs; q8 halves it near-losslessly, q4 halves it again at a real but small cost. Other architectures vary; the guide line is the field-test card's 8 GB.
Two consequences. First, budget for it. A model at 20 GB does not mean 24 GB of RAM is comfortable; the field's sizing rule is weights plus at least 50% headroom for cache, more for long contextest. Second, set the context length explicitly. Serving defaults are often small and truncate silently, and in an agent harness silent truncation looks like the model forgetting its tools mid-conversation. On a 128 GB box 32k context costs you nothing worth counting. Configure it, do not hope. On a small GPU the meter above is the whole game: 64k of fp16 cache is about 9 GB, more than the card itself.
Three shapes of machine appear on this page, and they differ in exactly one thing: where the fast memory sits relative to the compute.
Huge shelf, narrow road. Capacity is cheap here and bandwidth is not, so the model with the fewest active parameters wins.
One pool, no copies, no card to overflow. Buys capacity and bandwidth at once, which is why Apple and Strix Halo sit high on the ladder.
Fastest road, smallest shelf. Everything that fits flies. Everything that does not leaves over the thin PCIe strip at the bottom edge.
Here is the landscape, spec-sheet bandwidth unless noted. The ladder ranks the one number that sets speed. The map after it adds the axis a ladder cannot show, capacity, and that is where the intuition lives. Most of these machines are one click away in the calculator in section 01 and the simulator in section 08.
GB/s, logarithmic bar scale. Real means measured throughput; spec numbers are theoretical peaks. The gap between a shared VM and an H100 is roughly 70x, which is why the same model feels instant on one and broken on the other. Read the 2070 Super row twice: a six-year-old 8 GB card carries roughly 7x the bandwidth of desktop RAM. Whatever fits on the card flies. Whatever spills off it crawls.
Every device is a point in this plane, and the plane explains the market. Rightward means you can load more. Upward means it decodes faster. The corners carry the intuition: top right is where the money goes, bottom right is where reused enterprise hardware quietly disappoints, top left is small GPUs that fly only inside their VRAM. Click or tab to any point for its story. Double-ringed points are the machines I actually used.
Now pick a model and a target speed. The shaded corner is where that model both fits and hits the target. Every model demands its own distance from the zero corner, rightward for its working set, upward for its speed. Machines outside the corner dim. The scale on the right edge re-reads bandwidth as decode tok/s for the selected model, which is why it moves when the model does.
Corner assumptions: q4 weights, 32k context with q8 KV cache, decode at 65% of the formula ceiling; everything est until benched. Dimmed machines miss the fit, the speed, or both.
Click or tab to any point above to read what that device teaches, and what the reference model would do on it.
| Tier | Typical box | Fits (q4) | MoE 3B-active decode | Who runs this |
|---|---|---|---|---|
| Consumer desktop32-64 GB DDR5 | ~65 GB/s real | Up to ~30B-A3B class | 20-30 tok/s est | Hobbyists, first pilots |
| MacBook AirM4, 16-32 GB unified | 120 GB/s | 7-8B class; 30B MoE only at 32 GB, tight | 35-55 tok/s est | The surprisingly capable default laptop |
| MacBook ProM-Pro class, 24-64 GB unified; my tier | ~273 GB/s | Up to ~40B MoE class at 64 GB | 75-120 tok/s est | Serious local work on a laptop |
| Mac StudioM4 Max / M3 Ultra, 64-512 GB; the current Mac ceiling | 546-819 GB/s | Up to 100B+ MoE; 400B-class at 512 GB | 60-100+ tok/s est | Small teams; best perf per euro without a server room |
| CPU server, 128 GB+EPYC if lucky, shared VM if not | 40-576 GB/s | Everything up to ~120B MoE | 10-30 tok/s est | Enterprises reusing existing iron |
| GPU workstationone RTX 4090/5090, 24-32 GB VRAM | 1,000-1,800 GB/s | ~30B MoE or dense ~27B in VRAM | 100+ tok/s | The sweet spot if buying hardware is on the table |
| Datacenter GPUH100/A100/L40S, usually rented | 864-3,350 GB/s | Anything below frontier | Not the bottleneck | Production at real request volume |
One naming trap first. Qwen 3.8 is the August 2026 model generation, not Qwen3 at 8B. The open Qwen3.8 models are a dense 27B and a 2.4T-parameter monster, and nothing in that generation is CPU-shaped. The CPU-friendly mid-size MoE models live in the earlier Qwen3, 3.5 and 3.6 lines. All sizes below are 4-bit unless noted.
| Model | Total / active | Size | Decode @65 GB/s | Verdict |
|---|---|---|---|---|
| Qwen3-Coder-30B-A3Bor Qwen3-30B-A3B-Instruct-2507 | 30B / 3B | 18.6 GB | 20-30 tok/s est | Start here. Proven tool calling, huge headroom, fastest option. |
| Qwen3.6-35B-A3BApr 2026, hybrid attention MoE | 35B / ~3B | ~21-22 GB | 20-30 tok/s est | Newer alternative, same speed class. Newer architecture; check llama.cpp support is mature before committing. |
| gpt-oss-120bOpenAI, MXFP4 native | 117B / 5.1B | ~60-65 GB | 10-20 tok/s est | The stretch. Big-model quality, still MoE-fast. Only on 128 GB does this become a CPU option at all. |
| Qwen3.5-122B-A10Bopen weights, GGUF available | 122B / 10B | ~70 GB est | ~10 tok/s est | Quality ceiling, speed floor. Viable for low-volume, patience required. |
| Qwen3-8B densethe literal "Qwen3 8B" | 8B / 8B | 5.0 GB | ~10-13 tok/s est | Skip. Slower than the 30B MoE and dumber. Dense is the wrong shape here. |
| Qwen3.8-27B / dense 70B class | 27-70B dense | 17-40 GB | 2-4 tok/s est | Skip on CPU. Excellent models, GPU shapes. Qwen3.8-27B belongs on a 24 GB card. |
Why the MoE rows win is worth seeing rather than reading. Both blocks below hold a whole model. Only the lit part crosses the bus to make a token.
Left is the fixed reference: a dense 8B at q4, every block lit, 4.8 GB across the bus per token. Right is whatever you pick. The 30B-A3B lights a router plus a slice of experts and moves 1.8 GB, so it decodes about 2.7x faster than the smaller dense model on the same machine. The dense 27B lights everything and moves 16.2 GB. The lit area is N3 from section 01, and N3 is the only thing decode reads. Tile counts are proportional, not architectural: real expert counts differ per model, and the ratio is what carries. Rates at 65% of the formula ceilingest.
The pattern to keep: on CPU, total parameters are nearly free and active parameters are expensive. A 117B MoE outruns a dense 27B by 3-5x while sitting comfortably in your RAM. Every model worth running on this box says A3B or A10B or 5.1B active on its spec sheet. One honesty note cutting the other way: the active-parameter discount is a batch-one property. A server batching many users routes different requests to different experts and lights up nearly the whole model, so MoE stops being cheap at datacenter scale. It stays cheap for exactly the single-user box this page is about. And all of it assumes the CPU does the decoding. The moment even a small GPU enters, the ranking flips. The simulator below lets you watch that happen.
The whole page in one instrument. Pick a machine and a model, set the context, read the verdict: does the working set fit, where does it live, how fast does it decode. The first two machines are the real boxes from this project. The rest are reference tiers.
The scenario chips preload the sessions worth replaying: both rounds of the field test, the trap and then the healthy fit, the CPU path with the MoE and with the gpt-oss stretch, the Mac's unified-memory answer, and the used-3090 upgrade that unlocks the dense 27B. Touch any control and you leave the scenario. The chips get you back.
The question behind all the numbers: what work can you actually hand this machine? Comfortable here means three things at once. The model fits in fast memory, it answers at a usable pace, and the model itself is capable enough for the job. The matrix computes all three from the same formula as the simulator, for every machine and model on this page. Click any cell for the reasoning behind its four verdicts.
Click or tab to any cell above to see the per-task reasoning: fit, speed, and whether the model itself is up to the job.
Assumptions: q4 weights, q8 KV cache, decode at 65% of the formula ceiling; context per task: 8k short text, 16k coding, 32k long documents and agent loops. Speed bands from the field doctrine (15+ tok/s comfortable, under 10 painful; coding wants 20+). The capability caps, such as a 4B not being a coding model no matter the hardware, are my judgment and are named in the cell details. Everything est. A bench run on your machine (section 13) outranks every cell here.
In August 2026 the scenario met a real machine: a small-enterprise Windows box with a Ryzen CPU, 128 GB of RAM, and an RTX 2070 Super with 8 GB of VRAM, driving an agent harness in a thin Linux VM against a network analytics platform over MCP. Everything below I watched happen. None of it is projected.
Round one: the shortlist pick, split across memories. The 30B-A3B MoE is 18.6 GB at q4 and cannot fit an 8 GB card, so the server split it. A slice of layers in VRAM, the rest in system RAM. It worked and it crawled. Multi-minute turnsobs, most of that prefill over the harness's ~20k tokens of tool schemas. The split ran at the speed of its slowest memory, and the CPU side of a desktop board is 65 GB/s on a good day.
Round two: a smaller model, all on the card. A dense 4B instruct model at 2.5 GB fits entirely in VRAM with room left over for a quantized KV cache. After the cache fix in the next section the box reached its healthy state: GPU pinned at 100%, CPU at 7%, shared GPU memory at zeroobs. Turns went from painful to workable, and the model still drove the MCP tool chain correctly, which for reporting work is the entire job.
The three numbers did not change. Their ranking did. On CPU, active bytes (N3) dominate, so MoE wins. On a small GPU, total footprint (N1) dominates, because the card's 448 GB/s only applies to what actually fits on it. A small dense model living entirely in VRAM beats a bigger, smarter MoE that spills. Fit first, then speed, then size.
Honesty note: I never benchmarked decode rates on this box, so this page quotes no measured tok/s for it. Painful and workable are the observed states. The bench run in section 13 is what turns them into numbers.
The most valuable thing the field test taught me. When weights plus KV cache exceed dedicated VRAM on Windows, the NVIDIA driver does not error. It silently pages the overflow into shared GPU memory, which is ordinary system RAM reached across the PCIe bus. Nothing in the serving logs mentions it. The only symptom is that everything is inexplicably slow, and it is slower than an honest CPU split, because now every token commutes across the bus. The simulator's dashed overflow segment is exactly this state.
Three lanes, three speeds. Drag the spill and watch which one sets the pace.
Lane thickness is drawn on the log of bandwidth, so the PCIe strip looks as thin as it behaves. Worked example: a dense 8B at q4, 4.8 GB of active weights per token, on the field-test card. All of it in VRAM at 448 GB/s decodes at about 61 tok/sest; all of it arriving over PCIe 4.0 x16 at roughly 32 GB/sest decodes at about 4, which is worse than the CPU path's 65 GB/s. That is the whole trap: the driver moves bytes to the slowest lane on the board and reports nothing. The model here is serial, each token waiting for its bytes from wherever they live; real splits overlap some of that, so read the shape, not the decimal. PCIe figure is the one-way spec rate for a 4.0 x16 link, not a measurement.
Task Manager, GPU tab: dedicated memory pinned near its cap (7.5 of 8.0 GBobs) and shared GPU memory above zero (2 GBobs). That pairing is the diagnosis. Dedicated alone running high is healthy. Shared above zero during inference is the trap.
The fix on the field-test box was to shrink the KV cache until the whole working set fit on the card:
# Windows host serving via Ollama. Set, then FULLY quit and restart
# the server process (tray quit; verify with: tasklist | findstr ollama)
setx OLLAMA_FLASH_ATTENTION "1"
setx OLLAMA_KV_CACHE_TYPE "q8_0"Re-run a real request and read the same Task Manager panel: shared GPU memory back to ~0 while the GPU sits near 100% and the CPU idlesobs. That readout is the pass or fail for the whole setup. Not vibes, not logs. The shared-memory line.
If q8 is not enough, the next screws in order: KV cache at q4_0, which halves it again at a small quality cost on long contexts, or a smaller serving window. A 22k-token working session fits comfortably under a 32k window, and there is no reason to pay 64k of cache for headroom you never touch. The KV quant select in the simulator is these screws, live. Flip the 4B on the 2070 Super from fp16 to q8 at 64k context and watch the overflow disappear.
Every failure in the field test happened below the model. The model, once fed correctly, just worked. These three cost the most hours.
Ollama's desktop app carries its own settings, including a context-length slider that defaults to 4k, and the GUI value overrides OLLAMA_CONTEXT_LENGTH set via setx. The symptom is an HTTP 400 saying the request at 20,800 tokens exceeds an available context size of 4,096, while you are certain you configured 65,536. The only source of truth is ollama ps and its CONTEXT column, which reports what the loaded instance actually got. Env vars are a request. The GUI is a decision.
Model names carry their training recipe. base is a raw text completer. -instruct chats and emits standard JSON tool calls with no thinking phase. -thinking reasons before answering. -coder specializes in code. Dated refreshes, the 2507 style, can split a hybrid line into separate instruct and thinking models and extend context. The expensive one: some coder lines emit an XML tool-call dialect that OpenAI-compatible harnesses cannot parse, while the plain instruct lines of the same family speak standard JSON. If tool calls render as text instead of executing, suspect the dialect before you suspect the harness.
Some agent harnesses enforce a minimum context window, 64k in the one I tested, and check it against the model's reported maximum rather than your serving window. Models reporting 32-40k get refused outright. A 4B that reports 256k sails through. Two numbers to check before pulling gigabytes: what the model reports, and what the harness demands.
Every decode number above, slider outputs included, is an estimate until it runs on the actual machine, because a shared cloud VM delivers whatever bandwidth the neighbors leave. Twenty minutes of measurement turns this page from theory into a quote you can stand behind.
# 0. If serving via Ollama: what did the loaded model ACTUALLY get?
ollama ps # CONTEXT column = the truth
ollama run <model> --verbose "test" # eval rate = decode tok/s
# 1. What is this machine? Core count and CPU generation drive prefill.
lscpu | grep -E "Model name|^CPU\(s\)|NUMA"
# 2. Measured memory bandwidth. This one number predicts decode speed.
sysbench memory --memory-block-size=1M --memory-total-size=32G \
--memory-oper=read --threads=$(nproc) run
# 3. The real test: llama-bench with an agent-sized prompt.
# pp8192 = prefill tok/s (first-token pain), tg128 = decode tok/s.
llama-bench -m qwen3-coder-30b-a3b-q4_k_m.gguf -p 8192 -n 128 -t $(nproc)
# 4. End to end: serve, then time a realistic harness turn.
llama-server -m model.gguf -c 32768 --jinja &
time curl -s localhost:8080/v1/chat/completions -d @sample-agent-turn.jsonRead it like this. tg128 under ~10 tok/s means agent turns will feel broken; go find a better box. 15+ is workable, 25+ is comfortable. If pp8192 is painful, confirm prefix caching is doing its job by sending the same request twice; the second must be dramatically faster. On any Windows GPU host, keep Task Manager's GPU panel open during the run: dedicated high is healthy, shared above zero is section 11. Once you have a measured bandwidth number, feed it into the calculator in section 01. From then on this page computes with your machine's truth instead of spec sheets.
For a single-user agent querying one data source a few hundred times a day, the field test says the smallest model that fits entirely in fast memory and drives tools correctly is very likely the end of the story. The 128 GB was never the constraint. It turned out the 8 GB was not either, once everything respected it. If a box ever serves a team instead of one agent, the story changes along section 04's roofline: server engines built around continuous batching, vLLM and SGLang and TensorRT-LLM, exist to push decode toward the compute ceiling. For one user, llama.cpp and Ollama stay the right size.
I build AI into small and mid-sized companies: automations, agent systems, self-hosted models where the data cannot leave the building. Things that subtract repetitive work instead of adding another tool. I also run workshops for teams, after which it is clear what can be built and what is an expensive promise.
We usually start with a short conversation about where the repetitive work sits in your week.
Get in touchNumbers marked est are estimates from the bandwidth formula or community reports, not measurements; every interactive output on this page inherits that label. Numbers marked obs were observed on the field-test box in August 2026 (Task Manager GPU readouts, serving logs, harness token counters); its decode rate remains unbenchmarked and is deliberately not quoted. Per-model KV-cache rates in the simulator are derived from public model configs and rounded; treat them as sizing aids. Everything else is from spec sheets and model cards, checked August 2026: Qwen3-30B-A3B GGUF, Qwen3-8B GGUF, Qwen3.6 run guide, gpt-oss announcement, Qwen3.5-122B-A10B GGUF, Qwen3.8 run guide. Machine-map bandwidth figures are vendor spec sheets; the two 128 GB unified mini-PCs and their measured gpt-oss-120b rates were cross-checked August 2026 against Tom's Hardware and The Register. Estimates assume 4-bit quantization and the formula's 50-80% realization rate; your bench run replaces them. Section 04's roofline framing and several rules of thumb (the ridge-point example, the precision-step gain, the KV-sensitivity ordering, the VRAM headroom rule, the MoE batch caveat, the speculative-decoding mechanics) follow the free book Inference Engineering, read August 2026; where its figures are worked examples rather than spec sheets they carry est here.
Written and built by Mateusz Sawka · more at komai.pl.