Can I run it?
Pick your hardware and context length to see which models fit, at the best quality that fits, with the memory each needs and a rough generation speed. Memory counts the weights, the context cache (from each model's real attention layout) and runtime buffers. Each model page has copy-paste commands for Ollama, llama.cpp, LM Studio and MLX.
RTX 4060 Ti · 16 GB: about 15 GB usable for a model. The best fit at 8K context is Nanbeige4.2 3B (index 67.2) at 8-bit, needing 5.8 GB and generating roughly 46 tokens/s.
| Model | Memory | Speed |
|---|---|---|
| Ternary Bonsai 2 27B Index 71.2 · Q2_0 (as released) · 27.4B | 9.3 GB | ≈ 26 tok/s |
| Nanbeige4.2 3B Index 67.2 · 8-bit · 4.2B | 5.8 GB | ≈ 46 tok/s |
| Spark-X2.6 4B Index 66.5 · 8-bit · 4.1B | 5.3 GB | ≈ 46 tok/s |
| Spark-X2.5 4B Index 65.9 · 8-bit · 4.1B | 5.3 GB | ≈ 46 tok/s |
| gpt-oss-20b Index 64.7 · 4-bit · 20.9B | 14 GB | ≈ 46 tok/s |
| Qwen3.5 9B Index 64.1 · 8-bit · 9B | 11 GB | ≈ 21 tok/s |
| Apriel 1.6 15B Thinker Index 63.3 · 6-bit · 14.9B | 15 GB | ≈ 17 tok/s |
| Granite 4.2 8B Index 62.7 · 8-bit · 8B | 11 GB | ≈ 24 tok/s |
| Ministral 3 14B Index 61.6 · 6-bit · 14B | 14 GB | ≈ 18 tok/s |
| LFM3 1.7B Index 61.2 · 8-bit · 1.7B | 2.5 GB | ≈ 110 tok/s |
| MiniCPM5 2B Index 60.9 · 8-bit · 2.5B | 3.6 GB | ≈ 75 tok/s |
| Mellum2.1 12B-A2.5B Thinking Index 60.8 · 8-bit · 12.2B | 14 GB | ≈ 38 tok/s |
| Qwen3 4B Thinking 2507 Index 59.8 · 8-bit · 4B | 6.0 GB | ≈ 47 tok/s |
| Qwen3.5 4B Index 59.7 · 8-bit · 4B | 5.1 GB | ≈ 47 tok/s |
| Nemotron Labs Dory 1.0 4B Index 58.6 · 8-bit · 4B | 5.2 GB | ≈ 48 tok/s |
| Falcon H1R 7B Index 58.6 · 8-bit · 7.6B | 9.1 GB | ≈ 25 tok/s |
| Gemma 4 12B Index 58.3 · 8-bit · 11.9B | 14 GB | ≈ 16 tok/s |
| Ministral 3 8B Index 57.7 · 8-bit · 8B | 10 GB | ≈ 24 tok/s |
| Granite 4.2 3B Index 57.3 · 8-bit · 3B | 4.4 GB | ≈ 63 tok/s |
| Phi-4-reasoning-plus Index 57.2 · 6-bit · 14.7B | 15 GB | ≈ 17 tok/s |
| MiniCPM-SALA Index 56.9 · 8-bit · 9.5B | 11 GB | ≈ 20 tok/s |
| DeepSeek-R1-0528-Qwen3-8B Index 56.3 · 8-bit · 8.2B | 11 GB | ≈ 23 tok/s |
| Hunyuan 7B Instruct Index 55.8 · 8-bit · 7.5B | 9.7 GB | ≈ 25 tok/s |
| Ling 3.0 Tiny Index 54.9 · 8-bit · 7.9B | 9.3 GB | ≈ 68 tok/s |
| Qwen3 8B Index 53.9 · 8-bit · 8.2B | 11 GB | ≈ 23 tok/s |
| Ministral 3 3B Index 53.8 · 8-bit · 3B | 4.6 GB | ≈ 63 tok/s |
| Step3-VL-10B Index 53.2 · 8-bit · 10B | 13 GB | ≈ 19 tok/s |
| Nemotron Nano 9B v2 Index 52.9 · 8-bit · 8.9B | 10 GB | ≈ 21 tok/s |
| Spark-X2.6 1.7B Index 52.3 · 8-bit · 1.7B | 2.5 GB | ≈ 111 tok/s |
| OLMo 3 7B Think Index 52.3 · 8-bit · 7.3B | 11 GB | ≈ 26 tok/s |
| Edge0 8B-A1B (preview) Index 50.1 · 8-bit · 7.9B | 9.3 GB | ≈ 68 tok/s |
| Tri-21B Think Index 47.2 · 4-bit · 20.7B | 15 GB | ≈ 16 tok/s |
| MiMo-7B-RL Index 45.8 · 8-bit · 7.8B | 10 GB | ≈ 24 tok/s |
| DeepSeek-R1-Distill-Qwen-7B Index 45.6 · 8-bit · 7.6B | 9.2 GB | ≈ 25 tok/s |
| LFM2.5 2.6B Index 44.7 · 8-bit · 2.7B | 3.6 GB | ≈ 70 tok/s |
| Ling-mini-2.0 Index 43.0 · 6-bit · 16.3B | 15 GB | ≈ 88 tok/s |
| EXAONE 4.0 1.2B Index 42.5 · 8-bit · 1.3B | 2.4 GB | ≈ 148 tok/s |
| Gemma 4 E4B Index 40.9 · 8-bit · 8B | 9.3 GB | ≈ 42 tok/s |
| LFM2.5 8B-A1B Index 40.1 · 8-bit · 8.5B | 9.8 GB | ≈ 95 tok/s |
| Mellum2 12B-A2.5B Instruct Index 39.2 · 8-bit · 12.2B | 14 GB | ≈ 38 tok/s |
| Gemma 4 E2B Index 37.4 · 8-bit · 5.1B | 6.1 GB | ≈ 82 tok/s |
| SmolLM3 3B Index 36.8 · 8-bit · 3.1B | 4.4 GB | ≈ 62 tok/s |
| LLaDA2.2 mini Index 36.2 · 6-bit · 16.3B | 15 GB | ≈ 88 tok/s |
| Qwen3.5 2B Index 35.9 · 8-bit · 2.3B | 3.1 GB | ≈ 82 tok/s |
| LFM2.5 1.2B Thinking Index 34.7 · 8-bit · 1.2B | 1.9 GB | ≈ 162 tok/s |
| Gemma 3 12B Index 29.5 · 8-bit · 12.2B | 15 GB | ≈ 16 tok/s |
| MiniCPM5 1B Index 29.4 · 8-bit · 1.1B | 1.9 GB | ≈ 176 tok/s |
| Phi-4 Index 26.2 · 6-bit · 14.7B | 15 GB | ≈ 17 tok/s |
| LFM2.5 1.2B Instruct Index 25.4 · 8-bit · 1.2B | 1.9 GB | ≈ 162 tok/s |
| MiniCPM-V 4.6 Index 24.7 · 8-bit · 1.3B | 2.0 GB | ≈ 146 tok/s |
| Gemma 3n E4B Index 24.6 · 8-bit · 7.8B | 9.1 GB | ≈ 47 tok/s |
| Apertus 1.5 8B Index 23.4 · 8-bit · 8.9B | 11 GB | ≈ 21 tok/s |
| Gemma 3n E2B Index 20.9 · 8-bit · 5.4B | 6.5 GB | ≈ 95 tok/s |
| Llama 3.1 8B Index 20.3 · 8-bit · 8B | 10 GB | ≈ 24 tok/s |
| Gemma 3 4B Index 19.4 · 8-bit · 4.3B | 5.4 GB | ≈ 44 tok/s |
| Llama 3.2 3B Index 19.1 · 8-bit · 3.2B | 4.9 GB | ≈ 59 tok/s |
| LFM2.5 350M Index 16.6 · 8-bit · 350M | 1.0 GB | ≈ 542 tok/s |
| Phi-4-mini Index 15.8 · 8-bit · 3.8B | 5.7 GB | ≈ 50 tok/s |
| Qwen3.5 0.8B Index 14.3 · 8-bit · 900M | 1.6 GB | ≈ 211 tok/s |
| Gemma 3 270M Index 12.8 · 8-bit · 270M | 0.8 GB | ≈ 703 tok/s |
| Gemma 3 1B Index 8.9 · 8-bit · 1B | 1.6 GB | ≈ 190 tok/s |
| Llama 3.2 1B Index 8.7 · 8-bit · 1.2B | 2.1 GB | ≈ 158 tok/s |
| Baguettotron No index · 8-bit · 320M | 1.1 GB | ≈ 593 tok/s |
| ERNIE 4.5 0.3B No index · 8-bit · 360M | 1.0 GB | ≈ 527 tok/s |
| MiniCPM-V 4.6 Thinking No index · 8-bit · 1.3B | 2.0 GB | ≈ 146 tok/s |
| MiniCPM-V 4.7 1B No index · 8-bit · 1.6B | 2.3 GB | ≈ 122 tok/s |
| LFM3-VL 2B No index · 8-bit · 1.9B | 2.7 GB | ≈ 99 tok/s |
| LFM3-Omni 2B No index · 8-bit · 2.2B | 3.0 GB | ≈ 87 tok/s |
| LFM2.5-VL 3B No index · 8-bit · 3.1B | 4.0 GB | ≈ 61 tok/s |
| Nemotron 3 Nano 4B No index · 8-bit · 4B | 5.6 GB | ≈ 47 tok/s |
| AgentCPM-Explore 4B No index · 8-bit · 4B | 6.1 GB | ≈ 47 tok/s |
| FrogNano 4B No index · 8-bit · 4.7B | 5.8 GB | ≈ 41 tok/s |
| Teuken 7B Instruct v0.6 No index · 8-bit · 7.5B | 8.7 GB | ≈ 25 tok/s |
| Salamandra 7B Instruct No index · 8-bit · 7.8B | 10.0 GB | ≈ 24 tok/s |
| Nemotron 3.5 Bolt E4B No index · 8-bit · 8B | 9.3 GB | ≈ 42 tok/s |
| Luciole 8B Instruct No index · 8-bit · 8.1B | 9.4 GB | ≈ 23 tok/s |
| MiMo-VL-7B-RL No index · 8-bit · 8.3B | 11 GB | ≈ 23 tok/s |
| Rnj-1.5 Instruct No index · 8-bit · 8.3B | 11 GB | ≈ 23 tok/s |
| GLM-4.6V-Flash No index · 8-bit · 9B | 11 GB | ≈ 21 tok/s |
| EuroLLM 9B Instruct (2512) No index · 8-bit · 9.2B | 12 GB | ≈ 21 tok/s |
| MiniCPM-o 4.5 No index · 8-bit · 9.4B | 12 GB | ≈ 20 tok/s |
| MiMo-V2.6-Distill-Qwen-9B No index · 8-bit · 9.4B | 12 GB | ≈ 20 tok/s |
| HyperCLOVA X SEED Omni 8B No index · 8-bit · 10.7B | 13 GB | ≈ 18 tok/s |
| Bielik 11B v3 Instruct No index · 8-bit · 11.2B | 14 GB | ≈ 17 tok/s |
| Phi-4-reasoning-vision-15B No index · 6-bit · 15B | 15 GB | ≈ 16 tok/s |
| Ring-mini-2.0 No index · 6-bit · 16.3B | 15 GB | ≈ 88 tok/s |
| Kimi-VL-A3B-Thinking-2506 No index · 6-bit · 16.4B | 15 GB | ≈ 44 tok/s |
| gpt-oss-safeguard-20b No index · 4-bit · 20.9B | 14 GB | ≈ 46 tok/s |
| ERNIE 4.5 21B-A3B Thinking No index · 4-bit · 21.8B | 15 GB | ≈ 55 tok/s |
Estimates for llama.cpp-style GGUF inference (Ollama, LM Studio and llama.cpp itself). Macs reserve part of their memory for the system by default; phones leave about half of it to apps. For per-device picks on phones, Raspberry Pi and Jetson, see LLMs for edge devices. Vision encoders and very long prompts add more on top. See the methodology.
Best models by device
Questions
How much memory does an LLM need to run locally?
Three parts add up: the weights (parameters × bits per weight ÷ 8), the context cache, which grows with every token in the conversation, and about half a gigabyte of runtime buffers. At 4-bit (Q4_K_M, about 4.85 bits per weight) an 8B model's weights take about 4.8 GB, and Llama 3.1 8B needs another 4.3 GB for a 32K-token context at 16-bit.
Why does the context length change what fits?
Each token in the context keeps keys and values for every attention layer. Llama 3.1 8B stores about 128 KB per token, so 128K tokens need 17 GB, more than its 4-bit weights. Newer hybrid designs such as Qwen3.8 27B use full attention in only some layers, so their cache is far smaller: 8.6 GB at 128K. An 8-bit context cache halves these numbers.
Can I run a model that is bigger than my GPU memory?
Yes, with llama.cpp or Ollama part of the model can stay in system RAM, at a cost in speed. It works best for Mixture-of-Experts models, which read only a few experts per token: Qwen3.6 35B-A3B has 35B parameters but reads only 3B per token, about 1.8 GB at 4-bit. Turn on "Spill into system RAM" above to see it.
How accurate are the speed estimates?
Generating text is limited by memory bandwidth: every new token reads the active weights once. The estimate divides the hardware's bandwidth by that amount and applies a typical efficiency, so expect real speeds within about a third either way. It covers generation with a short prompt; reading a long prompt and very full contexts are slower.