Can I run it? / RTX 4090 (24 GB)
Best LLMs to run on an RTX 4090 (24 GB)
Updated November 2026
As of November 2026, the best model to run on an RTX 4090 (24 GB) is Qwen3.8 27B by Alibaba (Qwen) (Small LM Index 74.6), at about 37 tokens/s with 5-bit weights.
24 GB fits 27–32B models at 4-bit, or Mixture-of-Experts models around 30B that run very fast because they read only a few billion parameters per token. About 23 GB is usable for a model, at 1008 GB/s. The figures below assume the best quality that fits (8-bit down to 4-bit) and a 8K-token context, and only include models that generate at least 5 tokens per second.
Models that run well on an RTX 4090 (24 GB)
The 25 best of the 94 models that run at 5+ tokens/s, ranked by the Small LM Index.
| # | Model | Index | Memory | Speed |
|---|---|---|---|---|
| 1 | Qwen3.8 27B Alibaba (Qwen) · 5-bit · Apache-2.0 | 74.6 | 21 GB | ≈ 37 tok/s |
| 2 | Ternary Bonsai 2 27B PrismML · Q2_0 (as released) · Apache-2.0 | 71.2 | 9.3 GB | ≈ 92 tok/s |
| 3 | Nanbeige4.3 33B Nanbeige (Kanzhun) · 4-bit · Apache-2.0 | 70.9 | 23 GB | ≈ 35 tok/s |
| 4 | Muse Glimmer 30B Meta · 5-bit · Apache-2.0 | 70.1 | 22 GB | ≈ 34 tok/s |
| 5 | Qwen3.6 27B Alibaba (Qwen) · 5-bit · Apache-2.0 | 69.7 | 21 GB | ≈ 37 tok/s |
| 6 | Qwen3.6 35B-A3B Alibaba (Qwen) · 4-bit · Apache-2.0 | 68.3 | 23 GB | ≈ 194 tok/s |
| 7 | Qwen3.5 27B Alibaba (Qwen) · 5-bit · Apache-2.0 | 68.3 | 21 GB | ≈ 37 tok/s |
| 8 | Nanbeige4.2 3B Nanbeige (Kanzhun) · 8-bit · Apache-2.0 | 67.2 | 5.8 GB | ≈ 159 tok/s |
| 9 | Qwen3.5 35B-A3B Alibaba (Qwen) · 4-bit · Apache-2.0 | 66.9 | 23 GB | ≈ 194 tok/s |
| 10 | Spark-X2.6 4B SparkLLM (XHToken) · 8-bit · Apache-2.0 | 66.5 | 5.3 GB | ≈ 162 tok/s |
| 11 | Gemma 4 31B Google · 4-bit · Apache-2.0 | 66.2 | 22 GB | ≈ 38 tok/s |
| 12 | GLM-4.7-Flash Z.ai (GLM) · 4-bit · MIT | 66.0 | 20 GB | ≈ 194 tok/s |
| 13 | Spark-X2.5 4B SparkLLM (XHToken) · 8-bit · Apache-2.0 | 65.9 | 5.3 GB | ≈ 162 tok/s |
| 14 | Granite 4.2 30B IBM (Granite) · 4-bit · Apache-2.0 | 65.8 | 21 GB | ≈ 39 tok/s |
| 15 | Qwen3 30B-A3B Thinking 2507 Alibaba (Qwen) · 4-bit · Apache-2.0 | 65.4 | 20 GB | ≈ 176 tok/s |
| 16 | Gemma 4 26B-A4B Google · 6-bit · Apache-2.0 | 64.8 | 22 GB | ≈ 113 tok/s |
| 17 | North Mini Code Cohere · 4-bit · Unverified | 64.7 | 21 GB | ≈ 194 tok/s |
| 18 | gpt-oss-20b OpenAI · 8-bit · Apache-2.0 | 64.7 | 23 GB | ≈ 92 tok/s |
| 19 | Nemotron 3 Nano 30B-A3B NVIDIA · 4-bit · NVIDIA Open Model | 64.4 | 20 GB | ≈ 166 tok/s |
| 20 | Qwen3.5 9B Alibaba (Qwen) · 8-bit · Apache-2.0 | 64.1 | 11 GB | ≈ 74 tok/s |
| 21 | Nemotron 3.5 Lightning NVIDIA · 4-bit · OpenMDW-1.1 | 63.3 | 22 GB | ≈ 162 tok/s |
| 22 | Apriel 1.6 15B Thinker ServiceNow (Apriel) · 8-bit · MIT | 63.3 | 18 GB | ≈ 45 tok/s |
| 23 | LFM2.5 24B-A2B Liquid AI · 6-bit · LFM Open License | 63.1 | 21 GB | ≈ 215 tok/s |
| 24 | Edge0 35B-A3B (preview) Edge0 · 4-bit · Apache-2.0 | 63.0 | 22 GB | ≈ 194 tok/s |
| 25 | Sarvam 30B Sarvam AI · 4-bit · Apache-2.0 | 62.8 | 21 GB | ≈ 242 tok/s |
Change the context length or quantization, or see models that only just fit, in Can I run it? for this device.
How to run Qwen3.8 27B on an RTX 4090 (24 GB)
- Easiest: Ollama or LM Studio. Both use the GPU automatically on Windows and Linux.
ollama run hf.co/ggml-org/Qwen3.8-27B-GGUF:Q4_K_M
- With llama.cpp. Starts a local server with a chat window at http://localhost:8080.
llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M
Every model page lists the same commands for its own files. Qwen3.8 27B's page also has a memory table for other context lengths.
Questions
What is the best LLM for an RTX 4090 (24 GB)?
As of November 2026, the best model to run on an RTX 4090 (24 GB) is Qwen3.8 27B by Alibaba (Qwen) (Small LM Index 74.6), at about 37 tokens/s with 5-bit weights.
What is the biggest model that runs on an RTX 4090 (24 GB)?
MiniCPM-V 4.7 35B-A3B (35.2B parameters, 3B active) fits at 4-bit, needing 23 GB and generating about 194 tokens/s. Bigger models may fit at lower quality or run more slowly; try them in the Can I run it? tool.
How much memory can a model use on an RTX 4090 (24 GB)?
About 23 GB of the 24 GB. 24 GB fits 27–32B models at 4-bit, or Mixture-of-Experts models around 30B that run very fast because they read only a few billion parameters per token.
How accurate are these speeds?
They are estimates from memory bandwidth (1008 GB/s here), which limits how fast a model generates text, with a typical efficiency for this kind of hardware. Expect real speeds within about a third either way. Long prompts take longer to read before the first word appears.