A single 24GB card is the practical floor for serious local inference. It is enough for genuinely capable models, and small enough to sit on one GPU. An RTX 3090 or RTX 4090 both land in this tier. The card you own matters less than the models you pick for it.
The old hobbyist move was to squeeze the biggest 70B quant onto the card. That advice is now outdated. The stronger 2026 strategy uses modern 20B–35B-class models that fit cleanly. These leave room for context, and still respond fast enough for coding, chat, and agents. This guide covers the models that actually fit, why each is worth running, and how 24GB gets spent.
How 24GB of VRAM actually gets spent
Three things consume memory during inference. Getting the split right decides whether a model fits.
The first is model weights. Their size depends on parameter count and quantization. At Q4_K_M, a common home-inference default, each parameter costs roughly 0.58 bytes. A 32B model therefore needs about 18–20GB in weights alone. Mixtral-style Mixture-of-Experts (MoE) models are the common trap here. Every expert stays resident in VRAM even when only a few route per token. So you size MoE memory by total parameters, never active parameters.
The second is the KV cache, which grows with context length. Longer prompts and longer sessions eat more VRAM. The third is runtime overhead from the serving stack. A safe rule of thumb adds roughly 1–2GB for the KV cache and runtime at short context.
Quantization is the lever that makes 24GB workable. Q4_K_M is the standard balance of quality and footprint. Q5_K_M and Q6_K raise quality at a memory cost. Q8_0 and BF16 are usually too large for 30B-class models on a single 24GB card. The interactive tool below lets you switch quantization and see each model’s fit change live.
The six best local LLMs for a 24GB GPU
Every model below carries a permissive license, either Apache 2.0 or MIT. All of them fit a single 24GB card at Q4_K_M with room for context. They are grouped by the job each one does best.
Qwen3.6-27B — best all-around and agentic coding
Alibaba’s Qwen3.6-27B is the strongest single default for the card. It is a dense 27B model released in April 2026 under Apache 2.0. The release focuses on agentic coding, repository-level reasoning, and frontend workflows. At Q4_K_M it needs roughly 16GB, leaving comfortable headroom for context. Full model cards and the changelog sit in the Qwen3.6 GitHub repository.
Qwen3.6-35B-A3B — fastest general-purpose option
The Qwen3.6-35B-A3B is a Mixture-of-Experts model with 35B total parameters and about 3B active per token. It decodes far faster than a dense 35B model because only a fraction of weights fire each step. The memory footprint still tracks total parameters, so plan for roughly 20GB at Q4_K_M. That makes it a tight but valid fit, best when you want speed for general chat and tool use.
Gemma 4 26B — multimodal and multilingual
Google DeepMind released Gemma 4 on April 2, 2026, under Apache 2.0. It was the first Gemma generation to ship with a fully open license, per the DeepMind Gemma page. The family spans edge models, a 12B unified multimodal model, a 26B MoE with 3.8B active, and a larger dense flagship. The 26B MoE is the natural 24GB pick when you need vision input and 140+ language coverage.
Mistral Small 3.2 24B — polished daily assistant
Mistral’s Small line targets the everyday assistant workload with low latency. Mistral Small 3.1 added multimodal input and a longer context window, and Small 3.2 refined instruction following. It is a 24B dense model under Apache 2.0, and the lightest footprint in this list at roughly 14GB at Q4_K_M. That headroom lets you push context further than the heavier 32B options allow. In 2026 Mistral also shipped larger successors, but those sit above the single-card tier.
gpt-oss-20b — the easy reasoning fallback
OpenAI’s gpt-oss-20b is an open-weight reasoning model under Apache 2.0. It is a Mixture-of-Experts design with 21B total parameters and 3.6B active per token. It ships in a native MXFP4 4-bit format, loading in roughly 14GB with generous headroom. It is strong at structured reasoning and tool use, and weaker on broad world knowledge.
DeepSeek-R1-Distill-Qwen-32B — deepest reasoning, tightest fit
DeepSeek distilled its R1 reasoning traces into smaller dense models. The DeepSeek-R1-Distill-Qwen-32B is the 32B variant, built on a Qwen2.5 base and released under an MIT license. At Q4_K_M it uses about 18–20GB, which is the tightest fit in this guide. It exposes its chain of thought through visible reasoning tokens, which is useful for slow, deliberate problems. Ready-made GGUF builds are available from bartowski on Hugging Face.
What does not fit on 24GB
The frontier open models of 2026 are large sparse MoE systems. They are quite good in performance and reasoning, but they do not run on one consumer card. GLM-5.2 from Z.ai is a roughly 753B-total MoE. Moonshot’s Kimi K2.7 is around 1T total. DeepSeek shipped V4 as a public preview in April 2026, with a V4-Pro checkpoint near 1.6T parameters. Alibaba’s Qwen3.5-397B and Mistral Large 3 sit in the same server-class range.
Because MoE memory tracks total parameters, all of these need multi-GPU rigs or high-memory unified systems. They are worth knowing as API options. They do not change what runs on a single 24GB card.
How to run these models locally
Three runtimes cover almost every setup. Ollama is the simplest path, with automatic quantization selection and an OpenAI-compatible API. llama.cpp gives fine control over GGUF quantization and offload. vLLM is the throughput-focused server for heavier concurrent workloads.
Start with one model that matches your main job, not the biggest file you can load. Keep context under control, and let the card do what it does best. Run serious local AI without sending a single token to a cloud endpoint.
Key Takeaways
- 24GB is the practical floor: run right-sized 20B–35B models, not the biggest 70B quant you can squeeze in.
- Qwen3.6-27B is the strongest single default; DeepSeek-R1-Distill-Qwen-32B is the tightest fit at ~18–20GB.
- MoE memory tracks total parameters, so every expert stays resident even when only a few route per token.
- Quantization is the fit lever: Q4_K_M is the home default; Q8_0 and BF16 rarely fit a 30B model on one card.
- GLM-5.2, Kimi K2.7, DeepSeek V4, and Mistral Large 3 are server-class and will not run on a single 24GB GPU.
Michal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.
Credit: Source link
