The best open source LLM for most self-hosters in 2026 is Qwen3-30B-A3B because its 15 GB quantized weights, 3B active parameters, Apache-2.0 license, and 128K context fit a practical 24 GB GPU setup. Smaller systems should consider Qwen3 8B or Gemma 3 4B, while Mistral Small 4 suits multi-GPU servers that need stronger multimodal and agent capabilities.
Which open source LLM is best for self-hosting in 2026?
Qwen3-30B-A3B offers the strongest overall balance for self-hosting in 2026, combining 30B total parameters, 3B active parameters, a 128K-token context window, and roughly 15 GB of 4-bit weights. A 24 GB GPU provides a realistic starting point for inference, runtime allocations, and a moderately sized context.
An open source LLM is a language model whose downloadable weights and license permit inspection, local deployment, and some degree of modification or redistribution. The label is used loosely, however. Apache-2.0 and MIT models provide familiar open-source permissions, while Gemma and Llama models use custom terms with added conditions.
Qwen’s 2025 documentation recommends vLLM or SGLang for serving Qwen3-30B-A3B. For local use, the project identifies Ollama, LM Studio, MLX, llama.cpp, and KTransformers as supported options. That breadth matters because you can start with a desktop interface, then move the same model family into a production serving stack.
For a smaller machine, Qwen3’s 2025 dense range covers 0.6B, 1.7B, 4B, 8B, 14B, and 32B parameters. The 4-bit weights range from under 1 GB to about 16 GB before overhead. Choosing a compact checkpoint can be smarter than chasing a flagship model, a trade-off examined further in this guide to why smaller language models make practical sense.
How much GPU memory does an open source LLM need?
An open source LLM generally needs more memory than the downloaded weight file alone suggests. A 15 GB 4-bit checkpoint may fit within a 24 GB GPU, but the inference engine, key-value cache, context length, and concurrent requests consume the remaining capacity. Long advertised context windows are therefore limits, not free operating modes.
The basic weight calculation is straightforward. Four-bit quantization stores each parameter in roughly half a byte, so a 30B-parameter dense model has a theoretical floor near 15 GB: 30 billion multiplied by 0.5 bytes. Quantization metadata and runtime allocations push actual requirements higher.
Mixture-of-experts models add a frequently missed complication. Qwen3-30B-A3B activates about 3B parameters per token in 2025, yet you still need to store the complete 30B-parameter model. Low active-parameter counts reduce computation; they don’t shrink downloaded weights to the size of the active subset.
The following comparison uses model cards and official documentation published in 2025 or 2026. Storage figures describe approximate 4-bit weights before runtime and key-value-cache overhead.
| Model | Parameters | Context | Approx. 4-bit weights | Practical hardware tier | License |
|---|---|---|---|---|---|
| Qwen3-30B-A3B (2025) | 30B total, 3B active | 128K tokens | 15 GB | 24 GB GPU | Apache 2.0 |
| Mistral Small 3.1 (2025) | 24B | 128K tokens | 12 GB | 24 GB GPU | Apache 2.0 |
| Gemma 3 12B (2025) | 12B | 128K tokens | 6 GB | 16 GB GPU | Gemma terms |
| Gemma 3 27B (2025) | 27B | 128K tokens | 13.5 GB | 24 GB GPU | Gemma terms |
| DeepSeek-R1 Distill 14B (2025) | 14B | 128K tokens | 7 GB | Allow more than 8 GB | MIT |
| Mistral Small 4 (2026) | 119B total, 6.5B active | 256K tokens | 60 GB | Multi-GPU or high-memory unified system | Apache 2.0 |
Context can become the hidden budget killer. Loading 15 GB of weights onto a 24 GB card leaves roughly 9 GB before engine overhead, and that remainder must accommodate the cache and requests. Honestly, a 24 GB card is the sensible floor for Qwen3-30B-A3B, not a promise that every 128K workload will run comfortably.
Which models fit an 8 GB, 16 GB, or 24 GB GPU?
Gemma 3 4B is a practical 8 GB choice in 2025, Gemma 3 12B aligns with a 16 GB tier, and Qwen3-30B-A3B or Mistral Small 3.1 suits a 24 GB GPU. These tiers include headroom beyond approximate weight storage, although usable context and concurrency still depend on the serving configuration.
Google’s 2025 Gemma 3 model card lists 1B, 4B, 12B, and 27B variants. The 1B edition is text-only with 32K context; the larger editions are multimodal, support more than 140 languages, and provide 128K context. Approximate 4-bit weights are 2 GB, 6 GB, and 13.5 GB for the 4B, 12B, and 27B versions.
Mistral Small 3.1 is especially attractive at the 24 GB level. Mistral’s 2025 announcement specifies 24B parameters, multimodal input, function calling, 128K context, and Apache-2.0 licensing. Its approximately 12 GB of 4-bit weights leave more operating room than several similarly capable alternatives.
Use these criteria when matching a model to your machine:
- Reserve memory beyond the quantized weight file for runtime allocations and the key-value cache.
- Choose a context target based on real documents, not the model card’s maximum.
- Check whether image input, tool calling, coding, or multilingual output is actually required.
- Estimate simultaneous users because concurrency multiplies cache and compute pressure.
- Confirm the exact checkpoint license before commercial deployment.
Is Mistral Small 4 better than Qwen3-30B-A3B?
Mistral Small 4 is the more capable infrastructure-class option in 2026, while Qwen3-30B-A3B is the more accessible self-hosting choice. Mistral Small 4 provides multimodal input, function calling, 256K context, and 6.5B active parameters, but its approximately 60 GB of 4-bit weights generally require multiple GPUs or high-memory unified hardware.
By comparison, the Qwen model needs about 15 GB for 4-bit weights and starts making practical sense on one 24 GB GPU. Both use Apache 2.0, so the decision rests largely on workload and budget rather than a restrictive model-specific license.
Mistral Small 4 is better suited to private assistants, coding agents, document and image analysis, and demanding reasoning services with server hardware. Qwen3-30B-A3B targets many of the same jobs with lower storage and compute demands. If agents are the goal, understanding how coding agents select tools will matter as much as raw model size.
At the desktop-server boundary, Qwen is my pick. Mistral Small 4 becomes compelling when you already own the hardware and need its 2026 multimodal or 256K capabilities. Buying several GPUs solely to claim a larger context window is difficult to justify unless your documents genuinely demand it.
Are all downloadable LLMs really open source?
Downloadable LLM weights are not automatically open source under a standard license. In 2026, Qwen3 and Mistral Small 4 use Apache 2.0, while DeepSeek-R1 uses MIT. Google Gemma 3 and Meta Llama 4 use model-specific terms, so commercial rights and redistribution conditions require separate review.
Google requires users to accept Gemma terms before obtaining model files. Meta’s 2025 Llama 4 Scout and Maverick releases use the Llama 4 Community License rather than Apache 2.0 or MIT. Both are multimodal mixture-of-experts models with 17B active parameters, and Scout has about 109B total parameters.
License names also shouldn’t be generalized across a vendor’s catalog. Mistral’s August 12, 2026 guidance says most open models use Apache 2.0, but some carry modified MIT terms requiring companies above $20 million in monthly revenue to obtain a commercial license or use Mistral Studio. Check each model card.
Security deserves the same checkpoint-level scrutiny. Local deployment keeps prompts under your control, but tool-enabled agents can still execute unsafe instructions or ingest hostile repository files. The reported risks from malicious AGENTS.md files targeting coding agents show why self-hosting isn’t a substitute for sandboxing and permission boundaries.
When should you use DeepSeek-R1 or wait for a newer model?
DeepSeek-R1 distillations are strong 2026 choices for local mathematics, coding, and complex reasoning, but the full 671B-total-parameter model remains a data-center deployment. The 14B and 32B distillations require roughly 7 GB and 16 GB of 4-bit weight storage respectively, before runtime overhead.
DeepSeek’s 2025 repository documents 37B active parameters and a 128K context window for full DeepSeek-R1, plus distillations at 1.5B, 7B, 8B, 14B, 32B, and 70B. The MIT license is unusually straightforward for commercial self-hosting.
Full deployment is another matter. The 2026 vLLM recipe lists eight H200-class GPUs for FP8 or four B200 GPUs for NVFP4. That recipe supports a 163,840-token configuration, exceeding the original checkpoint documentation’s 128K figure, but neither setup belongs in an ordinary workstation budget.
Waiting can also backfire. Mistral announced the 1-trillion-parameter Mistral Large 4 in public preview on October 6, 2026, with 52B active parameters and a reported 1M-token context. Downloadable weights had not appeared by October 27, 2026, so it wasn’t yet a verifiable self-hosting recommendation.
Frequently asked questions
Can I run a local LLM without a GPU?
A local LLM can run on a CPU or a unified-memory computer, especially at 0.6B to 4B parameters, but generation is generally slower than GPU inference. Quantized Qwen3 and Gemma 3 editions are the most realistic candidates from the 2025 model ranges covered here.
Does a 128K context window mean I can always process 128K tokens?
A 128K context window states the model’s documented maximum in 2025, not a guarantee for every local configuration. Key-value-cache memory, inference software, quantization, and available GPU memory can force a much smaller operating limit.
Is Ollama suitable for production self-hosting?
Ollama is convenient for local evaluation and smaller deployments, and Qwen identified it as a local option in 2025. Qwen recommends vLLM or SGLang for serving Qwen3-30B-A3B where higher concurrency and production-oriented operation matter.
Which self-hosted model is best for image understanding?
Mistral Small 3.1, Mistral Small 4, and Gemma 3 models above 1B all support multimodal input according to their 2025 or 2026 documentation. Gemma 3 4B is the lowest-memory option among them, with about 2 GB of 4-bit weights before overhead.


