Open-weight model downloads
Download a 4-bit GGUF file to run on your own hardware. Each Source link opens the original model page on Hugging Face.
This selection covers 9 models, from 0.8B parameters to MiMo V2.6 Pro at 1.02T. M means million, B means billion and T means trillion parameters. Size classes follow the published model names; parameter counts are rounded.
Model repositories and specifications checked . Hardware figures are our estimates, rather than measured requirements.
Some models also have a version with refusals removed, made by other people on Hugging Face. They will answer requests the original declines; you are responsible for how you use them.
Read the community rules.
How to read the figures
Each Download button links to the named GGUF file on Hugging Face. Sizes use decimal GB (1 GB = 1,000,000,000 bytes); exact byte counts appear below them. For split downloads, the size covers all parts: Download opens part 1 and All parts opens the file list. Get every part in the same folder. Multimodal projectors are separate files in that list; the download size excludes them.
The 4-bit column uses the named quant. QAT means quantization-aware training. Hardware and disk figures below are estimates; the file sizes beside Download come from Hugging Face. Check that your runner supports the architecture and quant.
All hardware numbers are about values in GB. VRAM assumes all weights on the GPU, one chat and about 4,096 tokens of total context, with some working space. RAM means system RAM alongside a dedicated GPU. Disk covers one set of weights, including a vision projector where applicable; allow extra space if keeping originals or converting files.
The 16-bit column estimates BF16 / FP16 weights and memory, including expanded weights for models published in FP8, MXFP4 or FP4 / FP8. It is a comparison, not a promise of a separate 16-bit download. A quant mixes precisions, so file sizes are larger than a simple four-bits-per-parameter calculation.
Large VRAM figures are totals across compatible GPUs; splitting weights and caches also depends on the runner. With partial GPU offload, leave enough system RAM for weights kept on the CPU. On a unified-memory computer, the GPU and CPU share one memory pool, so these columns cannot be added as separate capacities.
Context is the model card or configuration limit in tokens. The memory estimates do not cover those maximum lengths. Images, longer chats, larger batches and concurrent requests need more memory.
The listed models use Apache License 2.0 or MIT. Read each linked licence and preserve required notices when redistributing modified weights.
Tiny — under 2B
Small downloads for short tasks and trying a local runner.
| Model and parameters | Licence | Context tokens | 4-bit VRAM / RAM / disk | 16-bit VRAM / RAM / disk |
|---|---|---|---|---|
| Qwen 3.5 0.8B0.8B text + vision parametersImages and document reading.
Download
GGUF by unsloth
Refusals removed
Made by huihui-ai · GGUF by mradermacher
|
Apache License 2.0 | 262,144 | Q4_K_MVRAM about 4 GB RAM about 8 GB Disk about 1 GB |
VRAM about 4 GB RAM about 8 GB Disk about 2 GB |
| MiniCPM5 1B1.08B text parametersText reasoning and tool use.
Refusals removed
Made by huihui-ai · GGUF by mradermacher
|
Apache License 2.0 | 131,072 | Q4_K_MVRAM about 4 GB RAM about 8 GB Disk about 1 GB |
VRAM about 4 GB RAM about 8 GB Disk about 3 GB |
Small — 2–5B
A starting point for a laptop or a modest desktop GPU.
| Model and parameters | Licence | Context tokens | 4-bit VRAM / RAM / disk | 16-bit VRAM / RAM / disk |
|---|---|---|---|---|
| MiniCPM5 2B2.52B text parametersText reasoning and agents.
Refusals removed
Made by huihui-ai · GGUF by mradermacher
|
Apache License 2.0 | 131,072 | Q4_K_MVRAM about 4 GB RAM about 8 GB Disk about 2 GB |
VRAM about 8 GB RAM about 16 GB Disk about 6 GB |
| Qwen 3.5 4B4B text + vision parametersImages and video.
Download
GGUF by unsloth
|
Apache License 2.0 | 262,144 | Q4_K_MVRAM about 8 GB RAM about 16 GB Disk about 4 GB |
VRAM about 16 GB RAM about 24 GB Disk about 10 GB |
Medium — 7–14B
More room for conversation and coding, with a larger memory footprint.
| Model and parameters | Licence | Context tokens | 4-bit VRAM / RAM / disk | 16-bit VRAM / RAM / disk |
|---|---|---|---|---|
| Qwen 3.5 9B9B text + vision parametersGeneral reasoning and coding.
Download
GGUF by unsloth
Refusals removed
Made by huihui-ai · GGUF by mradermacher
|
Apache License 2.0 | 262,144 | Q4_K_MVRAM about 12 GB RAM about 24 GB Disk about 7 GB |
VRAM about 32 GB RAM about 48 GB Disk about 20 GB |
| Gemma 4 12B IT11.95B text, vision and audio parametersSpeech recognition and translation.
Refusals removed
Made by huihui-ai · GGUF by mradermacher
|
Apache License 2.0 | 262,144 | Q4_0 (QAT)VRAM about 12 GB RAM about 24 GB Disk about 8 GB |
VRAM about 32 GB RAM about 48 GB Disk about 25 GB |
Large — 20–40B
Check the complete weight size and context memory before loading.
| Model and parameters | Licence | Context tokens | 4-bit VRAM / RAM / disk | 16-bit VRAM / RAM / disk |
|---|---|---|---|---|
| Qwen 3.8 27B27B text + vision parametersCoding, reasoning and visual tasks.
Download
GGUF by ggml-org
Refusals removed
Made by huihui-ai · GGUF by mradermacher
|
Apache License 2.0 | 262,144Up to 1,000,000 with scaling. | Q4_K_MVRAM about 24 GB RAM about 48 GB Disk about 20 GB |
VRAM about 64 GB RAM about 96 GB Disk about 56 GB |
Very large — 70B+ / MoE
MoE means mixture of experts. Only some experts run for each token, but all stored weights still need space. MoE models of every size are grouped here. The largest entries need multiple GPUs or a server cluster for full GPU loading.
| Model and parameters | Licence | Context tokens | 4-bit VRAM / RAM / disk | 16-bit VRAM / RAM / disk |
|---|---|---|---|---|
| DeepSeek V4.1 Flash 1017763B stored parameters; MoE. Published in FP8; vision projector separate.Coding and tool-driven tasks.
Download
GGUF by unsloth
|
MIT License | 1,048,576 | UD-Q4_K_XLVRAM about 640 GB RAM about 256 GB Disk about 520 GB |
VRAM about 1,700 GB RAM about 512 GB Disk about 1,540 GB |
| MiMo V2.6 Pro RL1.02T total; 42B active parameters. MoE; image, video and audio input.General reasoning and multimodal work.
Download
GGUF by ggml-org
|
MIT LicenseDeclared on the publisher’s model card. | 1,048,576 | MXFP4VRAM about 640 GB RAM about 256 GB Disk about 560 GB |
VRAM about 2,300 GB RAM about 512 GB Disk about 2,100 GB |
Running models well
- Start with a model and quant that fit your available VRAM with room for the context cache and the runner. Q4_K_M is a common starting point. Try a higher precision if it fits and improves your own tasks.
- Start with a short context, such as 4,096 tokens, and increase it only as needed. Input, chat history and generated text share that window. Longer contexts use more memory; a model’s maximum context is not a sensible default for every computer.
- Use GPU offload where supported. Keeping all weights on the GPU usually helps speed. Partial offload can fit a larger model, but moving work through system RAM can slow it down. LM Studio can estimate memory and adjust GPU offload before loading.
- llama.cpp supports GGUF files and CPU / GPU inference. Ollama and LM Studio offer local model management and chat. Check that your chosen version supports the model’s architecture and quant; these tools do not all support every entry.
- For a server handling several requests, vLLM offers batching and memory controls. Set the context and concurrency to fit the hardware, and use a quantization format supported by that model and backend.
- Use the publisher’s chat template and sampling settings. Vision models may also need a matching vision projector. Prefer a runner with built-in support; review any repository code before enabling remote-code execution.
- Try the same few tasks when comparing quants or models. Check useful answers and memory use as well as speed. Keep important source material handy: local models can still make mistakes.