Assistant 🤖
Here's the detailed situation and instructions based on my research.
Ollama does NOT support MTP (Multi-Token Prediction) yet. From the official GGUF page on HuggingFace:
Ollama: Does not load — No speculative decoding support yet
So you have two paths:
Just the standard install — easy, but MTP won't work:
# 1. Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# 2. Pull Qwen3.6 27B
ollama pull qwen3.6:27b
# 3. Run it
ollama run qwen3.6:27b
Ollama auto-selects a quantization that fits your 48GB Mac. To force a specific one:
ollama run qwen3.6:27b-q4_K_M
✅ Pros: Dead simple, OpenAI-compatible API at http://localhost:11434/v1
❌ Cons: No MTP speedup (~1.5–2× slower generation than with MTP)
This is what you want if you specifically want MTP. llama.cpp b9180+ has built-in MTP support. Here's the full guide:
brew install llama.cpp
Verify the version (needs b9180+):
llama-server --version
Pick a quantization that fits your 48GB MacBook Pro. Based on the compatibility table:
|
RAM |
Best Quant |
KV Cache |
Max Context |
Total Used |
Vision |
|
48 GB |
Q6_K |
q8_0 |
262K (full) |
32.3 GB |
✅ |
|
48 GB |
Q5_K_M |
— |
262K (full) |
36.5 GB |
✅ |
|
48 GB |
Q8_0 |
q8_0 |
243K |
38.0 GB |
✅ |
My recommendation for your 48GB Mac: Q5_K_M — best sweet spot of quality, full 262K context, and room to spare. Or Q6_K with q8_0 KV cache for even better quality while still getting full context.
Download from froggeric/Qwen3.6-27B-MTP-GGUF:
# Create a models directory
mkdir -p ~/models/qwen3.6-27b-mtp && cd ~/models/qwen3.6-27b-mtp
# Download Q5_K_M (18 GB) — the sweet spot for 48GB
wget https://huggingface.co/froggeric/Qwen3.6-27B-MTP-GGUF/resolve/main/Qwen3.6-27B-Q5_K_M-mtp.gguf
# Download vision encoder (885 MB) — only if you want vision support
wget https://huggingface.co/froggeric/Qwen3.6-27B-MTP-GGUF/resolve/main/mmproj-Qwen3.6-27B-f16.gguf
(Or use your browser to download — the files are on HuggingFace)
For coding/chat (with MTP speedup):
llama-server \
-m ~/models/qwen3.6-27b-mtp/Qwen3.6-27B-Q5_K_M-mtp.gguf \
--mmproj ~/models/qwen3.6-27b-mtp/mmproj-Qwen3.6-27B-f16.gguf \
--spec-type draft-mtp --spec-draft-n-max 3 \
-c 262144 -fa off --n-predict -1 \
--temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.0 \
-ngl 99 --port 8081
Flag explanations: | Flag | What it does | |------|-------------| | --spec-type draft-mtp | Enables MTP (the magic sauce) | | --spec-draft-n-max 3 | Draft 3 tokens per step (optimal for most quants) | | -c 262144 | Full 262K native context | | -fa off | Disable Flash Attention (37–53% faster prefill on Apple Silicon!) | | -ngl 99 | Offload all layers to GPU (Metal on Apple Silicon) | | --mmproj ... | Vision encoder (omit if you don't need vision) |
Once the server is running, it exposes: - OpenAI-compatible API at http://localhost:8081/v1/chat/completions - Anthropic-compatible API at http://localhost:8081/v1/messages
Test it:
curl http://localhost:8081/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen","messages":[{"role":"user","content":"Write hello world in Python"}]}'
ANTHROPIC_BASE_URL=http://127.0.0.1:8081 claude
|
Use Case |
Enable MTP? |
Why |
|
Coding / debugging |
✅ Yes |
79–89% acceptance rate → ~1.3–1.5× faster |
|
Factual Q&A |
✅ Yes |
62–70% acceptance |
|
Creative writing |
❌ No (Q4–Q6) |
Only 39–48% acceptance; can be slower at low quants |
|
Creative writing |
✅ Yes (Q8+) |
At Q8, even creative gets +48% speedup |
To disable MTP temporarily: --spec-type none
If you want more headroom, you can increase the available memory for GPU:
sudo sysctl iogpu.wired_limit_mb=45000
(Sets limit to ~44 GB, leaving ~4 GB for macOS)
|
What you want |
Do this |
|
Easiest setup, don't care about MTP |
ollama pull qwen3.6:27b |
|
Want MTP speedup (~1.5–2× faster) |
Use llama.cpp (Path 2 above) with Q5_K_M quant |
|
Maximum quality on 48GB |
Q6_K with q8_0 KV cache → full 262K context + vision |
|
Best bang for buck |
Q5_K_M → full 262K context, 36.5 GB, vision, great quality |
Want me to write any of this into a script for you, or help with any specific step?