📋 voyage-4-nano

🤖 Assistant

OK, I've done a thorough deep-dive. Here's the full picture:

---

🟢 Viability: HIGHLY VIABLE — Go for it!

This is actually surprisingly well-supported already. Here's the breakdown:

---

1. ✅ GGUF Conversion — Already Done

The repo jsonMartin/voyage-4-nano-gguf has a ready-to-use conversion script (make_voyage4_gguf.py) and pre-built GGUF files on HuggingFace at jsonMartin/voyage-4-nano-gguf:

File	Size	Quality
voyage-4-nano-f16.gguf	695 MB	1.000000 cosine sim (identical)
voyage-4-nano-q8_0.gguf	372 MB	0.999903 cosine sim (excellent)
voyage-4-nano-linear.pt	~4 MB	Linear projection 1024→2048

The conversion uses standard convert_hf_to_gguf.py from llama.cpp. The model is Qwen3-based with bidirectional attention.

2. ✅ Q4 Quantization — Fully Possible

llama-quantize supports all K-quant formats from Q2_K through Q8_0. You can produce a Q4_K_M quant easily:

bash

./llama-quantize voyage-4-nano-f16.gguf voyage-4-nano-q4_k_m.gguf q4_k_m

Estimated size for Q4_K_M: ~250 MB (the model is 340M params, Q8_0 is 372MB). Perfectly reasonable for CPU inference.

3. ✅ Rust GGUF Inference — The llama-gguf Crate

The Lexmata/llama-gguf crate (v0.14.0, crates.io) is a full Rust reimplementation of llama.cpp. It:

• ✅ Supports Qwen3 architecture (what voyage-4-nano is based on)
• ✅ CPU-only inference with SIMD (AVX2, AVX-512, NEON) — no GPU needed
• ✅ All quant formats (Q2_K through Q8_0, F16/F32)
• ✅ Embedding/pooling support — mean pooling, L2 normalization
• ✅ Use as a library: just add llama-gguf = "0.14" to your Cargo.toml
• ✅ 50K+ lines of battle-tested Rust

4. 🧠 How It Would Integrate Into Moofile

Here's the architecture thinking:

┌─────────────────────────────────────────────┐

│ Moofile Rust Core │

│ │

│ ┌─────────────┐ ┌──────────────────────┐ │

│ │ Document │ │ Embedding Engine │ │

│ │ Store + │ │ (llama-gguf) │ │

│ │ Indexes │ │ │ │

│ │ │ │ voyage-4-nano-q4 │ │

│ │ text_search │ │ .gguf (250 MB) │ │

│ │ vector_search│ │ + linear.weight │ │

│ │ hybrid_search│ │ (4 MB, Rust-native) │ │

│ └──────┬───────┘ └──────────┬───────────┘ │

│ │ │ │

│ └─────────┬───────────┘ │

│ │ │

│ Auto-embed docs on insert │

└─────────────────────────────────────────────┘

The key integration points:

1. Add llama-gguf dependency to moofile-core/Cargo.toml
2. On Collection open (if embedding model configured), load the GGUF file + linear projection weights
3. Auto-embedding on insert: When a document is inserted with text fields for embedding, run the model and store the resulting vector
4. The linear projection (1024→2048) is a simple matrix multiply followed by L2 normalization — trivially implementable in Rust with ndarray or raw Vec<f32> operations
5. The llama-embedding CLI works like this (for reference on what flags are needed):

bash

./llama-embedding \

-m voyage-4-nano-q8_0.gguf \

--pooling mean \

--attention non-causal \

--embd-normalize 2 \

-p "Your text here"

5. ⚡️ CPU Performance Estimate

With a 340M param model in Q4:

• Memory: ~250 MB for weights + ~100 MB for runtime buffers ≈ 350 MB total
• Speed: On a modern x86 CPU with AVX2, expect roughly 50-100 tokens/second per document
• At 32K context, embedding a short doc (~100 tokens) would take ~1-2 seconds

6. ⚠️ Gotchas to Watch For

1. Bidirectional attention — The --attention non-causal flag is critical. The llama-gguf crate needs to support this for correct embeddings. The README lists Qwen3 support so it should work.
2. Linear projection layer — The GGUF model only outputs 1024-dim embeddings. To get the full 2048-dim output, you need the linear.weight matrix (shape 2048×1024). You'd need to store this as a separate weights file in GGUF or as raw binary and load it in Rust.
3. Tokenizer — voyage-4-nano uses Qwen3's tokenizer (151936 vocab size), which llama-gguf should handle natively.
4. Prompt format — Query needs "Represent the query for retrieving supporting documents: " prefix, documents need "Represent the document for retrieval: " prefix.

🏁 Bottom Line

Absolutely viable. The stars have aligned unusually well:

1. ✅ Model is Apache 2.0 licensed
2. ✅ Pre-built GGUF files exist
3. ✅ A mature Rust GGUF inference crate (llama-gguf) exists with Qwen3 support
4. ✅ Model is tiny (340M params, Q4 = ~250MB)
5. ✅ Moofile already has a Rust core and vector search infra
6. ✅ CPU-only inference is a first-class target

The integration work would involve:

• Adding llama-gguf as a Rust dependency (~1 day)
• Implementing the model loading & embedding pipeline (~2-3 days)
• Wiring it into Moofile's insert/query lifecycle (~1-2 days)
• Storing the linear projection weights in GGUF metadata or as a separate embedded file (~0.5 day)

Want me to prototype a basic Rust embedding module that loads the voyage-4-nano-gguf and generates embeddings?