Question 1

How much GPU VRAM do I need to run Ornith-1.5-35B-A3B locally?

Accepted Answer

To run Ornith-1.5-35B-A3B (35B parameters) smoothly in Q4_K_M quantization with an 8K context window, we recommend at least 24 GB of dedicated GPU VRAM (after accounting for standard 1.5 GB OS overhead).

Question 2

How do I run Ornith-1.5-35B-A3B in Ollama or vLLM?

Accepted Answer

You can run Ornith-1.5-35B-A3B locally in one command using Ollama: `ollama run ornith-ai-ornith-15-35b-a3b` or via vLLM: `vllm serve ornith-ai-ornith-15-35b-a3b`.

Question 3

Can I run Ornith-1.5-35B-A3B on Apple Silicon Mac (M1/M2/M3/M4)?

Accepted Answer

Yes. Apple Silicon Macs use Unified Memory shared between CPU and GPU. For Ornith-1.5-35B-A3B, ensure your Mac has at least 32GB to 48GB of Unified Memory, accounting for macOS's ~25% system safety reserve.

Question 4

Does Ornith-1.5-35B-A3B support CPU offloading?

Accepted Answer

Yes, inference runtimes like llama.cpp and Ollama allow offloading transformer layers to system RAM if your GPU VRAM is tight, though full GPU VRAM execution is recommended for peak token generation speed.