An LLM server for a single RTX 5090, built for agent workloads: tool calls, long conversations, reasoning, and many requests at once. Consistently faster than llama.cpp on the same models, with the numbers in the repo. Written end to end by Claude Code.
cpp cuda quantization inference-engine mixture-of-experts blackwell llm safetensors llama-cpp vllm local-llm llm-inference qwen gguf fp4 rtx-5090 sm120 nvfp4 rtx-pro-6000 gated-deltanet
-
Updated
Aug 26, 2026 - Cuda