8000

#

turboquant

Here are 142 public repositories matching this topic...

turbovec

RyanCodrai / turbovec

A vector index built on TurboQuant, written in Rust with Python bindings

python rust neon embeddings simd nearest-neighbor quant ann quantization avx512 embedding faiss rag vector-search turboquant

Updated Aug 21, 2026
Rust

jaylfc / taOS

Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).

raspberry-pi privacy offline-first distributed-computing self-hosted orange-pi ai-agents data-sovereignty ai-platform local-first agent-framework apple-silicon llm vllm local-llm llm-inference kv-cache-quantization rockchip-npu turboquant

Updated Aug 23, 2026
Python

quantumaikr / quant.cpp

LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.

embeddable transformer pure-c quantization delta-compression kv-cache llm llm-inference gguf turboquant

Updated Apr 26, 2026
C

AtomicBot-ai / atomic-llama-cpp-turboquant

llama.cpp fork with TurboQuant WHT-rotated KV cache & weight compression + Gemma 4 MTP and Qwen 3.6 NextN speculative decoding (+30-50% throughput).

Updated Aug 6, 2026
C++

PacifAIst / Quansloth

Based on the implementation of Google's TurboQuant (ICLR 2026) — Quansloth brings elite KV cache compression to local LLM inference. Quansloth is a fully private, air-gapped AI server that runs massive context models natively on consumer hardware with ease

cuda turboquant quansloth vram-wall

Updated May 13, 2026
Python

julianmb / q38rocm

Qwen 3.8 27B ROCmFP4 on AMD Strix Halo (Ryzen AI Max+ 395). Up to 36 tok/s via MTP Speculation, TurboQuant & Mesa RADV Wave64.

vulkan mtp rocm llama-cpp qwen gguf speculative-decoding fp4 strix-halo amd-ryzen-ai radv turboquant

Updated Aug 23, 2026
Python

arozanov / turboquant-mlx

TurboQuant KV cache compression for MLX with fused Metal kernels. 4.6x compression at 98% FP16 speed.

metal quantization mlx kv-cache apple-silicon llm turboquant

Updated Apr 30, 2026
Python

Indras-Mirror / llama.cpp-turboq-mtp

Fused TBQ4 Flash Attention + MTP + Shared Tensors + Qwen35 SWA Hybrid for llama.cpp — 82+ tok/s, lossless 4.25 bpv KV cache, SWA-bounded deep-context decode (w/ long-range recall) on RTX 4090

cuda quantization mtp kv-cache fwht llama-cpp flash-attention qwen speculative-decoding rtx-4090 multi-token-prediction turboquant tbq4 tensor-sharing

Updated Aug 23, 2026
C++

manjunathshiva / turboquant-mlx

Extreme weight + KV cache compression for LLMs on Apple Silicon (MLX implementation of Google's TurboQuant)

quantization mlx kv-cache apple-silicon llm turboquant

Updated Aug 18, 2026
Python

unixsysdev / llama-turboquant

TurboQuant 3-bit KV-cache quantization for llama.cpp

quantization kv-cache llama-cpp llm-inference turboquant

Updated Jul 30, 2026
C++

bigmacfive / turbo-graph

TurboQuant-compatible vector search plus graph memory for constrained RAG.

python rust graph embeddings rag vector-search turboquant turbovec

Updated Jun 19, 2026
Rust

Alberto-Codes / turboquant-vllm

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

compression transformer triton quantization inference-optimization kv-cache llm vllm consumer-gpu turboquant

Updated Apr 10, 2026
Python

atomicmilkshake / llama-cpp-turboquant

llama.cpp fork with TurboQuant quantization (turbo2/3/4) and TriAttention GPU-accelerated KV cache pruning. 75 tok/s on Qwen3-8B / RTX 3080.

windows cuda inference quantization kv-cache llm llama-cpp ggml turboquant triattention

Updated Jul 2, 2026
C++

back2matching / turboquant

First open-source TurboQuant KV cache compression for LLM inference. Drop-in for HuggingFace. pip install turboquant.

machine-learning compression gpu transformers inference pytorch quantization vram huggingface kv-cache llm turboquant

Updated Apr 21, 2026
Python

aivrar / vllm-windows-build

Native Windows vLLM 0.27.1 wheels: Python 3.13, PyTorch 2.13 + CUDA 13.0, SM 7.5-12.0 for RTX 20/30/40/50, OpenAI-compatible serving, FlashAttention/Rust, 10 KV formats, Multi-TurboQuant, and experimental CPU/RAM/NVMe prompt-KV offload - no WSL or Docker.

Updated Aug 21, 2026
Python

croll83 / llama.cpp-dgx

llama.cpp fork optimized for NVIDIA DGX Spark / GB10 (Blackwell, SM 12.1) — TurboQuant weights + KV, NVFP4, DFlash MTP

b 10000 lackwell llama-cpp speculative-decoding gb10 nvfp4 dflash turboquant

Updated May 26, 2026
C++

Sggin1 / DGX-SPARK

DGX Spark research and tests - containers, benchmarks, and investigation notes for running models on GB10 (SM 12.1)

aarch64 blackwell kv-cache vllm nvfp4 dgx-spark mamba-ssm sm121 turboquant

Updated Aug 6, 2026
Python

artalis-io / bitnet.c

Minimal, zero-dependency LLM inference in pure C11. CPU-first with NEON/AVX2 SIMD. Flash MoE (pread + LRU expert cache). TurboQuant 3-bit KV compression (8.9x less memory per session). 20+ GGUF quant formats. Compiles to WASM.

c neon wasm inference simd moe avx2 quantization kv-cache cpu-inference llm gguf turboquant

Updated Aug 15, 2026
C

aivrar / multi-turboquant

Unified KV-cache compression for LLM inference: 12 Python-native methods, guarded add-on composition and routing, analytical capacity simulation, Godzilla KVarN/TriAttention with SM86/SM89 qualification, CUDA weight sharing, and multi-GPU planning.

Updated Aug 22, 2026
Python

carlosfundora / llama.cpp-1-bit-turbo

HIP/ROCm fork optimized for AMD RDNA2 (gfx1030) with PrismML Q1_0_G128 1-bit quant support, RotorQuant, TurboQuant, EAGLE3 and P-EAGLE speculative decoding, and full Wave32 kernel optimizations.

hip quantization bonsai rocm amd-gpu llama-cpp gguf rdna2 turboquant prismml gfx1030

Updated Jun 23, 2026
C++

Improve this page

Add a description, image, and links to the turboquant topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the turboquant topic, visit your repo's landing page and select "manage topics."

0