A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.
🏆 2026 SKKU NPU Challenge 최우수상 — Mobilint MLA100 기반 KIVI-style INT4 KV Cache 구현·실험
Run Qwen3.6-27B on four Tesla V100s at 366 tok/s using hand-written NVFP4 CUDA kernels and chain-MTP speculation.
Pure-Rust, CPU-only OCR engine for Baidu Unlimited-OCR (a DeepSeek-OCR-derived 3B MoE VLM). Five-model zoo, custom int8 kernels, no ML framework, no Python, no GPU.
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime
414 KB WASM runtime for Needle a 14M-parameter tool-calling transformer. Runs in browser, Cloudflare Workers, and Node.js. No backend required.
Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4-AutoRound vs the FP8 baseline, measured on one harness — including why the quantization advantage collapses to +0.2% by c16.
Hand-written NVFP4 W4A16 CUDA kernels for Volta
Browser-native LLM inference engine. Hand-written WGSL kernels, int4 weights, no ML framework. Qwen2.5-0.5B at 43 tok/s entirely client-side.
[ICML 2026] GRACE-VLM: deployable INT4 Qwen3-VL via quantization-aware distillation.
Transparent local LLM inference server on Jetson Orin Nano Super (8GB). Watches INT4 quantization, session-pinned KV cache reuse, and per-layer activation patterns all in one place.
⚡️ The fastest way to run local LLMs on Apple Silicon — sub-second model loads, beats Ollama on throughput, tail latency, and full-response time. OpenAI/Ollama-compatible. No cloud, no API keys.
Purpose-built inference engine for Mistral Shieldstral 1.0 3B safety classifier. Optimized for RTX 3050 4GB VRAM. Custom CUDA kernel, INT4 quantization, multimodal vision pipeline, FastAPI moderation API.
Modelo neuronal de lenguaje local de 2,3 M de parámetros para ESP32-WROOM-32, con firmware, REPL serial y chat web sin conexión.
纯 Python 大模型高效推理参考工具包：INT8/INT4 训练后量化 + 分页 KV-cache + 投机解码，离线可跑
Post-training weight compression for low-RAM machines: Q4/Q8 quantization, green-format repair, AVX2 CPU inference, optional CUDA. ~45% less RAM at ~99.9% quality.
VKNN — Vulkan Neural Network: dependency-free C++17 on-device inference engine that runs ONNX models entirely on Android GPUs. CNNs, YOLO, LLMs, VLMs, 3D Gaussian Splatting; int4 quantization; compile once to .vxm.
Custom PTX INT4 GEMM kernel with Tensor Cores (WMMA) and end-to-end Qwen2 integration on NVIDIA Turing (SM75). Register-level dequantization, mma.sync.aligned intrinsics, and automatic M-padding wrapper. Delivers 14.7% VRAM reduction on Qwen2-0.5B with coherent generation quality.
Experimental MicroZoom INT4 inference and LoRA training port validated on a 24 GB RTX 3090 Ti
Low-bit LLM inference engine in Rust + CUDA. Converts models to a 4-bit .wstone format (3.75x smaller) and runs them on consumer GPUs. Built for memory-bandwidth-bound decode on Turing, where bytes per weight — not TOPS — sets token throughput.
Add a description, image, and links to the int4 topic page so that developers can more easily learn about it.
To associate your repository with the int4 topic, visit your repo's landing page and select "manage topics."