E533
Skip to content
#

int4

Star

Here are 48 public repositories matching this topic...

Language: All
Filter by language
All 48 Python 32 Rust 5 C 3 C++ 3 Go 1 JavaScript 1 Jupyter Notebook 1 Swift 1 TypeScript 1
Sort: Most stars
Sort options
Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

0xBakeer / Qwen3.8-27B-4-bit-on-a-single-DGX-Spark

Star 42

Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4-AutoRound vs the FP8 baseline, measured on one harness — including why the quantization advantage collapses to +0.2% by c16.

benchmark nvidia quantization marlin int4 llm-serving vllm llm-inference qwen speculative-decoding autoround qwen3 gb10 nvfp4 dgx-spark
  • Updated Aug 19, 2026
  • Python

konjoai / squish

Star 19

⚡️ The fastest way to run local LLMs on Apple Silicon — sub-second model loads, beats Ollama on throughput, tail latency, and full-response time. OpenAI/Ollama-compatible. No cloud, no API keys.

macos rust metal self-hosted quantization mlx inference-engine kv-cache apple-silicon openai-api int4 llm local-llm local-ai ollama qwen llm-infernece on-device-inference ollama-alternative llama-cpp-alternative
  • Updated Aug 13, 2026
  • Python

NAME0x0 / AVA

Star 5

Research and training stack for AVA — a tool-using, memory-aware virtual assistant targeting 4 GB VRAM. Spans custom transformers, verifier-RL, external memory, multi-domain benchmarks, and Gemma 4 inference optimization.

ai reproducible-research benchmarks personal-assistant quantization external-memory gemma virtual-assistant distillation inference-optimization tool-use edge-inference int4 llm long-context small-language-models kv-cache-compression recurrent-depth 4gb-vram verifier-rl
  • Updated Jul 25, 2026
  • Python

Anonym0usWork1221 / Whetstone

Star 3

Low-bit LLM inference engine in Rust + CUDA. Converts models to a 4-bit .wstone format (3.75x smaller) and runs them on consumer GPUs. Built for memory-bandwidth-bound decode on Turing, where bytes per weight — not TOPS — sets token throughput.

rust machine-learning deep-learning gpu cuda transformers cuda-kernels gpu-computing quantization performance-optimization inference-engine model-compression edge-ai tensor-cores low-bit-quantization int4 llm gptq llm-inference qwen
  • Updated Jul 29, 2026
  • Rust

Improve this page

Add a description, image, and links to the int4 topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the int4 topic, visit your repo's landing page and select "manage topics."

Learn more

0