rust library to write integer types of any bit length into a buffer - from `i1` to `i64`.
-
Updated
Jul 16, 2024 - Rust
#
int4
Here are 48 public repositories matching this topic...
Quantize TinyLlama-1.1B-Chat from PyTorch to CoreML (float16, int8, int4) for efficient on-device inference on iOS 18+.
nlp mobile ai transformers pytorch llama quantization int8 coreml on-device huggingface apple-silicon int4 llm tinyllama ios18 mlpackage
-
Updated
May 18, 2025 - Python
🧬🔍 Vecgo is a pure Go, embeddable, hybrid vector database designed for high-performance production workloads. It combines commit-oriented durability with HNSW + DiskANN indexing for best-in-class performance.
golang vector zero-copy mvcc s3 embeddings simd bq mmap ann opq pq hnsw rrf vamana int4 vectorstore diskann rabitq sq8
-
Updated
Jan 19, 2026 - Go
Quantizes the unquantized LATCH-Qwen2.5-14B model to GGUF format with a strict perplexity delta constraint. Outputs a Markdown benchmark report comparing FP16 vs Q4_K_M accuracy.
-
Updated
Mar 26, 2026 - Python
Utilities to rewrite ONNX convolution patterns into MatMul f 7440 orms for optimal LLM-like int4 quantization (esp. Audio/Speech models).
-
Updated
Apr 17, 2026 - Python
Training-free fix for KV cache INT4 failures. Norm separation + per-channel quantization. Qwen2-7B: 744× improvement (ΔPPL +238 → +0.32). 12 models, 124M–40B. 4 lines of PyTorch.
reproducible-research pytorch open-science transformer outliers quantization memory-optimization kv-cache int4 llm-inference norm-separation per-channel-quantization
-
Updated
Apr 18, 2026 - Python
PyTorch implementation of TRIAD-PTQ (Trace-Router-Interaction-Aware Decomposition) — weight-only INT3/INT4 PTQ for compact LLMs and edge CNNs/ViTs, with real benchmarks on SmolLM/TinyLlama/MobileNetV2/EfficientNet-B0/MobileViT-S.
pytorch mps quantization model-compression mobilenet efficient-inference edge-ai post-training-quantization apple-silicon vision-transformer int4 ptq llm gptq weight-only-quantization
-
Updated
May 5, 2026 - Python
row-major matmul optimization
-
Updated
May 14, 2026 - C++
Training-free INT3 KV cache quantization: 5.09× compression, ~10 lines of Python, <5% WikiText-2 ΔPPL on 8 of 8 open-weight Transformers (GPT-J 2021 → Gemma-4 2026). No calibration, no codebook, no rotation, no adapter. +2.4% decode overhead with torch.compile (no custom CUDA).
reproducible-research transformers pytorch open-science quantization memory-optimization kv-cache int4 calibration-free llm-inference int3 norm-separation per-channel-quantization activation-outliers
-
Updated
May 21, 2026 - Python
Block-scaled FP8 / FP4 / INT4 tensor primitive with Triton scaled-matmul at FP32 parity on H100. NumPy / PyTorch / MLX / JAX backends.
machine-learning deep-learning numpy pytorch triton quantization mlx jax low-precision int4 fp8 fp4 torchao nvfp4 transformer-engine mxfp8
-
Updated
May 24, 2026 - Python
Backprop-free learning study: spiking (LIF) neurons + Forward-Forward + JEPA + int4 QAT, with a full ablation notebook.
deep-learning pytorch spiking-neural-networks quantization neuromorphic norse self-supervised-learning int4 forward-forward jepa
-
Updated
May 30, 2026 - Python
-
Updated
Jun 16, 2026 - Python
Analyzes per-layer quantization sensitivity in GPT-2 across 9 quantization configs (INT8/INT6/INT5/INT4/INT3/INT2 with group-wise and per-tensor). Key findings: INT8-g32 achieves 1.8x compression with +0.13 perplexity; group-wise quantization reduces INT4 degradation by 99%; mlp_proj layers dominate sensitivity.
python compression gpu transformers inference pytorch quantization int8 mixed-precision gpt2 weight-quantization int4 llm
-
Updated
Jul 10, 2026 - Python
Empirical profiling of FP32, FP16, INT8, and INT4 quantization on GPT-2 and GPT-2-medium: throughput, decode latency, model memory, and perplexity across batch sizes 1–16. FP16 wins on throughput; INT4 wins on memory with negligible quality loss.
memory transformers pytorch throughput quantization fp16 int8 gpt2 int4 llm-inference bitsandbytes gpu-profiling
-
Updated
Jul 13, 2026 - Python
Run Tencent Hy3 (295B MoE, 21B active) on a consumer machine — pure C, experts streamed from NVMe
-
Updated
Jul 15, 2026 - C
GLM-5.2, a 744 billion parameter mixture of experts model, in a pure C inference engine: quantized to int4, experts streamed from disk, deployed and benchmarked. Generates in 16 GB of RAM.
c machine-learning deep-learning cuda moe glm quantization systems-programming inference-engine consumer-hardware mixture-of-experts edge-ai cpu-inference int4 large-language-models llm llm-inference speculative-decoding
-
Updated
Jul 15, 2026 - C
Benchmarks five KV cache quantization schemes (FP16→INT4) on GPT-2 with per-layer sensitivity analysis. Measures memory reduction vs perplexity tradeoff across sequence lengths without modifying model weights.
machine-learning inference pytorch transformer attention quantization int8 memory-optimization gpt2 kv-cache int4 llm fp8 llm-inference
-
Updated
Jul 17, 2026 - Python
Modly extension for research-only Cube3D INT4 text-to-mesh generation with UI-managed weights and GLB output.
cuda pytorch mesh glb bfloat16 cube3d int4 research-only text-to-3d local-ai text-to-mesh modly modly-extension geometry-only
-
Updated
Jul 17, 2026 - Python
Improve this page
Add a description, image, and links to the int4 topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the int4 topic, visit your repo's landing page and select "manage topics."