token-counting

Here is 1 public repository matching this topic...

VirusAid / zerno

A single-header, zero-dependency BPE tokenizer for LLMs in C++17 — byte-exact with OpenAI's tiktoken (cl100k, o200k). Count GPT-4/GPT-4o tokens natively

nlp cpp tokenizer openai header-only cpp17 single-header bpe gpt-4 llm tiktoken token-counting

Updated Jul 15, 2026
C++

Improve this page

Add a description, image, and links to the token-counting topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the token-counting topic, visit your repo's landing page and select "manage topics."

Learn more