LLM Foundry is the source repository for the development of models, datasets, and other artifacts related to the development of Large Language Models. It bundles training, evaluation, post-training, data processing, and tokenization pipelines into a single, cluster-ready code base.
This repository contains all source code used for the development of the artifacts tied to the Polyglot project at the University of Bonn. It is primarily designed to run on both the Marvin cluster and Bender (University of Bonn), which have dual software stacks (AMD and Intel) that the code base is aware of. We also offer documentation to work with our code base on other clusters, such as JSC Jupiter and BAF (Bonn Analysis Facility).
- PyTorch-native distributed training — Full implementations for both DDP and FSDP2, giving you flexibility across single-node and multi-node setups. Our code base scales almost linearly across hundreds of GPUs.
- Flash Attention — Support up to Flash Attention 4 for Hopper/Blackwell GPUs, plus Liger Kernel integration for additional fused kernel efficiency.
- Expert Parallelism — First-class support for Mixture-of-Experts (MoE) training.
- Hybrid architectures — Ready for models combining attention with state-space blocks. Includes specialized kernels like flash-linear-attention and causal-conv1d.
- FP8 training — Native FP8 mixed-precision support on Hopper/Blackwell GPUs for faster, more memory-efficient pretraining.
- Modern optimizers — AdamW, plus a Muon + AdamW mixed optimizer setup for improved convergence.
- Experiment tracking — Log metrics and artifacts via Weights & Biases or TrackIO (online and offline modes).
- Carbon tracking — Built-in CodeCarbon integration to monitor and log the carbon footprint of your training runs.
- Multiple alignment strategies — From simple SFT and DPO to more advanced RL-based methods like GRPO with verifier-based rewards, and Reward Model training.
- Toy Gym — A lightweight environment for generating tasks with verifiable rewards/solutions (currently in Portuguese, easily adaptable to other languages).
- Large-scale text processing — Scalable, multi-node pipelines for processing massive corpora (e.g., Common Crawl dumps) via DataTrove. We provide implementations of the FineWeb-2 pipeline, which offers support for thousands of languages.
- Learned quality filters — Working implementations for training and running learned classifiers for data quality control.
- Tokenization & packing — Full support for pretraining and post-training tokenization with advanced features: BFD packing, selective loss masking for SFT samples, decontamination, and more.
- Tokenizer training — Train tokenizers with 🤗 Tokenizers or SentencePiece, with automatic conversion to Hugging Face-compatible objects.
- Scalable generation — Large-scale synthetic data pipelines for both pretraining and more elaborate workflows (e.g., Constitutional AI) via vLLM + DataTrove.
- Agent-based traces — Generate multi-step CodeAgent reasoning traces with smolagents for downstream training.
- Standardized evaluation — Run our full evaluation suite through the lm-evaluation-harness.
- Model merging — Simple merging strategies like tokenizer transplantation, which powers our continual pretraining workflows, via mergekit.
A rich collection of utilities for data downloading, dataset conversion to Hugging Face format, token counting, weight resetting, embedding layer resizing, checkpoint uploading, model inspection, inference testing, and more—everything we use day-to-day to develop models and run experiments.
All in a single repository.
- Repository Structure
- Installation
- Running the Tests
- How to Train a Model
- Code of Conduct
- Contributing
- License
- Acknowledgments
The code base is organized into the following main folders:
alignment/— Implementation of post-training techniques for alignment, including both Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), Reward Model training, and Group Relative Policy Optimization (GRPO) with verifier-based rewards.alignment/gym/— Scripts for training and evaluating language models on custom environments.
data/— Scripts for working with text preprocessing (i.e., filtering, tokenization, etc.).data/cc/— Scripts for working with Common Crawl data.data/filters/— Dataset filtering and annotation pipelines for text corpus curation.data/formatting/— Parsers for converting raw datasets into a standardized format or to perform stratification for evaluation.data/tokenization/— Tokenization, packing, decontamination, and validation split utilities for pretraining and SFT datasets.
distributed/— Scripts for training and evaluating language models with DDP and FSDP.evals/— Scripts for evaluating language models via thelm-evaluation-harness.merge/— Scripts for running different merging techniques viamergekit.shared/— Shared utility modules used across the major pipeline folders. Seeshared/README.md.synthetic/— Scripts for generating synthetic datasets with vLLM.synthetic/agents/— Agent-based trace generation using smolagents. Records multi-step CodeAgent reasoning traces for downstream training. Seesynthetic/agents/README.md.
tests/— Unit and integration tests for our code base.tokenizer/— Scripts for training and evaluating tokenizers.tools/— Miscellaneous tools and utilities for our code base.docs/— Cluster-specific documentation for running the code base on other systems.docs/baf/— How to run jobs on the BAF (Bonn Analysis Facility) cluster. Seedocs/baf/README.md.docs/jupiter/— How to run jobs on the JSC Jupiter booster. Seedocs/jupiter/README.md.
All of our codebase is designed primarily to run on Marvin or Bender, i.e., the University of Bonn HPC clusters. You will only need to set things up on the cluster itself - not on your local machine. For your local machine, you can just clone the repository and work with the files (e.g., editing code, writing new scripts, etc.) without worrying too much about dual stack setups or module loading.
On Marvin, we work with workspaces that are allocated with a specific file system.
Use tools/slurm/marvin_create_workspace.sh to allocate a workspace, clone the repository, and prepare the directory layout. Open the script first and edit the user customization section at the top (username, file_system, work_group, email, workspace_name) to match your account, then run it from a Marvin login node:
bash tools/slurm/marvin_create_workspace.shFor Bender users, /home/$USER is the default workspace directory, so you can just clone the repository there and start working.
Marvin and Bender have a dual software stack (AMD and Intel). The single .modules.sh file at the repository root loads the right build for you. It auto-detects the stack from the SLURM environment, so most of the time you can just source it and forget about it:
# Marvin:
# - Partitions with "gpu" in the name (e.g. sgpu, mlgpu) -> AMD stack
# - All other partitions -> Intel stack
#
# Bender:
# - Partition "a100" -> AMD stack
# - Partition "a40" -> Intel stack
source "$workdir/.modules.sh"You can also force a specific stack by setting the LLM_FOUNDRY_STACK environment variable before sourcing:
LLM_FOUNDRY_STACK=amd source "$workdir/.modules.sh" # GPU/training stack
LLM_FOUNDRY_STACK=intel source "$workdir/.modules.sh" # CPU/data stackSourcing prints whose stack was selected, why, and the resulting module list, so your job logs always show the resolved environment.
If you are working on JSC Jupiter, things work a little differently. See
docs/jupiter/README.mdfor JSC-specific module and installation scripts.If you are working in BAF, things also work a little differently. See
docs/baf/README.mdfor how to run jobs on it. BAF uses containers (HTCondor) instead of SLURM.
Use the pyproject.toml to install a specific set of dependencies. The available extras are:
data— For downloading and preprocessing datasets.tokenizer— For training and evaluating tokenizers with the pinned SentencePiece-compatible stack.distributed— For training language models with our DDP and FSDP implementations.synth— For generating synthetic samples with vLLM.agents— For generating CodeAgent execution traces with smolagents (LiteLLM, Transformers, or vLLM backends).trl— For post-training and alignment with TRL.tests— For running our test suite.
For example:
pip install -e "./llm-foundry/.[distributed]"NOTE: Some dependency stacks are hard to build. For example, when several libraries (e.g.,
torch,liger-kernel,flash-attn,causal-conv1d, etc.) have to live together and share the samenvcccompiler, building this environment can be a real nightmare. For some of the environments we use, we have dedicatedcreate_venvscripts that take care of all of the complexity of this build (e.g.,distributed/slurm/create_venv_marvin.sh,docs/jupiter/jupiter_installation_2026.sh,docs/baf/create_venv.sh). In these cases, we recommend using those scripts instead of praying to the gods thatpiporuvwill find their way through the forest of incompatibilities.
uv is an extremely fast Python package and project manager, written in Rust, that works as a drop-in replacement for pip. Install it with the standalone installer:
# On macOS and Linux.
curl -LsSf https://astral.sh/uv/install.sh | shAlternatively, install it from PyPI with pip install uv (or pipx install uv). On Windows, use powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex". If you used the standalone installer, you can keep it up to date with uv self update. See the uv installation docs for more options.
Then install the same extras as before with the uv pip interface:
uv pip install -e "./llm-foundry/.[distributed]"The uv pip commands mirror pip, so install -e works the same way and the extras listed above apply unchanged. You can also create a virtual environment first with uv venv (and activate it) if you prefer an isolated environment.
- IMPORTANT: We are constantly updating our dependencies so this stack is up to date with the latest developments of the LLM ecosystem.
If you are a developer or contributor, you can run our test suite to verify that your changes do not break anything. The tests are organized into unit and integration tests, and they cover the main functionalities of the code base.
Install the test dependencies first:
pip install -e "./llm-foundry/.[tests]"Run all test scripts in sequence:
python tests/Or run a specific script (e.g., the distributed training tests):
python tests/tests_distributed.pyFor a step-by-step walkthrough of the LLM Foundry—covering data collection, tokenization, evaluation harness setup, pretraining, and post-training/alignment—see HOWTO.md.
This project adheres to a Contributor Covenant Code of Conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to kluge@uni-bonn.de.
Contributions are welcome! Please see CONTRIBUTING.md for details on how to set up your development environment, the contribution workflow (forking, branching, squashing commits, opening a pull request), and the project's style guide.
This project is licensed under the Apache License 2.0. See LICENSE for the full license text.
Polyglot is a project funded by the Federal Ministry of Education and Research (BMBF) and the Ministry of Culture and Science of the State of North Rhine-Westphalia (MWK) as part of TRA Sustainable Futures (University of Bonn) and the Excellence Strategy of the federal and state governments.
We also gratefully acknowledge access to the Marvin and Bender clusters, hosted by the University of Bonn, and maintained by the university's High Performance Computing Team. We also appreciate the work of the support team that maintains the Bonn Analysis Facility (BAF) for providing constant support and maintenance to the infrastructure we all share.