A practical lab for intelligent LLM model selection using GitHub Models, focused on evaluation-driven decision-making for real-world GenAI systems.
This repository treats Large Language Models not as monolithic solutions, but as interchangeable system components whose selection is framed as a structured decision problem shaped by task requirements, system constraints, and explicit trade-offs.
This is not a leaderboard and not a training repository.
It is a decision-centric evaluation harness for real-world GenAI systems.
In real-world GenAI applications, choosing a model is a decision problem, not a popularity contest.
Typical trade-offs include:
- Schema adherence vs latency
- Determinism vs creativity
- Token efficiency vs output quality
- Consistency vs reasoning depth
This lab makes those trade-offs explicit, measurable, and reproducible.
- Task-level evaluation (e.g., JSON extraction, classification)
- System behavior benchmarks (latency, token efficiency)
- Reproducible prompts and datasets
- Artifact-based evaluation (every run is saved)
- Explicit model selection policy (rule-based baseline)
- Model training or fine-tuning
- Leaderboard-style SOTA claims
- Large-scale production load testing
- Provider cost optimization (only proxies)
llm-model-selection-lab/
├── models/ # Model registry and provider adapters
├── tasks/ # Task implementations
├── benchmarks/ # System benchmarks (latency, tokens)
├── evaluation/ # Metrics and validation logic
├── selection/ # Model selection policy (router)
├── prompts/ # Versioned prompt templates
├── datasets/ # Small JSONL datasets
├── scripts/ # CLI entry points
├── results/ # Run artifacts and summaries
└── docs/ # Design and interpretation docs
The system is designed around a clean separation of concerns:
- Tasks are model-agnostic
- Models are accessed through a single client interface
- Providers (GitHub Models) are hidden behind adapters
- Every execution produces explicit artifacts for analysis
flowchart LR
CLI[CLI Scripts<br/>scripts/*] --> TASKS[Tasks<br/>tasks/*]
CLI --> ROUTER[Model Selection Policy<br/>scripts/recommend_model.py]
TASKS --> CLIENT[Model Client<br/>models/client.py]
CLIENT --> REGISTRY[Model Registry<br/>models/registry.yaml]
CLIENT --> ADAPTER[GitHub Models Adapter<br/>models/github_models.py]
ADAPTER --> GH[GitHub Models<br/>Hosted Inference]
TASKS --> EVAL[Evaluation & Metrics<br/>evaluation/*]
EVAL --> RESULTS[Run Artifacts<br/>results/runs & results/benchmarks]
RESULTS --> SUMMARY[Aggregation<br/>scripts/summarize_results.py]
SUMMARY --> REPORTS[Summary Outputs<br/>results/summary.json]
ROUTER --> REGISTRY
This diagram reflects the current v1 design: task execution remains explicitly model-driven, while the selection policy operates as a standalone decision utility for structured comparison and future runtime integration.
- Python 3.10+ (validated in CI on Python 3.11)
- A GitHub account with access to GitHub Models
- A GitHub token with
models:readpermission uvinstalled
git clone https://github.com/<your-username>/llm-model-selection-lab.git
cd llm-model-selection-labDependencies are defined in pyproject.toml.
uv syncuv manages the project environment automatically, so no manual virtual environment activation is required.
Recommendation
If you’re using uv-only, it’s best to commit
uv.lockafter runninguv sync.
This ensures fully reproducible installs for reviewers and contributors.
cp config.example.env .envExport variables in your shell:
export GITHUB_TOKEN="your_github_personal_access_token"
# Optional: only set this if your GitHub Models access
# is provided via a GitHub organization
export GITHUB_ORG=""Note
GITHUB_ORG is optional.
Leave it empty unless your GitHub Models access is associated with an organization.
Never committokensto the repository.
Verify that GitHub Models access works:
uv run python -m scripts.smoke_test --model gpt-4.1-mini- A short model response printed to the console
- Latency metadata included in the output
- GITHUB_TOKEN is set correctly
- The model name exists in
models/registry.yaml
Evaluates schema adherence—one of the most common enterprise GenAI requirements.
uv run python -m scripts.run_task \
--task extract_json \
--model gpt-4.1-mini- JSON valid rate
- Required field completeness
- Extra keys rate
results/runs/<timestamp>__extract_json__<model>.json
Classification workflow is included in the repository and can be wired into the CLI as a follow-on extension.
Planned evaluation metrics:
- Accuracy
- Invalid output rate
Artifacts are written under results/runs/.
Measures P50 / P95 latency over a fixed prompt set.
uv run python -m scripts.run_benchmark \
--suite latency \
--model gpt-4.1-miniresults/benchmarks/<timestamp>__latency__<model>.json
Measures token usage when returned by the provider.
uv run python -m scripts.run_benchmark \
--suite token_efficiency \
--model mistral-smallNote Some models or providers may not return token usage. This is handled gracefully and recorded as null.
This repo includes a transparent, rule-based model selector.
Example
uv run python -m scripts.recommend_model \
--task extract_json \
--strict_schemaOutput
- Recommended model
- Human-readable rationale explaining the decision
The policy is intentionally simple and auditable in v1.
See:
docs/decision_policy.md
Aggregate all task runs into a single summary:
uv run python -m scripts.summarize_resultsOutput
results/summary.json
Each row contains:
- task
- model
- key metrics
Designed for downstream analysis or visualization.
- Prompts are versioned in
prompts/ - Datasets are small, deterministic JSONL files
- Every run writes a timestamped JSON artifact
- No hidden runtime state is maintained within the repository
- Latency measurements are relative comparisons, not production SLAs
- Token usage is a proxy for cost, not billing data
- The rule-based router is a baseline, not a learned policy
- Small datasets are intentional (signal > scale)
Ideas for v2:
- Fully integrated classification workflow with standarized evaluation and benchmarking
- Formal schema validation (JSON Schema / Pydantic)
- Learned model selection policy from results
- RAG task with faithfulness evaluation
- CI extensions with runtime smoke tests (using secrets)
MIT License. See LICENSE.
This repository is designed to answer one question clearly:
Given a task and constraints, which LLM should I choose — and why?