Tool-using ReAct + RAG agent (enterprise assistant) with a built-in evaluation harness scoring accuracy, tool selection, safety/refusal, latency and token cost. Live demo: https://reagent.symbai.dev
Python LLM eval framework with full OTel tracing, structured logs, and daily Welch's-t-test regression detection persisted to Postgres
TjoeReviewKit：tjoe 的本地离线工作流复盘检查工具；不运行任务、不联网、不接管工具调用、不采集生产日志
LLM self-consistency and sycophancy diagnostic — catches contradictions and false-premise acceptance hiding behind confident answers
Multi-agent customer-interview synthesis pipeline in n8n with LLM-as-judge eval, Slack human-in-the-loop approval, and drift monitoring.
A lightweight harness to test agent behaviour (tool-call correctness, injection refusal, cost ceilings) before deployment
Free Apache-2.0 skills for Claude Code, plus trigger-probe — the open-source tool we used to measure all 18 of our paid skills: 144/180 fires when it should, 180/180 stays quiet when it should, 8 failures published.
AI Observability & Evaluation
Does done mean done? Coding-agent honesty demo: verified pass vs false completion vs scope violation. Public demo split + selftest.
An AI agent that answers questions about U.S. campaign money against real FEC data. Source-cited, graded on a public eval scoreboard.
Automated groundedness scoring, hallucination detection, and regression tracking for RAG and agent outputs
SFX Lead Intelligence Command Center: local-LLM hub plus lead dashboard, quality lifted 61 to 99 percent via a ground-truth eval harness. Showcase; source private.
The prompt engineering, prompt management, and prompt evaluation tool for Kotlin.
RAG evaluation toolkit — faithfulness, answer relevance, context precision/recall, cost accounting, CI gates. Pairs with hybrid-rag-qdrant and agent-eval-harness.
Developer-first LLM/RAG safety audits for CI/CD — faithfulness, relevancy, prompt injection (BYOK OpenAI/xAI). pip install auditai-cli
Marketing site for book-mash and mash-core
中文 LLM 简历改写诚实度 benchmark：20 脱敏简历 × 3 模型 × 4 维度 · promptfoo + LLM-as-judge · 含在线报告
RL-style eval measuring intent/action divergence in frontier agents: model acknowledges a correction, then acts on the stale value anyway. 3 scenarios, 655 trials on claude-haiku-4-5, Sonnet 4.6, GPT-5.4, and Gemini 3.1 Pro Preview.
Persona-driven dynamic testing for conversational AI products. Focus groups for your agents.
Add a description, image, and links to the llm-eval topic page so that developers can more easily learn about it.
To associate your repository with the llm-eval topic, visit your repo's landing page and select "manage topics."