Catch LLM hallucinations before your users do. Verify any LLM output against ground truth using Claude Opus 4.7's deep reasoning. Math is computed. Code is executed. Citations are checked. Facts are reasoned through.
Every LLM hallucinates. Production apps ship wrong answers daily:
- A RAG bot invents quotes that don't exist in the source documents
- A coding assistant generates code that doesn't compile or silently misbehaves
- A reasoning chain says "since
200 + 50 = 251, therefore..." — and the user trusts it - An agent fabricates an API endpoint, a function signature, a date, a number
Schema validators (Pydantic, Guardrails) catch structural errors. None catch semantic errors. That is what oracle is for.
oracle reads the LLM's output, extracts every verifiable claim, and independently checks each one:
| Claim type | How it's verified |
|---|---|
🔢 Math (2+2=4, sqrt(144)=12) |
Computed via safe AST evaluator — no eval() |
| 💻 Code (Python blocks) | Executed in sandboxed subprocess with timeout + memory limits |
📚 Citations ("quote from doc") |
Exact + fuzzy-matched against provided context |
| 🧠 Facts & reasoning | Claude Opus 4.7 with extended thinking — multi-step verification |
Returns a VerifiedResponse with confidence score and a list of every claim with its verdict.
from oracle import Oracle
oracle = Oracle(client=None) # no API key needed for math/code/citation
text = "First, 17 * 23 = 391. Then 100 / 4 = 25. Final: 200 + 50 = 251."
result = oracle.verify(text, mode="math")
print(result.summary)
# ⚠ 1 hallucination(s) detected (math). Total claims: 3 (3 verifiable).
for r in result.incorrect_claims:
print(f" ✗ {r.claim.text} → correction: {r.correction}")
# ✗ 200 + 50 = 251 → correction: 200 + 50 = 250The fact and logic verifiers benefit most from Opus 4.7's reasoning depth:
- Extended thinking is enabled by default (8K thinking tokens) — the verifier reasons step-by-step about counter-examples and edge cases before issuing a verdict
- On multi-fact and reasoning-chain claims, Opus 4.7 catches errors smaller models miss, because verification itself requires the same reasoning capability that produced the (possibly wrong) original answer
- Calibrated confidence scores — Opus 4.7 reliably says "uncertain" instead of confidently guessing
Math, code, and citation verifiers are pure Python — no API key required for those modes.
pip install oracle-verifyfrom oracle import Oracle
oracle = Oracle()
result = oracle.verify("Multiplying: 47 * 53 = 2491. Adding: 100 + 200 = 350.", mode="math")
print(result.summary)
# ⚠ 1 hallucination(s) detected. Total claims: 2 (2 verifiable).result = oracle.verify("""
```python
for i in range(1, 16):
if i % 15 == 0: print("FizzBuzz")
elif i % 3 == 0: print("Fizz")
elif i % 5 == 0: print("Buzz")
else: print(i)""", mode="code") print(result.is_trustworthy) # True
### 3. Catch fabricated citations in RAG
```python
source = "Python was created by Guido van Rossum in 1991."
llm_output = '''
The doc says "Python was designed to replace Java in 2003." (fabricated!)
And "Python was created by Guido van Rossum in 1991." (real)
'''
result = oracle.verify(llm_output, context=source, mode="rag")
print(f"Hallucinations: {len(result.incorrect_claims)}") # 1
from anthropic import Anthropic
from oracle import Oracle, verified_call
client = Anthropic()
oracle = Oracle(client=client)
text, report = verified_call(
client=client,
verifier=oracle,
messages=[{"role": "user", "content": "What is 247 * 389?"}],
retry_on_hallucination=True, # re-prompt model with verifier's feedback
max_retries=2,
)
# text is now correct — model was given the verifier's feedback and revised# Pipe any LLM output
echo "Linux was first released by Linus Torvalds in 1989." | oracle verify
# Strict mode — exit non-zero if hallucinations found (use in CI)
oracle verify --file response.txt --strict
# RAG mode with context docs
oracle verify --file answer.txt --context source.md --context paper.txt
# No API key — math + code + citations only
oracle verify --file response.txt --no-llmWhere to plug oracle in |
What it catches |
|---|---|
| Before sending to user | Wrong math, broken code, hallucinated citations |
| In CI for prompt regressions | Use --strict mode to fail builds on hallucinations |
| Agent step validation | Verify each step before executing the next |
| RAG quality monitoring | Track hallucination rate across queries |
| Eval pipelines | Score model outputs at scale |
oracle/
├── verifier.py # Orchestrator — extracts claims, routes to verifier, aggregates
├── extractors.py # Pulls verifiable claims from text (regex + Opus 4.7)
├── types.py # Pydantic models (Claim, Verdict, VerifiedResponse)
├── wrap.py # verified_call() — drop-in self-correcting wrapper
├── cli.py # `oracle verify` command
└── verifiers/
├── math.py # Safe AST evaluator — computes the expression
├── code.py # Subprocess sandbox — executes Python with timeout + memory limits
├── citation.py # Exact + fuzzy (SequenceMatcher) match against context
└── fact.py # Opus 4.7 + extended thinking for fact/logic claims
| oracle | Guardrails AI | Pydantic | LLM-as-judge | |
|---|---|---|---|---|
| Structural validation | — | ✓ | ✓ | — |
| Math correctness | ✓ | — | — | ~ |
| Code execution check | ✓ | — | — | — |
| Citation grounding | ✓ | ~ | — | ✓ |
| Fact reasoning (Opus 4.7) | ✓ | — | — | ~ |
| Self-correcting loop | ✓ | — | — | ~ |
| Works offline (no API) | ✓¹ | ✓ | ✓ | — |
¹ Math, code, and citation modes only.
MIT © bhupendra05
If oracle saves you from shipping a hallucination, star ⭐ the repo.