Provenance Guard is a Flask backend that helps creative platforms label text submissions with attribution context. It classifies writing with multiple signals, returns a confidence score, stores structured audit evidence, rate limits submissions, and lets creators appeal a decision.
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtCreate .env in the project root:
GROQ_API_KEY=your_key_here.env is ignored by git. If GROQ_API_KEY is missing, the app still runs with
a clearly flagged local development fallback for the LLM signal. Real demos and
grading should use Groq so the first signal is an actual model-based classifier.
Run the app:
flask --app app:create_app runRun tests:
python -m unittestThe default SQLite database is stored under Flask's instance/ directory.
The current source implements the Milestone 5 production layer plus all four stretch features:
POST /submit- first detection signal: Groq LLM when
GROQ_API_KEYis set, with a local fallback for offline development - second detection signal: stylometric heuristics
- third detection signal: lexical pattern detection
- ensemble confidence scoring using
50% Groq + 30% stylometry + 20% lexical patterns - structured SQLite audit logging
- durable submission status storage
POST /appealandPOST /appealsappeal routes- per-IP submission rate limiting:
10 per minuteand100 per day - provenance certificate request, approval, lookup, and verified-human badge
content_type: "image_metadata"submissions- browser UI at
GET /uifor text, text-file, image-metadata submissions, score display, appeals, and dashboard access GET /dashboardanalytics view with detection pattern, appeal rate, average confidence, content types, recent submissions, and appealsGET /log
The dashboard and certificate routes are intentionally lightweight so they remain easy to demo and inspect for the course rubric.
Milestone 5 curl test:
curl -s -X POST http://localhost:5000/submit \
-H "Content-Type: application/json" \
-d '{"text": "The sun dipped below the horizon, painting the sky in hues of amber and rose. I sat on the porch, coffee in hand, watching the neighborhood slowly go quiet.", "creator_id": "test-user-1"}' | python -m json.toolExpected response fields include content_id, attribution, confidence,
confidence_score, ai_probability, label, content_type, metadata,
ensemble, creator_certificate, verified_human_badge, signal_1,
signal_2, signal_3, and signals.lexical_patterns. The same content_id is
written to the audit log with llm_score, stylometry_score,
lexical_pattern_score, label, status, content_type, metadata, and
appeal_filed.
Appeal test:
curl -s -X POST http://localhost:5000/appeal \
-H "Content-Type: application/json" \
-d '{"content_id": "PASTE-CONTENT-ID-HERE", "creator_reasoning": "I wrote this myself from personal experience. I am a non-native English speaker and my writing style may appear more formal than typical."}' | python -m json.toolPOST /appeals is kept as a compatibility alias. The appeal response returns an
appeal_id, the content_id, status: "under_review", the creator reasoning,
and the original classification summary. The audit log receives a structured
appeal_submitted event with the original signal payload.
Stretch demo flow:
curl -s -X POST http://localhost:5000/certificates/request \
-H "Content-Type: application/json" \
-d '{"creator_id": "creator-123", "verification_method": "portfolio_review", "evidence": {"portfolio_url": "https://example.com/creator-123"}}' | python -m json.tool
curl -s -X POST http://localhost:5000/certificates/PASTE-REQUEST-ID-HERE/approve \
-H "Content-Type: application/json" \
-d '{"reviewer_notes": "Draft history and portfolio reviewed."}' | python -m json.tool
curl -s -X POST http://localhost:5000/submit \
-H "Content-Type: application/json" \
-d '{"creator_id": "creator-123", "content": "This is a certificate-aware submission after manual verification."}' | python -m json.tool
curl -s -X POST http://localhost:5000/submit \
-H "Content-Type: application/json" \
-d '{"creator_id": "artist-88", "content_type": "image_metadata", "content": "A hand-drawn ink illustration of a city street at dusk.", "metadata": {"declared_creation_method": "hand drawn, scanned, color corrected", "alt_text": "Ink drawing of storefronts under evening light"}}' | python -m json.toolOpen http://127.0.0.1:5000/dashboard after generating a few submissions to see
the analytics dashboard.
Browser UI:
http://127.0.0.1:5000/ui
The UI can score pasted text, read text files into the form, submit image captions and metadata, show individual signal scores, submit appeals, and link to the dashboard. It does not upload or inspect raw image pixels.
Audit log demo:
curl -s http://localhost:5000/log | python -m json.toolFour-input scoring check used for Milestone 5 calibration:
| Input type | Groq/local score | Stylometry score | Lexical score | Combined AI probability | Result |
|---|---|---|---|---|---|
| Clearly AI-generated | 0.9263 |
0.7794 |
1.0000 |
0.8970 |
likely_ai |
| Clearly human-written | 0.0000 |
0.2212 |
0.0000 |
0.0664 |
likely_human |
| Borderline formal human writing | 0.4567 |
0.7796 |
0.0000 |
0.4982 |
uncertain |
| Borderline lightly edited AI output | 0.7305 |
0.5174 |
0.4333 |
0.6071 |
uncertain |
Rate-limit evidence from the Flask test client, using 11 rapid requests from the same IP:
201
201
201
201
201
201
201
201
201
201
429
A submitted piece of work enters through POST /submit with creator_id,
optional title, content, optional content_type, and optional metadata.
Flask validates the JSON body, checks required fields, and Flask-Limiter applies
the per-IP submission limits. The default content_type is text; the stretch
multi-modal path also supports image_metadata, where content is the image
description or caption and metadata stores structured context.
The validated content goes to the detection pipeline. The required baseline uses the Groq LLM signal and stylometric signal. The stretch ensemble adds a lexical pattern signal, so each submission can return three signal scores plus the documented ensemble weights. The scoring layer applies the classification thresholds and builds the transparency label. SQLite stores the submission, signal results, certificate context, and audit event before the API returns the structured response to the platform.
If a creator contests the result, POST /appeal or the POST /appeals alias
looks up the original submission, records the creator's reason, updates the
submission status to under_review, and logs the appeal with the original
decision.
The code can be refactored into smaller components for readability. The intended ownership is:
| File | Components |
|---|---|
app.py |
Minimal Flask entrypoint. |
provenance_guard/__init__.py |
App factory, route registration, database initialization, and limiter setup. |
provenance_guard/config.py |
Model names, rate limits, database path, thresholds, and signal weights. |
provenance_guard/db.py |
SQLite connection, schema, and database helpers. |
provenance_guard/labels.py |
Transparency label templates and label builder. |
provenance_guard/scoring.py |
Signal weighting, confidence calculation, and result thresholds. |
provenance_guard/audit.py |
Audit event creation and serialization. |
provenance_guard/certificates.py |
Verification requests, manual approvals, certificate lookup, and verified-human badge text. |
provenance_guard/analytics.py |
Dashboard metrics for detection counts, appeal rate, and average confidence. |
provenance_guard/modalities.py |
Content type validation and normalization for text and image_metadata. |
provenance_guard/validation.py |
Request validation, normalization, and JSON error helpers. |
provenance_guard/routes/submissions.py |
POST /submit and root service metadata endpoint. |
provenance_guard/routes/appeals.py |
POST /appeal, POST /appeals, and appeal status updates. |
provenance_guard/routes/logs.py |
GET /log audit-log endpoint. |
provenance_guard/routes/dashboard.py |
GET /dashboard HTML review page. |
provenance_guard/routes/certificates.py |
Certificate request, approval, and lookup endpoints. |
provenance_guard/signals/groq_signal.py |
Groq LLM classifier. |
provenance_guard/signals/stylometry.py |
Stylometric heuristic scorer. |
provenance_guard/signals/lexical_patterns.py |
Generic wording, stock transition, and template-language detector. |
provenance_guard/signals/fallback_signal.py |
Local development fallback signal. |
provenance_guard/templates/dashboard.html |
Dashboard template. |
tests/test_routes.py |
Endpoint, validation, appeal, audit, dashboard, and rate-limit tests. |
tests/test_scoring.py |
Threshold, confidence, and label tests. |
tests/test_signals.py |
Signal-specific tests. |
| Endpoint | Accepts | Returns |
|---|---|---|
POST /submit |
creator_id, content, optional title, optional content_type, optional metadata |
submission_id, content type, result, confidence, AI probability, label text, signals, ensemble details, optional certificate badge, status |
POST /appeal / POST /appeals |
content_id or submission_id, creator_reasoning or reason |
Appeal record and original decision summary |
GET /log |
Optional limit query parameter |
Structured decision and appeal audit events |
GET /dashboard |
No body | HTML analytics page with detection counts, appeal rate, average confidence, recent submissions, and appeals |
POST /certificates/request |
creator_id, verification_method, optional evidence |
Verification request with pending_review status |
POST /certificates/{certificate_id}/approve |
Reviewer/admin notes | Active certificate record |
GET /certificates/{creator_id} |
No body | Active certificate status or no active certificate |
Accepts a creator ID, content, optional title, optional content_type, and
optional metadata. content_type defaults to text; the stretch multi-modal
path also supports image_metadata. Returns the attribution result, confidence
score, AI probability, transparency label, individual signal outputs, ensemble
weights, optional certificate context, and content status.
curl -X POST http://127.0.0.1:5000/submit \
-H "Content-Type: application/json" \
-d '{
"creator_id": "creator-123",
"title": "Kitchen Window Draft",
"content": "I wrote the first draft beside the kitchen window while rain tapped the glass..."
}'Example response:
{
"submission_id": "7bbf4bb4-cb23-4492-81f7-753c17e7f718",
"creator_id": "creator-123",
"title": "Kitchen Window Draft",
"content_type": "text",
"result": "uncertain",
"confidence_score": 0.61,
"ai_probability": 0.61,
"transparency_label": "Provenance Guard: Authorship is uncertain. The detection signals were mixed or not strong enough, with 61% confidence. This should not be treated as an AI-generated label.",
"ensemble": {
"weights": {
"groq_llm": 0.5,
"stylometry": 0.3,
"lexical_patterns": 0.2
}
},
"signals": {
"groq_llm": {
"ai_probability": 0.58,
"confidence": 0.58,
"details": {
"source": "groq",
"model": "llama-3.3-70b-versatile",
"rationale": "Mixed evidence."
}
},
"stylometry": {
"ai_probability": 0.66,
"confidence": 0.72,
"details": {
"source": "stylometric_heuristics"
}
},
"lexical_patterns": {
"ai_probability": 0.62,
"confidence": 0.59,
"details": {
"source": "lexical_patterns",
"matched_patterns": ["balanced perspective"]
}
}
},
"creator_certificate": null,
"verified_human_badge": null,
"status": "classified"
}Image metadata submission example:
curl -X POST http://127.0.0.1:5000/submit \
-H "Content-Type: application/json" \
-d '{
"creator_id": "artist-88",
"title": "Gallery Upload Caption",
"content_type": "image_metadata",
"content": "A hand-drawn ink illustration of a city street at dusk.",
"metadata": {
"declared_creation_method": "hand drawn, scanned, color corrected",
"alt_text": "Ink drawing of storefronts under evening light"
}
}'Captures the creator's reasoning, logs it alongside the original decision, and
updates the submission status to under_review. POST /appeals is an alias for
the same handler.
curl -X POST http://127.0.0.1:5000/appeal \
-H "Content-Type: application/json" \
-d '{
"content_id": "7bbf4bb4-cb23-4492-81f7-753c17e7f718",
"creator_reasoning": "This is my original draft with edits from my notebook."
}'The route also accepts submission_id as an alias for content_id and reason
as an alias for creator_reasoning.
Returns structured audit events.
curl http://127.0.0.1:5000/logShows an HTML analytics page with detection counts by result, appeal rate, average confidence, recent submissions, certificate badges, statuses, and appeal counts.
Creators request manual verification through POST /certificates/request.
Reviewers approve a request through POST /certificates/{certificate_id}/approve.
Platforms can check a creator's active credential with
GET /certificates/{creator_id}.
For demos, the approval route can use the request ID in the URL:
POST /certificates/{request_id}/approve. In that case the app generates the
certificate ID. If you want a specific certificate ID, put that ID in the URL and
send the request_id in the JSON body.
Certificate request example:
curl -X POST http://127.0.0.1:5000/certificates/request \
-H "Content-Type: application/json" \
-d '{
"creator_id": "creator-123",
"verification_method": "portfolio_review",
"evidence": {
"portfolio_url": "https://example.com/creator-123",
"process_note": "Draft history and publication records available."
}
}'Certificate-aware submission response excerpt:
{
"submission_id": "7bbf4bb4-cb23-4492-81f7-753c17e7f718",
"creator_id": "creator-123",
"content_type": "text",
"result": "likely_human",
"confidence_score": 0.88,
"creator_certificate": {
"certificate_id": "cert-123",
"status": "active",
"verification_method": "portfolio_review"
},
"verified_human_badge": "Verified Human: This creator completed an additional Provenance Guard verification step. Certificate ID: cert-123."
}Provenance Guard's required baseline uses two distinct signals. The ensemble stretch feature adds a third signal and uses documented weights:
- Groq LLM classifier:
llama-3.3-70b-versatilereads the submitted text and returns an AI probability, confidence, rationale, and evidence. This captures holistic semantic and stylistic cues that are hard to express as hand-written rules. - Stylometric heuristics: pure Python statistics measure sentence length uniformity, type-token ratio, repetition, and punctuation variety. This captures structural properties of the text without relying on the LLM.
- Lexical pattern detector: phrase and pattern matching catches generic AI-like wording, stock transitions, and repeated template language.
The stretch final AI probability is weighted as
50% Groq + 30% stylometry + 20% lexical patterns. The weights sum to 1.0.
The LLM gets the largest weight because it can assess context and tone,
stylometry keeps the pipeline from depending on one signal, and lexical patterns
add a narrower check for reusable generated phrasing.
Signal blind spots:
| Signal | What it misses |
|---|---|
| Groq LLM classifier | It can be overconfident, influenced by prompt wording, unavailable if Groq is down or unconfigured, and wrong about unusual human styles. It cannot prove authorship. |
| Stylometric heuristics | It cannot understand meaning, intent, or revision history. Short text produces weak statistics, polished humans can look uniform, and AI can imitate messy human rhythm. |
| Lexical pattern detector | It can penalize legitimate formal writing, miss newer model phrasing, and cannot detect AI text that has been heavily edited or intentionally varied. |
The ensemble stretch feature incorporates three detection signals with a documented weighting approach. The implemented ensemble uses:
| Signal | Weight |
|---|---|
| Groq LLM classifier | 0.50 |
| Stylometric heuristics | 0.30 |
| Lexical pattern detector | 0.20 |
The POST /submit response includes an ensemble object with these weights and
the signal names used for the final score.
A creator can earn a verified-human credential through a manual verification request and reviewer approval. The certificate appears separately from the classification label so readers see both the work-level attribution result and the creator-level verification context.
Certificate display text:
"Verified Human: This creator completed an additional Provenance Guard verification step. Certificate ID: {certificate_id}."
The dashboard stretch feature implements GET /dashboard with detection
patterns by result, appeal rate, and average confidence. It also keeps recent
submissions, statuses, appeal counts, content-type counts, and certificate
badges visible for review.
The multi-modal stretch feature supports content_type: "image_metadata" in
addition to text through the same POST /submit route. The system does not
upload binary image files. Instead,
content contains the image description or caption, and metadata can include
structured context such as declared creation method, upload source, tool name, or
alt text. Audit logs preserve the content type and metadata.
Stretch data notes:
submissions.content_typedefaults totextand can also beimage_metadata.submissions.metadata_jsonstores optional structured metadata for multi-modal submissions.certificatesstores approved verified-human credentials.verification_requestsstores pending and reviewed certificate requests.- Certificate badges are separate from the three required transparency labels.
The system stores ai_probability as the combined probability that the text is
AI-generated. The displayed confidence_score is:
max(ai_probability, 1 - ai_probability)
Classification thresholds:
| AI probability | Result |
|---|---|
>= 0.85 |
likely_ai |
<= 0.25 |
likely_human |
| otherwise | uncertain |
The AI threshold is intentionally higher than the human threshold because a
false positive, labeling a creator's human work as AI-generated, is more harmful
than leaving a suspicious submission uncertain. A score near 0.51 produces an
uncertain label, while a score near 0.95 produces a high-confidence AI label.
Tests check those threshold differences directly.
False positive walkthrough: if a human writer submits a polished poem and the
signals mistakenly score it as AI-like, mixed evidence should fall into the
uncertain range instead of likely_ai. The uncertain label says the work
should not be treated as AI-generated. If the score still crosses the high AI
threshold, the label explicitly tells the creator they may appeal. The appeal
records the creator's reasoning, changes the submission status to
under_review, and keeps the original decision and appeal together in the audit
log for human review.
Short poems, haiku, and lyrical fragments with intentional repetition may be misread as AI-like. The stylometric signal treats repeated structure and low vocabulary diversity as possible generated-text evidence, even though those features are common in poetry and songwriting.
Polished academic, grant-style, or policy prose may score higher than intended. Those genres often use formal transitions such as "overall" or "it is important to note," which can trigger the lexical pattern detector even when a human wrote the passage.
Very short captions, titles, or two-sentence posts provide too little evidence for reliable scoring. Groq or the local fallback has limited context, stylometry cannot compute stable sentence and vocabulary patterns, and lexical matching may miss the text entirely or overreact to one phrase.
The exact label text is:
| Variant | Result | Exact text |
|---|---|---|
| High-confidence AI | likely_ai |
"Provenance Guard: This work is likely AI-generated. The detection score is {confidence_percent}% confidence. The creator may appeal this label." |
| High-confidence human | likely_human |
"Provenance Guard: This work is likely human-created. The detection score is {confidence_percent}% confidence." |
| Uncertain | uncertain |
"Provenance Guard: Authorship is uncertain. The detection signals were mixed or not strong enough, with {confidence_percent}% confidence. This should not be treated as an AI-generated label." |
Creators can contest a decision with POST /appeal or the POST /appeals
alias. An appeal requires:
content_idorsubmission_idcreator_reasoningorreason
The API looks up the original submission, inserts an appeal record, updates the
submission status to under_review, and writes an appeal_submitted audit event
that includes both the appeal reason and the original classification decision.
The system does not automatically reclassify appealed work.
POST /submit is limited per IP address:
| Limit | Reason |
|---|---|
10 per minute |
Allows normal creative workflows while blocking rapid spam or scripted probing. |
100 per day |
Gives active creators room to submit drafts, but caps abuse from one IP. |
Appeals and log reads are not rate limited in this version because the project
requirement specifically targets the submission endpoint. When the minute limit
is exceeded, Flask-Limiter returns HTTP 429.
Every decision and appeal is stored in SQLite as a structured audit event. Each decision event includes timestamp, content ID, attribution result, confidence score, AI probability, individual signal scores, ensemble weights, status, and label text. Each appeal event includes the creator's reason and the original decision. Stretch audit events will also record image metadata submissions, certificate requests, and certificate approvals.
Sample GET /log output with at least three entries:
{
"entries": [
{
"event_type": "appeal_submitted",
"appeal_id": "appeal-001",
"content_id": "sub-002",
"submission_id": "sub-002",
"creator_id": "creator-77",
"timestamp": "2026-06-28T16:24:30.123+00:00",
"attribution": "likely_ai",
"confidence": 0.897,
"ai_probability": 0.897,
"llm_score": 0.9263,
"stylometry_score": 0.7794,
"lexical_pattern_score": 1.0,
"label": "Provenance Guard: This work is likely AI-generated. The detection score is 90% confidence. The creator may appeal this label.",
"status": "under_review",
"appeal_filed": true,
"appeal_reasoning": "This poem was drafted by hand and revised from my journal.",
"original_classification": {
"event_type": "submission_classified",
"attribution": "likely_ai",
"confidence": 0.897,
"status": "classified"
}
},
{
"event_type": "submission_classified",
"content_id": "sub-002",
"submission_id": "sub-002",
"creator_id": "creator-77",
"timestamp": "2026-06-28T16:22:10.456+00:00",
"attribution": "likely_ai",
"confidence": 0.897,
"ai_probability": 0.897,
"llm_score": 0.9263,
"stylometry_score": 0.7794,
"lexical_pattern_score": 1.0,
"label": "Provenance Guard: This work is likely AI-generated. The detection score is 90% confidence. The creator may appeal this label.",
"status": "classified",
"appeal_filed": false,
"signals": {
"groq_llm": {"ai_probability": 0.9263},
"stylometry": {"ai_probability": 0.7794},
"lexical_patterns": {"ai_probability": 1.0}
}
},
{
"event_type": "submission_classified",
"content_id": "sub-001",
"submission_id": "sub-001",
"creator_id": "creator-14",
"timestamp": "2026-06-28T16:19:05.789+00:00",
"attribution": "likely_human",
"confidence": 0.9336,
"ai_probability": 0.0664,
"llm_score": 0.0,
"stylometry_score": 0.2212,
"lexical_pattern_score": 0.0,
"label": "Provenance Guard: This work is likely human-created. The detection score is 93% confidence.",
"status": "classified",
"appeal_filed": false
}
]
}The implementation diverged from the original required-feature plan in one important way: the required baseline was planned as two signals, Groq/local fallback plus stylometry, but the current source now runs a three-signal ensemble. The lexical pattern stretch signal was added before the other stretch features because it fit cleanly into the existing scoring pipeline and helped show individual signal scores alongside the combined result.
The implementation also changed the timing of the stretch work. Certificates, the analytics dashboard, and multi-modal support were documented before they were wired, then implemented after the Milestone 5 production layer was stable. The final stretch implementations stay lightweight: certificate verification is a manual approval flow, the dashboard is a SQLite-backed review page, and multi-modal support processes image descriptions and metadata rather than binary uploads.
- Refactor planning: I directed AI assistance to split the early single-file
Flask app into a package structure with an app factory, route modules,
signal modules, scoring, labels, audit, and database helpers. I revised the
output to keep
app.pyas a minimal entrypoint and delayed the stretch-only routes until the required production layer was working. - Detection and scoring implementation: I directed AI assistance to use
planning.mdto generate the stylometric signal, lexical pattern signal, and combined scoring logic. I checked and revised the generated behavior so the thresholds stayed at>= 0.85forlikely_ai,<= 0.25forlikely_human, and the ensemble weights stayed at50% Groq,30% stylometry, and20% lexical patterns. - Production layer: I directed AI assistance to add the final label behavior,
appeals workflow, rate limiting, and audit evidence. I revised the endpoint
contract to make
POST /appealthe primary route, keptPOST /appealsas a compatibility alias, and kept the production logic out ofapp.pyso the component ownership matched the planning document.
- Ensemble responses include at least three signals and documented weights.
- Ensemble weights sum to
1.0. - Text submissions remain backward compatible when
content_typeis omitted. - Image metadata submissions are accepted and audited with
content_type: "image_metadata". - Approved certificates appear in submission responses and dashboard rows.
- The dashboard displays detection counts, appeal rate, and average confidence.
- Appeals and certificate events are included in the audit log.
- All four stretch features are implemented.
- Multi-modal support means image descriptions or structured metadata, not binary image upload.
- Certificate verification uses manual reviewer approval.
- The verified-human badge is separate from the three required transparency label variants.