🏢 Enterprise Document Intelligence System

Production-ready multi-agent AI system for automated document compliance, risk assessment, and PII detection using CrewAI

📖 Overview

An enterprise-grade document intelligence system powered by 6 specialized AI agents working collaboratively to:

🔍 Detect sensitive information (PII/PHI) across 6 pattern types
⚖️ Validate compliance with GDPR, HIPAA, and SOC2 regulations
📊 Calculate risk scores (0-100) with weighted algorithms
📝 Generate audit-ready reports with actionable recommendations
👤 Implement human-in-the-loop governance for critical cases

Perfect for: Healthcare, Finance, Legal, HR, and any industry handling sensitive data.

✨ Key Features

🤖 Multi-Agent Architecture

6 Specialized Agents working in sequential workflow
YAML-based configuration for easy customization
CrewAI orchestration with modern best practices

🛡️ Security & Compliance

PII/PHI Detection: SSN, MRN, DOB, Email, Phone, Credit Cards
Multi-Framework Support: GDPR, HIPAA, SOC2
Risk-Based Processing: Automatic routing based on sensitivity
Human Oversight: Required for high-risk documents (score ≥ 75)

🔧 Production-Ready

Custom Tools with Pydantic validation
Natural Language Parsing for flexible agent inputs
Comprehensive Logging and error handling
Sample Documents for testing all risk levels

🏗️ System Architecture

┌─────────────┐
│   Document  │
└──────┬──────┘
       │
       ▼
┌─────────────────────────────────────────────┐
│         6-Agent Sequential Pipeline         │
├─────────────────────────────────────────────┤
│ 1. Intake Specialist    → Extract & Validate│
│ 2. Classifier           → Type & Sensitivity│
│ 3. PII Scanner          → Detect Sensitive  │
│ 4. Compliance Checker   → Regulatory Review │
│ 5. Risk Assessor        → Calculate Score   │
│ 6. Report Generator     → Audit Report      │
└──────┬──────────────────────────────────────┘
       │
       ▼
┌─────────────┐
│   Report    │
│ Risk: 85/100│
│ CRITICAL    │
└─────────────┘

🚀 Quick Start

Prerequisites

Python 3.8 or higher
OpenAI API key
pip or conda for package management

Installation

Clone the repository

git clone https://github.com/YOUR_USERNAME/enterprise-doc-intelligence.git
cd enterprise-doc-intelligence

Create virtual environment (recommended)

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Install dependencies
```
pip install -r requirements.txt
```

Set up environment variables

cp .env.example .env
# Edit .env and add your OpenAI API key:
# OPENAI_API_KEY=your-api-key-here

Running Your First Analysis

cd src
python main.py ../sample_documents/medical_record_sample.txt

Expected output:

6 agents execute in sequence
PII detection: 5 types found (SSN, MRN, DOB, Email, Phone)
Risk score: 85/100 (CRITICAL)
Recommendation: Human review required
Report saved to outputs/compliance_report.md

📁 Project Structure

enterprise-doc-intelligence/
├── src/
│   ├── config/
│   │   ├── agents.yaml          # Agent definitions (roles, goals, backstory)
│   │   └── tasks.yaml           # Task definitions (descriptions, outputs)
│   ├── tools/
│   │   ├── pii_detector.py      # 6-pattern PII/PHI detection
│   │   ├── compliance_checker.py# GDPR/HIPAA/SOC2 validation
│   │   ├── risk_calculator.py   # Weighted risk scoring
│   │   ├── pdf_extractor.py     # PDF text extraction
│   │   └── text_extractor.py    # TXT file reader
│   ├── crew.py                  # CrewAI orchestration
│   └── main.py                  # Entry point
├── sample_documents/            # Test files (3 risk levels)
├── docs/
│   ├── COMPLETE_GUIDE.md        # Step-by-step tutorial
│   └── YAML_QUICKSTART.md       # YAML configuration guide
├── requirements.txt
├── .env.example
└── README.md

🤖 The 6 Agents

Agent	Role	Tools	Output
Intake Specialist	Document Validator	Text/PDF Extractor	Clean text + metadata
Classifier	Type Analyzer	LLM Reasoning	Document type + sensitivity
PII Scanner	Data Detector	PII Detector	Sensitive data list + score
Compliance Checker	Regulatory Auditor	Compliance Engine	Violations + remediation
Risk Assessor	Risk Calculator	Risk Scorer	0-100 score + level
Report Generator	Documentation Writer	LLM Synthesis	Audit-ready report

All agents are configured in src/config/agents.yaml using CrewAI's modern declarative YAML approach.

🔧 Custom Tools

1. PII Detector

Patterns: SSN, Email, Phone, MRN, Credit Cards, DOB
Risk Weights: 3-10 points per type
Output: Detection list + total risk score

2. Compliance Checker

Frameworks: GDPR, HIPAA, SOC2
Rules: 15+ compliance checks per framework
Output: Violations + recommended controls

3. Risk Calculator

Inputs: PII score + Classification + Violations
Algorithm: Weighted sum (max 100)
Output: Risk level (LOW/MEDIUM/HIGH/CRITICAL)

4. PDF/Text Extractors

Supported: PDF, TXT files
Output: Text content + metadata (size, line count)

📊 Sample Documents

Three sample documents demonstrate different risk levels:

Document	Risk Level	PII Types	Use Case
`medical_record_sample.txt`	CRITICAL (85)	SSN, MRN, DOB, Email, Phone	Healthcare PHI
`invoice_sample.txt`	MEDIUM (45)	Email, Phone	Business documents
`public_announcement_sample.txt`	LOW (10)	None	Public communications

🎯 Usage Examples

Basic Usage

cd src
python main.py /path/to/document.txt

Process Medical Records

python main.py ../sample_documents/medical_record_sample.txt

Batch Processing

for file in ../sample_documents/*.txt; do
    python main.py "$file"
done

View Output

cat outputs/compliance_report.md

📈 Sample Output

# Compliance Report

## Executive Summary
- Document Type: Medical Record
- Classification: RESTRICTED
- Risk Score: 85/100 (CRITICAL)

## PII/PHI Findings
- SSN: 123-45-6789
- MRN: MRN-987654
- DOB: 03/15/1985
- Email: patient@email.com
- Phone: 555-123-4567

## Risk Assessment
⚠️ CRITICAL RISK - Requires human review

## Recommendations
1. [HUMAN REVIEW] Require approval before processing
2. [SECURITY] Apply enhanced PII protection
3. [ACTION] Implement RBAC and encryption

🔐 Security & Privacy

✅ No data storage: Documents processed in-memory only
✅ API key protection: Environment variables for credentials
✅ Audit logging: Complete activity trail
✅ Configurable: Adjust risk thresholds and rules
✅ Extensible: Add custom compliance frameworks

🛠️ Configuration

Customize Agents

Edit src/config/agents.yaml:

pii_detection_agent:
  role: "Sensitive Data 
8000
Scanner"
  goal: "Detect all PII and PHI in documents"
  backstory: "Expert in data privacy with 15 years experience..."

Customize Tasks

Edit src/config/tasks.yaml:

pii_detection_task:
  description: "Scan the document for PII/PHI"
  expected_output: "Complete PII detection results with risk scores"

Adjust Risk Weights

Edit src/tools/risk_calculator.py:

CLASSIFICATION_WEIGHT = {
    'public': 0,
    'internal': 10,
    'confidential': 25,
    'restricted': 40
}

📚 Documentation

Complete Guide - Comprehensive tutorial with diagrams
YAML Quickstart - Agent/task configuration guide
Test Results - System validation report
Verification - Tool testing results

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

How to Contribute

Fork the repository
Create a feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add some AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

Ideas for Contributions

🔹 Additional compliance frameworks (PCI-DSS, ISO 27001)
🔹 More PII patterns (passport numbers, tax IDs)
🔹 Support for more file formats (DOCX, XLSX)
🔹 Web UI for document upload
🔹 Database integration for report storage
🔹 Multi-language support
🔹 Performance optimizations
🔹 Unit tests and integration tests

🐛 Known Issues & Limitations

PDF extraction requires PyPDF2 (included in requirements)
Risk calculator currently supports text-based analysis only
LLM calls can be slow for large documents (>10 pages)
Requires OpenAI API key (costs apply)

🗺️ Roadmap

Add support for DOCX and XLSX files
Implement caching for repeated analyses
Add FastAPI REST API endpoint
Create web-based UI
Support for custom PII patterns via config
Integration with document management systems
Performance benchmarking suite
Docker containerization

💡 Use Cases

Healthcare

Patient record processing
PHI compliance validation
HIPAA audit preparation

Finance

Customer data governance
PCI-DSS compliance
Fraud detection support

Legal

Contract analysis
Discovery document review
Client data protection

Human Resources

Employee file processing
Background check compliance
GDPR data mapping

🙏 Acknowledgments

CrewAI - For the excellent multi-agent framework
OpenAI - For GPT-4 language models
Community - For feedback and contributions

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📞 Contact & Support

Issues: GitHub Issues
Discussions: GitHub Discussions
LinkedIn: Girijesh Prasad
GitHub: girijesh-ai

⭐ Star History

If you find this project useful, please consider giving it a star! It helps others discover the project.

Built with ❤️ using CrewAI, OpenAI, and Python

📊 Project Stats

Agents: 6 specialized agents
Tools: 5 custom tools
Frameworks: 3 compliance frameworks
PII Types: 6 detection patterns
Code: ~3,000 lines
Documentation: Comprehensive guides included

Name		Name	Last commit message	Last commit date
docs		docs
sample_documents		sample_documents
src		src
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
TEST_RESULTS.md		TEST_RESULTS.md
VERIFICATION_RESULTS.md		VERIFICATION_RESULTS.md
requirements.txt		requirements.txt
test_imports.py		test_imports.py

Latest commit

History

Folders and files

Repository files navigation