PyRead: Production-Ready Document to Plain Text Converter

A fast, accurate, and maintainable Python CLI utility that converts documents (PDF, EPUB, TXT, HTML) to standardized, clean plain text.

Features

Lightning-fast PDF extraction using pypdfium2 (fastest available)
Intelligent fallback system (pypdfium2 → PyMuPDF → OCR)
Clean UTF-8 output with NFKC normalization and intelligent encoding handling
Multiple format support: PDF, EPUB, TXT, HTML
Optional OCR for scanned PDFs (via pytesseract)
Parallel batch processing with progress bars
Robust error handling - never crashes on corrupt files
Beautiful CLI with rich formatting and progress reporting
Preserves directory structure in batch mode
Automatic versioning - creates versioned files (file_v2.txt) instead of overwriting

What's New

Automatic File Versioning: PyRead now creates versioned output files (document_v2.txt, document_v3.txt) when the output already exists, instead of skipping or overwriting. This makes it easier to compare different extraction settings or methods. Use --overwrite to replace files instead. See VERSIONING.md for details.

Quick Start

Installation

# Clone or download pyread.py
pip install -r requirements.txt

Basic Usage

# Convert a single file
python pyread.py document.pdf

# Process a directory with output folder
python pyread.py docs/ --output-dir output/

# Recursive processing with OCR for scanned PDFs
python pyread.py scanned_docs/ -r --ocr

# Parallel processing with 8 workers
python pyread.py large_batch/ -w 8 --verbose

📋 Requirements

Core Dependencies:

pypdfium2>=4.0.0        # Fast PDF extraction (primary)
pymupdf>=1.23.0         # PDF/EPUB fallback
python-magic>=0.4.27    # Format detection
ftfy>=6.1.0             # Encoding repair
chardet>=5.0.0          # Encoding detection
typer>=0.9.0            # CLI framework
rich>=13.0.0            # Beautiful terminal output
beautifulsoup4>=4.12.0  # HTML parsing
lxml>=4.9.0             # HTML parser backend
ebooklib>=0.18          # EPUB support

Optional (for OCR):

pytesseract>=0.3.10     # OCR support
Pillow>=10.0.0          # Image processing

Note: For OCR to work, you also need to install Tesseract OCR engine.

📖 Detailed Usage

Command-Line Options

Usage: pyread.py [OPTIONS] INPUT_PATH

Arguments:
  INPUT_PATH              Input file or directory to process [required]

Options:
  -o, --output-dir PATH   Output directory (default: same as input)
  -r, --recursive         Process directories recursively
  -w, --workers INTEGER   Number of parallel workers (1-32) [default: 4]
  --ocr                   Use OCR for scanned PDFs (requires tesseract)
  --overwrite             Overwrite existing output files (instead of versioning)
  --force-ascii           Remove non-ASCII characters from output
  -v, --verbose           Enable verbose logging
  --log-file PATH         Write logs to file
  --help                  Show this message and exit

Default Behavior: If an output file already exists, PyRead automatically creates a versioned filename (e.g., document_v2.txt, document_v3.txt) instead of overwriting. Use --overwrite to replace existing files.

Examples

Single File Processing

# Basic conversion (creates document.txt)
python pyread.py document.pdf
# Output: document.txt (same directory)

# Run again (creates versioned file automatically)
python pyread.py document.pdf
# Output: document_v2.txt (doesn't overwrite original)

# With custom output directory
python pyread.py document.pdf --output-dir ./output/
# Output: ./output/document.txt

# Overwrite existing files instead of versioning
python pyread.py document.pdf --overwrite
# Output: document.txt (replaces existing file)

# Force ASCII-only output
python pyread.py unicode_doc.pdf --force-ascii

Batch Processing

# Process all supported files in a directory
python pyread.py ./documents/ --output-dir ./text_output/

# Recursive processing (includes subdirectories)
python pyread.py ./documents/ -r --output-dir ./text_output/
# Preserves directory structure in output

# With parallel workers
python pyread.py ./documents/ -r -w 8
# Uses 8 parallel workers for faster processing

OCR for Scanned PDFs

# Enable OCR for scanned documents
python pyread.py scanned.pdf --ocr

# Batch OCR processing
python pyread.py ./scanned_docs/ -r --ocr -w 4

Advanced Usage

# Overwrite existing files + verbose logging
python pyread.py ./docs/ -r --overwrite --verbose

# Save logs to file
python pyread.py ./docs/ -r --log-file processing.log

# Maximum performance (8 workers, no OCR)
python pyread.py ./large_dataset/ -r -w 8 --overwrite

🎯 How It Works

Text Processing Pipeline

Format Detection
- Uses python-magic for MIME type detection
- Falls back to file extension + content sniffing
- Supports: PDF, EPUB, TXT, HTML
Text Extraction
- PDF: pypdfium2 (fastest) → PyMuPDF (better layout) → OCR (if enabled)
- EPUB: PyMuPDF → ebooklib fallback
- HTML: BeautifulSoup4 with lxml parser
- TXT: Encoding detection (UTF-8 → chardet → Latin-1)
Text Cleaning
- Fix mojibake using ftfy
- NFKC Unicode normalization
- Remove BOM, zero-width spaces, control characters
- Collapse multiple spaces/newlines (preserve paragraph structure)
- Strip leading/trailing whitespace
- Optional ASCII-only mode
Output
- UTF-8 encoded .txt files
- Preserves directory structure (batch mode)
- Skips existing files by default (configurable)

PDF Extraction Strategy

┌─────────────┐
│  Input PDF  │
└──────┬──────┘
       │
       ▼
┌──────────────────┐
│ Try pypdfium2    │  ◄── Fastest, lowest memory
│ (page-by-page)   │
└────────┬─────────┘
         │
    Good text? ─── Yes ──► Clean & Output
         │
         No
         ▼
┌──────────────────┐
│ Try PyMuPDF      │  ◄── Better layout fidelity
│ (with layout)    │
└────────┬─────────┘
         │
    Good text? ─── Yes ──► Clean & Output
         │
         No
         ▼
┌──────────────────┐
│ Try OCR          │  ◄── For scanned PDFs
│ (if --ocr flag)  │      (slowest, requires tesseract)
└────────┬─────────┘
         │
         ▼
   Clean & Output

🔧 Configuration

Supported Formats

Format	Extensions	Primary Library	Fallback
PDF	`.pdf`	pypdfium2	PyMuPDF, OCR
EPUB	`.epub`	PyMuPDF	ebooklib
HTML	`.html`, `.htm`	BeautifulSoup4	-
Text	`.txt`, `.text`	stdlib	chardet

Performance Characteristics

100-page PDF: ~3-5 seconds (pypdfium2)
Memory usage: Low (page-by-page processing)
Parallel scaling: Near-linear up to CPU count
OCR: ~2-3 seconds per page (depends on resolution)

🐛 Troubleshooting

Common Issues

1. "No PDF extraction library available"

pip install pypdfium2
# OR
pip install pymupdf

2. "tesseract not found" (when using --ocr)

Install Tesseract OCR engine:
- Windows: https://github.com/UB-Mannheim/tesseract/wiki
- macOS: brew install tesseract
- Linux: sudo apt-get install tesseract-ocr

3. "python-magic not working" (Windows)

pip install python-magic-bin

4. "Low quality extraction from PDF"

Try enabling OCR: --ocr
For scanned PDFs, OCR is usually necessary

5. "Files are being skipped" or "Too many versions created"

Default behavior: PyRead creates versioned files (file_v2.txt, file_v3.txt) when output exists
Use --overwrite to replace existing files instead of versioning
See VERSIONING.md for details on automatic versioning

6. Check file permissions

Ensure you have write access to the output directory

Debugging

# Enable verbose logging to see detailed processing info
python pyread.py input.pdf --verbose

# Save logs to file for analysis
python pyread.py input/ -r --log-file debug.log --verbose

🏗️ Architecture

Code Structure

pyread.py
├── Constants & Configuration
├── Data Structures (FileFormat, ExtractionResult, ProcessingStats)
├── TextCleaner (normalization pipeline)
├── FormatDetector (magic + extension detection)
├── Extractors
│   ├── TxtExtractor (encoding detection)
│   ├── PypdfiumExtractor (fast PDF)
│   ├── PymupdfExtractor (PDF/EPUB fallback)
│   ├── OcrExtractor (scanned PDFs)
│   ├── HtmlExtractor (BeautifulSoup)
│   └── EpubExtractor (PyMuPDF/ebooklib)
├── ExtractorFactory (selection + fallback logic)
├── OutputManager (path generation, writing)
├── Processing Functions (single file, batch)
├── Logging Setup
└── CLI Application (Typer)

Design Principles

Speed First: Use fastest libraries (pypdfium2), process page-by-page
Fail Gracefully: Skip corrupt files, never crash in batch mode
Clean Separation: Each component has single responsibility
Easy to Extend: Add new formats by implementing Extractor protocol
User-Friendly: Clear error messages, progress bars, helpful defaults

🧪 Testing

Manual Testing

# Test single file
echo "Hello, World!" > test.txt
python pyread.py test.txt

# Test batch processing
mkdir test_docs
cp *.pdf test_docs/
python pyread.py test_docs/ --output-dir test_output/ -v

# Test error handling (corrupt file)
echo "Not a PDF" > fake.pdf
python pyread.py fake.pdf -v
# Should fail gracefully with error message

Sample Files

Create test files for each format:

sample.txt - Plain text
sample.pdf - PDF with text
scanned.pdf - Scanned PDF (for OCR testing)
sample.epub - EPUB ebook
sample.html - HTML file

📝 Development

Adding a New Format

Create an extractor class:

class MyFormatExtractor:
    @staticmethod
    def extract(file_path: Path, **kwargs) -> ExtractionResult:
        # Your extraction logic
        return ExtractionResult(success=True, text=extracted_text)

Update FileFormat enum:

class FileFormat(str, Enum):
    MYFORMAT = "myformat"

Add to FormatDetector.detect():

if ext == 'myext':
    return FileFormat.MYFORMAT

Update ExtractorFactory.get_extractor():

elif file_format == FileFormat.MYFORMAT:
    return MyFormatExtractor()

Code Style

Python 3.10+ with type hints
Docstrings for all public functions
Keep functions focused and testable
Use descriptive variable names

📄 License

MIT License - feel free to use in your projects!

🙏 Acknowledgments

Built with excellent libraries:

pypdfium2 - Fast PDF processing
PyMuPDF - PDF/EPUB extraction
ftfy - Text encoding repair
Typer - CLI framework
Rich - Beautiful terminal output

🚀 Version History

v1.0.0 - Initial release

PDF, EPUB, TXT, HTML support
Parallel batch processing
Optional OCR
Clean text normalization

Made with ❤️ for fast, reliable document conversion

Name		Name	Last commit message	Last commit date
test_samples		test_samples
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
EXAMPLES.md		EXAMPLES.md
FINAL_SUMMARY.md		FINAL_SUMMARY.md
GITHUB_SETUP.md		GITHUB_SETUP.md
INSTALL.md		INSTALL.md
PROJECT_SUMMARY.md		PROJECT_SUMMARY.md
PUSH_TO_GITHUB.txt		PUSH_TO_GITHUB.txt
QUICKSTART.md		QUICKSTART.md
README.md		README.md
VERSIONING.md		VERSIONING.md
generate_test_files.py		generate_test_files.py
pyproject.toml		pyproject.toml
pyread.py		pyread.py
requirements.txt		requirements.txt
test_pyread.py		test_pyread.py
test_sample.txt		test_sample.txt

Latest commit

History

Folders and files

Repository files navigation