Handling Tables and Scanned PDFs

Part 11 flagged scanned PDFs as a real, honest gap. This part covers that directly, plus a second, genuinely common real problem — tables, which plain text extraction handles poorly.

Detecting whether a real PDF actually needs OCR

from pypdf import PdfReader

def needs_ocr(filepath: str) -> bool:
    reader = PdfReader(filepath)
    text = "".join(page.extract_text() for page in reader.pages)
    # a real, practical heuristic: a genuine, text-based PDF should
    # extract a meaningful amount of real text per page
    return len(text.strip()) < 100 * len(reader.pages)

This real, simple heuristic checks whether extract_text() genuinely returned meaningful content — a scanned PDF with no real, actual text layer returns empty or near-empty text, since there's no text to extract at all, only pixel data.

Real OCR for scanned documents

pip install pytesseract pdf2image

import pytesseract
from pdf2image import convert_from_path

def extract_text_with_ocr(filepath: str) -> str:
    images = convert_from_path(filepath)
    full_text = ""
    for image in images:
        full_text += pytesseract.image_to_string(image) + "\n"
    return full_text

pytesseract is a real, widely used Python wrapper around Tesseract, a genuine, open-source OCR (Optical Character Recognition) engine — convert_from_path() first turns each real PDF page into an actual image, then image_to_string() recognizes real, actual text within that image, a genuinely different process from part 11's direct text-layer extraction.

Combining both into one real, robust function

def extract_text_smart(filepath: str) -> str:
    if needs_ocr(filepath):
        return extract_text_with_ocr(filepath)
    return extract_text_from_pdf(filepath)  # part 11's real version

This real, automatic routing is genuinely useful — GreenDesk's real document set likely includes both natively text-based contracts and scanned, signed paper agreements, and this function handles both correctly without a person having to manually identify which type each real file is.

Real, practical table extraction

pip install pdfplumber

import pdfplumber

def extract_tables(filepath: str) -> list[list[list[str]]]:
    all_tables = []
    with pdfplumber.open(filepath) as pdf:
        for page in pdf.pages:
            tables = page.extract_tables()
            all_tables.extend(tables)
    return all_tables

Plain text extraction genuinely mangles real tabular data — a real pricing table's rows and columns collapse into a confusing, unstructured jumble of text when extracted as plain prose. pdfplumber's extract_tables() instead returns real, structured data (a list of rows, each a list of real cell values), preserving the actual tabular relationships.

Converting a real, extracted table into RAG-friendly text

def table_to_text(table: list[list[str]]) -> str:
    header = table[0]
    rows_text = []
    for row in table[1:]:
        row_desc = ", ".join(f"{header[i]}: {cell}" for i, cell in enumerate(row))
        rows_text.append(row_desc)
    return "\n".join(rows_text)

Real, structured table row: ["Enterprise", "$800/mo", "Unlimited users"]
Real, converted text: "Plan: Enterprise, Price: $800/mo, Users: Unlimited users"

This real conversion is genuinely important for RAG specifically — chunking and embedding a real, raw table's collapsed, jumbled text (from plain extraction) produces poor, unreliable embeddings; converting each real row into an explicit, labeled sentence first produces chunks that embed and retrieve far more reliably, directly connecting back to part 5's chunking-quality discussion.

A real, honest combined pipeline

def index_pdf_completely(filepath: str, document_id: str) -> None:
    text = extract_text_smart(filepath)
    tables = extract_tables(filepath)
    table_texts = [table_to_text(t) for t in tables if t]

    all_chunks = chunk_document(text) + table_texts
    # ...proceed with part 11's real indexing, using all_chunks

This real, combined function handles the genuine range of real documents GreenDesk actually deals with — scanned or native, with or without pricing tables — rather than the simplified, single-path version part 11 started with.

Next: building a real website AI chatbot — an embeddable, real widget bringing this series' RAG work directly onto Bright Leaf Coffee's actual site.