Python on

Upgrading My PDF Converter to IBM's Docling

Sat, 02 May 2026 00:00:00 +0000

When My Own Tool Couldn’t Handle My Work

The error message was easy to dismiss: RapidOCR returned empty result!. It appeared twice in the terminal, then silence — a blank .md file where a 40-page Oracle HCM implementation guide should have been. The PDF had come straight from Oracle’s support portal, the same format I use for every triage session. But this one stored its pages as images, and PyMuPDF4LLM had nothing to work with.

That was one category of failure. The other was quieter. For documents that did convert, I started noticing the tables were wrong — not corrupted, just structurally dissolved. An eligibility matrix that should have had six clearly labeled columns came back as a run of loosely connected text. Useful for nothing.

I had built this tool to serve my Oracle work. Then my Oracle work showed me exactly where it fell short.

The Problem with PyMuPDF4LLM

If you’ve followed this series, you know that PyMuPDF4LLM was a solid choice when I first built the converter . It handled text-based PDFs cleanly, installed without friction, and required almost no configuration. For research papers and simple documentation, it worked well.

But Oracle HCM documentation is a different category of document. Oracle’s guides are dense with tables: configuration reference grids, eligibility matrices, step-and-action setup tables. These are not decorative — they carry most of the meaning. When PyMuPDF4LLM dissolved those tables into unstructured text, it was silently degrading the most important parts of the document.

The image-based PDF problem was a hard wall. If a document was captured as page images rather than extractable text, the converter returned nothing. No partial output, no warning — just empty files.

Discovering Docling

IBM Research Zurich’s AI for Knowledge team open-sourced Docling in July 2024. The project has a specific focus: turning complex documents into structured, AI-ready output. In April 2025, IBM donated it to the Linux Foundation AI & Data, and it now powers data ingestion for Red Hat Enterprise Linux AI. As of this writing it has over 24,000 GitHub stars.

What makes Docling different is that it treats document conversion as a computer vision problem, not just a text extraction problem.

Layout analysis: Docling uses an RT-DETR-derived model trained on DocLayNet — IBM’s human-annotated dataset of real-world documents — to detect and classify every region on the page: tables, figures, headers, footers, section titles, body text. It knows the structure before it extracts any content.

Table reconstruction: This is where Docling earns its place for Oracle documentation. It uses a vision transformer called TableFormer that predicts row/column structure and header roles directly from the page image. The result is a proper Markdown table, not a stream of cell values.

Image-based PDFs: For documents stored as page images, Docling integrates OCR into its pipeline natively. The same converter handles text-based and image-based PDFs without any changes on your end.

The Switch

The API change was minimal. The old code:

import pymupdf4llm

md_text = pymupdf4llm.to_markdown(pdf_path)

The new code:

from docling.document_converter import DocumentConverter

converter = DocumentConverter()
result = converter.convert(pdf_path)
md_text = result.document.export_to_markdown()

Three lines instead of one, but the extra structure pays dividends: DocumentConverter can be initialized once and reused across an entire batch, which matters when processing a folder of 50 Oracle guides.

A note on startup: The first time you run Docling, it downloads its ML models from Hugging Face. You will see this:

Loading weights: 100%|██████████| 770/770 [00:00<00:00, 1656.35it/s]

This is normal. The models cache locally after the first download and subsequent runs start immediately. If you see a warning about HF_TOKEN, that is also expected — Docling works without one, but setting a token removes the rate-limit warning:

echo 'export HF_TOKEN="hf_your_token_here"' >> ~/.zshrc

What Changed in Practice

Oracle documentation: Tables that previously collapsed into text now render as proper Markdown tables. A 6-column configuration reference comes back with headers intact and every row correctly aligned.

AI books: My knowledge base includes dense technical books on LLM engineering and machine learning. These have complex layouts — sidebars, multi-column sections, figures with captions. Docling’s layout model handles these significantly better than PyMuPDF4LLM’s heuristic approach.

Image-based PDFs: Documents that previously produced empty output now convert cleanly. The two-step workaround (ocrmypdf → pdf2md) is no longer necessary for most cases.

Two Other Improvements

While I was updating the engine, I added two things that were overdue:

DOCX support. The converter now handles Word documents using pandoc as a backend. The same pdf2md command works for both file types. This matters for Oracle support exports and study notes from my reMarkable.

Batch manifest. When processing a large folder, the converter now writes a manifest file tracking which files have been converted and their checksums. Re-running on the same folder skips files that haven’t changed. A --force flag overrides this when you need a fresh conversion.

pdf2md --batch ~/oracle-pdfs/ # skips already-converted
pdf2md --batch ~/oracle-pdfs/ --force # reconverts everything

What’s Next

The web UI — which I added in the last post — has also been updated to use Docling. Drag a PDF onto it, click Convert, and the same deep-learning pipeline runs behind the scenes.

The next thing I want to add is direct output to the Obsidian inbox. Right now the flow is: convert → download ZIP → move to vault. A toggle that sends output directly to ~/projects/obsidian-vault/00-inbox/ would cut that manual step entirely.

The tool is doing what I originally wanted: converting my Oracle documentation and AI library into clean, searchable Markdown. Docling is what makes that reliable for the documents that actually matter.

Adding a Web UI to My PDF to Markdown Converter

Sat, 28 Mar 2026 00:00:00 +0000

The Promise I Made to Myself

In my last post about building the PDF to Markdown converter , I listed some “what’s next” ideas at the end. One of them was:

FastAPI wrapper: Create an HTTP API for web apps to use

Well, I did it. And I went a step further — I built a full drag-and-drop web UI on top of it.

The CLI still works exactly as before. This is an addition, not a replacement. But now when I want to convert a batch of PDFs without thinking about terminal commands, I just open a browser tab.

What the UI Does

The interface is intentionally minimal:

Drag-and-drop zone — drop one PDF or fifty onto it
Browse button — if you prefer clicking
Convert button — kicks off the conversion
Per-file progress bars — live updates as each file converts
Individual download — each completed file gets its own Download button
Download all as ZIP — one click to grab everything
Clear — resets the session and cleans up temp files server-side

Everything runs locally. Files go to a temp directory on your machine, get converted, and are served back to you. Nothing hits an external API.

The Stack

I kept it as simple as possible:

Backend: FastAPI + uvicorn

FastAPI was the obvious choice — it handles file uploads cleanly, has first-class async support, and the python-multipart library makes multi-file form handling trivial. The conversion logic is unchanged from the CLI: pymupdf4llm.to_markdown() doing the heavy lifting.

Progress updates: Server-Sent Events (SSE)

This is the part I found most interesting. When you hit Convert, the browser opens a persistent connection to /progress/{job_id} and receives a stream of JSON events — one every 400ms — until the job finishes. No polling loop, no WebSocket complexity. SSE is perfect for this: unidirectional, simple, and built into every modern browser.

async def event_stream():
 while True:
 data = json.dumps({"progress": job["progress"], "done": job["done"]})
 yield f"data: {data}\n\n"
 if job["done"]:
 break
 await asyncio.sleep(0.4)

return StreamingResponse(event_stream(), media_type="text/event-stream")

On the frontend, consuming it is three lines:

const eventSource = new EventSource(`/progress/${jobId}`);
eventSource.onmessage = e => {
 const { progress, done } = JSON.parse(e.data);
 // update the UI...
};

Threading: The conversion itself is synchronous (PyMuPDF4LLM blocks while it processes pages). To keep the FastAPI event loop from freezing during conversion, each job runs in a ThreadPoolExecutor:

asyncio.get_event_loop().run_in_executor(executor, _convert_job, job_id, job_dir)

Four workers by default — enough to handle several simultaneous conversions without overloading the machine.

Frontend: Vanilla JS, no build step

I deliberately avoided React, Vue, or any framework. The whole UI is a single static/index.html file. It loads instantly, has no dependencies to install, and is easy to read and modify. For a local tool that one person uses, this is the right call.

Project Structure

Here’s what changed from the original CLI project:

pdf-to-markdown/
 pdf2md — original CLI (unchanged)
 app.py — FastAPI server (new)
 static/
 index.html — drag-drop UI (new)
 serve — start script (new)
 requirements.txt — updated with FastAPI deps
 venv/ — existing venv, three new packages added

The serve script is just:

#!/usr/bin/env bash
cd "$(dirname "$0")"
source venv/bin/activate
uvicorn app:app --host 0.0.0.0 --port 8765 --reload

Run it once, open http://localhost:8765, and you have a working converter in your browser.

One Gotcha: PyMuPDF4LLM is Synchronous

This tripped me up briefly. pymupdf4llm.to_markdown() does not return a coroutine — it’s a blocking call that can take 10–30 seconds on a large document. If you call it directly in an async FastAPI route handler, you freeze the entire event loop while it runs. No other requests get handled. The SSE stream stops updating.

The fix is the ThreadPoolExecutor pattern above — push the blocking work off the event loop entirely. The async route returns immediately, the SSE stream keeps ticking, and the conversion runs in a thread pool where it belongs.

The Download Endpoints

Three endpoints handle output:

GET /download/{job_id}/{filename} — single .md file
GET /download-all/{job_id} — all .md files as a ZIP
DELETE /job/{job_id} — clean up temp files

The ZIP is built in memory using Python’s zipfile module and streamed directly to the browser — no intermediate file on disk:

buf = io.BytesIO()
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
 for f in md_files:
 zf.write(f, f.name)
buf.seek(0)
return StreamingResponse(buf, media_type="application/zip", ...)

What This Unlocks

The CLI was already useful. The web UI adds a few things the CLI cannot easily do:

Non-terminal users. Anyone on my network can now use this converter by visiting http://my-machine:8765. No Python knowledge required.

Bulk drop workflows. Dragging 20 PDFs from Finder into a browser window and clicking Convert is significantly faster than constructing a --batch command with the right paths.

Visual feedback. The progress bars are not just cosmetic. For large PDFs that take 20–30 seconds, knowing the conversion is running (and roughly how far along it is) removes the anxiety of staring at a terminal cursor.

What’s Next

The original roadmap item was “FastAPI wrapper.” That’s done. The next one I’m eyeing:

Auto-feed to Obsidian inbox. Right now the flow is: convert in the web UI, download the ZIP, unzip, move to Obsidian. I’d like to add a toggle: “Send output directly to ~/projects/obsidian-vault/00-inbox/” — one less manual step.

That’s a small addition to the backend. Coming soon.

Running It

cd ~/projects/pdf-to-markdown
./serve
# Open http://localhost:8765

The first run installs nothing new — the three new packages (fastapi, uvicorn, python-multipart) are already in the venv. It just works.

Building an AI Conference Directory That Populates Itself

Sat, 14 Mar 2026 00:00:00 +0000

The Problem: AI Conferences Are Everywhere and Nowhere

If you’ve ever tried to find a comprehensive list of upcoming AI conferences, you know the pain. There’s no single source. AAAI has their page. NeurIPS has theirs. ICML posts deadlines on OpenReview. Half the emerging summits only exist on LinkedIn event pages or buried in Reddit threads.

I wanted a simple, searchable directory of AI conferences — one site where I could see what’s coming up, filter by topic, and get the key details. But I didn’t want to manually curate it. I’ve seen too many “awesome lists” on GitHub that are lovingly maintained for three months and then abandoned.

What I wanted was a system that populates itself.

So I built one. And with Claude Code running through my PAI system, the whole pipeline — from search to database to website — came together over a few focused sessions.

Here’s the full story.

The Architecture: Three Layers, Zero Manual Data Entry

The final system has three layers, each handling a distinct responsibility:

SearXNG (search engine)
→ conference_tracker.py (discovery)
→ Airtable (database)
→ fetch-events.mjs (build-time fetch)
→ React + Vite site on Netlify

Each layer is independently useful, loosely coupled, and replaceable. Let’s walk through them.

Layer 1: The Tracker — Finding Conferences Automatically

The foundation is a Python script called conference_tracker.py. Its job is simple: search the web for AI conferences and store what it finds.

Search: SearXNG Instead of Google

Rather than hitting the Google API (with its quotas and billing), I use SearXNG — an open-source, self-hosted meta-search engine. It aggregates results from Google, Bing, DuckDuckGo, and others without API keys or rate limits.

The tracker runs a curated list of search queries defined in config.yaml:

search_queries:
 - "AI conference 2026"
 - "artificial intelligence conference 2026"
 - "machine learning conference 2026"
 - "NeurIPS 2026"
 - "ICML 2026"
 - "AAAI 2026"
 - "AI summit 2026"
 - "deep learning conference 2026"
 - "computer vision conference 2026 CVPR"
 - "natural language processing conference 2026"

Each query returns up to 10 results. The tracker extracts the title, URL, and snippet from each result, deduplicates against what’s already in the database, and stores new finds.

Storage: Airtable as the Source of Truth

Why Airtable? Because it’s a real database with an API, but it also has a spreadsheet-like UI for manual review. When you’re building a pipeline that discovers data automatically, you want a way to eyeball the results and clean up noise — and Airtable is perfect for that.

The tracker writes five fields per record: title, websiteUrl, description, Source Query, and Date Found. That’s it. Just the raw discovery data. The structured details come later.

The deduplication is URL-based — normalized and lowercased. If we’ve already stored neurips.cc/2026, we don’t store it again even if it appears in a different search query.

def extract_conference_info(result, source_query):
 return {
 "title": result["title"][:200],
 "websiteUrl": result["url"],
 "description": result["snippet"][:1000],
 "Source Query": source_query,
 "Date Found": datetime.now(timezone.utc).strftime("%Y-%m-%d"),
 }

After one run, we had 87 unique conference records. The real stuff — NeurIPS, ICML, CVPR, AAAI — alongside smaller but interesting events like the Quantum AI and NLP Conference, Deep Learning Indaba, and the Wharton Human-AI Research summit.

Layer 2: The Website — React + Vite on Netlify

The directory itself is a React app built with Vite and deployed on Netlify. It’s a single-page app with search, tag filtering, and individual event pages.

The key architectural decision: data is fetched at build time, not runtime. A prebuild script (fetch-events.mjs) pulls conference data from the database and writes it to a data.ts file that Vite bundles into the site. This means:

No API keys exposed in the browser
No CORS issues
Instant page loads (data is already in the bundle)
The site works even if Airtable is temporarily down

The prebuild hook in package.json makes this automatic:

{
 "scripts": {
 "fetch-events": "bun scripts/fetch-events.mjs",
 "prebuild": "bun scripts/fetch-events.mjs",
 "build": "vite build"
 }
}

Every time Netlify builds the site, it automatically fetches the latest data from Airtable. Fresh data on every deploy.

The Middleman Problem: Cutting Google Sheets

Here’s where the story gets interesting.

The original pipeline had an extra step: Airtable → Google Sheets → website. The fetch-events.mjs script was pulling from a published Google Sheet CSV. Why? Because when I first prototyped the site, I started with a spreadsheet. It was quick and easy.

But once the conference tracker was writing directly to Airtable, Google Sheets became a middleman with no purpose. Data had to be synced from Airtable to Sheets (manually or via Zapier), and that sync was another thing that could break.

The fix was straightforward: teach fetch-events.mjs to talk directly to the Airtable API.

Airtable’s REST API

The Airtable API is clean. A single GET request returns records as JSON:

const url = new URL(`https://api.airtable.com/v0/${baseId}/${tableId}`);
const resp = await fetch(url.toString(), {
 headers: { Authorization: `Bearer ${pat}` },
});
const data = await resp.json();
// data.records = [{ id, fields: { title, date, ... } }]

The one gotcha: Airtable paginates at 100 records. You need to follow the offset token:

async function fetchFromAirtable(pat, baseId, tableId) {
 const allRecords = [];
 let offset = null;

 do {
 const url = new URL(`https://api.airtable.com/v0/${baseId}/${tableId}`);
 if (offset) url.searchParams.set('offset', offset);

 const resp = await fetch(url.toString(), {
 headers: { Authorization: `Bearer ${pat}` },
 });
 const data = await resp.json();
 allRecords.push(...data.records);
 offset = data.offset || null;
 } while (offset);

 return allRecords;
}

Graceful Fallback

I kept the Google Sheets path as a fallback. The main() function uses a priority chain:

Airtable — if AIRTABLE_PAT, AIRTABLE_BASE_ID, AIRTABLE_TABLE_ID are set
Google Sheets — if GOOGLE_SHEET_CSV_URL is set
Fallback events — hardcoded sample data so the build never fails

This means you can’t break the site by misconfiguring a data source. The build always succeeds.

Layer 3: The Enrichment — AI-Powered Data Extraction

This is where things got really interesting.

After cutting Google Sheets, I had 87 conference records in Airtable. But they only had three useful fields: title, description, and URL. No dates. No locations. No tags. The site worked, but every event card was sparse — no way to filter by date or location, no tags to browse by topic.

Filling in 87 records by hand? No thanks.

The Idea: Visit Each URL and Ask AI to Extract the Data

The approach: for each conference record, fetch its web page, extract the text content, and use AI inference to pull out structured fields like date, location, organizer, and tags.

I built an enrichment script — enrich_conferences.py — that sits alongside the tracker in the same project.

Step 1: Fetch and Clean the Page

Each conference URL gets fetched with requests, then cleaned with BeautifulSoup. Navigation, footers, scripts, and styling get stripped, leaving just the text content:

def fetch_page_text(url, timeout=15):
 resp = requests.get(url, headers=headers, timeout=timeout)
 soup = BeautifulSoup(resp.text, "html.parser")

 for tag in soup(["script", "style", "nav", "footer", "header", "aside"]):
 tag.decompose()

 text = soup.get_text(separator="\n", strip=True)
 lines = [line.strip() for line in text.splitlines() if line.strip()]
 return "\n".join(lines)

Step 2: AI Extraction via PAI Inference

The cleaned text gets sent to Claude (via PAI’s Inference tool) with a structured extraction prompt. The prompt is specific about what to extract and what format to use:

Given text from a conference web page, extract these fields as JSON:
{
"date": "human-readable date like 'May 5-6, 2026'",
"endDate": "ISO end date like '2026-05-06'",
"location": "City, State/Country",
"venue": "venue name",
"price": "ticket price or 'Free'",
"organizer": "organizing body",
"tags": "comma-separated topic tags (max 4)"
}

One critical addition: if the page is a list of conferences (like “Top 10 AI Conferences of 2026”), the AI returns {"is_list_page": true} and the script skips it. This was essential — about 15% of our URLs were aggregator pages, not individual conference pages.

Step 3: Write Back to Airtable

Non-empty extracted fields get PATCHed back to Airtable. The script only writes fields that actually exist in the table schema — a lesson learned the hard way when venue and imageUrl threw 422 errors because those columns hadn’t been created yet.

def build_patch_fields(extracted, allowed_fields):
 if extracted.get("is_list_page"):
 return None
 patch = {}
 for key in ["date", "endDate", "location", "venue", "price", "organizer", "tags"]:
 if key not in allowed_fields:
 continue
 val = extracted.get(key, "")
 if isinstance(val, str) and val.strip():
 patch[key] = val.strip()
 return patch if patch else None

The Results

Running the enrichment script across all 87 records:

Outcome	Count
Records enriched	48
List/aggregator pages (correctly skipped)	12
No extractable fields (social media, OpenReview, etc.)	11
Errors (timeouts, HTTP 403s)	16

After enrichment:

Field	Records populated
Date	42
Location	41
Tags	47
Organizer	27
Price	4

From zero structured data to a directory where most events have dates, locations, and topic tags — without opening a single conference website manually.

Some highlights from the extraction:

NeurIPS 2026: December 6-12, Sydney, Australia — Deep Learning, Research, Algorithms, LLMs
CVPR 2026: June 3-7, Denver, CO — Computer Vision, Deep Learning, Research
ICML 2026: July 6-11, Seoul, South Korea — LLMs, Computer Vision, NLP, Robotics
AI Council 2026: May 12-14, San Francisco, CA — Generative AI, ML Ops, AI Safety
MIDL 2026: July 8-10, Taipei — Deep Learning, Healthcare AI, Computer Vision

The Pipeline Today

Here’s what the full system looks like now:

SearXNG (self-hosted search)
→ conference_tracker.py (Python — discovers conferences)
→ Airtable (source of truth — 87 records)
→ enrich_conferences.py (Python — AI-powered field extraction)
→ Airtable (now with dates, locations, tags)
→ fetch-events.mjs (Node — build-time data fetch)
→ data.ts (bundled into the site)
→ React + Vite app on Netlify

The tracker discovers. The enricher structures. The fetcher delivers. The site displays. Each piece runs independently and can be re-run at any time.

The enrichment script is idempotent — it only processes records where the date field is empty, so running it again only touches new or previously-failed records.

What I’d Do Differently (And What’s Next)

The Timeout Problem

About 16 records hit the 25-second inference timeout. The fast tier (Haiku) is quick but occasionally chokes on pages with dense, complex content. A retry mechanism using the standard tier (Sonnet) for failed records would catch most of these.

Missing Table Columns

The venue and imageUrl fields don’t exist in the Airtable table yet. The enrichment script extracts venue names beautifully (The Venetian for Ai4, COEX Convention Center for ICML, Dongguk University for AAAI Summer), but the data gets dropped because the columns aren’t there. A quick table schema update in the Airtable UI fixes this.

Scheduled Runs

Right now, both the tracker and enricher are manual. The natural next step is scheduling — run the tracker daily to discover new conferences, the enricher on new records, and trigger a Netlify deploy afterward. The Netlify build hook is already configured; it just needs a cron job or GitHub Action to call it.

Data Quality

Some records are noise — Reddit discussion threads, Amazon Science blog posts, Twitter/X profiles. A quality filter (either rule-based on URL patterns or AI-powered) would clean the dataset before enrichment runs.

Lessons Learned

1. Eliminate Middlemen Early

Google Sheets added zero value once Airtable was in the picture. But it lingered because it was the “original” approach. Every extra hop in a pipeline is a thing that can break, a thing that needs syncing, and a thing that slows you down. Cut it.

2. Build-Time Data Fetching Is Underrated

Pulling data at build time instead of runtime means no API keys in the browser, no loading spinners, and no CORS headaches. For data that changes daily (not per-second), this is the right architecture.

3. AI Extraction Beats Manual Curation

Using AI to extract structured data from unstructured web pages isn’t perfect — we got 48 out of 87 records enriched, not 87 out of 87. But it took 20 minutes of runtime versus what would have been hours of manual work. And the script is re-runnable. Improvement is incremental.

4. Detect Your Data’s Shape Before Writing

The Airtable 422 errors on venue were entirely preventable. The enrichment script now probes the table schema at startup and only writes to fields that exist. Defensive coding at system boundaries saves debugging time.

5. List Page Detection Is Essential for Web Scraping Pipelines

When you’re scraping URLs from search results, a significant percentage will be aggregator pages (“Top 10 Best AI Conferences”) rather than individual event pages. If you don’t detect and skip these, you’ll corrupt your dataset with merged data from multiple events. The is_list_page flag in the AI extraction prompt was one of the highest-value additions to the whole pipeline.

The Bigger Picture

This project is a miniature version of a pattern I keep coming back to: systems that compound.

The tracker runs once and discovers 87 conferences. The enricher runs once and structures 48 of them. The next time the tracker runs, it discovers only new conferences (deduplication handles the rest). The next time the enricher runs, it only processes records it hasn’t touched yet.

Every run makes the dataset better without redoing previous work. That’s the whole point of building infrastructure instead of doing things manually — you invest upfront so the system improves over time with minimal additional effort.

Working with Claude through PAI made each layer come together faster than I expected. The tracker, the Airtable integration, the Google Sheets elimination, the enrichment script — each was a focused session where the AI handled the implementation details while I focused on architecture decisions.

That’s the augmented part of Augmented Resilience. Not replacing the thinking — amplifying it.

When Your PDF Workflow Breaks - Building a Markdown Converter with Claude Code

Wed, 18 Feb 2026 00:00:00 +0000

The Problem: PDFs Are Knowledge Prisons

You know that feeling when you download a brilliant research paper, only to realize you can’t easily feed it into your AI workflow? Or when you want to add documentation to your knowledge base, but it’s locked in a format that doesn’t play well with version control or LLM tools?

Yeah, I was there last week.

I had just downloaded a fascinating 1.3MB research paper on Generative Engine Optimization and wanted to process it with my AI tools. But PDFs are terrible for this. They’re designed for printing, not for processing. What I needed was Markdown—clean, portable, AI-friendly Markdown.

So I built a converter. And with Claude Code as my copilot through the PAI (Personal AI Infrastructure) system, the whole thing took less than 30 minutes.

Here’s how it went down.

Why Markdown is Better Than PDF for LLMs

Before diving into the build, let’s answer the obvious question: why bother converting? Can’t LLMs just read PDFs directly?

Technically, yes. But the results are significantly worse, and the reasons are fundamental to how PDFs work.

PDFs Are Layout-First, Not Structure-First

PDFs were designed to describe where things appear on a page, not what they mean. As Steven Howard explains in Why PDFs Fail Under LLM Parsing :

“Table cells with wrapped text insert hard line breaks that fragment token continuity and break logical row recognition. Headers and footers simply add noise to the context when used with LLMs. Sentences are split with arbitrary CR/LFs making it very difficult to find paragraph boundaries.”

This architectural mismatch — a format designed for printing being fed into a system designed for understanding — causes cascading problems downstream.

The Token Efficiency Problem

Every token your LLM processes costs money and consumes context window space. PDF extraction wastes both.

According to analysis from MarkdownConverters , Markdown saves up to 70% more tokens compared to extracted PDF text for the same content. The culprit: PDF extraction introduces formatting artifacts, metadata noise, headers/footers, and encoding remnants that all consume tokens without adding semantic value.

To put that in practical terms: a PDF that would use 10,000 tokens might only need 3,000 tokens when properly converted to Markdown. At scale, this compounds dramatically.

The RAG Performance Problem

If you’re building Retrieval Augmented Generation (RAG) systems — using documents as a knowledge base for AI — document format directly impacts answer quality.

The research here is compelling:

Academic validation: A 2024 paper on arXiv (Revolutionizing RAG with Enhanced PDF Structure Recognition ) found that “the low accuracy of PDF parsing significantly impacts the effectiveness of professional knowledge-based QA.”
Industry validation: NVIDIA’s technical blog documents how their NeMo Retriever pipeline converts extracted content to Markdown specifically because it “preserves row/column relationships in an LLM-native format, significantly reducing numeric hallucination” — and reduces incorrect answers by 50%. (NVIDIA: Approaches to PDF Data Extraction for Information Retrieval )
Chunking quality: Analysis from Towards Data Science shows that Markdown’s heading structure (#, ##, ###) produces semantically meaningful chunks, while PDF-based chunking relies on arbitrary page breaks and heuristics.
Retrieval failure rates: Unstructured.io’s research on contextual chunking — tested across 5,563 question-answer pairs — showed an 84% reduction in retrieval failure rates when using structure-aware chunking (the kind Markdown enables natively).
Real-world outcomes: The 2025 Semrush AI Index, cited by Webex Developers Blog , found that 72% of top AI-indexed articles used Markdown or Markdown-like structures, achieving 34% higher retrieval accuracy across ChatGPT, Perplexity, and Gemini.

The Bottom Line

Metric	Impact
Token reduction	Up to 70% fewer tokens vs PDF extraction
Incorrect answers in RAG	50% reduction (NVIDIA NeMo)
Retrieval failure rates	84% reduction (Unstructured.io)
Retrieval accuracy	34% higher (Semrush AI Index 2025)

Markdown isn’t just more convenient — it’s meaningfully better for AI. Converting your document libraries is one of the highest-ROI steps you can take before building any LLM-powered workflow.

The First Failure: When Bleeding-Edge Python Bites Back

I’m running Python 3.14.2—the latest release, barely a few weeks old. Modern, shiny, cutting-edge. Perfect, right?

Not quite.

My first instinct was to use marker-pdf, a high-performance converter optimized for scientific papers and books. It looked perfect on paper (pun intended). But when I tried to install it:

Building wheel for Pillow (pyproject.toml): finished with status 'error'

Ugh.

Turns out, marker-pdf depends on Pillow (the Python imaging library), and Pillow hasn’t built binary wheels for Python 3.14 yet. I could have downgraded Python. I could have fought with source compilation. But why?

This is where working with Claude Code really shines. Instead of going down a rabbit hole trying to force marker-pdf to work, Claude suggested pivoting to PyMuPDF4LLM—a mature, actively maintained library specifically designed for AI/LLM workflows.

And it just worked.

The Solution: PyMuPDF4LLM

PyMuPDF4LLM turned out to be exactly what I needed:

Works flawlessly with Python 3.14 (no compilation errors)
Fast and accurate conversion
Built specifically for feeding documents into LLMs
Clean, simple API
Actively maintained by the PyMuPDF team

The installation was literally:

pip install pymupdf4llm

Five seconds later, I was ready to go.

Building the Tool: First Principles Thinking

As someone new to the CLI world, I’ve been learning to think through project structure from first principles. Where should this live? How should it be organized?

With Claude’s guidance, I chose /Users/dsa/projects/pdf-to-markdown/ for a few key reasons:

Separation of Concerns: Tool projects should be separate from my main workspace
Discoverability: Clear, descriptive naming means I’ll find it again in 6 months
Reusability: This structure works both as a CLI tool AND as a library I could import later

The project structure ended up simple but complete:

pdf-to-markdown/
├── README.md # Documentation
├── venv/ # Isolated Python environment
├── input/ # Test PDFs
├── output/ # Generated markdown
├── pdf2md # CLI wrapper script
└── requirements.txt # Dependencies

The Code: A Simple but Powerful CLI

I wanted a tool I could actually use—something with a clean command-line interface that handles the common cases elegantly. Working with Claude through PAI, we created a Python script that does exactly that:

#!/usr/bin/env python3
"""
PDF to Markdown Converter
A simple CLI tool to convert PDF files to Markdown using PyMuPDF4LLM
"""

import sys
import os
from pathlib import Path
import pymupdf4llm
import pymupdf
from tqdm import tqdm

def convert_pdf_to_markdown(pdf_path: str, output_path: str = None) -> str:
 """Convert a PDF file to Markdown format."""

 if not os.path.exists(pdf_path):
 raise FileNotFoundError(f"PDF file not found: {pdf_path}")

 # Get page count for progress bar
 doc = pymupdf.open(pdf_path)
 page_count = doc.page_count
 doc.close()

 print(f"Converting: {pdf_path}")
 with tqdm(total=page_count, unit="page", desc="Processing", colour="blue") as bar:
 md_text = pymupdf4llm.to_markdown(pdf_path, page_chunks=False)
 bar.n = page_count
 bar.refresh()

 if output_path is None:
 output_path = Path(pdf_path).with_suffix('.md')

 with open(output_path, 'w', encoding='utf-8') as f:
 f.write(md_text)

 print(f"✓ Done: {output_path} ({len(md_text):,} characters)")
 return str(output_path)

def batch_convert(input_dir: str, output_dir: str = None) -> None:
 """Convert all PDFs in a directory to Markdown."""
 input_path = Path(input_dir)
 if not input_path.is_dir():
 raise NotADirectoryError(f"Not a directory: {input_dir}")

 pdfs = sorted(input_path.glob("*.pdf"))
 if not pdfs:
 print(f"No PDF files found in: {input_dir}")
 sys.exit(0)

 if output_dir:
 output_dir = Path(output_dir)
 else:
 output_dir = input_path.parent / "output"
 output_dir.mkdir(parents=True, exist_ok=True)

 total = len(pdfs)
 succeeded = 0
 failed = 0

 print(f"\nBatch mode: {total} PDF(s) found in '{input_dir}'")
 print(f"Output folder: {output_dir}\n")

 for i, pdf_path in enumerate(pdfs, start=1):
 print(f"[{i}/{total}] {pdf_path.name}")
 output_path = output_dir / pdf_path.with_suffix('.md').name
 try:
 convert_pdf_to_markdown(str(pdf_path), str(output_path))
 succeeded += 1
 except Exception as e:
 print(f" ✗ Failed: {e}")
 failed += 1
 print()

 print("─" * 40)
 print(f"Batch complete: {succeeded} converted, {failed} failed")
 print(f"Output folder: {output_dir}")

def main():
 """Main CLI entry point"""
 args = sys.argv[1:]

 if not args:
 print("Usage:")
 print(" pdf2md <input.pdf> [output.md] # Convert a single PDF")
 print(" pdf2md --batch <folder/> # Convert all PDFs in a folder")
 print(" pdf2md --batch <folder/> --output <out_folder/> # Batch with custom output dir")
 print("\nExamples:")
 print(" pdf2md document.pdf # Creates document.md")
 print(" pdf2md document.pdf custom.md # Creates custom.md")
 print(" pdf2md --batch input/ # Converts all PDFs in input/")
 print(" pdf2md --batch ~/documents/pdfs/ --output ~/knowledge-base/docs/")
 sys.exit(1)

 if args[0] == "--batch":
 input_dir = args[1]
 output_dir = None
 if "--output" in args:
 idx = args.index("--output")
 output_dir = args[idx + 1]
 batch_convert(input_dir, output_dir)
 else:
 pdf_path = args[0]
 output_path = args[1] if len(args) > 1 else None
 convert_pdf_to_markdown(pdf_path, output_path)

if __name__ == "__main__":
 main()

What I love about this code:

Smart defaults: If you don’t specify an output path, it just replaces .pdf with .md
Progress bars: tqdm gives you a blue progress bar with page count
Batch mode: --batch processes an entire folder at once, with optional --output target
Helpful errors: Clear messages when things go wrong
Flexible usage: Works with relative paths, absolute paths, custom output names

Make it executable:

chmod +x pdf2md

And now it’s a proper command-line tool.

The Moment of Truth: Testing with Real Data

Theory is great. But does it actually work?

I grabbed that 1.3MB research paper on Generative Engine Optimization and ran:

python pdf2md input/test.pdf output/test.md

The output:

Converting input/test.pdf to Markdown...
Processing: 100%|████████████████| 12/12 [00:02<00:00, 5.8 pages/s]
✓ Done: output/test.md (73,463 characters)

1.3MB PDF → 74KB of clean Markdown in seconds.

I opened the output file, and there it was—perfectly formatted markdown:

## **GEO: Generative Engine Optimization**

Pranjal Aggarwal [∗]
Indian Institute of Technology Delhi
New Delhi, India
pranjal2041@gmail.com

Ashwin Kalyan
Independent
Seattle, USA
asaavashwin@gmail.com
...

Headers, formatting, structure—all preserved. No manual cleanup needed.

Success.

What This Unlocks

Now that I have PDFs converting to Markdown reliably, a whole world of possibilities opens up:

AI Workflows

Feed research papers and documentation directly into Claude or other LLMs
Build RAG (Retrieval Augmented Generation) pipelines backed by your document library
Process technical documentation at scale without losing structure

Knowledge Management

Import PDFs into your Obsidian vault automatically
Version control document content (because it’s now plain text in git)
Full-text search across your entire converted document library

Automation Ideas

Watch folder that auto-converts any dropped PDFs
Batch process entire directories of reports, papers, or manuals
Feed converted markdown directly into a vector database
API wrapper to convert PDFs via HTTP requests

Lessons Learned (Especially for CLI Beginners)

1. Virtual Environments Are Non-Negotiable

Every Python project should live in its own virtual environment. Always:

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip

This keeps dependencies isolated and projects reproducible.

2. Bleeding-Edge Isn’t Always Better

Python 3.14 is awesome, but sometimes mature tooling (like PyMuPDF) that “just works” beats bleeding-edge alternatives. Don’t be afraid to pivot when something doesn’t work.

3. Test With Real Data

I didn’t test with “hello.pdf” containing two sentences. I tested with a 1.3MB research paper. Real data reveals real issues (or in this case, confirms it works beautifully).

4. Document As You Build

Writing the README alongside the code made the project immediately understandable. Future-me will thank present-me.

5. Claude Code + PAI = Superpowers

Working with Claude through the PAI infrastructure meant I had a senior developer helping me think through:

Project structure (first principles)
Library selection (when to pivot)
Code organization (clean, maintainable)
Real-world usage patterns

This wasn’t just coding faster—it was learning better patterns while building.

Usage Examples

Basic Conversion

# Activate environment first (always!)
source venv/bin/activate

# Convert a PDF
python pdf2md document.pdf

# Custom output name
python pdf2md research.pdf my-notes.md

# Full paths
python pdf2md ~/Downloads/paper.pdf ~/Documents/notes.md

Batch Processing

Convert an entire folder of PDFs:

source venv/bin/activate

# Convert all PDFs in a folder (output goes to output/ by default)
python pdf2md --batch ~/documents/pdfs/

# Convert to a specific knowledge base directory
python pdf2md --batch ~/documents/pdfs/ --output ~/knowledge-base/docs/

Add to PATH (Optional)

To use pdf2md from anywhere:

# Add to ~/.zshrc
export PATH="/Users/dsa/projects/pdf-to-markdown:$PATH"

# Then run from anywhere
pdf2md ~/Downloads/paper.pdf ~/Documents/paper.md

What’s Next?

This tool works great as-is, but there are some exciting enhancements on the roadmap:

Immediate Improvements

Better layout analysis: Install pymupdf_layout for improved structure detection on complex documents
Recursive batch mode: Process nested folder structures, not just flat directories

Future Integrations

RAG pipeline: Auto-feed converted markdown into a vector database
Obsidian plugin: Detect PDFs in vault and convert automatically
FastAPI wrapper: Create an HTTP API for web apps to use
Electron/Tauri app: Build a desktop GUI for non-technical users

The Bigger Picture: Why This Matters

This project is tiny—roughly 100 lines of Python, 30 minutes of work. But it represents something bigger:

The ability to build tools that solve your actual problems.

I had a workflow friction (PDFs don’t work well with AI tools). I built a solution. Now that friction is gone, and I can focus on higher-level work.

And the data is clear: converting your document library to Markdown isn’t a nice-to-have. It’s a multiplier on every AI workflow that follows. Up to 70% fewer tokens consumed. 84% fewer retrieval failures. 50% fewer incorrect answers. These aren’t marginal improvements—they’re transformational.

Working with Claude Code through PAI accelerated all of this. It’s like having a patient senior developer sitting next to you, suggesting better approaches, catching errors before they happen, and explaining why certain patterns work.

Resources

PyMuPDF4LLM Docs: https://pymupdf.readthedocs.io/en/latest/pymupdf4llm/
PyMuPDF GitHub: https://github.com/pymupdf/PyMuPDF

Citations: Markdown vs PDF for LLMs

Why PDFs Fail Under LLM Parsing — Steven Howard, Untethered AI: https://untetheredai.substack.com/p/why-pdfs-fail-under-llm-parsing
PDF vs Markdown for AI: Token Efficiency — MarkdownConverters: https://markdownconverters.com/blog/pdf-vs-markdown-ai-tokens
Revolutionizing RAG with Enhanced PDF Structure Recognition — arXiv:2401.12599 (2024): https://arxiv.org/abs/2401.12599
Approaches to PDF Data Extraction for Information Retrieval — NVIDIA Technical Blog: https://developer.nvidia.com/blog/approaches-to-pdf-data-extraction-for-information-retrieval/
Improved RAG Document Processing With Markdown — Dr. Leon Eversberg, Towards Data Science: https://medium.com/data-science/improved-rag-document-processing-with-markdown-426a2e0dd82b
Contextual Chunking: Boost Your RAG Retrieval Accuracy — Unstructured.io: https://unstructured.io/blog/contextual-chunking-in-unstructured-platform-boost-your-rag-retrieval-accuracy
Boosting AI Performance: The Power of LLM-Friendly Content in Markdown — Webex Developers Blog: https://developer.webex.com/blog/boosting-ai-performance-the-power-of-llm-friendly-content-in-markdown

Happy converting!

Deploying a Hugo Site to Namecheap with PAI

Sun, 15 Feb 2026 00:00:00 +0000

I recently deployed my Hugo blog to Namecheap shared hosting, using Obsidian as my content editor and Claude Code with PAI (Personal AI) as my copilot. Here’s a walkthrough of every step, from fixing build errors to setting up a fully automated pipeline that goes from Obsidian to live site in a single command.

The Starting Point

I created a Hugo blog project called Augmented Resilience and used the re-terminal theme, a Namecheap shared hosting account, and a GitHub repository. I used Claude Code in VS Code editor and leveraged Daniel Miessler’s Personal AI infrastructure. The goal: get the site live at augmentedresilience.com with a push-to-deploy workflow.

For context, the Personal AI Infrastructure System (PAI) from Daniel Miessler (see resources below) is an open-source framework that wraps around Claude Code and turns it into a structured problem-solving system. Instead of just chatting with an AI, PAI runs every request through a 7-phase algorithm — observe, think, plan, build, execute, verify, learn — so nothing gets skipped. It maintains persistent memory across sessions (so it remembers my project structure, preferences, and past decisions), automatically selects specialized agents for different tasks (security review, architecture, engineering), and enforces verification criteria before declaring anything “done.” For this project, PAI handled everything from debugging Hugo build errors to writing the deploy script to catching sensitive data I accidentally left in this blog post before it went live. It wasn’t just an AI assistant — it was the entire workflow engine. I found it easier to use it within VS Code (still getting used to using the command line interface).

Step 1: Fixing the Hugo Build

The first issue was a build error:

module "hugo-theme-re-terminal" not found

The problem was a mismatch between the theme name in hugo.toml and the actual directory name. The theme was installed as a git submodule at themes/re-terminal/, but the config referenced hugo-theme-re-terminal.

Fix: Change the theme name in hugo.toml:

theme = "re-terminal"

After that, hugo built the site successfully, generating the public/ folder with all the static files.

Step 2: Setting Up the GitHub Repository

I initialized the repo and connected it to GitHub:

git init
git remote add origin git@github.com:dsacosta/Augmented-Resilience.git
git add .
git commit -m "my first commit"
git push origin main

One gotcha: I initially typed orgin instead of origin in the remote add command. Typos happen — double-check your remote names with git remote -v.

Step 3: Connecting Namecheap to GitHub via SSH

This was the trickiest part. Namecheap shared hosting needs an SSH key to clone from a private GitHub repo. Here’s what worked:

Generate an SSH Key on Namecheap

Log into cPanel on Namecheap
Go to SSH Access → Manage SSH Keys → Generate a New Key
Generate an RSA key (I used the default settings)

Remove the Passphrase

This is critical. cPanel’s Git Version Control runs non-interactively, so it can’t prompt for a passphrase. I opened cPanel Terminal and ran:

ssh-keygen -p -f ~/.ssh/id_rsa

Enter the old passphrase, then press Enter twice for no new passphrase.

Add the Public Key to GitHub

On Namecheap’s cPanel Terminal, run: cat ~/.ssh/id_rsa.pub
Copy the output
Go to your GitHub repo → Settings → Deploy Keys → Add deploy key
Paste the public key and save

Verify the Connection

From cPanel Terminal:

ssh -T git@github.com

You should see: Hi dsacosta/Augmented-Resilience! You've successfully authenticated...

Step 4: Clone the Repo on Namecheap

In cPanel, go to Git Version Control → Create
Toggle Clone a Repository on
Enter the clone URL: git@github.com:dsacosta/Augmented-Resilience.git
Set the repository path (I used /home/yourusername/your-repo)
Click Create

Important: Don’t clone directly into public_html or your domain folder — it likely already has files and will error out. Clone to a separate directory and use deployment to copy files over.

Step 5: Auto-Deployment with .cpanel.yml

cPanel supports automatic deployment tasks via a .cpanel.yml file in the repo root. This file tells cPanel what to do after each pull:

---
deployment:
 tasks:
 - export DEPLOYPATH=/home/yourusername/yourdomain.com/
 - /bin/cp -R public/* $DEPLOYPATH

This copies everything from the public/ folder (Hugo’s build output) into the live site directory.

After pushing this file to GitHub:

Go to Git Version Control → Manage your repo
Click the Pull or Deploy tab
Click Update from Remote to pull the latest
Click Deploy HEAD Commit to trigger the .cpanel.yml tasks

Your site should now be live.

Step 6: Fully Automated Deploys with GitHub Actions

To eliminate the manual “pull and deploy” step in cPanel, I set up a GitHub Actions workflow that SSHs into Namecheap and triggers the pull automatically on every push.

Generate a Deploy Key

On your local machine, generate a key pair with no passphrase:

ssh-keygen -t ed25519 -C "github-actions-deploy" -f ~/.ssh/deploy_key -N ""

Add the public key to Namecheap:

# In cPanel Terminal on Namecheap:
echo "ssh-ed25519 AAAA...your-key-here github-actions-deploy" >> ~/.ssh/authorized_keys

Add Secrets to GitHub

Go to your repo → Settings → Secrets and variables → Actions and add:

Secret	Value
`NC_HOST`	`augmentedresilience.com`
`NC_USER`	Your cPanel username
`NC_PORT`	Your SSH port (check cPanel)
`NC_SSH_KEY`	The full private key (including BEGIN/END lines)

Create the Workflow

Add .github/workflows/deploy.yml to your repo:

name: Deploy to Namecheap

on:
 push:
 branches: [main]

jobs:
 deploy:
 runs-on: ubuntu-latest
 steps:
 - name: Deploy via SSH
 uses: appleboy/ssh-action@v1
 with:
 host: ${{ secrets.NC_HOST }}
 username: ${{ secrets.NC_USER }}
 key: ${{ secrets.NC_SSH_KEY }}
 port: ${{ secrets.NC_PORT }}
 script: |
 cd ~/your-repo && git pull origin main && /bin/cp -R public/* ~/yourdomain.com/

Now every push to main automatically deploys to your live site.

Step 7: One-Command Deploy Script

Five manual commands every time you publish? That’s not a workflow — that’s a chore. I had Claude write a Python script that handles everything in one shot:

#!/usr/bin/env python3
"""One-command deploy: Obsidian → Hugo → GitHub → Live site."""

import subprocess
import sys
from datetime import datetime

PROJECT_DIR = "~/Documents/Augmented-Resilience"
OBSIDIAN_POSTS = "~/projects/obsidian-vault/30-projects/augmented-resilience-posts"
HUGO_POSTS = f"{PROJECT_DIR}/content/posts"


def run(cmd, description, cwd=PROJECT_DIR):
 """Run a command and print status."""
 print(f"\n{'='*50}")
 print(f" {description}")
 print(f"{'='*50}")
 result = subprocess.run(cmd, shell=True, cwd=cwd)
 if result.returncode != 0:
 print(f"\n FAILED: {description}")
 sys.exit(1)
 return result


def main():
 msg = " ".join(sys.argv[1:]) if len(sys.argv) > 1 else f"Site update {datetime.now().strftime('%Y-%m-%d %H:%M')}"

 run(f'rsync -av --delete "{OBSIDIAN_POSTS}" "{HUGO_POSTS}"', "Syncing posts from Obsidian")
 run(f"python3 {PROJECT_DIR}/images.py", "Processing images")
 run("hugo", "Building site with Hugo")
 run("git add .", "Staging changes")

 result = subprocess.run("git diff --cached --quiet", shell=True, cwd=PROJECT_DIR)
 if result.returncode == 0:
 print("\n No changes to commit. Site is up to date.")
 return

 run(f'git commit -m "{msg}"', "Committing")
 run("git push origin main", "Pushing to GitHub")

 print(f"\n{'='*50}")
 print(" DEPLOYED! Your site will be live shortly.")
 print(f"{'='*50}\n")


if __name__ == "__main__":
 main()

Save this as deploy.py in your project root. Now the entire workflow is:

# Default timestamped commit message
python3 deploy.py

# Or with a custom message
python3 deploy.py "Add new blog post about deployment"

The script runs every step in sequence — syncs from Obsidian, converts image links, builds with Hugo, commits, and pushes. If any step fails, it stops immediately so you don’t push a broken build. Combined with the GitHub Actions workflow from Step 6, pushing triggers the auto-deploy to Namecheap. One command, fully live.

Lessons Learned

SSH passphrases break cPanel automation. Always remove the passphrase from keys used by cPanel’s Git Version Control.
Theme names must match directory names. Hugo looks for the theme in themes/<theme-name>/, so the theme value in your config must match exactly.
Don’t clone into the live site directory. Clone to a separate folder and use .cpanel.yml to copy the built files over.
GitHub Actions + SSH is the cleanest auto-deploy for shared hosting. No webhooks, no cron jobs — just a simple SSH action that runs on every push.
Claude Code with PAI made this possible in a single session. From debugging build errors to SSH key troubleshooting to writing GitHub Actions workflows, having an AI pair programmer turned what could have been hours of Stack Overflow rabbit holes into a smooth, guided process.

Tools Used

Hugo — Static site generator
re-terminal theme — Hugo theme
Namecheap — Shared hosting with cPanel
GitHub Actions — CI/CD automation
Claude Code — AI pair programmer
Personal AI Infrastructure (PAI) — Workflow engine
Obsidian — Content authoring
VS Code - Development environment; used as the workspace for Claude Code sessions

Resources

I started a blog…..in 2024 (why you should too) — The YouTube video that inspired me to start this blog
Building a Personal AI Infrastructure (PAI) — Daniel Miessler’s guide to building your own Personal AI system