Rag on

Connecting PAI to NotebookLM via MCP: Your Research Becomes a Live Knowledge Layer

Sun, 15 Mar 2026 00:00:00 +0000

Connecting PAI to NotebookLM via MCP: Your Research Becomes a Live Knowledge Layer

I’ve been using Google’s NotebookLM for a while to manage research. Drop in a PDF, a few URLs, some YouTube transcripts — and suddenly I have a knowledge base I can interrogate with natural language. It answers questions grounded entirely in what I gave it, with citations to the exact source, no hallucinations.

The problem is it’s a separate tool. NotebookLM over here. PAI over there. My research couldn’t feed into my workflows, and my workflows didn’t know my research existed.

The Model Context Protocol changed that.

What MCP Actually Does (the short version)

The Model Context Protocol is a standard that lets AI systems connect to external tools and data sources through a defined interface — think of it as an API contract that any MCP-compatible client (like Claude Code) can speak without needing custom integration code for every new service.

When you wire an MCP server into Claude Code’s configuration, that server’s capabilities become available as tools inside every conversation. It’s not a plugin or a browser extension. It’s a live connection — authenticated, persistent, callable inside the same session where the Algorithm is running.

For NotebookLM, this means the boundary between “my research” and “my AI workflow” effectively disappears.

The Setup

The integration runs through a local MCP server binary at /Users/dsa/.local/bin/notebooklm-mcp. Authentication works through a Chrome browser profile — the server captures your active NotebookLM session (cookies, CSRF token, session ID) and caches it so every subsequent request is already authenticated. One notebooklm-mcp-auth command handles the initial handshake; after that, sessions persist across restarts.

In Claude Code’s configuration, it’s registered as a named MCP server:

"notebooklm": {
 "command": "/Users/dsa/.local/bin/notebooklm-mcp"
}

That’s the entire wiring. Claude Code sees the server at startup, the PAI NotebookLM skill knows how to invoke it, and the connection is live in every session from that point forward.

What the NotebookLM Skill Can Do

With the MCP bridge active, the NotebookLM skill exposes six workflows:

Workflow	What It Does
QueryNotebook	Ask a natural language question; get a citation-backed answer from your notebook sources
ListNotebooks	Show all notebooks with IDs and titles
CreateNotebook	Create a new notebook for a topic or project
AddSource	Add URLs, PDFs, YouTube videos, Google Drive files, or pasted text to a notebook
GenerateAudio	Create a podcast-style audio overview of a notebook’s contents
SyncSources	Refresh stale sources (Drive files, dynamic URLs)

The routing is intent-based, same as every other PAI skill. I don’t address the skill directly — I just describe what I need:

“What does my AI Governance notebook say about data lineage requirements?”

That hits the QueryNotebook workflow, fires the MCP query, and returns an answer with citations to the exact source sections that grounded it.

The Real Benefit: Grounded Answers Inside the Algorithm

Here’s what changes when NotebookLM is callable from inside PAI’s Algorithm.

In the standard PAI research flow, the THINK phase selects capabilities — often Research agents that go out to the web, synthesize content, and return findings. Those findings are model-generated. They’re high quality, but they’re inferences from training data and web retrieval. They can be wrong. They can drift from your actual source material.

NotebookLM answers don’t work that way. Every response is grounded in documents you explicitly added to that notebook. The model is constrained to those sources. It can’t invent facts that aren’t in them. When it tells you that a compliance framework requires a specific control, it points you to the exact paragraph in the exact document where that requirement lives.

When that kind of answer is callable from the THINK phase — as an input to ISC criteria, as evidence in the VERIFY phase, as a reference check in the EXECUTE phase — the entire workflow becomes more reliable. You’re not asking PAI to remember what a standard says. You’re asking it to look it up in the document you provided.

Scenarios Where This Changes Things

AI Governance Certification Study

I’m working through an AI Security & Governance certification — 8 modules, each with detailed technical and regulatory content. The study notes from each module live in my NotebookLM certification notebook.

When I’m reviewing or need to quiz myself, I don’t have to context-switch to the NotebookLM web UI. From inside PAI, I can ask:

“Query my AI Governance notebook: what are the key principles covered in module 3 around model risk management?”

The answer comes back cited to specific sections of the source material. I can follow up immediately within the same workflow. I can ask PAI to generate flashcard prompts based on the cited content. The research stays in NotebookLM where it lives. The workflow stays in PAI where it runs. The MCP bridge connects them without forcing me to copy-paste between tools.

Security Research Accumulation

Every time I add a research paper, a security advisory, or a threat report to a NotebookLM notebook, it becomes a queryable asset in PAI’s research layer. During an OSINT or reconnaissance workflow, instead of relying solely on real-time web retrieval, I can query my curated security research base for context that I’ve already vetted and accumulated.

“Does my security research notebook have anything on SSRF exploitation chains through cloud metadata endpoints?”

That’s my own research library answering me, not a model guessing.

Blog Content Drafting

For this blog — Augmented Resilience — I’m building a notebook that captures posts, ideas, and reader questions. Before drafting a new post, I can query:

“Does my Augmented Resilience notebook have any prior content on MCP integration?”

No more accidentally retreading ground I’ve already covered. No more losing track of connected ideas across posts. The notebook becomes an editorial memory that the Algorithm can access during the build phase.

The Audio Overview Feature Is Worth Its Own Mention

One capability that doesn’t have an obvious parallel in most AI tools: NotebookLM can generate a podcast-style audio overview of an entire notebook. Two AI voices discuss the material in a conversational format — synthesizing themes, surfacing key points, connecting ideas across sources.

Through the GenerateAudio workflow, I can trigger this from PAI:

“Generate an audio overview of my AI Governance notebook”

The result is a produced audio file I can listen to during a commute or while doing something else. It’s NotebookLM’s synthesis capability — which is genuinely impressive at extracting narrative threads from dense technical material — accessible through the same interface I use for everything else.

Knowledge That Compounds

The deeper benefit of this integration isn’t any single query — it’s the compounding effect of building curated notebooks over time and having them available in every PAI session.

Every source I add to NotebookLM becomes part of a retrieval layer that gets richer with every addition. The AI Governance notebook grows as I work through modules. The security research notebook grows as I read papers. The Oracle HCM notebook grows as I document fixes and configurations.

PAI already has a memory system for capturing what I do — completed work, learned patterns, quality signals. NotebookLM handles the complementary layer: the source material that grounds what I know. Together, they’re not two tools running side by side. They’re two layers of the same system — one remembering what I’ve done, the other grounding what I know.

MCP is just the wire between them.

RAG, Agents, and Skills: The Three Pillars Inside My Personal AI

Tue, 24 Feb 2026 00:00:00 +0000

RAG, Agents, and Skills: The Three Pillars Inside My Personal AI

This site — Augmented Resilience — didn’t get built the way most blogs do. There was no staring at blank Hugo config files, no manually hunting down Namecheap SSH docs, no scrambling to remember whether the deploy script needed the public/ folder cleaned before each build.

Instead, I described what I wanted. The AI knew my hosting setup (Namecheap shared hosting), my stack (Hugo with the re-terminal theme), my repo (GitHub, SSH-keyed), and my editor (Obsidian). When a build error surfaced — a theme name mismatch between hugo.toml and the actual directory — it was diagnosed and fixed before I had time to Google it. When the deploy script needed writing, it was scaffolded against my specific environment. When I accidentally left sensitive data in an early draft, it caught it before the commit.

None of that context lived in the prompt. It lived in the infrastructure.

The system behind it is called PAI (Personal AI Infrastructure) — an open-source framework I run locally on top of Claude Code. And the reason it could handle an entire site build end-to-end without constant hand-holding comes down to three architectural pillars: RAG, Agents, and Skills.

What Is PAI?

PAI is an open-source personal AI infrastructure system that runs on top of Claude Code. It’s not a SaaS product — it’s a framework you install on your own machine. The system is built around a central idea: AI systems need structure to be reliable. Like scaffolding supports construction, PAI provides the architectural patterns that make AI assistance consistent, contextual, and capable of compounding over time.

There are 34 skills installed on my system, 17 event hooks, 141 workflows, and a memory system that learns from every interaction. But none of that would matter without three core mechanisms working in concert.

The PAI statusline — live system stats showing version (v2.4), algorithm (ALG:v0.2.25), skill count (SK: 34), workflows (WF: 141), hooks (17), context usage (48%), memory signals (144 ratings), and a rolling quality score trend.

Pillar 1: RAG — Your Personal Knowledge Base In Every Response

RAG (Retrieval-Augmented Generation) is the pattern of retrieving relevant documents from a knowledge store and augmenting the AI’s prompt with that context before generating a response. In enterprise AI, this is how you get a chatbot that can answer questions about your internal policies without hallucinating.

In PAI, RAG is the engine that makes the AI feel like it knows you.

How It Works in PAI

When a session starts, PAI’s hook system loads a foundational context layer: my identity, my name, the current date, and the core behavioral rules (the Algorithm). This is the retrieval index — a lightweight map of everything the system knows how to find.

When I make a request, the system retrieves additional context on demand:

Skills frontmatter — Each of the 34 skills has a description field with a USE WHEN clause. These descriptions load at startup as a routing index. When my request matches a skill’s intent, the full skill content loads. This is retrieval — pulling in the right expertise document for the task.
USER/ context files — There’s a structured personal knowledge base living at ~/.claude/skills/PAI/USER/. It contains my resume, my TELOS life goals, my contacts, my projects, my tech stack preferences. When I ask a question where my professional background is relevant, that context gets retrieved and injected.
MEMORY/ directory — Every session, every correction, every insight gets captured in a structured memory system organized into WORK/, LEARNING/, SIGNALS/, and RESEARCH/ directories. Past work items, completed tasks, and quality signals from previous interactions can all be retrieved to inform the current one.
Hook-injected context — Event hooks fire at specific moments (session start, before each prompt, after tool use) and inject dynamic context — things like the current depth classification, relevant behavioral rules, or system state.

Practical Scenario: Building Augmented Resilience

When I was setting up this site and ran into the Hugo theme mismatch error, here’s what PAI retrieved without me explaining any of it:

My tech stack preferences from the USER context — Hugo, GitHub, Namecheap, Obsidian as editor
The WebSavant skill loaded automatically (matched “Hugo site”, “deployment” intent), bringing with it Hugo-specific knowledge about theme configuration, hugo.toml structure, and build pipelines
My project context from MEMORY — the repo name, the hosting environment, decisions made in prior sessions about the deploy workflow

By the time I described the error, PAI already knew the environment it was debugging. It didn’t need me to explain what kind of hosting I had, which theme I was using, or what my folder structure looked like. The retrieval layer had already assembled that context before a single word of the solution was written.

That’s the difference RAG makes — not smarter AI, but contextually equipped AI.

Pillar 2: Skills — Domain Expertise That Activates Itself

If RAG is how PAI knows context, Skills are how PAI does work. A skill is a self-contained expertise module that activates automatically based on intent, routes to the right workflow, and executes a structured procedure.

Think of each skill as a senior specialist on call — and you never have to explicitly page them.

The Anatomy of a Skill

Every skill follows the same structure:

SkillName/
├── SKILL.md ← Routing layer (loads on invocation)
├── Workflows/ ← Step-by-step execution procedures
│ └── Create.md
│ └── Update.md
└── Tools/ ← CLI automation scripts (TypeScript)
└── Generate.ts

The SKILL.md file has two parts:

YAML frontmatter with a USE WHEN clause — this is how Claude Code knows when to activate the skill
Workflow routing table — once activated, this routes the request to the correct workflow file

The magic is in USE WHEN. Here’s a simplified example from the OracleHCM skill:

description: Expert Oracle HCM Cloud troubleshooting and guidance. USE WHEN user
 mentions Oracle HCM, HCM Cloud, HDL, HCM Data Loader, Journey, Checklist,
 workflow approvals, autocomplete rules, fast formulas, security profiles...

I never have to say “use the Oracle HCM skill.” I just describe my problem in natural language. The intent matching system routes it.

Practical Scenario: Configuring SEO and GEO for augmentedresilience.com

Before the site went live, I needed proper SEO and GEO — Open Graph tags for social sharing, meta descriptions for search, canonical URLs, a sitemap, and schema.org structured data so AI-powered search engines like Perplexity, ChatGPT, and Claude could understand and cite the content accurately. None of that comes configured out of the box with the re-terminal theme. I said:

“Set up SEO and apply Generative Engine Optimization to augmentedresilience.com.”

The system:

Activated the WebSavant skill (matched “SEO” + “GEO” + “site” intent — no skill named, no flags set)
Routed to the SEO and AddSchema workflows inside that skill
Created the correct Hugo partial override at layouts/partials/extended_head.html — the exact injection point the re-terminal theme exposes without touching any theme files
Added the full Open Graph tag set (og:title, og:description, og:image, og:url, og:type) wired to Hugo’s page variables
Injected schema.org JSON-LD for every page type: WebSite and Person on the homepage, Article and BreadcrumbList on every post, and AboutPage on /about — giving AI crawlers a machine-readable knowledge graph of the site
Created robots.txt explicitly permitting GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers — with the sitemap URL wired in
Configured hugo.toml for canonical URL generation and enabled the built-in sitemap output

Without the skill, this is a day of Hugo documentation, schema.org spec-reading, and trial-and-error. With it, the full SEO and GEO stack was complete in a single pass — because the skill had already encoded where everything goes in Hugo’s directory structure, which schema types matter for which page contexts, and how to wire Hugo’s template variables into valid JSON-LD that AI search engines can actually parse.

The 34 Skills I Have Installed

My current skill roster includes tools for Oracle HCM support, security recon, OSINT research, browser automation, art generation, document processing, code generation, red teaming, and more. Each one is a packaged capability that activates without friction.

The system is also designed to be extended — building a new skill means writing a SKILL.md with a USE WHEN clause, a workflow routing table, and the workflow files. The CreateSkill skill handles the scaffolding automatically.

Pillar 3: Agents — Parallel Specialized Brains

Skills handle individual domain expertise. Agents handle scale and specialization when a task is too complex for a single pass or requires multiple perspectives simultaneously.

PAI has a three-tier agent system:

Tier 1: Task Tool Subagents (Internal Workhorses)

These are pre-built specialist agents that skills and workflows invoke internally: Engineer, Architect, Explore, QATester, Pentester, ClaudeResearcher, GeminiResearcher, GrokResearcher, and others.

When I ask PAI to research something deeply, it doesn’t just run one search. It can fan out to multiple research agents simultaneously — Claude, Gemini, and Grok each investigating from different angles — then synthesize the results with a “spotcheck” agent that verifies consistency.

This is parallel processing that would take me hours of manual work, running in minutes.

Tier 2: Named Agents (Persistent Specialists)

Named agents are recurring characters with rich backstories, persistent identities, and unique voices via ElevenLabs text-to-speech. They build relationship continuity across sessions.

My installed named agents include:

Serena Blackwood — Architect. Long-term system design decisions.
Marcus Webb — Engineer. Strategic technical leadership.
Rook Blackburn — Pentester. Security testing with a distinct personality.
Ava Sterling — Researcher (Claude). Strategic deep-dive analysis.
Alex Rivera — Researcher (Gemini). Multi-perspective comprehensive analysis.

When Rook runs a security assessment, he doesn’t just return findings — he announces them in his own voice through my speakers. It sounds minor. It’s not. Distinct voices make it cognitively easier to understand who did what work and why you should trust it.

Tier 3: Custom Agents (On-Demand Compositions)

For tasks that don’t fit a named agent, PAI can compose agents dynamically from trait combinations:

Expertise traits: security, legal, finance, medical, research, technical, creative
Personality traits: skeptical, enthusiastic, analytical, contrarian, meticulous
Approach traits: thorough, rapid, systematic, adversarial, synthesizing

Each unique trait combination maps to a different ElevenLabs voice. A security + adversarial agent gets Callum’s edgy voice. An analytical + meticulous agent gets Charlotte’s precise cadence.

The trait system means I can spin up a custom agent for any edge case without writing a new agent from scratch.

Practical Scenario: Pre-Launch Validation of Augmented Resilience

Before pushing the first real commit to augmentedresilience.com, I wasn’t going to just cross my fingers and run deploy.py. I asked PAI to validate the site was actually ready. What happened next wasn’t a single check — it was a parallel review board.

PAI spawned three agents simultaneously:

Rook Blackburn (Pentester) scanned the entire repo for credentials, API keys, and sensitive data I might have accidentally left in a config file or draft post — and announced his findings in his own voice through my speakers
A QA agent opened the Hugo local preview, walked every page, verified links weren’t broken, images loaded, and the deploy pipeline produced a clean public/ build
A Researcher agent audited the site’s meta tags, Open Graph data, and hugo.toml settings against SEO best practices for a new blog

A fourth spotcheck agent then reviewed all three outputs for conflicts — did Rook’s findings overlap with anything the QA agent flagged? Were there config issues that touched both SEO and security?

The result was a single consolidated pre-launch checklist. Two issues surfaced: a leftover draft post with personal notes still marked draft: false, and a missing og:image tag. Both fixed before the first visitor ever landed.

The site you’re reading right now went live clean because three agents checked it in parallel before I touched the deploy button. That’s the difference between asking a question and deploying intelligence.

When All Three Work Together

The real power of PAI isn’t any single pillar — it’s the composition.

Here’s what happened when I needed to go from “Obsidian draft” to “live on augmentedresilience.com” without a manual process I’d eventually forget or skip.

RAG assembled the context before I described the problem. From MEMORY it already knew: Namecheap shared hosting doesn’t support native git pull — content has to be pushed via FTP through GitHub Actions. It knew the Obsidian vault was the content source, that images.py had to run before hugo to convert image links, and that Python was the right tool for the orchestration script. Not one of those constraints was in my prompt.

Skills handled the architecture. WebSavant recognized a Hugo deployment pipeline request and routed to a workflow already aware of the full delivery chain: sync posts from Obsidian → convert images → hugo build → git commit → git push → GitHub Actions → Namecheap FTP. It knew the sequence. It knew why each step had to happen in that order.

Agents built it. An Engineer agent wrote deploy.py — the script that runs the whole sequence in a single command. An Architect agent designed the GitHub Actions workflow that picks up after the push and handles the Namecheap delivery step automatically. Two agents, two distinct responsibilities, running the job that a solo developer would have spent an afternoon piecing together from Stack Overflow answers.

The result:

python3 deploy.py "Add new post"

That’s it. One command. Every post that has ever gone live on this site — including this one — passed through a pipeline that RAG, Skills, and Agents built together. It’s not running because I set it up manually. It’s running because three systems knew what the job required before I finished explaining it.

The Compounding Effect

What makes this architecture meaningful over time isn’t any single interaction — it’s that the system gets better at helping you with every session.

The MEMORY system captures learnings. The SIGNALS directory tracks your implicit feedback. When something goes wrong, it logs the full context under LEARNING/FAILURES. When a workflow produces a 9-10 response, that signal is captured too. The system adjusts.

Generic AI starts fresh every time. PAI compounds.

I’m still early in this — the personal profile files still have template placeholders I haven’t filled in, and there are skills I’ve barely touched. But even at partial configuration, the system already thinks more like a senior colleague than a search engine.

That’s what RAG, Agents, and Skills make possible together: an AI that knows your context, activates the right expertise automatically, and scales parallel intelligence for complex work — all without you having to manage the machinery.

When Your PDF Workflow Breaks - Building a Markdown Converter with Claude Code

Wed, 18 Feb 2026 00:00:00 +0000

The Problem: PDFs Are Knowledge Prisons

You know that feeling when you download a brilliant research paper, only to realize you can’t easily feed it into your AI workflow? Or when you want to add documentation to your knowledge base, but it’s locked in a format that doesn’t play well with version control or LLM tools?

Yeah, I was there last week.

I had just downloaded a fascinating 1.3MB research paper on Generative Engine Optimization and wanted to process it with my AI tools. But PDFs are terrible for this. They’re designed for printing, not for processing. What I needed was Markdown—clean, portable, AI-friendly Markdown.

So I built a converter. And with Claude Code as my copilot through the PAI (Personal AI Infrastructure) system, the whole thing took less than 30 minutes.

Here’s how it went down.

Why Markdown is Better Than PDF for LLMs

Before diving into the build, let’s answer the obvious question: why bother converting? Can’t LLMs just read PDFs directly?

Technically, yes. But the results are significantly worse, and the reasons are fundamental to how PDFs work.

PDFs Are Layout-First, Not Structure-First

PDFs were designed to describe where things appear on a page, not what they mean. As Steven Howard explains in Why PDFs Fail Under LLM Parsing :

“Table cells with wrapped text insert hard line breaks that fragment token continuity and break logical row recognition. Headers and footers simply add noise to the context when used with LLMs. Sentences are split with arbitrary CR/LFs making it very difficult to find paragraph boundaries.”

This architectural mismatch — a format designed for printing being fed into a system designed for understanding — causes cascading problems downstream.

The Token Efficiency Problem

Every token your LLM processes costs money and consumes context window space. PDF extraction wastes both.

According to analysis from MarkdownConverters , Markdown saves up to 70% more tokens compared to extracted PDF text for the same content. The culprit: PDF extraction introduces formatting artifacts, metadata noise, headers/footers, and encoding remnants that all consume tokens without adding semantic value.

To put that in practical terms: a PDF that would use 10,000 tokens might only need 3,000 tokens when properly converted to Markdown. At scale, this compounds dramatically.

The RAG Performance Problem

If you’re building Retrieval Augmented Generation (RAG) systems — using documents as a knowledge base for AI — document format directly impacts answer quality.

The research here is compelling:

Academic validation: A 2024 paper on arXiv (Revolutionizing RAG with Enhanced PDF Structure Recognition ) found that “the low accuracy of PDF parsing significantly impacts the effectiveness of professional knowledge-based QA.”
Industry validation: NVIDIA’s technical blog documents how their NeMo Retriever pipeline converts extracted content to Markdown specifically because it “preserves row/column relationships in an LLM-native format, significantly reducing numeric hallucination” — and reduces incorrect answers by 50%. (NVIDIA: Approaches to PDF Data Extraction for Information Retrieval )
Chunking quality: Analysis from Towards Data Science shows that Markdown’s heading structure (#, ##, ###) produces semantically meaningful chunks, while PDF-based chunking relies on arbitrary page breaks and heuristics.
Retrieval failure rates: Unstructured.io’s research on contextual chunking — tested across 5,563 question-answer pairs — showed an 84% reduction in retrieval failure rates when using structure-aware chunking (the kind Markdown enables natively).
Real-world outcomes: The 2025 Semrush AI Index, cited by Webex Developers Blog , found that 72% of top AI-indexed articles used Markdown or Markdown-like structures, achieving 34% higher retrieval accuracy across ChatGPT, Perplexity, and Gemini.

The Bottom Line

Metric	Impact
Token reduction	Up to 70% fewer tokens vs PDF extraction
Incorrect answers in RAG	50% reduction (NVIDIA NeMo)
Retrieval failure rates	84% reduction (Unstructured.io)
Retrieval accuracy	34% higher (Semrush AI Index 2025)

Markdown isn’t just more convenient — it’s meaningfully better for AI. Converting your document libraries is one of the highest-ROI steps you can take before building any LLM-powered workflow.

The First Failure: When Bleeding-Edge Python Bites Back

I’m running Python 3.14.2—the latest release, barely a few weeks old. Modern, shiny, cutting-edge. Perfect, right?

Not quite.

My first instinct was to use marker-pdf, a high-performance converter optimized for scientific papers and books. It looked perfect on paper (pun intended). But when I tried to install it:

Building wheel for Pillow (pyproject.toml): finished with status 'error'

Ugh.

Turns out, marker-pdf depends on Pillow (the Python imaging library), and Pillow hasn’t built binary wheels for Python 3.14 yet. I could have downgraded Python. I could have fought with source compilation. But why?

This is where working with Claude Code really shines. Instead of going down a rabbit hole trying to force marker-pdf to work, Claude suggested pivoting to PyMuPDF4LLM—a mature, actively maintained library specifically designed for AI/LLM workflows.

And it just worked.

The Solution: PyMuPDF4LLM

PyMuPDF4LLM turned out to be exactly what I needed:

Works flawlessly with Python 3.14 (no compilation errors)
Fast and accurate conversion
Built specifically for feeding documents into LLMs
Clean, simple API
Actively maintained by the PyMuPDF team

The installation was literally:

pip install pymupdf4llm

Five seconds later, I was ready to go.

Building the Tool: First Principles Thinking

As someone new to the CLI world, I’ve been learning to think through project structure from first principles. Where should this live? How should it be organized?

With Claude’s guidance, I chose /Users/dsa/projects/pdf-to-markdown/ for a few key reasons:

Separation of Concerns: Tool projects should be separate from my main workspace
Discoverability: Clear, descriptive naming means I’ll find it again in 6 months
Reusability: This structure works both as a CLI tool AND as a library I could import later

The project structure ended up simple but complete:

pdf-to-markdown/
├── README.md # Documentation
├── venv/ # Isolated Python environment
├── input/ # Test PDFs
├── output/ # Generated markdown
├── pdf2md # CLI wrapper script
└── requirements.txt # Dependencies

The Code: A Simple but Powerful CLI

I wanted a tool I could actually use—something with a clean command-line interface that handles the common cases elegantly. Working with Claude through PAI, we created a Python script that does exactly that:

#!/usr/bin/env python3
"""
PDF to Markdown Converter
A simple CLI tool to convert PDF files to Markdown using PyMuPDF4LLM
"""

import sys
import os
from pathlib import Path
import pymupdf4llm
import pymupdf
from tqdm import tqdm

def convert_pdf_to_markdown(pdf_path: str, output_path: str = None) -> str:
 """Convert a PDF file to Markdown format."""

 if not os.path.exists(pdf_path):
 raise FileNotFoundError(f"PDF file not found: {pdf_path}")

 # Get page count for progress bar
 doc = pymupdf.open(pdf_path)
 page_count = doc.page_count
 doc.close()

 print(f"Converting: {pdf_path}")
 with tqdm(total=page_count, unit="page", desc="Processing", colour="blue") as bar:
 md_text = pymupdf4llm.to_markdown(pdf_path, page_chunks=False)
 bar.n = page_count
 bar.refresh()

 if output_path is None:
 output_path = Path(pdf_path).with_suffix('.md')

 with open(output_path, 'w', encoding='utf-8') as f:
 f.write(md_text)

 print(f"✓ Done: {output_path} ({len(md_text):,} characters)")
 return str(output_path)

def batch_convert(input_dir: str, output_dir: str = None) -> None:
 """Convert all PDFs in a directory to Markdown."""
 input_path = Path(input_dir)
 if not input_path.is_dir():
 raise NotADirectoryError(f"Not a directory: {input_dir}")

 pdfs = sorted(input_path.glob("*.pdf"))
 if not pdfs:
 print(f"No PDF files found in: {input_dir}")
 sys.exit(0)

 if output_dir:
 output_dir = Path(output_dir)
 else:
 output_dir = input_path.parent / "output"
 output_dir.mkdir(parents=True, exist_ok=True)

 total = len(pdfs)
 succeeded = 0
 failed = 0

 print(f"\nBatch mode: {total} PDF(s) found in '{input_dir}'")
 print(f"Output folder: {output_dir}\n")

 for i, pdf_path in enumerate(pdfs, start=1):
 print(f"[{i}/{total}] {pdf_path.name}")
 output_path = output_dir / pdf_path.with_suffix('.md').name
 try:
 convert_pdf_to_markdown(str(pdf_path), str(output_path))
 succeeded += 1
 except Exception as e:
 print(f" ✗ Failed: {e}")
 failed += 1
 print()

 print("─" * 40)
 print(f"Batch complete: {succeeded} converted, {failed} failed")
 print(f"Output folder: {output_dir}")

def main():
 """Main CLI entry point"""
 args = sys.argv[1:]

 if not args:
 print("Usage:")
 print(" pdf2md <input.pdf> [output.md] # Convert a single PDF")
 print(" pdf2md --batch <folder/> # Convert all PDFs in a folder")
 print(" pdf2md --batch <folder/> --output <out_folder/> # Batch with custom output dir")
 print("\nExamples:")
 print(" pdf2md document.pdf # Creates document.md")
 print(" pdf2md document.pdf custom.md # Creates custom.md")
 print(" pdf2md --batch input/ # Converts all PDFs in input/")
 print(" pdf2md --batch ~/documents/pdfs/ --output ~/knowledge-base/docs/")
 sys.exit(1)

 if args[0] == "--batch":
 input_dir = args[1]
 output_dir = None
 if "--output" in args:
 idx = args.index("--output")
 output_dir = args[idx + 1]
 batch_convert(input_dir, output_dir)
 else:
 pdf_path = args[0]
 output_path = args[1] if len(args) > 1 else None
 convert_pdf_to_markdown(pdf_path, output_path)

if __name__ == "__main__":
 main()

What I love about this code:

Smart defaults: If you don’t specify an output path, it just replaces .pdf with .md
Progress bars: tqdm gives you a blue progress bar with page count
Batch mode: --batch processes an entire folder at once, with optional --output target
Helpful errors: Clear messages when things go wrong
Flexible usage: Works with relative paths, absolute paths, custom output names

Make it executable:

chmod +x pdf2md

And now it’s a proper command-line tool.

The Moment of Truth: Testing with Real Data

Theory is great. But does it actually work?

I grabbed that 1.3MB research paper on Generative Engine Optimization and ran:

python pdf2md input/test.pdf output/test.md

The output:

Converting input/test.pdf to Markdown...
Processing: 100%|████████████████| 12/12 [00:02<00:00, 5.8 pages/s]
✓ Done: output/test.md (73,463 characters)

1.3MB PDF → 74KB of clean Markdown in seconds.

I opened the output file, and there it was—perfectly formatted markdown:

## **GEO: Generative Engine Optimization**

Pranjal Aggarwal [∗]
Indian Institute of Technology Delhi
New Delhi, India
pranjal2041@gmail.com

Ashwin Kalyan
Independent
Seattle, USA
asaavashwin@gmail.com
...

Headers, formatting, structure—all preserved. No manual cleanup needed.

Success.

What This Unlocks

Now that I have PDFs converting to Markdown reliably, a whole world of possibilities opens up:

AI Workflows

Feed research papers and documentation directly into Claude or other LLMs
Build RAG (Retrieval Augmented Generation) pipelines backed by your document library
Process technical documentation at scale without losing structure

Knowledge Management

Import PDFs into your Obsidian vault automatically
Version control document content (because it’s now plain text in git)
Full-text search across your entire converted document library

Automation Ideas

Watch folder that auto-converts any dropped PDFs
Batch process entire directories of reports, papers, or manuals
Feed converted markdown directly into a vector database
API wrapper to convert PDFs via HTTP requests

Lessons Learned (Especially for CLI Beginners)

1. Virtual Environments Are Non-Negotiable

Every Python project should live in its own virtual environment. Always:

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip

This keeps dependencies isolated and projects reproducible.

2. Bleeding-Edge Isn’t Always Better

Python 3.14 is awesome, but sometimes mature tooling (like PyMuPDF) that “just works” beats bleeding-edge alternatives. Don’t be afraid to pivot when something doesn’t work.

3. Test With Real Data

I didn’t test with “hello.pdf” containing two sentences. I tested with a 1.3MB research paper. Real data reveals real issues (or in this case, confirms it works beautifully).

4. Document As You Build

Writing the README alongside the code made the project immediately understandable. Future-me will thank present-me.

5. Claude Code + PAI = Superpowers

Working with Claude through the PAI infrastructure meant I had a senior developer helping me think through:

Project structure (first principles)
Library selection (when to pivot)
Code organization (clean, maintainable)
Real-world usage patterns

This wasn’t just coding faster—it was learning better patterns while building.

Usage Examples

Basic Conversion

# Activate environment first (always!)
source venv/bin/activate

# Convert a PDF
python pdf2md document.pdf

# Custom output name
python pdf2md research.pdf my-notes.md

# Full paths
python pdf2md ~/Downloads/paper.pdf ~/Documents/notes.md

Batch Processing

Convert an entire folder of PDFs:

source venv/bin/activate

# Convert all PDFs in a folder (output goes to output/ by default)
python pdf2md --batch ~/documents/pdfs/

# Convert to a specific knowledge base directory
python pdf2md --batch ~/documents/pdfs/ --output ~/knowledge-base/docs/

Add to PATH (Optional)

To use pdf2md from anywhere:

# Add to ~/.zshrc
export PATH="/Users/dsa/projects/pdf-to-markdown:$PATH"

# Then run from anywhere
pdf2md ~/Downloads/paper.pdf ~/Documents/paper.md

What’s Next?

This tool works great as-is, but there are some exciting enhancements on the roadmap:

Immediate Improvements

Better layout analysis: Install pymupdf_layout for improved structure detection on complex documents
Recursive batch mode: Process nested folder structures, not just flat directories

Future Integrations

RAG pipeline: Auto-feed converted markdown into a vector database
Obsidian plugin: Detect PDFs in vault and convert automatically
FastAPI wrapper: Create an HTTP API for web apps to use
Electron/Tauri app: Build a desktop GUI for non-technical users

The Bigger Picture: Why This Matters

This project is tiny—roughly 100 lines of Python, 30 minutes of work. But it represents something bigger:

The ability to build tools that solve your actual problems.

I had a workflow friction (PDFs don’t work well with AI tools). I built a solution. Now that friction is gone, and I can focus on higher-level work.

And the data is clear: converting your document library to Markdown isn’t a nice-to-have. It’s a multiplier on every AI workflow that follows. Up to 70% fewer tokens consumed. 84% fewer retrieval failures. 50% fewer incorrect answers. These aren’t marginal improvements—they’re transformational.

Working with Claude Code through PAI accelerated all of this. It’s like having a patient senior developer sitting next to you, suggesting better approaches, catching errors before they happen, and explaining why certain patterns work.

Resources

PyMuPDF4LLM Docs: https://pymupdf.readthedocs.io/en/latest/pymupdf4llm/
PyMuPDF GitHub: https://github.com/pymupdf/PyMuPDF

Citations: Markdown vs PDF for LLMs

Why PDFs Fail Under LLM Parsing — Steven Howard, Untethered AI: https://untetheredai.substack.com/p/why-pdfs-fail-under-llm-parsing
PDF vs Markdown for AI: Token Efficiency — MarkdownConverters: https://markdownconverters.com/blog/pdf-vs-markdown-ai-tokens
Revolutionizing RAG with Enhanced PDF Structure Recognition — arXiv:2401.12599 (2024): https://arxiv.org/abs/2401.12599
Approaches to PDF Data Extraction for Information Retrieval — NVIDIA Technical Blog: https://developer.nvidia.com/blog/approaches-to-pdf-data-extraction-for-information-retrieval/
Improved RAG Document Processing With Markdown — Dr. Leon Eversberg, Towards Data Science: https://medium.com/data-science/improved-rag-document-processing-with-markdown-426a2e0dd82b
Contextual Chunking: Boost Your RAG Retrieval Accuracy — Unstructured.io: https://unstructured.io/blog/contextual-chunking-in-unstructured-platform-boost-your-rag-retrieval-accuracy
Boosting AI Performance: The Power of LLM-Friendly Content in Markdown — Webex Developers Blog: https://developer.webex.com/blog/boosting-ai-performance-the-power-of-llm-friendly-content-in-markdown

Happy converting!