Claude-Code on

Your AI Isn't Going Off the Rails. It Never Had Any.

Sun, 31 May 2026 00:00:00 +0000

Your AI Isn’t Going Off the Rails. It Never Had Any.

The most common complaint I hear from people using AI is some version of the same sentence: “It goes off the rails.” It drifts. It forgets what I told it. It invents things. It has no direction.

I understand the frustration, but the phrasing hides the actual problem. Going off the rails implies there were rails to begin with. There weren’t. A model in a blank chat window has no memory of who you are, no rules about how it should behave, and no defined process for doing the work. It is not drifting away from a plan. There was never a plan for it to drift from.

So when people tell me their AI lacks direction, what they are really describing is a missing system. The fix is almost never a better prompt. It is more structure. And the amount of structure you give the AI is the single biggest variable in whether it behaves like a reliable partner or a clever stranger who resets every morning.

The clearest way I have found to explain this is as three tiers. Each one solves a bigger slice of the “no direction” problem than the last.

Tier 1: Claude Chat — The Conversation

This is where almost everyone starts, and where most people stay. You open a chat window, you type, it responds. Each conversation is mostly a blank slate.

The defining trait of this tier is amnesia. A new chat forgets everything. Whatever context you want the model to have, you provide manually, in the prompt, every single time. The direction comes entirely from you. The model cannot touch your files, run anything, or reach your systems. It talks, and that is all it does.

This is genuinely useful. For a quick question, a brainstorm, a first draft, or thinking out loud, a chat window is fast and frictionless. But it is also exactly why it feels directionless on anything bigger. Nothing constrains it. There are no rules, no persistent goal beyond your last message. If your prompt is vague, the output is vague. It is a brilliant intern with total amnesia, and you are re-explaining the entire job every morning.

People at this tier blame the model. The model is rarely the issue. The issue is that nothing is holding it on a track, because there is no track.

Tier 2: Claude Code — The Operator

The second tier is a different kind of tool entirely. Claude Code is an agent that lives in your terminal. It does not just talk about work — it does the work. It reads and writes real files, runs commands, searches the web, and operates on your actual environment instead of an imagined one.

Two things change the moment you move here.

First, it gets a working memory of your project. A CLAUDE.md file holds persistent instructions for that codebase or project, so the model arrives already knowing the conventions, the goals, and the rules you have written down. You stop re-explaining the project on every session.

Second, and more importantly, it works in a loop: act, observe the result, correct. It writes a file and sees whether the change worked. It runs a command and reads the actual output. That feedback loop is what kills the drift. The model is not imagining what might happen — it is looking at what did happen and adjusting. Operating on real artifacts instead of guesses is most of the discipline.

The honest limitation: this discipline is per-project and largely manual. You set up each project’s instructions yourself. The memory does not follow you from one project to the next, and there is no consistent persona or process spanning everything you do. It is a sharp, capable operator — but one you have to brief fresh for every new job.

Tier 3: AI Infrastructure

The third tier is the one that actually fixes “no direction” at the root, because it stops treating each session as a fresh start.

AI Infrastructure is a system that wraps Claude Code and gives it a permanent identity, a rule set, a knowledge base, and a defined process. The jump from Tier 2 to Tier 3 is the difference between hiring a contractor and building an operations department. This post itself is a small example: it was written inside an AI Infrastructure that already knew my blog’s voice, my formatting conventions, and where the file should be saved, without my having to say any of it.

Three things work together here, and they are what make drift structurally hard.

The first is persistent identity and memory. The infrastructure does not forget who I am, what I work on, or how I want things done. When I correct it, the correction sticks across every future session, not just the current chat. The knowledge lives in files I own, not inside a conversation that disappears when I close the tab.

The second is a defined process. Every non-trivial request gets classified and routed through a structured sequence: understand the request, plan the approach, do the work, verify it against explicit criteria. The model cannot freewheel, because a process governs the response before it starts. That is the literal opposite of going off the rails — the rails are built in.

The third is context routing. Instead of me pasting the right background into every prompt, the system pulls the relevant knowledge automatically based on what I am doing. The model arrives oriented, every time.

None of this makes the underlying model smarter. It makes the environment around the model disciplined. That is the whole trick.

The Same Symptom, Mapped to the Fix

When someone describes their AI as directionless, the specific complaint usually tells you exactly which tier they are stuck on and what would move them up.

If it forgets what you told it, you are in a chat window and you need persistent instructions — that is the move to Claude Code.

If it hallucinates instead of using your real data, you need to let it read your actual files — again, the move to an operator that touches your environment.

If you keep re-explaining your preferences across projects, you have outgrown per-project memory and need persistent identity — the move to infrastructure.

And if it has no consistent process from one task to the next, you need a defined algorithm governing how every request gets handled — the same move.

Notice the pattern. Every one of these is solved by adding structure, not by writing a cleverer sentence into the prompt box.

The Real Shift

Direction is not something you nag the AI for inside each prompt. It is something you build into the system once.

That reframing is the entire jump from chatting to infrastructure. A chat window is equally capable on day one and day three hundred, because nothing accumulates. An operator gets more useful per project, as long as you keep briefing it. An infrastructure compounds — every rule you add, every preference it learns, every process you refine makes the next session start further ahead than the last.

If your AI feels like it has no direction, it is not malfunctioning. It is doing exactly what an unstructured system does. The rails were never the model’s job to build. They are yours.

If a Script Can Do It, Don't Ask the LLM

Mon, 20 Apr 2026 00:00:00 +0000

If a Script Can Do It, Don’t Ask the LLM

When your AI system is spending inference tokens to parse date fields, strip PDF formatting artifacts, or convert file syntax — that is not an AI problem. That is an architecture problem. Code can handle all of those tasks deterministically, faster, cheaper, and more accurately than any language model. If your token bill is high and your outputs are inconsistent, the problem is usually not the model. It is what you are handing the model.

PAI treats every token as a resource allocation decision. If a script can do it, a script does it. The model gets the part that actually requires reasoning.

The Structural Tax

Most AI workflows are built around convenience. You have a PDF — you upload it. You have a session log — you paste it in. You have a directory of files — you point the model at the folder. The model figures it out.

That approach works well enough for one-off tasks. It breaks down at scale, in recurring workflows, and anywhere precision matters.

Raw PDFs carry structural overhead that has nothing to do with the content you care about: page headers, footers, column artifacts, encoding noise, redundant whitespace, and metadata that made sense in the original layout but becomes garbage in a text stream. A language model consuming that document is spending a meaningful portion of its context window on formatting debris. That cost compounds across every document in every workflow.

The same pattern shows up everywhere. Raw session transcripts contain tool call metadata, timestamps, and infrastructure details that are irrelevant to extracting what was learned. Full database contents are too large to fit in context, so the model guesses at relevance instead of reasoning over curated results. Obsidian-flavored Markdown with custom image syntax breaks rendering in any system that was not built for it.

Every one of these is a structural problem. None of them require intelligence to solve. They require code.

What PAI Does Instead

PAI operates on a two-phase model: scripts handle structure, the LLM handles reasoning. The handoff only happens after the mechanical work is done.

A few examples of how this plays out in practice.

PDF to Markdown conversion. When I ingest Oracle HCM documentation — dense, multi-column PDFs that can run hundreds of pages — nothing goes to the LLM in PDF form. A Python script converts the document to clean Markdown first, stripping formatting artifacts and preserving the content hierarchy. The model sees structured text, not a scanned layout. The difference in token consumption is substantial, and the difference in output quality is larger still.

Session harvesting. Claude Code sessions produce raw JSON transcripts containing every tool call, every response, and a lot of infrastructure noise. SessionHarvester.ts parses those transcripts before any analysis happens. It extracts structured learning signals — what worked, what failed, what decisions were made — and writes them to a clean format. The LLM receives a curated summary, not a raw dump.

Activity parsing. ActivityParser.ts does the same for daily activity logs: reads session files, extracts structured change records, and produces a clean representation of what changed in the PAI system. No model inference required until there is something worth reasoning about.

Knowledge base search. The cybersecurity knowledge base holds over fifty books worth of content in a local PostgreSQL vector store. When a query comes in, CyberSecKB.ts runs a similarity search and returns the most relevant passages. The LLM reasons over that curated result set — not the entire library. Retrieval is deterministic. Reasoning is probabilistic. They are separate steps by design.

Image pipeline. Obsidian uses ![Image Description](/images/filename.png) syntax for embedded images. Hugo uses standard Markdown. images.py handles the conversion and copies files to the correct static directory at publish time. No LLM is involved at any point in that pipeline. It is a string transformation and a file copy — exactly the kind of task code should own.

The Boundary Is the Design

The two-phase pattern is not a workaround. It is the architecture.

Every AI workflow has a boundary between deterministic computation and probabilistic reasoning. The question is where you draw it. Drawing it too early means the model is doing structural work it is not suited for: burning tokens on format conversion, wasting context on irrelevant metadata, producing inconsistent output because the input was inconsistent. Drawing it correctly means the model receives exactly what it needs to do the work only it can do.

This is a different problem than the one prompt engineering was designed to solve. Prompt engineering is about communicating intent clearly to a model. The code-first pattern is about not asking the model to earn the right to see your data by first organizing it. That work should already be done.

The scripts in PAI are not wrappers around LLM calls. They are the preparation layer — the part of the system that exists specifically so the model does not have to deal with structure. Once that layer is in place, the model’s job gets narrower and cleaner. It reasons. It synthesizes. It makes judgment calls. It does not convert file formats.

The Result

When the pipeline is built correctly, token consumption goes down and output quality goes up — not because the model improved, but because the input did. The model is working with clean data in a consistent format, scoped to exactly what is relevant. The structural variables are removed before inference begins.

That is the return on building the prep layer. Less waste. More signal. A model that can focus on the part of the problem that actually requires a model.

AI Writes Code Fast. Here's Why Security-First Thinking Matters More Than Ever.

Sun, 19 Apr 2026 00:00:00 +0000

Speed Is the Feature. Unexamined Speed Is the Liability.

One of the most impressive things about using AI to write code is how fast it moves. You describe an endpoint, a data model, a feature — and within seconds you have working code. Not a sketch. Not pseudocode. Actual, runnable implementation.

That speed is real, and it is genuinely useful. I use it every day.

But here is something I noticed the longer I worked with AI-generated code: it writes to the happy path. AI is extraordinarily good at making code that works when everything goes as expected. It is considerably less reliable at writing code that stays safe when things go wrong — when someone sends unexpected input, when a dependency is compromised, when a secret accidentally surfaces in a log, when a logged-in user tries to access someone else’s data.

This is not a criticism of AI models. It is a structural observation about how code generation works. AI learns from what code looks like, not from what happens to systems that run it. The attack surface is invisible at generation time.

Security-first thinking is the discipline that fills that gap. And building it into how you work with AI is one of the highest-leverage things you can do as a developer.

What AI Gets Wrong By Default

Before we get to principles, it helps to see the failure modes concretely. These are patterns I started noticing after reviewing AI-generated code more carefully.

Hardcoded secrets. Ask AI to write a function that connects to a database or calls an external API, and it will often produce something like this:

db = connect(host="prod-db.company.com", user="admin", password="Sup3rS3cr3t!")
client = OpenAI(api_key="sk-proj-abc123...")

The code works. It will also put your credentials in git history forever. A secret committed to a repository — even briefly, even to a private repo — must be treated as compromised. Git history is permanent. The only remediation is rotation.

Missing input validation. AI tends to write code that trusts request data. An endpoint that receives req.body.quantity will often use it directly, skipping the check for whether it is a positive integer, whether it is within expected bounds, whether it contains what the code assumes it contains.

Fetch-then-check authorization. This one is subtle. AI commonly writes authorization logic like this:

const order = await db.orders.findById(req.params.id);
if (order.userId !== req.user.id) return res.status(403).send();
return res.json(order);

That looks right. But it fetches the record first and checks ownership second. A slightly better pattern is to include the user ID in the query itself, so that non-owned records simply return null — and you return a 404, not a 403. Returning 403 confirms to an attacker that the resource exists. It is a small thing that compounds at scale.

Weak cryptography by familiarity. Ask AI to hash a password and it might reach for SHA-256. SHA-256 is a solid hash function — for data integrity. It is the wrong tool for passwords because it is fast. Fast means a GPU can test billions of candidate passwords per second against a leaked hash. bcrypt and Argon2 are deliberately slow. That slowness is the security property.

No rate limiting on authentication. AI generates login endpoints without the defensive scaffolding that should always accompany them: rate limiting per IP, account lockout after repeated failures, uniform response times to prevent user enumeration.

None of these are exotic edge cases. They are the everyday failure modes that show up in security audits and breach post-mortems, again and again.

The Mindset Shift: From Checklist to First Principles

Here is where I want to push back against the usual framing. Security is often presented as a checklist — OWASP Top 10, compliance requirements, “use HTTPS.” Checklists have their place, but they do not produce secure code. They produce code that passes the checklist.

Security-first thinking is different. It is a way of looking at code that asks one question at every boundary: who or what is being trusted here, and has that trust been earned?

Every class of vulnerability — SQL injection, XSS, CSRF, IDOR, broken authentication, insecure deserialization — reduces to a single failure: untrusted data crossing a trust boundary with the authority of trusted data.

SQL injection happens when user input is trusted to be SQL-safe before it reaches the database. XSS happens when user-generated content is trusted to be display-safe before it enters the DOM. IDOR happens when a request parameter is trusted to represent a resource the current user is allowed to access.

Once you see security through this lens — trust boundaries and their enforcement — you stop asking “did I remember the checklist item?” and start asking “where are the trust boundaries in this code, and what enforces them?” That question applies to every system, every language, every framework. It does not go stale.

This is what I mean by security-first thinking as a mindset rather than a procedure. It is not about knowing rules. It is about developing the habit of seeing trust assumptions in code.

The Eight Principles

When I worked through this with Luna recently — building a security knowledge base from 50 cybersecurity books — we distilled the trust boundary framework down to eight irreducible properties that AI-generated code must satisfy.

I want to walk through each one because they are not independent rules. They are facets of the same underlying principle.

1. Input is untrusted by default. Every value arriving from outside your process — HTTP body, query parameters, headers, cookies, file uploads — is hostile until validated. Client-side validation is UX. Server-side validation is security. They cannot substitute for each other.

2. Output is encoded for its context. A string displayed in HTML needs HTML encoding. A value interpolated into SQL needs parameterization. A value passed to a shell command needs to go through an argument array, not string concatenation. The encoding is not about the string itself — it is about the context it will be interpreted in.

3. Secrets never live in code. Passwords, API keys, tokens, certificates — these belong in environment variables or a secrets manager, never in source files. This is absolute. Git history is permanent.

4. Authentication is verified, not assumed. Every protected route, every protected function, checks identity on that request. Being logged in at some prior point does not carry forward.

5. Authorization is checked at the data layer. Being authenticated is not the same as being authorized to access a specific resource. Ownership checks belong in the query itself, not as a post-fetch condition.

6. Least privilege is the default. Database connections, service accounts, IAM roles, file operations — each gets only the access its specific function requires. Nothing more. A compromised least-privilege component fails safely. A compromised over-privileged one does not.

7. Cryptography is never homegrown. Cipher and hash choices go to battle-tested libraries: bcrypt or Argon2 for passwords, AES-GCM for encryption, HMAC-SHA256 for message authentication. The failure modes of hand-rolled crypto are subtle and catastrophic.

8. Dependencies are trust extensions. Every package you add is code you are trusting. Its transitive dependencies are code you are trusting. Supply chain attacks are real. Lock files, audits, and version pinning are not paranoia — they are basic hygiene.

These eight properties are not a checklist to run through at the end. They are filters to apply while generating code. The question during every code-writing session is: which of these are relevant to what I am building right now?

Baking It Into Your AI Workflow

Knowing principles is not the same as applying them consistently. The reason I started building infrastructure around this is that consistency requires more than intention — it requires systems.

Here is what I built, and it came directly from the work of turning 50 cybersecurity books into a searchable knowledge base.

Tier 1 topic files. Each of the eight principles above now lives in a concise reference file — 8 to 12KB each — distilled from the source material. input-validation.md, secrets-management.md, authentication.md, authorization.md, cryptography.md, and the others. Each file is dense but scannable: concrete patterns, code examples, anti-patterns, quick checklists.

Engineer PREFERENCES. I wired those topic files into the Engineer agent through a PREFERENCES file. The trigger table maps code categories to topic files: anything involving SQL gets xss-injection-prevention.md loaded. Anything involving authentication loads authentication.md. Anything involving secrets loads secrets-management.md. The agent loads the relevant files silently before writing code and runs the associated checklist before declaring work done.

The result is that security context is present in every relevant code-generation session without me having to ask for it. The principles are not something I have to remember to invoke. They are part of the workflow infrastructure.

This is what I keep coming back to with PAI: the value is not in any single AI interaction. It is in building infrastructure that makes the right thing the default thing. Security-first code used to require conscious effort and specialized knowledge in the moment. Now it is loaded context — always present, reliably applied.

Security and AI Are Not in Tension

I want to end on this because it is easy to come away from a security conversation feeling like the message is “AI is dangerous, slow down, be careful.” That is not the message.

AI writing code fast is a genuine capability improvement. Being able to go from specification to working implementation in minutes changes what is possible for a small team or a solo developer. That is real.

Security-first thinking is not a constraint on that capability. It is the thing that makes the output of that capability trustworthy. Code that moves fast and ships vulnerable systems is not actually faster in any meaningful sense — it is accumulating debt that will be paid with interest.

The combination is the point. AI that knows how to think about trust boundaries, that loads security context automatically, that applies principles rather than just patterns — that is a force multiplier, not a liability. You get the speed and you get the rigor.

Building toward that combination is one of the more interesting engineering problems I have worked on. The knowledge base, the topic files, the wiring into the workflow — it is all aimed at the same thing: making security-first thinking something that happens automatically, not something that requires a specialist in the room.

The specialist knowledge exists. We built it into 77KB of reference material drawn from 50 books. Now it is always in the room.

Part of the PAI series on building infrastructure that makes AI more useful, more reliable, and more trustworthy. The security knowledge base and topic files referenced in this post were built using Claude Code, PostgreSQL, pgvector, and source material from O’Reilly’s security catalog.

Stop Prompt Engineering. Start Building Infrastructure.

Sun, 19 Apr 2026 00:00:00 +0000

Stop Prompt Engineering. Start Building Infrastructure.

Last week I opened a terminal, typed six words, and watched PAI spend the next three minutes processing a set of handwritten study notes exported from my reMarkable tablet. It converted the file format, extracted key concepts, generated structured review questions, cross-referenced my existing knowledge base, and saved everything to the correct directories in Obsidian — organized by module, tagged correctly, ready to use. I did not write a prompt. I did not explain what certification I was studying. I did not describe the output structure I wanted. I just named the module.

Eighteen months ago, that same task would have started with a paragraph explaining what the reMarkable export format was, what the certification covered, how I organized notes in Obsidian, what level of detail I wanted in the summary, and what format the quiz questions should follow — and another paragraph if I wanted the output saved to a specific location. Every single session. From scratch.

That gap is the entire argument for building an AI harness instead of staying in a chat window.

The Chat Window Tax

Prompt engineering emerged as a discipline because LLMs are stateless by default. Every conversation starts with a blank model. If you want the model to know who you are, what you work on, how you like your outputs formatted, and which approach you prefer for recurring problems — you have to tell it. Every time.

That is a tax. Not a feature. A tax.

The people who got good at prompt engineering got skilled at paying that tax efficiently — writing shorter context dumps, using system prompts in API playgrounds, building prompt libraries they paste from. It helped. But it never made the tax go away. It just made each payment slightly cheaper.

In 2026, paying that tax is a choice. The tools exist to stop paying it entirely.

What a Harness Actually Does

A harness is infrastructure wrapped around your AI runtime. In my case, that is PAI — Personal AI Infrastructure — running on top of Claude Code in the terminal. The architecture has three layers.

Memory is persistent context that survives across sessions. PAI knows my role (HRIS analyst), my platform (Oracle HCM Cloud), my Oracle triage methodology, my blog’s writing conventions, my active projects, and my preferences for output formats. None of that gets re-entered. It gets loaded automatically at session start.

Skills are pre-built, parameterized workflows. When I say “process my study notes,” a skill handles that — reading from the right directory, converting the format, saving to the right Obsidian path, cross-referencing the knowledge base. The skill is the prompt, written once, tested, improved over time. I do not craft it fresh every time.

The Algorithm is a structured execution framework. When the work is complex — multi-step, multi-file, non-trivial — PAI runs through a defined process: observe, think, plan, build, execute, verify, learn. The output is consistent because the process is consistent.

Taken together, these three things mean the model is never starting from zero. It arrives at each session already oriented.

The Token Economy Hidden Inside the Infrastructure

There is a practical angle to this that does not get talked about enough: token consumption.

Every message in a chat session burns tokens — your context, the model’s reasoning, the output, and whatever you paste in to re-establish state. The longer and more complex the session, the faster you burn toward usage limits. When you are re-explaining your role, your project, and your preferences at the start of each conversation, you are spending tokens on re-orientation, not on actual work.

A harness changes the math.

PAI loads persistent context at session start through hooks — but those are structured files read by the runtime, not large prompt blocks the model has to reason through. The model arrives oriented. The working token budget goes toward the task.

More importantly, PAI externalizes logic that would otherwise live inside the conversation. The skills are pre-written workflows. The Algorithm is a structured execution framework. The session hooks handle routing and context injection. A significant portion of what would normally require the model to think its way through — “what directory does this go in?”, “what format does this certification use?”, “what’s the right next step in this process?” — is already answered in scripts and configuration files that run before the model responds.

That is not just more efficient. It changes your usage ceiling. When the model is not spending context budget on re-orientation or derivable decisions, more of each session goes toward meaningful work. You hit limits later, do more per session, and run longer chains of complex tasks without interruption.

Prompt engineering optimizes the prompt. Infrastructure optimizes the budget.

CLI vs. Chat: It Is Architecture, Not Preference

This is the part that took me a while to articulate. The preference for CLI over chat window is not aesthetic — it is structural.

A chat window is a conversation interface. Conversations are ephemeral. They have no persistent state, no programmable hooks, no way to inject context at session start, no way to trigger workflows, no way to store outputs in structured memory. The UX is polished. The architecture is a dead end for anything requiring continuity.

A CLI is a programmable runtime. Session start hooks can load context files. Commands can trigger skills. Outputs can write back to memory. Different agents can be spawned with different contexts and run in parallel. The AI operates inside an environment you built, not inside a box you are renting.

That difference compounds. A chat window is equally capable on day one and day three hundred. A harness gets more capable every time you add a skill, improve the memory, or refine the algorithm.

Before and After: The Same Problem, Two Environments

Chat window, eight months ago:

“I have study notes from a certification I’m working through, exported as a Word document from my tablet. I organize my notes in Obsidian under a folder structure by certification and module number. I need you to convert the content to clean markdown, extract the key concepts as a structured summary, generate quiz questions with answers, and format everything to match my existing note structure. The certification is [name], this is module [N], and here’s an example of how my other notes look: [paste example]…”

Then the session ended. Next time I had notes to process — same context dump, from scratch.

With PAI, today:

“Process my study notes for module 4.”

PAI already knows the certification, the Obsidian directory structure, the naming conventions, the quiz format, and which knowledge base to cross-reference. Processing starts immediately. The notes land in the right place in the right format.

The eight-month gap between those two experiences is not better prompting. It is infrastructure.

2026: Where the Power Users Went

The practitioners who were deep into prompt engineering two years ago have largely moved on — not to better prompts, but to better systems. They are building skills, writing memory schemas, wiring session hooks, running structured execution algorithms on complex work. The prompt engineer persona is being quietly replaced by the AI infrastructure builder.

This is not about being technical. It is about thinking one level up. Instead of asking how to get a better response to this prompt, you ask what a system would need to know to handle this reliably, every time.

Your Knowledge Doesn’t Live in the Model

One of the less obvious benefits of building infrastructure rather than relying on chat conversations: your knowledge is not locked to any LLM.

When everything lives in a chat window, switching models means starting over. Your context, your conversation history, your accumulated session knowledge — gone. The model you were using knew who you were because you kept telling it. A different model knows nothing.

With PAI, the knowledge lives in files you own. The memory is markdown on your machine. The skills are scripts in a directory. The algorithm is a structured process your runtime executes. None of it is stored inside Claude, or any other model. The AI is the engine, not the warehouse.

That distinction matters more than it sounds. LLMs are evolving fast. A model that is the best choice today may not be the best choice in six months. If your entire working context is entangled with one provider’s chat history, migration is painful. If your context lives in a portable, file-based system, switching the underlying model is a configuration change — not a rebuild.

I run PAI on Claude today because it is the best fit for how I work right now. But the memory schema, the skill library, the algorithm — all of it would transfer to a different model without losing a session’s worth of context. That portability is a deliberate design choice, and it is one of the most underappreciated properties of building on open infrastructure rather than inside a walled chat product.

Credit Where It’s Due

PAI did not emerge from a vacuum. A significant part of the thinking behind it — the idea that AI should be augmenting structured, intentional human systems rather than replacing ad-hoc conversations — traces directly to the work of Daniel Miessler .

Daniel has been articulating the case for AI infrastructure thinking longer than most. His Fabric project, his writing on augmented intelligence, and his broader framing of what it means to build systems that extend human capability rather than just answer questions — all of it shaped how PAI was conceived and how it continues to evolve.

The shift from “better prompts” to “better systems” is not a new idea. It just needed enough tooling to become practical. Daniel saw that early.

Where to Start

PAI is open-source. Claude Code is free to start — it is Anthropic’s official CLI, available to any Claude user. The distance between using AI in a chat window and running it inside a harness is smaller than it looks, and the compounding return starts from the first session where PAI remembers something you did not have to re-enter.

If you are still re-explaining yourself every time you open a new tab, that is the problem worth solving.

Adding a Web UI to My PDF to Markdown Converter

Sat, 28 Mar 2026 00:00:00 +0000

The Promise I Made to Myself

In my last post about building the PDF to Markdown converter , I listed some “what’s next” ideas at the end. One of them was:

FastAPI wrapper: Create an HTTP API for web apps to use

Well, I did it. And I went a step further — I built a full drag-and-drop web UI on top of it.

The CLI still works exactly as before. This is an addition, not a replacement. But now when I want to convert a batch of PDFs without thinking about terminal commands, I just open a browser tab.

What the UI Does

The interface is intentionally minimal:

Drag-and-drop zone — drop one PDF or fifty onto it
Browse button — if you prefer clicking
Convert button — kicks off the conversion
Per-file progress bars — live updates as each file converts
Individual download — each completed file gets its own Download button
Download all as ZIP — one click to grab everything
Clear — resets the session and cleans up temp files server-side

Everything runs locally. Files go to a temp directory on your machine, get converted, and are served back to you. Nothing hits an external API.

The Stack

I kept it as simple as possible:

Backend: FastAPI + uvicorn

FastAPI was the obvious choice — it handles file uploads cleanly, has first-class async support, and the python-multipart library makes multi-file form handling trivial. The conversion logic is unchanged from the CLI: pymupdf4llm.to_markdown() doing the heavy lifting.

Progress updates: Server-Sent Events (SSE)

This is the part I found most interesting. When you hit Convert, the browser opens a persistent connection to /progress/{job_id} and receives a stream of JSON events — one every 400ms — until the job finishes. No polling loop, no WebSocket complexity. SSE is perfect for this: unidirectional, simple, and built into every modern browser.

async def event_stream():
 while True:
 data = json.dumps({"progress": job["progress"], "done": job["done"]})
 yield f"data: {data}\n\n"
 if job["done"]:
 break
 await asyncio.sleep(0.4)

return StreamingResponse(event_stream(), media_type="text/event-stream")

On the frontend, consuming it is three lines:

const eventSource = new EventSource(`/progress/${jobId}`);
eventSource.onmessage = e => {
 const { progress, done } = JSON.parse(e.data);
 // update the UI...
};

Threading: The conversion itself is synchronous (PyMuPDF4LLM blocks while it processes pages). To keep the FastAPI event loop from freezing during conversion, each job runs in a ThreadPoolExecutor:

asyncio.get_event_loop().run_in_executor(executor, _convert_job, job_id, job_dir)

Four workers by default — enough to handle several simultaneous conversions without overloading the machine.

Frontend: Vanilla JS, no build step

I deliberately avoided React, Vue, or any framework. The whole UI is a single static/index.html file. It loads instantly, has no dependencies to install, and is easy to read and modify. For a local tool that one person uses, this is the right call.

Project Structure

Here’s what changed from the original CLI project:

pdf-to-markdown/
 pdf2md — original CLI (unchanged)
 app.py — FastAPI server (new)
 static/
 index.html — drag-drop UI (new)
 serve — start script (new)
 requirements.txt — updated with FastAPI deps
 venv/ — existing venv, three new packages added

The serve script is just:

#!/usr/bin/env bash
cd "$(dirname "$0")"
source venv/bin/activate
uvicorn app:app --host 0.0.0.0 --port 8765 --reload

Run it once, open http://localhost:8765, and you have a working converter in your browser.

One Gotcha: PyMuPDF4LLM is Synchronous

This tripped me up briefly. pymupdf4llm.to_markdown() does not return a coroutine — it’s a blocking call that can take 10–30 seconds on a large document. If you call it directly in an async FastAPI route handler, you freeze the entire event loop while it runs. No other requests get handled. The SSE stream stops updating.

The fix is the ThreadPoolExecutor pattern above — push the blocking work off the event loop entirely. The async route returns immediately, the SSE stream keeps ticking, and the conversion runs in a thread pool where it belongs.

The Download Endpoints

Three endpoints handle output:

GET /download/{job_id}/{filename} — single .md file
GET /download-all/{job_id} — all .md files as a ZIP
DELETE /job/{job_id} — clean up temp files

The ZIP is built in memory using Python’s zipfile module and streamed directly to the browser — no intermediate file on disk:

buf = io.BytesIO()
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
 for f in md_files:
 zf.write(f, f.name)
buf.seek(0)
return StreamingResponse(buf, media_type="application/zip", ...)

What This Unlocks

The CLI was already useful. The web UI adds a few things the CLI cannot easily do:

Non-terminal users. Anyone on my network can now use this converter by visiting http://my-machine:8765. No Python knowledge required.

Bulk drop workflows. Dragging 20 PDFs from Finder into a browser window and clicking Convert is significantly faster than constructing a --batch command with the right paths.

Visual feedback. The progress bars are not just cosmetic. For large PDFs that take 20–30 seconds, knowing the conversion is running (and roughly how far along it is) removes the anxiety of staring at a terminal cursor.

What’s Next

The original roadmap item was “FastAPI wrapper.” That’s done. The next one I’m eyeing:

Auto-feed to Obsidian inbox. Right now the flow is: convert in the web UI, download the ZIP, unzip, move to Obsidian. I’d like to add a toggle: “Send output directly to ~/projects/obsidian-vault/00-inbox/” — one less manual step.

That’s a small addition to the backend. Coming soon.

Running It

cd ~/projects/pdf-to-markdown
./serve
# Open http://localhost:8765

The first run installs nothing new — the three new packages (fastapi, uvicorn, python-multipart) are already in the venv. It just works.

I Turned 50 Cybersecurity Books Into a Searchable Brain

Sat, 21 Mar 2026 00:00:00 +0000

The Problem With Security Books

I have a lot of cybersecurity books. PDFs from Humble Bundles, O’Reilly downloads, books I’ve bought and never finished, reference material I collected “just in case.” Like most people, they lived in a folder I rarely opened.

The reason is friction. When I needed to look something up — say, how SQL injection payloads work, or the steps for privilege escalation on Linux — I’d have to remember which book covered it, open it, and search inside. Or just Google it and hope Stack Overflow had something decent.

That’s not a knowledge base. That’s a graveyard.

So I built something better: a local semantic search engine over all of them, powered by PostgreSQL, pgvector, and OpenAI embeddings. Now I ask questions in plain English and get back the exact passages — with the book and chapter — that answer them. The whole thing runs locally on my machine.

Here’s how I built it, and why it’s become one of the most useful tools in my PAI (Personal AI Infrastructure) stack.

What Semantic Search Actually Means

Traditional search is keyword matching. You type “SQL injection” and it finds documents containing those exact words.

Semantic search is different. It converts your query and your documents into vectors — lists of numbers that represent meaning in high-dimensional space. Similar concepts cluster together regardless of exact wording. Ask “how to bypass database input validation” and you’ll surface the same SQL injection content, even though you never typed “SQL injection.”

This matters enormously for a security knowledge base. Security concepts have dozens of names. “Privilege escalation,” “privesc,” “root access,” “vertical privilege abuse” — these all mean the same thing. Semantic search finds all of them.

The Stack

PostgreSQL 17 — the database
pgvector 0.8.2 — vector similarity search extension for Postgres
OpenAI text-embedding-3-small — converts text chunks to 1536-dimensional vectors
CyberSecKB.ts — a custom Bun/TypeScript CLI I built to tie it all together

Everything runs locally. The only external call is to OpenAI’s embedding API (which runs once at ingest time, not at query time).

The Pipeline: From PDF to Searchable Knowledge

Step 1: Convert PDFs to Markdown

Raw PDFs are terrible for text processing. I convert everything to Markdown first using a pdf2md Python tool:

cd ~/projects/pdf-to-markdown
source venv/bin/activate

# Text-based PDFs (most books):
python pdf2md input/mybook.pdf

# Image-based or scanned PDFs (use OCR first):
ocrmypdf --force-ocr input/mybook.pdf /tmp/ocr.pdf
python pdf2md /tmp/ocr.pdf output/mybook.md

# Move to library:
mv output/mybook.md ~/projects/cybersecurity-library/books/

Step 2: Ingest into the Database

TOOL=~/.claude/skills/PAI/USER/KNOWLEDGE/CYBERSECURITY/Tools/CyberSecKB.ts

# Single book with topics tagged:
bun $TOOL ingest \
 --file ~/projects/cybersecurity-library/books/mybook.md \
 --title "My Book Title" \
 --topics web,network,linux

# Or load everything at once:
bun $TOOL ingest --batch ~/projects/cybersecurity-library/books/

The ingest process:

Reads the Markdown file
Splits it into ~800-token chunks, preserving chapter headings
Sends chunks to OpenAI’s embedding API in batches
Stores chunks + their vector embeddings in PostgreSQL

Step 3: Search

# Plain English query:
bun $TOOL search "how do attackers bypass WAF rules for SQL injection"

# Filter by topic:
bun $TOOL search "privilege escalation" --topics linux --limit 5

# Check what's in the KB:
bun $TOOL list
bun $TOOL stats

What It Looks Like in Practice

Here’s a real query. I asked:

bun $TOOL search "SQL injection bypass techniques" --limit 3

Result:

━━━ [63.3%] Web Penetration Testing With Kali Linux → Detecting and Exploiting Injection-Based Flaws
The `;` metacharacter in a SQL statement is used similarly to how it's used
in command injection to combine multiple queries on the same line...
━━━ [62.5%] Web Penetration Testing With Kali Linux → Detecting and Exploiting Injection-Based Flaws
If user input is used without prior validation, and it is concatenated
directly into a SQL query, a user can inject different data...
━━━ [60.4%] Web Penetration Testing With Kali Linux → Detecting and Exploiting Injection-Based Flaws
Input taken from cookies, input forms, and URL variables is used to build
SQL statements that are passed back to the database...

Each result shows the similarity score, book title, chapter, and a preview. I can immediately tell which book to go deeper in.

Another query — privilege escalation:

bun $TOOL search "privilege escalation linux" --limit 3

━━━ [66.1%] Cybersecurity Attack And Defense Strategies → Privilege Escalation
Most systems are built using the least privilege concept — users are
purposefully given the least privileges they need to perform their work...
━━━ [65.9%] Kali Linux Cookbook → Privilege Escalation
CVE-2015-1328: overlayfs vulnerability affecting Ubuntu where it does not
do proper checking of file creation in the upper filesystem area...
━━━ [65.8%] Cybersecurity Attack And Defense Strategies → Privilege Escalation
On Linux, vertical escalation allows attackers to have root privileges
that enable them to modify systems and programs...

This is the power of the system: I asked about a concept, not a keyword, and got specific, sourced, actionable results from three different books.

The Current State of the KB

After the initial batch ingest:

50 books indexed
11,757 chunks stored and embedded
Coverage spans: penetration testing, malware analysis, forensics, identity and access, cloud security, social engineering, cryptography, threat modeling, and more

Some of what’s in there:

Practical Malware Analysis (620 chunks)
Cybersecurity Threats, Malware Trends and Strategies (552 chunks)
Cybersecurity Attack and Defense Strategies (460 chunks)
Security Chaos Engineering (387 chunks)
Hardware Hacking Handbook (378 chunks)
Modern Data Protection (338 chunks)

Why This Fits Into PAI

This knowledge base is part of my PAI system — Personal AI Infrastructure. The idea behind PAI is to build infrastructure that amplifies what I can do with AI, rather than using AI one prompt at a time.

The Security KB is a perfect example. It’s not about asking ChatGPT “explain SQL injection.” It’s about having my own curated library, chunked, embedded, and ready to surface exactly the passage I need — from books I trust, with sources I can trace back.

When I’m working through a security challenge or studying for a certification, I can query the KB directly. Luna (my PAI assistant) can also query it as part of a larger workflow — search the KB, pull context into the prompt, and answer questions grounded in my actual library rather than generic training data.

Building It With Claude Code

The entire CyberSecKB tool was built using Claude Code through PAI. The process:

Described what I wanted: ingest markdown books, chunk by section, embed with OpenAI, store in pgvector
Claude Code scaffolded the TypeScript CLI
We hit a few real-world issues along the way:
- The OpenAI project key needed embedding model access enabled separately
- Batch size of 2048 hit the 300k token/request limit — tuned down to 200
- The 1M tokens/minute rate limit required adding a 15-second delay between batches
- A SQL type error in the search function when no topics filter was passed

Each issue was diagnosed and fixed in the same conversation. The tool went from concept to 50 books indexed in a single session.

What’s Next

A few things I want to add:

Tag all books with proper topics — the batch ingest skipped topic assignment; I’ll tag each book so --topics web or --topics linux filters actually work
Tier 1 topic files — condensed 5-15KB reference files for the most-used topics (SQLi, XSS, privilege escalation, etc.) that load directly into context
AI Security KB integration — the AI Security research KB shares the same database; queries cross both domains automatically

The knowledge base is live. The friction is gone. Now the books actually get used.

Built with PAI, Claude Code, PostgreSQL, pgvector, and OpenAI embeddings. All processing runs locally except the embedding API calls at ingest time.

Claude Code Has Skills. PAI Has a Skill System. Here's the Difference.

Sun, 15 Mar 2026 00:00:00 +0000

Claude Code Has Skills. PAI Has a Skill System. Here’s the Difference.

There’s a word that shows up in both Claude Code’s documentation and in PAI’s architecture: skills. And because they share the same word — and even the same file conventions — it’s easy to assume they’re roughly equivalent. One is just a slightly fancier version of the other.

They’re not. The relationship is closer to HTTP and a web framework. Claude Code’s skill mechanism is the protocol. PAI is the framework built on top of it.

Understanding that distinction changed how I think about what I’ve actually built on my machine.

Start Here: What Claude Code’s Skill Mechanism Actually Is

Before explaining what PAI adds, it’s worth being precise about what Claude Code provides natively — because it’s both more minimal and more elegant than most people realize.

Claude Code’s skill system works like this:

At startup, Claude reads every SKILL.md file it finds under ~/.claude/skills/
The description field in each skill’s YAML frontmatter determines when that skill activates — it’s pure intent matching. Anthropic caps this at 1024 characters.
When a skill matches your request, the Skill tool injects the full SKILL.md content into Claude’s context window
Claude follows the instructions in that file

That’s the entire mechanism. It’s a context injection system with a routing layer.

The USE WHEN clause in a skill description is the key piece. Here’s a simplified example from my OracleHCM skill:

---
name: OracleHCM
description: Expert Oracle HCM Cloud troubleshooting. USE WHEN user mentions
 Oracle HCM, HCM Cloud, HDL, HCM Data Loader, Journey, Checklist, workflow
 approvals, autocomplete rules, fast formulas, security profiles...
---

When I describe an Oracle HCM problem in natural language, Claude Code matches my intent against that description and loads the skill. I never have to say “use the Oracle HCM skill.” The intent matching handles it.

Elegant. Minimal. And — on its own — surprisingly limited.

The Gap Between “Context Injection” and “Operational Capability”

Imagine a skill that’s just a long markdown file. When it loads, Claude reads the instructions and tries to follow them. If the instructions are clear and the task is simple, that works fine. But for anything complex — something that involves multiple steps, personalized behavior, CLI tooling, external APIs, or parallel agents — a single markdown file loaded into context starts to break down.

The instructions get long. They can’t be personalized without making the skill personal (and therefore un-shareable). There’s no way to dispatch to a sub-procedure. There’s no tooling layer. There’s no way to say “if the user wants to create a blog post, follow this procedure; if they want to publish, follow that one.”

This is the gap PAI fills.

What PAI Builds on Top: Nine Layers

PAI’s SKILLSYSTEM.md defines a canonical structure that every skill must follow. It’s not a suggestion — it’s enforced by convention and by the CreateSkill skill that scaffolds new ones. Here’s what each layer adds.

Layer 1 — Canonical Structure

Claude Code just needs a SKILL.md. PAI requires a specific directory layout:

SkillName/
├── SKILL.md ← minimal routing only (40-50 lines)
├── Workflows/ ← execution procedures, one per task
│ ├── Create.md
│ └── Update.md
├── Tools/ ← TypeScript CLI tools (always present)
│ └── Generate.ts
└── ApiReference.md ← context files loaded on demand

SKILL.md stays minimal. Complexity lives in workflows and context files that load when actually needed.

Layer 2 — Workflow Routing

This is the most immediately useful layer. Claude Code routes to a skill. PAI routes within a skill.

The routing table in every SKILL.md dispatches sub-tasks to specific workflow files:

| Workflow | Trigger | File |
|-------------|----------------------------|---------------------------|
| **Create** | "write a post" | `Workflows/Create.md` |
| **Publish** | "publish", "deploy" | `Workflows/Publish.md` |
| **Header** | "create header image" | `Workflows/Header.md` |

“Write a post” and “publish the site” both activate the same skill, but they route to completely different procedures. Without this, a skill that handles multiple operations becomes one giant file Claude has to navigate by itself.

Layer 3 — The Personalization Layer

Every system skill in PAI checks for user overrides before executing:

~/.claude/skills/PAI/USER/SKILLCUSTOMIZATIONS/{SkillName}/
├── EXTEND.yaml ← merge strategy (append | override | deep_merge)
└── PREFERENCES.md ← user-specific behavior

The system skill stays generic and shareable. My preferences — color palettes for the Art skill, voice configurations for the Agents skill, output format defaults for Research — live separately and merge in at runtime. The skill author never needs to know about my preferences. I never need to fork the skill to add my own.

Layer 4 — System vs Personal Skill Separation

PAI enforces a hard naming convention that determines portability:

TitleCase (Research, Browser, OracleHCM) = system skills, no personal data, shareable via PAI Packs
_ALLCAPS (_BLOGGING, _MAQINA) = personal skills, private by convention, never exported

Claude Code has no concept of skill visibility. PAI makes it structural.

Layer 5 — CLI Tooling Convention

Every PAI skill has a Tools/ directory. When a workflow needs to do something repeatable — generate an image, manage a server, sync a repository — it calls a TypeScript CLI tool instead of embedding logic in the workflow markdown itself.

Tools use #!/usr/bin/env bun, expose configuration via flags, and have .help.md documentation files. This keeps workflows simple (intent routing) and tools encapsulated (execution). You can test a tool independently of its workflow.

Layer 6 — AI-Powered Hooks

PAI runs 17 event hooks that fire at specific moments: session start, prompt submission, pre-tool, post-tool, and others. The most important one for response quality is FormatReminder — it runs AI inference on your raw prompt before Claude even starts responding, classifies the depth required (FULL / ITERATION / MINIMAL), and injects that classification as authoritative context.

This is hooks doing real work, not just shell scripts appending text to prompts.

Layer 7 — The Algorithm

Every response PAI generates runs through a 7-phase problem-solving framework: OBSERVE → THINK → PLAN → BUILD → EXECUTE → VERIFY → LEARN.

This isn’t decorative structure. The OBSERVE phase reverse-engineers your actual intent. The THINK phase selects capabilities and validates skill choices against the problem. The VERIFY phase uses TaskCreate/TaskUpdate to track measurable success criteria. The LEARN phase captures what to improve next time.

Skills feed into this framework — they’re not parallel to it. When a skill activates, it executes inside the Algorithm, with its results held accountable to the ISC criteria created in OBSERVE.

Layer 8 — Agent Composition Patterns

PAI skills can spawn specialized subagents and compose them using named patterns:

Pattern	Shape	When
Pipeline	A → B → C	Sequential domain handoff
TDD Loop	Engineer ↔ QATester	Build-verify cycle
Fan-out	→ [A, B, C]	Multiple perspectives needed
Gate	A → check → B or retry	Quality gate before progression

A skill that just loads into context can’t orchestrate parallel agents. A PAI skill that routes to a workflow that invokes a Fan-out pattern can research, build, and verify in parallel — with a spotcheck agent at the end synthesizing results.

Layer 9 — Dynamic Loading

Large skills use deferred loading. Only the SKILL.md loads on invocation. Reference documents, API guides, and style specs load only when the specific workflow that needs them runs. This actively manages token budget rather than blowing it on context that might not be needed.

The Feature Gap in One Table

Feature	Claude Code (Native)	PAI System
Skill discovery	YAML `description` at startup	Same + `USE WHEN` intent parsing
Sub-routing	None	Workflow routing table
Personalization	None	`SKILLCUSTOMIZATIONS` layer
Skill visibility	All equal	System vs Personal convention
Tooling	None	TypeScript CLI tools
Hooks	Basic	17 AI-powered hooks
Response structure	Free-form	Algorithm (7 phases, ISC, verify)
Agents	None	15+ specialized subagent types
Memory	None	File-based cross-session memory
Dynamic loading	Full file loaded	Context files on demand
Portability	No convention	PAI Packs

Why This Matters Practically

The single most useful shift in mental model: Claude Code skills are context. PAI skills are operational units.

When I ask my system to publish a blog post, the publishing skill doesn’t just remind Claude how publishing works. It dispatches to the Publish workflow, which runs image conversion, calls hugo, commits, pushes to GitHub, and triggers the Actions pipeline that deploys to Namecheap FTP — all as a structured procedure with steps that can fail, be verified, and be corrected independently.

That’s not context injection. That’s execution.

The 34 skills on my system aren’t 34 long markdown files. They’re 34 capabilities, each with their own routing logic, personalization layer, tooling, and agent integration. Claude Code’s mechanism made them possible. PAI’s framework made them reliable.

Where to Go from Here

If you’re new to PAI and want to understand the broader architecture this sits inside — the memory system, the agent tiers, how RAG ties everything together — the prior post RAG, Agents, and Skills: The Three Pillars Inside My Personal AI covers the full picture.

If you want to go deeper on the skill system itself, the canonical reference is ~/.claude/skills/PAI/SYSTEM/SKILLSYSTEM.md — it’s the document all skills are built against, and it explains every convention described here in precise detail.

PAI is open source at github.com/danielmiessler/PAI .

Building an AI Conference Directory That Populates Itself

Sat, 14 Mar 2026 00:00:00 +0000

The Problem: AI Conferences Are Everywhere and Nowhere

If you’ve ever tried to find a comprehensive list of upcoming AI conferences, you know the pain. There’s no single source. AAAI has their page. NeurIPS has theirs. ICML posts deadlines on OpenReview. Half the emerging summits only exist on LinkedIn event pages or buried in Reddit threads.

I wanted a simple, searchable directory of AI conferences — one site where I could see what’s coming up, filter by topic, and get the key details. But I didn’t want to manually curate it. I’ve seen too many “awesome lists” on GitHub that are lovingly maintained for three months and then abandoned.

What I wanted was a system that populates itself.

So I built one. And with Claude Code running through my PAI system, the whole pipeline — from search to database to website — came together over a few focused sessions.

Here’s the full story.

The Architecture: Three Layers, Zero Manual Data Entry

The final system has three layers, each handling a distinct responsibility:

SearXNG (search engine)
→ conference_tracker.py (discovery)
→ Airtable (database)
→ fetch-events.mjs (build-time fetch)
→ React + Vite site on Netlify

Each layer is independently useful, loosely coupled, and replaceable. Let’s walk through them.

Layer 1: The Tracker — Finding Conferences Automatically

The foundation is a Python script called conference_tracker.py. Its job is simple: search the web for AI conferences and store what it finds.

Search: SearXNG Instead of Google

Rather than hitting the Google API (with its quotas and billing), I use SearXNG — an open-source, self-hosted meta-search engine. It aggregates results from Google, Bing, DuckDuckGo, and others without API keys or rate limits.

The tracker runs a curated list of search queries defined in config.yaml:

search_queries:
 - "AI conference 2026"
 - "artificial intelligence conference 2026"
 - "machine learning conference 2026"
 - "NeurIPS 2026"
 - "ICML 2026"
 - "AAAI 2026"
 - "AI summit 2026"
 - "deep learning conference 2026"
 - "computer vision conference 2026 CVPR"
 - "natural language processing conference 2026"

Each query returns up to 10 results. The tracker extracts the title, URL, and snippet from each result, deduplicates against what’s already in the database, and stores new finds.

Storage: Airtable as the Source of Truth

Why Airtable? Because it’s a real database with an API, but it also has a spreadsheet-like UI for manual review. When you’re building a pipeline that discovers data automatically, you want a way to eyeball the results and clean up noise — and Airtable is perfect for that.

The tracker writes five fields per record: title, websiteUrl, description, Source Query, and Date Found. That’s it. Just the raw discovery data. The structured details come later.

The deduplication is URL-based — normalized and lowercased. If we’ve already stored neurips.cc/2026, we don’t store it again even if it appears in a different search query.

def extract_conference_info(result, source_query):
 return {
 "title": result["title"][:200],
 "websiteUrl": result["url"],
 "description": result["snippet"][:1000],
 "Source Query": source_query,
 "Date Found": datetime.now(timezone.utc).strftime("%Y-%m-%d"),
 }

After one run, we had 87 unique conference records. The real stuff — NeurIPS, ICML, CVPR, AAAI — alongside smaller but interesting events like the Quantum AI and NLP Conference, Deep Learning Indaba, and the Wharton Human-AI Research summit.

Layer 2: The Website — React + Vite on Netlify

The directory itself is a React app built with Vite and deployed on Netlify. It’s a single-page app with search, tag filtering, and individual event pages.

The key architectural decision: data is fetched at build time, not runtime. A prebuild script (fetch-events.mjs) pulls conference data from the database and writes it to a data.ts file that Vite bundles into the site. This means:

No API keys exposed in the browser
No CORS issues
Instant page loads (data is already in the bundle)
The site works even if Airtable is temporarily down

The prebuild hook in package.json makes this automatic:

{
 "scripts": {
 "fetch-events": "bun scripts/fetch-events.mjs",
 "prebuild": "bun scripts/fetch-events.mjs",
 "build": "vite build"
 }
}

Every time Netlify builds the site, it automatically fetches the latest data from Airtable. Fresh data on every deploy.

The Middleman Problem: Cutting Google Sheets

Here’s where the story gets interesting.

The original pipeline had an extra step: Airtable → Google Sheets → website. The fetch-events.mjs script was pulling from a published Google Sheet CSV. Why? Because when I first prototyped the site, I started with a spreadsheet. It was quick and easy.

But once the conference tracker was writing directly to Airtable, Google Sheets became a middleman with no purpose. Data had to be synced from Airtable to Sheets (manually or via Zapier), and that sync was another thing that could break.

The fix was straightforward: teach fetch-events.mjs to talk directly to the Airtable API.

Airtable’s REST API

The Airtable API is clean. A single GET request returns records as JSON:

const url = new URL(`https://api.airtable.com/v0/${baseId}/${tableId}`);
const resp = await fetch(url.toString(), {
 headers: { Authorization: `Bearer ${pat}` },
});
const data = await resp.json();
// data.records = [{ id, fields: { title, date, ... } }]

The one gotcha: Airtable paginates at 100 records. You need to follow the offset token:

async function fetchFromAirtable(pat, baseId, tableId) {
 const allRecords = [];
 let offset = null;

 do {
 const url = new URL(`https://api.airtable.com/v0/${baseId}/${tableId}`);
 if (offset) url.searchParams.set('offset', offset);

 const resp = await fetch(url.toString(), {
 headers: { Authorization: `Bearer ${pat}` },
 });
 const data = await resp.json();
 allRecords.push(...data.records);
 offset = data.offset || null;
 } while (offset);

 return allRecords;
}

Graceful Fallback

I kept the Google Sheets path as a fallback. The main() function uses a priority chain:

Airtable — if AIRTABLE_PAT, AIRTABLE_BASE_ID, AIRTABLE_TABLE_ID are set
Google Sheets — if GOOGLE_SHEET_CSV_URL is set
Fallback events — hardcoded sample data so the build never fails

This means you can’t break the site by misconfiguring a data source. The build always succeeds.

Layer 3: The Enrichment — AI-Powered Data Extraction

This is where things got really interesting.

After cutting Google Sheets, I had 87 conference records in Airtable. But they only had three useful fields: title, description, and URL. No dates. No locations. No tags. The site worked, but every event card was sparse — no way to filter by date or location, no tags to browse by topic.

Filling in 87 records by hand? No thanks.

The Idea: Visit Each URL and Ask AI to Extract the Data

The approach: for each conference record, fetch its web page, extract the text content, and use AI inference to pull out structured fields like date, location, organizer, and tags.

I built an enrichment script — enrich_conferences.py — that sits alongside the tracker in the same project.

Step 1: Fetch and Clean the Page

Each conference URL gets fetched with requests, then cleaned with BeautifulSoup. Navigation, footers, scripts, and styling get stripped, leaving just the text content:

def fetch_page_text(url, timeout=15):
 resp = requests.get(url, headers=headers, timeout=timeout)
 soup = BeautifulSoup(resp.text, "html.parser")

 for tag in soup(["script", "style", "nav", "footer", "header", "aside"]):
 tag.decompose()

 text = soup.get_text(separator="\n", strip=True)
 lines = [line.strip() for line in text.splitlines() if line.strip()]
 return "\n".join(lines)

Step 2: AI Extraction via PAI Inference

The cleaned text gets sent to Claude (via PAI’s Inference tool) with a structured extraction prompt. The prompt is specific about what to extract and what format to use:

Given text from a conference web page, extract these fields as JSON:
{
"date": "human-readable date like 'May 5-6, 2026'",
"endDate": "ISO end date like '2026-05-06'",
"location": "City, State/Country",
"venue": "venue name",
"price": "ticket price or 'Free'",
"organizer": "organizing body",
"tags": "comma-separated topic tags (max 4)"
}

One critical addition: if the page is a list of conferences (like “Top 10 AI Conferences of 2026”), the AI returns {"is_list_page": true} and the script skips it. This was essential — about 15% of our URLs were aggregator pages, not individual conference pages.

Step 3: Write Back to Airtable

Non-empty extracted fields get PATCHed back to Airtable. The script only writes fields that actually exist in the table schema — a lesson learned the hard way when venue and imageUrl threw 422 errors because those columns hadn’t been created yet.

def build_patch_fields(extracted, allowed_fields):
 if extracted.get("is_list_page"):
 return None
 patch = {}
 for key in ["date", "endDate", "location", "venue", "price", "organizer", "tags"]:
 if key not in allowed_fields:
 continue
 val = extracted.get(key, "")
 if isinstance(val, str) and val.strip():
 patch[key] = val.strip()
 return patch if patch else None

The Results

Running the enrichment script across all 87 records:

Outcome	Count
Records enriched	48
List/aggregator pages (correctly skipped)	12
No extractable fields (social media, OpenReview, etc.)	11
Errors (timeouts, HTTP 403s)	16

After enrichment:

Field	Records populated
Date	42
Location	41
Tags	47
Organizer	27
Price	4

From zero structured data to a directory where most events have dates, locations, and topic tags — without opening a single conference website manually.

Some highlights from the extraction:

NeurIPS 2026: December 6-12, Sydney, Australia — Deep Learning, Research, Algorithms, LLMs
CVPR 2026: June 3-7, Denver, CO — Computer Vision, Deep Learning, Research
ICML 2026: July 6-11, Seoul, South Korea — LLMs, Computer Vision, NLP, Robotics
AI Council 2026: May 12-14, San Francisco, CA — Generative AI, ML Ops, AI Safety
MIDL 2026: July 8-10, Taipei — Deep Learning, Healthcare AI, Computer Vision

The Pipeline Today

Here’s what the full system looks like now:

SearXNG (self-hosted search)
→ conference_tracker.py (Python — discovers conferences)
→ Airtable (source of truth — 87 records)
→ enrich_conferences.py (Python — AI-powered field extraction)
→ Airtable (now with dates, locations, tags)
→ fetch-events.mjs (Node — build-time data fetch)
→ data.ts (bundled into the site)
→ React + Vite app on Netlify

The tracker discovers. The enricher structures. The fetcher delivers. The site displays. Each piece runs independently and can be re-run at any time.

The enrichment script is idempotent — it only processes records where the date field is empty, so running it again only touches new or previously-failed records.

What I’d Do Differently (And What’s Next)

The Timeout Problem

About 16 records hit the 25-second inference timeout. The fast tier (Haiku) is quick but occasionally chokes on pages with dense, complex content. A retry mechanism using the standard tier (Sonnet) for failed records would catch most of these.

Missing Table Columns

The venue and imageUrl fields don’t exist in the Airtable table yet. The enrichment script extracts venue names beautifully (The Venetian for Ai4, COEX Convention Center for ICML, Dongguk University for AAAI Summer), but the data gets dropped because the columns aren’t there. A quick table schema update in the Airtable UI fixes this.

Scheduled Runs

Right now, both the tracker and enricher are manual. The natural next step is scheduling — run the tracker daily to discover new conferences, the enricher on new records, and trigger a Netlify deploy afterward. The Netlify build hook is already configured; it just needs a cron job or GitHub Action to call it.

Data Quality

Some records are noise — Reddit discussion threads, Amazon Science blog posts, Twitter/X profiles. A quality filter (either rule-based on URL patterns or AI-powered) would clean the dataset before enrichment runs.

Lessons Learned

1. Eliminate Middlemen Early

Google Sheets added zero value once Airtable was in the picture. But it lingered because it was the “original” approach. Every extra hop in a pipeline is a thing that can break, a thing that needs syncing, and a thing that slows you down. Cut it.

2. Build-Time Data Fetching Is Underrated

Pulling data at build time instead of runtime means no API keys in the browser, no loading spinners, and no CORS headaches. For data that changes daily (not per-second), this is the right architecture.

3. AI Extraction Beats Manual Curation

Using AI to extract structured data from unstructured web pages isn’t perfect — we got 48 out of 87 records enriched, not 87 out of 87. But it took 20 minutes of runtime versus what would have been hours of manual work. And the script is re-runnable. Improvement is incremental.

4. Detect Your Data’s Shape Before Writing

The Airtable 422 errors on venue were entirely preventable. The enrichment script now probes the table schema at startup and only writes to fields that exist. Defensive coding at system boundaries saves debugging time.

5. List Page Detection Is Essential for Web Scraping Pipelines

When you’re scraping URLs from search results, a significant percentage will be aggregator pages (“Top 10 Best AI Conferences”) rather than individual event pages. If you don’t detect and skip these, you’ll corrupt your dataset with merged data from multiple events. The is_list_page flag in the AI extraction prompt was one of the highest-value additions to the whole pipeline.

The Bigger Picture

This project is a miniature version of a pattern I keep coming back to: systems that compound.

The tracker runs once and discovers 87 conferences. The enricher runs once and structures 48 of them. The next time the tracker runs, it discovers only new conferences (deduplication handles the rest). The next time the enricher runs, it only processes records it hasn’t touched yet.

Every run makes the dataset better without redoing previous work. That’s the whole point of building infrastructure instead of doing things manually — you invest upfront so the system improves over time with minimal additional effort.

Working with Claude through PAI made each layer come together faster than I expected. The tracker, the Airtable integration, the Google Sheets elimination, the enrichment script — each was a focused session where the AI handled the implementation details while I focused on architecture decisions.

That’s the augmented part of Augmented Resilience. Not replacing the thinking — amplifying it.

RAG, Agents, and Skills: The Three Pillars Inside My Personal AI

Tue, 24 Feb 2026 00:00:00 +0000

RAG, Agents, and Skills: The Three Pillars Inside My Personal AI

This site — Augmented Resilience — didn’t get built the way most blogs do. There was no staring at blank Hugo config files, no manually hunting down Namecheap SSH docs, no scrambling to remember whether the deploy script needed the public/ folder cleaned before each build.

Instead, I described what I wanted. The AI knew my hosting setup (Namecheap shared hosting), my stack (Hugo with the re-terminal theme), my repo (GitHub, SSH-keyed), and my editor (Obsidian). When a build error surfaced — a theme name mismatch between hugo.toml and the actual directory — it was diagnosed and fixed before I had time to Google it. When the deploy script needed writing, it was scaffolded against my specific environment. When I accidentally left sensitive data in an early draft, it caught it before the commit.

None of that context lived in the prompt. It lived in the infrastructure.

The system behind it is called PAI (Personal AI Infrastructure) — an open-source framework I run locally on top of Claude Code. And the reason it could handle an entire site build end-to-end without constant hand-holding comes down to three architectural pillars: RAG, Agents, and Skills.

What Is PAI?

PAI is an open-source personal AI infrastructure system that runs on top of Claude Code. It’s not a SaaS product — it’s a framework you install on your own machine. The system is built around a central idea: AI systems need structure to be reliable. Like scaffolding supports construction, PAI provides the architectural patterns that make AI assistance consistent, contextual, and capable of compounding over time.

There are 34 skills installed on my system, 17 event hooks, 141 workflows, and a memory system that learns from every interaction. But none of that would matter without three core mechanisms working in concert.

The PAI statusline — live system stats showing version (v2.4), algorithm (ALG:v0.2.25), skill count (SK: 34), workflows (WF: 141), hooks (17), context usage (48%), memory signals (144 ratings), and a rolling quality score trend.

Pillar 1: RAG — Your Personal Knowledge Base In Every Response

RAG (Retrieval-Augmented Generation) is the pattern of retrieving relevant documents from a knowledge store and augmenting the AI’s prompt with that context before generating a response. In enterprise AI, this is how you get a chatbot that can answer questions about your internal policies without hallucinating.

In PAI, RAG is the engine that makes the AI feel like it knows you.

How It Works in PAI

When a session starts, PAI’s hook system loads a foundational context layer: my identity, my name, the current date, and the core behavioral rules (the Algorithm). This is the retrieval index — a lightweight map of everything the system knows how to find.

When I make a request, the system retrieves additional context on demand:

Skills frontmatter — Each of the 34 skills has a description field with a USE WHEN clause. These descriptions load at startup as a routing index. When my request matches a skill’s intent, the full skill content loads. This is retrieval — pulling in the right expertise document for the task.
USER/ context files — There’s a structured personal knowledge base living at ~/.claude/skills/PAI/USER/. It contains my resume, my TELOS life goals, my contacts, my projects, my tech stack preferences. When I ask a question where my professional background is relevant, that context gets retrieved and injected.
MEMORY/ directory — Every session, every correction, every insight gets captured in a structured memory system organized into WORK/, LEARNING/, SIGNALS/, and RESEARCH/ directories. Past work items, completed tasks, and quality signals from previous interactions can all be retrieved to inform the current one.
Hook-injected context — Event hooks fire at specific moments (session start, before each prompt, after tool use) and inject dynamic context — things like the current depth classification, relevant behavioral rules, or system state.

Practical Scenario: Building Augmented Resilience

When I was setting up this site and ran into the Hugo theme mismatch error, here’s what PAI retrieved without me explaining any of it:

My tech stack preferences from the USER context — Hugo, GitHub, Namecheap, Obsidian as editor
The WebSavant skill loaded automatically (matched “Hugo site”, “deployment” intent), bringing with it Hugo-specific knowledge about theme configuration, hugo.toml structure, and build pipelines
My project context from MEMORY — the repo name, the hosting environment, decisions made in prior sessions about the deploy workflow

By the time I described the error, PAI already knew the environment it was debugging. It didn’t need me to explain what kind of hosting I had, which theme I was using, or what my folder structure looked like. The retrieval layer had already assembled that context before a single word of the solution was written.

That’s the difference RAG makes — not smarter AI, but contextually equipped AI.

Pillar 2: Skills — Domain Expertise That Activates Itself

If RAG is how PAI knows context, Skills are how PAI does work. A skill is a self-contained expertise module that activates automatically based on intent, routes to the right workflow, and executes a structured procedure.

Think of each skill as a senior specialist on call — and you never have to explicitly page them.

The Anatomy of a Skill

Every skill follows the same structure:

SkillName/
├── SKILL.md ← Routing layer (loads on invocation)
├── Workflows/ ← Step-by-step execution procedures
│ └── Create.md
│ └── Update.md
└── Tools/ ← CLI automation scripts (TypeScript)
└── Generate.ts

The SKILL.md file has two parts:

YAML frontmatter with a USE WHEN clause — this is how Claude Code knows when to activate the skill
Workflow routing table — once activated, this routes the request to the correct workflow file

The magic is in USE WHEN. Here’s a simplified example from the OracleHCM skill:

description: Expert Oracle HCM Cloud troubleshooting and guidance. USE WHEN user
 mentions Oracle HCM, HCM Cloud, HDL, HCM Data Loader, Journey, Checklist,
 workflow approvals, autocomplete rules, fast formulas, security profiles...

I never have to say “use the Oracle HCM skill.” I just describe my problem in natural language. The intent matching system routes it.

Practical Scenario: Configuring SEO and GEO for augmentedresilience.com

Before the site went live, I needed proper SEO and GEO — Open Graph tags for social sharing, meta descriptions for search, canonical URLs, a sitemap, and schema.org structured data so AI-powered search engines like Perplexity, ChatGPT, and Claude could understand and cite the content accurately. None of that comes configured out of the box with the re-terminal theme. I said:

“Set up SEO and apply Generative Engine Optimization to augmentedresilience.com.”

The system:

Activated the WebSavant skill (matched “SEO” + “GEO” + “site” intent — no skill named, no flags set)
Routed to the SEO and AddSchema workflows inside that skill
Created the correct Hugo partial override at layouts/partials/extended_head.html — the exact injection point the re-terminal theme exposes without touching any theme files
Added the full Open Graph tag set (og:title, og:description, og:image, og:url, og:type) wired to Hugo’s page variables
Injected schema.org JSON-LD for every page type: WebSite and Person on the homepage, Article and BreadcrumbList on every post, and AboutPage on /about — giving AI crawlers a machine-readable knowledge graph of the site
Created robots.txt explicitly permitting GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers — with the sitemap URL wired in
Configured hugo.toml for canonical URL generation and enabled the built-in sitemap output

Without the skill, this is a day of Hugo documentation, schema.org spec-reading, and trial-and-error. With it, the full SEO and GEO stack was complete in a single pass — because the skill had already encoded where everything goes in Hugo’s directory structure, which schema types matter for which page contexts, and how to wire Hugo’s template variables into valid JSON-LD that AI search engines can actually parse.

The 34 Skills I Have Installed

My current skill roster includes tools for Oracle HCM support, security recon, OSINT research, browser automation, art generation, document processing, code generation, red teaming, and more. Each one is a packaged capability that activates without friction.

The system is also designed to be extended — building a new skill means writing a SKILL.md with a USE WHEN clause, a workflow routing table, and the workflow files. The CreateSkill skill handles the scaffolding automatically.

Pillar 3: Agents — Parallel Specialized Brains

Skills handle individual domain expertise. Agents handle scale and specialization when a task is too complex for a single pass or requires multiple perspectives simultaneously.

PAI has a three-tier agent system:

Tier 1: Task Tool Subagents (Internal Workhorses)

These are pre-built specialist agents that skills and workflows invoke internally: Engineer, Architect, Explore, QATester, Pentester, ClaudeResearcher, GeminiResearcher, GrokResearcher, and others.

When I ask PAI to research something deeply, it doesn’t just run one search. It can fan out to multiple research agents simultaneously — Claude, Gemini, and Grok each investigating from different angles — then synthesize the results with a “spotcheck” agent that verifies consistency.

This is parallel processing that would take me hours of manual work, running in minutes.

Tier 2: Named Agents (Persistent Specialists)

Named agents are recurring characters with rich backstories, persistent identities, and unique voices via ElevenLabs text-to-speech. They build relationship continuity across sessions.

My installed named agents include:

Serena Blackwood — Architect. Long-term system design decisions.
Marcus Webb — Engineer. Strategic technical leadership.
Rook Blackburn — Pentester. Security testing with a distinct personality.
Ava Sterling — Researcher (Claude). Strategic deep-dive analysis.
Alex Rivera — Researcher (Gemini). Multi-perspective comprehensive analysis.

When Rook runs a security assessment, he doesn’t just return findings — he announces them in his own voice through my speakers. It sounds minor. It’s not. Distinct voices make it cognitively easier to understand who did what work and why you should trust it.

Tier 3: Custom Agents (On-Demand Compositions)

For tasks that don’t fit a named agent, PAI can compose agents dynamically from trait combinations:

Expertise traits: security, legal, finance, medical, research, technical, creative
Personality traits: skeptical, enthusiastic, analytical, contrarian, meticulous
Approach traits: thorough, rapid, systematic, adversarial, synthesizing

Each unique trait combination maps to a different ElevenLabs voice. A security + adversarial agent gets Callum’s edgy voice. An analytical + meticulous agent gets Charlotte’s precise cadence.

The trait system means I can spin up a custom agent for any edge case without writing a new agent from scratch.

Practical Scenario: Pre-Launch Validation of Augmented Resilience

Before pushing the first real commit to augmentedresilience.com, I wasn’t going to just cross my fingers and run deploy.py. I asked PAI to validate the site was actually ready. What happened next wasn’t a single check — it was a parallel review board.

PAI spawned three agents simultaneously:

Rook Blackburn (Pentester) scanned the entire repo for credentials, API keys, and sensitive data I might have accidentally left in a config file or draft post — and announced his findings in his own voice through my speakers
A QA agent opened the Hugo local preview, walked every page, verified links weren’t broken, images loaded, and the deploy pipeline produced a clean public/ build
A Researcher agent audited the site’s meta tags, Open Graph data, and hugo.toml settings against SEO best practices for a new blog

A fourth spotcheck agent then reviewed all three outputs for conflicts — did Rook’s findings overlap with anything the QA agent flagged? Were there config issues that touched both SEO and security?

The result was a single consolidated pre-launch checklist. Two issues surfaced: a leftover draft post with personal notes still marked draft: false, and a missing og:image tag. Both fixed before the first visitor ever landed.

The site you’re reading right now went live clean because three agents checked it in parallel before I touched the deploy button. That’s the difference between asking a question and deploying intelligence.

When All Three Work Together

The real power of PAI isn’t any single pillar — it’s the composition.

Here’s what happened when I needed to go from “Obsidian draft” to “live on augmentedresilience.com” without a manual process I’d eventually forget or skip.

RAG assembled the context before I described the problem. From MEMORY it already knew: Namecheap shared hosting doesn’t support native git pull — content has to be pushed via FTP through GitHub Actions. It knew the Obsidian vault was the content source, that images.py had to run before hugo to convert image links, and that Python was the right tool for the orchestration script. Not one of those constraints was in my prompt.

Skills handled the architecture. WebSavant recognized a Hugo deployment pipeline request and routed to a workflow already aware of the full delivery chain: sync posts from Obsidian → convert images → hugo build → git commit → git push → GitHub Actions → Namecheap FTP. It knew the sequence. It knew why each step had to happen in that order.

Agents built it. An Engineer agent wrote deploy.py — the script that runs the whole sequence in a single command. An Architect agent designed the GitHub Actions workflow that picks up after the push and handles the Namecheap delivery step automatically. Two agents, two distinct responsibilities, running the job that a solo developer would have spent an afternoon piecing together from Stack Overflow answers.

The result:

python3 deploy.py "Add new post"

That’s it. One command. Every post that has ever gone live on this site — including this one — passed through a pipeline that RAG, Skills, and Agents built together. It’s not running because I set it up manually. It’s running because three systems knew what the job required before I finished explaining it.

The Compounding Effect

What makes this architecture meaningful over time isn’t any single interaction — it’s that the system gets better at helping you with every session.

The MEMORY system captures learnings. The SIGNALS directory tracks your implicit feedback. When something goes wrong, it logs the full context under LEARNING/FAILURES. When a workflow produces a 9-10 response, that signal is captured too. The system adjusts.

Generic AI starts fresh every time. PAI compounds.

I’m still early in this — the personal profile files still have template placeholders I haven’t filled in, and there are skills I’ve barely touched. But even at partial configuration, the system already thinks more like a senior colleague than a search engine.

That’s what RAG, Agents, and Skills make possible together: an AI that knows your context, activates the right expertise automatically, and scales parallel intelligence for complex work — all without you having to manage the machinery.

When Your PDF Workflow Breaks - Building a Markdown Converter with Claude Code

Wed, 18 Feb 2026 00:00:00 +0000

The Problem: PDFs Are Knowledge Prisons

You know that feeling when you download a brilliant research paper, only to realize you can’t easily feed it into your AI workflow? Or when you want to add documentation to your knowledge base, but it’s locked in a format that doesn’t play well with version control or LLM tools?

Yeah, I was there last week.

I had just downloaded a fascinating 1.3MB research paper on Generative Engine Optimization and wanted to process it with my AI tools. But PDFs are terrible for this. They’re designed for printing, not for processing. What I needed was Markdown—clean, portable, AI-friendly Markdown.

So I built a converter. And with Claude Code as my copilot through the PAI (Personal AI Infrastructure) system, the whole thing took less than 30 minutes.

Here’s how it went down.

Why Markdown is Better Than PDF for LLMs

Before diving into the build, let’s answer the obvious question: why bother converting? Can’t LLMs just read PDFs directly?

Technically, yes. But the results are significantly worse, and the reasons are fundamental to how PDFs work.

PDFs Are Layout-First, Not Structure-First

PDFs were designed to describe where things appear on a page, not what they mean. As Steven Howard explains in Why PDFs Fail Under LLM Parsing :

“Table cells with wrapped text insert hard line breaks that fragment token continuity and break logical row recognition. Headers and footers simply add noise to the context when used with LLMs. Sentences are split with arbitrary CR/LFs making it very difficult to find paragraph boundaries.”

This architectural mismatch — a format designed for printing being fed into a system designed for understanding — causes cascading problems downstream.

The Token Efficiency Problem

Every token your LLM processes costs money and consumes context window space. PDF extraction wastes both.

According to analysis from MarkdownConverters , Markdown saves up to 70% more tokens compared to extracted PDF text for the same content. The culprit: PDF extraction introduces formatting artifacts, metadata noise, headers/footers, and encoding remnants that all consume tokens without adding semantic value.

To put that in practical terms: a PDF that would use 10,000 tokens might only need 3,000 tokens when properly converted to Markdown. At scale, this compounds dramatically.

The RAG Performance Problem

If you’re building Retrieval Augmented Generation (RAG) systems — using documents as a knowledge base for AI — document format directly impacts answer quality.

The research here is compelling:

Academic validation: A 2024 paper on arXiv (Revolutionizing RAG with Enhanced PDF Structure Recognition ) found that “the low accuracy of PDF parsing significantly impacts the effectiveness of professional knowledge-based QA.”
Industry validation: NVIDIA’s technical blog documents how their NeMo Retriever pipeline converts extracted content to Markdown specifically because it “preserves row/column relationships in an LLM-native format, significantly reducing numeric hallucination” — and reduces incorrect answers by 50%. (NVIDIA: Approaches to PDF Data Extraction for Information Retrieval )
Chunking quality: Analysis from Towards Data Science shows that Markdown’s heading structure (#, ##, ###) produces semantically meaningful chunks, while PDF-based chunking relies on arbitrary page breaks and heuristics.
Retrieval failure rates: Unstructured.io’s research on contextual chunking — tested across 5,563 question-answer pairs — showed an 84% reduction in retrieval failure rates when using structure-aware chunking (the kind Markdown enables natively).
Real-world outcomes: The 2025 Semrush AI Index, cited by Webex Developers Blog , found that 72% of top AI-indexed articles used Markdown or Markdown-like structures, achieving 34% higher retrieval accuracy across ChatGPT, Perplexity, and Gemini.

The Bottom Line

Metric	Impact
Token reduction	Up to 70% fewer tokens vs PDF extraction
Incorrect answers in RAG	50% reduction (NVIDIA NeMo)
Retrieval failure rates	84% reduction (Unstructured.io)
Retrieval accuracy	34% higher (Semrush AI Index 2025)

Markdown isn’t just more convenient — it’s meaningfully better for AI. Converting your document libraries is one of the highest-ROI steps you can take before building any LLM-powered workflow.

The First Failure: When Bleeding-Edge Python Bites Back

I’m running Python 3.14.2—the latest release, barely a few weeks old. Modern, shiny, cutting-edge. Perfect, right?

Not quite.

My first instinct was to use marker-pdf, a high-performance converter optimized for scientific papers and books. It looked perfect on paper (pun intended). But when I tried to install it:

Building wheel for Pillow (pyproject.toml): finished with status 'error'

Ugh.

Turns out, marker-pdf depends on Pillow (the Python imaging library), and Pillow hasn’t built binary wheels for Python 3.14 yet. I could have downgraded Python. I could have fought with source compilation. But why?

This is where working with Claude Code really shines. Instead of going down a rabbit hole trying to force marker-pdf to work, Claude suggested pivoting to PyMuPDF4LLM—a mature, actively maintained library specifically designed for AI/LLM workflows.

And it just worked.

The Solution: PyMuPDF4LLM

PyMuPDF4LLM turned out to be exactly what I needed:

Works flawlessly with Python 3.14 (no compilation errors)
Fast and accurate conversion
Built specifically for feeding documents into LLMs
Clean, simple API
Actively maintained by the PyMuPDF team

The installation was literally:

pip install pymupdf4llm

Five seconds later, I was ready to go.

Building the Tool: First Principles Thinking

As someone new to the CLI world, I’ve been learning to think through project structure from first principles. Where should this live? How should it be organized?

With Claude’s guidance, I chose /Users/dsa/projects/pdf-to-markdown/ for a few key reasons:

Separation of Concerns: Tool projects should be separate from my main workspace
Discoverability: Clear, descriptive naming means I’ll find it again in 6 months
Reusability: This structure works both as a CLI tool AND as a library I could import later

The project structure ended up simple but complete:

pdf-to-markdown/
├── README.md # Documentation
├── venv/ # Isolated Python environment
├── input/ # Test PDFs
├── output/ # Generated markdown
├── pdf2md # CLI wrapper script
└── requirements.txt # Dependencies

The Code: A Simple but Powerful CLI

I wanted a tool I could actually use—something with a clean command-line interface that handles the common cases elegantly. Working with Claude through PAI, we created a Python script that does exactly that:

#!/usr/bin/env python3
"""
PDF to Markdown Converter
A simple CLI tool to convert PDF files to Markdown using PyMuPDF4LLM
"""

import sys
import os
from pathlib import Path
import pymupdf4llm
import pymupdf
from tqdm import tqdm

def convert_pdf_to_markdown(pdf_path: str, output_path: str = None) -> str:
 """Convert a PDF file to Markdown format."""

 if not os.path.exists(pdf_path):
 raise FileNotFoundError(f"PDF file not found: {pdf_path}")

 # Get page count for progress bar
 doc = pymupdf.open(pdf_path)
 page_count = doc.page_count
 doc.close()

 print(f"Converting: {pdf_path}")
 with tqdm(total=page_count, unit="page", desc="Processing", colour="blue") as bar:
 md_text = pymupdf4llm.to_markdown(pdf_path, page_chunks=False)
 bar.n = page_count
 bar.refresh()

 if output_path is None:
 output_path = Path(pdf_path).with_suffix('.md')

 with open(output_path, 'w', encoding='utf-8') as f:
 f.write(md_text)

 print(f"✓ Done: {output_path} ({len(md_text):,} characters)")
 return str(output_path)

def batch_convert(input_dir: str, output_dir: str = None) -> None:
 """Convert all PDFs in a directory to Markdown."""
 input_path = Path(input_dir)
 if not input_path.is_dir():
 raise NotADirectoryError(f"Not a directory: {input_dir}")

 pdfs = sorted(input_path.glob("*.pdf"))
 if not pdfs:
 print(f"No PDF files found in: {input_dir}")
 sys.exit(0)

 if output_dir:
 output_dir = Path(output_dir)
 else:
 output_dir = input_path.parent / "output"
 output_dir.mkdir(parents=True, exist_ok=True)

 total = len(pdfs)
 succeeded = 0
 failed = 0

 print(f"\nBatch mode: {total} PDF(s) found in '{input_dir}'")
 print(f"Output folder: {output_dir}\n")

 for i, pdf_path in enumerate(pdfs, start=1):
 print(f"[{i}/{total}] {pdf_path.name}")
 output_path = output_dir / pdf_path.with_suffix('.md').name
 try:
 convert_pdf_to_markdown(str(pdf_path), str(output_path))
 succeeded += 1
 except Exception as e:
 print(f" ✗ Failed: {e}")
 failed += 1
 print()

 print("─" * 40)
 print(f"Batch complete: {succeeded} converted, {failed} failed")
 print(f"Output folder: {output_dir}")

def main():
 """Main CLI entry point"""
 args = sys.argv[1:]

 if not args:
 print("Usage:")
 print(" pdf2md <input.pdf> [output.md] # Convert a single PDF")
 print(" pdf2md --batch <folder/> # Convert all PDFs in a folder")
 print(" pdf2md --batch <folder/> --output <out_folder/> # Batch with custom output dir")
 print("\nExamples:")
 print(" pdf2md document.pdf # Creates document.md")
 print(" pdf2md document.pdf custom.md # Creates custom.md")
 print(" pdf2md --batch input/ # Converts all PDFs in input/")
 print(" pdf2md --batch ~/documents/pdfs/ --output ~/knowledge-base/docs/")
 sys.exit(1)

 if args[0] == "--batch":
 input_dir = args[1]
 output_dir = None
 if "--output" in args:
 idx = args.index("--output")
 output_dir = args[idx + 1]
 batch_convert(input_dir, output_dir)
 else:
 pdf_path = args[0]
 output_path = args[1] if len(args) > 1 else None
 convert_pdf_to_markdown(pdf_path, output_path)

if __name__ == "__main__":
 main()

What I love about this code:

Smart defaults: If you don’t specify an output path, it just replaces .pdf with .md
Progress bars: tqdm gives you a blue progress bar with page count
Batch mode: --batch processes an entire folder at once, with optional --output target
Helpful errors: Clear messages when things go wrong
Flexible usage: Works with relative paths, absolute paths, custom output names

Make it executable:

chmod +x pdf2md

And now it’s a proper command-line tool.

The Moment of Truth: Testing with Real Data

Theory is great. But does it actually work?

I grabbed that 1.3MB research paper on Generative Engine Optimization and ran:

python pdf2md input/test.pdf output/test.md

The output:

Converting input/test.pdf to Markdown...
Processing: 100%|████████████████| 12/12 [00:02<00:00, 5.8 pages/s]
✓ Done: output/test.md (73,463 characters)

1.3MB PDF → 74KB of clean Markdown in seconds.

I opened the output file, and there it was—perfectly formatted markdown:

## **GEO: Generative Engine Optimization**

Pranjal Aggarwal [∗]
Indian Institute of Technology Delhi
New Delhi, India
pranjal2041@gmail.com

Ashwin Kalyan
Independent
Seattle, USA
asaavashwin@gmail.com
...

Headers, formatting, structure—all preserved. No manual cleanup needed.

Success.

What This Unlocks

Now that I have PDFs converting to Markdown reliably, a whole world of possibilities opens up:

AI Workflows

Feed research papers and documentation directly into Claude or other LLMs
Build RAG (Retrieval Augmented Generation) pipelines backed by your document library
Process technical documentation at scale without losing structure

Knowledge Management

Import PDFs into your Obsidian vault automatically
Version control document content (because it’s now plain text in git)
Full-text search across your entire converted document library

Automation Ideas

Watch folder that auto-converts any dropped PDFs
Batch process entire directories of reports, papers, or manuals
Feed converted markdown directly into a vector database
API wrapper to convert PDFs via HTTP requests

Lessons Learned (Especially for CLI Beginners)

1. Virtual Environments Are Non-Negotiable

Every Python project should live in its own virtual environment. Always:

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip

This keeps dependencies isolated and projects reproducible.

2. Bleeding-Edge Isn’t Always Better

Python 3.14 is awesome, but sometimes mature tooling (like PyMuPDF) that “just works” beats bleeding-edge alternatives. Don’t be afraid to pivot when something doesn’t work.

3. Test With Real Data

I didn’t test with “hello.pdf” containing two sentences. I tested with a 1.3MB research paper. Real data reveals real issues (or in this case, confirms it works beautifully).

4. Document As You Build

Writing the README alongside the code made the project immediately understandable. Future-me will thank present-me.

5. Claude Code + PAI = Superpowers

Working with Claude through the PAI infrastructure meant I had a senior developer helping me think through:

Project structure (first principles)
Library selection (when to pivot)
Code organization (clean, maintainable)
Real-world usage patterns

This wasn’t just coding faster—it was learning better patterns while building.

Usage Examples

Basic Conversion

# Activate environment first (always!)
source venv/bin/activate

# Convert a PDF
python pdf2md document.pdf

# Custom output name
python pdf2md research.pdf my-notes.md

# Full paths
python pdf2md ~/Downloads/paper.pdf ~/Documents/notes.md

Batch Processing

Convert an entire folder of PDFs:

source venv/bin/activate

# Convert all PDFs in a folder (output goes to output/ by default)
python pdf2md --batch ~/documents/pdfs/

# Convert to a specific knowledge base directory
python pdf2md --batch ~/documents/pdfs/ --output ~/knowledge-base/docs/

Add to PATH (Optional)

To use pdf2md from anywhere:

# Add to ~/.zshrc
export PATH="/Users/dsa/projects/pdf-to-markdown:$PATH"

# Then run from anywhere
pdf2md ~/Downloads/paper.pdf ~/Documents/paper.md

What’s Next?

This tool works great as-is, but there are some exciting enhancements on the roadmap:

Immediate Improvements

Better layout analysis: Install pymupdf_layout for improved structure detection on complex documents
Recursive batch mode: Process nested folder structures, not just flat directories

Future Integrations

RAG pipeline: Auto-feed converted markdown into a vector database
Obsidian plugin: Detect PDFs in vault and convert automatically
FastAPI wrapper: Create an HTTP API for web apps to use
Electron/Tauri app: Build a desktop GUI for non-technical users

The Bigger Picture: Why This Matters

This project is tiny—roughly 100 lines of Python, 30 minutes of work. But it represents something bigger:

The ability to build tools that solve your actual problems.

I had a workflow friction (PDFs don’t work well with AI tools). I built a solution. Now that friction is gone, and I can focus on higher-level work.

And the data is clear: converting your document library to Markdown isn’t a nice-to-have. It’s a multiplier on every AI workflow that follows. Up to 70% fewer tokens consumed. 84% fewer retrieval failures. 50% fewer incorrect answers. These aren’t marginal improvements—they’re transformational.

Working with Claude Code through PAI accelerated all of this. It’s like having a patient senior developer sitting next to you, suggesting better approaches, catching errors before they happen, and explaining why certain patterns work.

Resources

PyMuPDF4LLM Docs: https://pymupdf.readthedocs.io/en/latest/pymupdf4llm/
PyMuPDF GitHub: https://github.com/pymupdf/PyMuPDF

Citations: Markdown vs PDF for LLMs

Why PDFs Fail Under LLM Parsing — Steven Howard, Untethered AI: https://untetheredai.substack.com/p/why-pdfs-fail-under-llm-parsing
PDF vs Markdown for AI: Token Efficiency — MarkdownConverters: https://markdownconverters.com/blog/pdf-vs-markdown-ai-tokens
Revolutionizing RAG with Enhanced PDF Structure Recognition — arXiv:2401.12599 (2024): https://arxiv.org/abs/2401.12599
Approaches to PDF Data Extraction for Information Retrieval — NVIDIA Technical Blog: https://developer.nvidia.com/blog/approaches-to-pdf-data-extraction-for-information-retrieval/
Improved RAG Document Processing With Markdown — Dr. Leon Eversberg, Towards Data Science: https://medium.com/data-science/improved-rag-document-processing-with-markdown-426a2e0dd82b
Contextual Chunking: Boost Your RAG Retrieval Accuracy — Unstructured.io: https://unstructured.io/blog/contextual-chunking-in-unstructured-platform-boost-your-rag-retrieval-accuracy
Boosting AI Performance: The Power of LLM-Friendly Content in Markdown — Webex Developers Blog: https://developer.webex.com/blog/boosting-ai-performance-the-power-of-llm-friendly-content-in-markdown

Happy converting!

Deploying a Hugo Site to Namecheap with PAI

Sun, 15 Feb 2026 00:00:00 +0000

I recently deployed my Hugo blog to Namecheap shared hosting, using Obsidian as my content editor and Claude Code with PAI (Personal AI) as my copilot. Here’s a walkthrough of every step, from fixing build errors to setting up a fully automated pipeline that goes from Obsidian to live site in a single command.

The Starting Point

I created a Hugo blog project called Augmented Resilience and used the re-terminal theme, a Namecheap shared hosting account, and a GitHub repository. I used Claude Code in VS Code editor and leveraged Daniel Miessler’s Personal AI infrastructure. The goal: get the site live at augmentedresilience.com with a push-to-deploy workflow.

For context, the Personal AI Infrastructure System (PAI) from Daniel Miessler (see resources below) is an open-source framework that wraps around Claude Code and turns it into a structured problem-solving system. Instead of just chatting with an AI, PAI runs every request through a 7-phase algorithm — observe, think, plan, build, execute, verify, learn — so nothing gets skipped. It maintains persistent memory across sessions (so it remembers my project structure, preferences, and past decisions), automatically selects specialized agents for different tasks (security review, architecture, engineering), and enforces verification criteria before declaring anything “done.” For this project, PAI handled everything from debugging Hugo build errors to writing the deploy script to catching sensitive data I accidentally left in this blog post before it went live. It wasn’t just an AI assistant — it was the entire workflow engine. I found it easier to use it within VS Code (still getting used to using the command line interface).

Step 1: Fixing the Hugo Build

The first issue was a build error:

module "hugo-theme-re-terminal" not found

The problem was a mismatch between the theme name in hugo.toml and the actual directory name. The theme was installed as a git submodule at themes/re-terminal/, but the config referenced hugo-theme-re-terminal.

Fix: Change the theme name in hugo.toml:

theme = "re-terminal"

After that, hugo built the site successfully, generating the public/ folder with all the static files.

Step 2: Setting Up the GitHub Repository

I initialized the repo and connected it to GitHub:

git init
git remote add origin git@github.com:dsacosta/Augmented-Resilience.git
git add .
git commit -m "my first commit"
git push origin main

One gotcha: I initially typed orgin instead of origin in the remote add command. Typos happen — double-check your remote names with git remote -v.

Step 3: Connecting Namecheap to GitHub via SSH

This was the trickiest part. Namecheap shared hosting needs an SSH key to clone from a private GitHub repo. Here’s what worked:

Generate an SSH Key on Namecheap

Log into cPanel on Namecheap
Go to SSH Access → Manage SSH Keys → Generate a New Key
Generate an RSA key (I used the default settings)

Remove the Passphrase

This is critical. cPanel’s Git Version Control runs non-interactively, so it can’t prompt for a passphrase. I opened cPanel Terminal and ran:

ssh-keygen -p -f ~/.ssh/id_rsa

Enter the old passphrase, then press Enter twice for no new passphrase.

Add the Public Key to GitHub

On Namecheap’s cPanel Terminal, run: cat ~/.ssh/id_rsa.pub
Copy the output
Go to your GitHub repo → Settings → Deploy Keys → Add deploy key
Paste the public key and save

Verify the Connection

From cPanel Terminal:

ssh -T git@github.com

You should see: Hi dsacosta/Augmented-Resilience! You've successfully authenticated...

Step 4: Clone the Repo on Namecheap

In cPanel, go to Git Version Control → Create
Toggle Clone a Repository on
Enter the clone URL: git@github.com:dsacosta/Augmented-Resilience.git
Set the repository path (I used /home/yourusername/your-repo)
Click Create

Important: Don’t clone directly into public_html or your domain folder — it likely already has files and will error out. Clone to a separate directory and use deployment to copy files over.

Step 5: Auto-Deployment with .cpanel.yml

cPanel supports automatic deployment tasks via a .cpanel.yml file in the repo root. This file tells cPanel what to do after each pull:

---
deployment:
 tasks:
 - export DEPLOYPATH=/home/yourusername/yourdomain.com/
 - /bin/cp -R public/* $DEPLOYPATH

This copies everything from the public/ folder (Hugo’s build output) into the live site directory.

After pushing this file to GitHub:

Go to Git Version Control → Manage your repo
Click the Pull or Deploy tab
Click Update from Remote to pull the latest
Click Deploy HEAD Commit to trigger the .cpanel.yml tasks

Your site should now be live.

Step 6: Fully Automated Deploys with GitHub Actions

To eliminate the manual “pull and deploy” step in cPanel, I set up a GitHub Actions workflow that SSHs into Namecheap and triggers the pull automatically on every push.

Generate a Deploy Key

On your local machine, generate a key pair with no passphrase:

ssh-keygen -t ed25519 -C "github-actions-deploy" -f ~/.ssh/deploy_key -N ""

Add the public key to Namecheap:

# In cPanel Terminal on Namecheap:
echo "ssh-ed25519 AAAA...your-key-here github-actions-deploy" >> ~/.ssh/authorized_keys

Add Secrets to GitHub

Go to your repo → Settings → Secrets and variables → Actions and add:

Secret	Value
`NC_HOST`	`augmentedresilience.com`
`NC_USER`	Your cPanel username
`NC_PORT`	Your SSH port (check cPanel)
`NC_SSH_KEY`	The full private key (including BEGIN/END lines)

Create the Workflow

Add .github/workflows/deploy.yml to your repo:

name: Deploy to Namecheap

on:
 push:
 branches: [main]

jobs:
 deploy:
 runs-on: ubuntu-latest
 steps:
 - name: Deploy via SSH
 uses: appleboy/ssh-action@v1
 with:
 host: ${{ secrets.NC_HOST }}
 username: ${{ secrets.NC_USER }}
 key: ${{ secrets.NC_SSH_KEY }}
 port: ${{ secrets.NC_PORT }}
 script: |
 cd ~/your-repo && git pull origin main && /bin/cp -R public/* ~/yourdomain.com/

Now every push to main automatically deploys to your live site.

Step 7: One-Command Deploy Script

Five manual commands every time you publish? That’s not a workflow — that’s a chore. I had Claude write a Python script that handles everything in one shot:

#!/usr/bin/env python3
"""One-command deploy: Obsidian → Hugo → GitHub → Live site."""

import subprocess
import sys
from datetime import datetime

PROJECT_DIR = "~/Documents/Augmented-Resilience"
OBSIDIAN_POSTS = "~/projects/obsidian-vault/30-projects/augmented-resilience-posts"
HUGO_POSTS = f"{PROJECT_DIR}/content/posts"


def run(cmd, description, cwd=PROJECT_DIR):
 """Run a command and print status."""
 print(f"\n{'='*50}")
 print(f" {description}")
 print(f"{'='*50}")
 result = subprocess.run(cmd, shell=True, cwd=cwd)
 if result.returncode != 0:
 print(f"\n FAILED: {description}")
 sys.exit(1)
 return result


def main():
 msg = " ".join(sys.argv[1:]) if len(sys.argv) > 1 else f"Site update {datetime.now().strftime('%Y-%m-%d %H:%M')}"

 run(f'rsync -av --delete "{OBSIDIAN_POSTS}" "{HUGO_POSTS}"', "Syncing posts from Obsidian")
 run(f"python3 {PROJECT_DIR}/images.py", "Processing images")
 run("hugo", "Building site with Hugo")
 run("git add .", "Staging changes")

 result = subprocess.run("git diff --cached --quiet", shell=True, cwd=PROJECT_DIR)
 if result.returncode == 0:
 print("\n No changes to commit. Site is up to date.")
 return

 run(f'git commit -m "{msg}"', "Committing")
 run("git push origin main", "Pushing to GitHub")

 print(f"\n{'='*50}")
 print(" DEPLOYED! Your site will be live shortly.")
 print(f"{'='*50}\n")


if __name__ == "__main__":
 main()

Save this as deploy.py in your project root. Now the entire workflow is:

# Default timestamped commit message
python3 deploy.py

# Or with a custom message
python3 deploy.py "Add new blog post about deployment"

The script runs every step in sequence — syncs from Obsidian, converts image links, builds with Hugo, commits, and pushes. If any step fails, it stops immediately so you don’t push a broken build. Combined with the GitHub Actions workflow from Step 6, pushing triggers the auto-deploy to Namecheap. One command, fully live.

Lessons Learned

SSH passphrases break cPanel automation. Always remove the passphrase from keys used by cPanel’s Git Version Control.
Theme names must match directory names. Hugo looks for the theme in themes/<theme-name>/, so the theme value in your config must match exactly.
Don’t clone into the live site directory. Clone to a separate folder and use .cpanel.yml to copy the built files over.
GitHub Actions + SSH is the cleanest auto-deploy for shared hosting. No webhooks, no cron jobs — just a simple SSH action that runs on every push.
Claude Code with PAI made this possible in a single session. From debugging build errors to SSH key troubleshooting to writing GitHub Actions workflows, having an AI pair programmer turned what could have been hours of Stack Overflow rabbit holes into a smooth, guided process.

Tools Used

Hugo — Static site generator
re-terminal theme — Hugo theme
Namecheap — Shared hosting with cPanel
GitHub Actions — CI/CD automation
Claude Code — AI pair programmer
Personal AI Infrastructure (PAI) — Workflow engine
Obsidian — Content authoring
VS Code - Development environment; used as the workspace for Claude Code sessions

Resources

I started a blog…..in 2024 (why you should too) — The YouTube video that inspired me to start this blog
Building a Personal AI Infrastructure (PAI) — Daniel Miessler’s guide to building your own Personal AI system