This continues the story of switching my external brain’s search to semantic search. Last time, I noticed that keyword search on my vault was missing things, so I added local embeddings via Ollama for semantic search. Now even a vague query like “how did I set up gitleaks again” pulls up the right daily log.

That made pulling specifics much better. But as I kept using it, I noticed a different gap. There was no entry point for grasping the whole picture broadly.

You Can Pull Points, But Not See the Whole Plane

What semantic search gives you is the ability to pinpoint a single past moment. You have a question, and it digs up the note closest to it. That’s a “point” motion.

But when a human actually recalls something, they don’t jump straight to a point. First they grasp the whole situation — “I’m in this kind of situation right now, working on this kind of thing, it’s roughly about this” — and only then dig into specifics.

Claude Code’s external brain didn’t have this “grasp broadly” side. When I checked the aggregated profile it’s supposed to read on startup, it turned out to be essentially empty. Who I am, what I’m working on, what’s been happening lately — none of that was pulled together into one place. The facts were scattered across personal-info notes, mistakes, individual projects, and daily logs, but nothing handed over the big picture the moment it started up.

So every time, at the start of a session, I’d hit semantic search a few times and reconstruct the situation from fragments. Pulling several points and guessing at the plane from them. It would be much simpler to just hand over the plane itself from the start.

Recall needs two layers: one that grasps broadly, and one that pulls specifics. I’ll call the former macro and the latter micro. What I built with semantic search was only micro. This time, I’m adding macro.

Stealing the Philosophy a Second Time

Last time, I wrote about evaluating TencentDB-Agent-Memory, a tool for giving agents long-term memory, and ended up only borrowing the one idea of semantic search and building it myself. That tool has more to it: it distills conversations through four layers — facts, scenarios, and persona. The topmost layer is a picture of “this is who this user is.”

What I wanted for macro was exactly that. Something that pulls out only the essentials from scattered facts and condenses them into a single picture of who I am and where I currently stand.

But I wasn’t going to bring the whole thing over. Same as last time. That tool auto-extracts memory from conversations. But in my very first post about the external brain, I decided that “what to remember shouldn’t be left to the AI.” Auto-extraction runs head-on into that principle.

So I stole only the philosophy. I took the idea of “distilling a person’s profile into a single note.” I threw out the implementation of “having the AI do it automatically.” What I do is just pull out the essentials, and only build it from handwritten primary-source notes.

Persona: A Single Note Distilled From What I Wrote by Hand

What I built is a single note called persona.md. It sits at the root of the vault, read first on startup alongside mistakes and TODOs.

The contents: who I am, my working style and preferences to respect, what’s currently in motion, recent developments, and entry points for digging into the past. I didn’t try to cram everything into a summary — for details, I link out to personal-info notes and individual project notes. Macro gives you the grasp, and from there you can drop down into micro if needed.

I built a command called /persona-refresh to generate it. All it does is read handwritten primary-source notes and summarize them. No auto-extraction from conversation logs. Source links are mandatory, and filling in gaps with guesswork is forbidden. The only facts allowed in are ones I wrote down somewhere myself, by hand. The AI distills, but it doesn’t invent material.

One problem: with this kind of manual regeneration, it’s never clear when you’re supposed to run it. Forget to run it, and it just rots out of date. So on startup, it checks the date — if persona.md doesn’t exist, or if it’s been more than seven days since the last update, it auto-regenerates before being read. I can still trigger it manually anytime, and if I leave it alone, it refreshes itself within a week regardless. I keep manual control without giving it up, while decay gets stopped automatically.

MOC: The “Plane” Search Can’t Pull Up

Alongside persona, I also put in a MOC.md. Map of Content — a map of my notes.

This is a longstanding idea in the Obsidian community: build a single handwritten index by domain that ties all your notes together. Divided into categories like self, environment and tools, blog, projects, and logs, with links out to each hub note.

Why do you need an index when you already have search? Because search is a tool for pulling “points.” You throw a query, and the closest matches come back. But “what kinds of clusters even exist in this vault in the first place” isn’t something search can grasp. I wanted to survey the terrain itself before even framing a question. Only a handwritten map can do that. Where search pulls a point, MOC surveys the plane.

Now the external brain’s recall is two-tiered. On startup, persona and MOC first give a broad grasp of “here’s who you are, what you’re working on, and here’s the map.” From there, individual questions get answered by vault-search’s semantic search pulling specific points. Grasp broadly, then drop to a point. This is the motion I originally wanted.

While I Was At It, I Evaluated One More

While building macro, I also looked at another memory-related OSS project. graphify, a tool that turns code and documentation into a knowledge graph.

I didn’t need the whole thing here either. It comes with a graph database and HTML visualization — overkill for a personal vault. But there was one good idea in its design. It labels extracted relationships with a confidence level: facts explicitly stated, relationships connected by inference, and things suspicious enough that a human should look at them. It distinguishes these three tiers and surfaces the suspicious ones in a report for a human to judge.

My external brain had the same hole. Up to now, whenever Claude Code wrote knowledge into the vault, a verified fact and an in-the-moment guess sat there wearing the same face. My future self, in the next session, would trust both equally as fact and act on them. A guess sits there posing as a fact, and I build more mistakes on top of it. Memory quietly gets contaminated.

So I added labels to the writing rules. Same three tiers. Anything backed by verification stays unmarked. An unverified guess gets marked “guess.” Something that hasn’t been backed up and is waiting on judgment gets marked “needs confirmation.” When reading, anything marked “needs confirmation” doesn’t get taken at face value — it gets verified before use. I don’t need the graph database, but I do need this one dial. Another case of taking only the philosophy and discarding the implementation.

Looking Back

I evaluated two existing memory OSS projects, stole only the philosophy from each, and got by with a self-built implementation in plain Markdown. From Tencent’s project: “distill a person’s profile into a single note.” From graphify: “label memory with confidence levels.” Bring either one in wholesale, and you get a dedicated database and AI auto-extraction along with it. I don’t need that part.

What runs through all of this, unchanged since the first post, is “don’t leave what to remember up to the AI.” I distill, I label, but deciding what counts as material, and judging confidence levels, stays on the human side. The more convenient the automation, the more quietly it steals control over memory. That’s the one thing I won’t hand over.

The external brain can now pull specific points via semantic search (micro), and grasp the whole picture via persona and MOC (macro). Recall is now two-tiered. From here, I’ll keep using it and probably keep noticing what’s still missing. It’s not something you build once and finish — every time something’s inconvenient, you add another layer. I suspect that’s just going to keep repeating.