I want to try new personal agents as they emerge. My personal agents without the context were useless.
In this post I explain how I created a portable memory that can make sure all the agents know everything about me, so I can use them all together https://x.com/i/article/2111981779032965120
Hermes, Grok Bot, and Instinct know everything about me thanks to a portable memory I built
I use multiple AI agents, and I hate having to explain myself again every time I switch.
Who I work with. What we decided last week. Which recommendations I saved. What's happening with a project. The context that makes an assistant useful usually gets trapped inside whichever product I happened to use that day.
I wanted that context to belong to me.
So I built a personal wiki that lives outside any individual model or agent. Hermes maintains it, and I expose the same knowledge through authenticated MCP and REST interfaces so my other agents can use it.
The storage is mostly Markdown. The search is hybrid. The ingestion is automated, with rules about what gets saved and where.
Here's the setup.
What's in the wiki
The wiki has four sections.
CRM
This is my relationship memory: people, companies, our interactions, introductions, commitments and relevant personal context.
It has everything about all my work and personal relationships. What we talked about, where, what they asked for, what I promised, what changed at their company and whether something is still unresolved.
The CRM keeps concise summaries with source references, rather than turning someone's profile into a giant pasted email thread.
It also has synthesis and concept pages for context that spans multiple interactions. For example, it created a meta note with all the VCs I've sent deals to invest and those that have sent me companies to advise.
Notes
Notes holds the things I want to remember or come back to: recommendations, preferences, products, places, ideas and useful reference material.
Some of these have structured metadata. Something can be a book I want to read, a show I haven't finished or a place I want to visit.
I want to be able to ask "What books what I was recommended that I haven't read?" or "What country VPN do I need to use to buy cheaper flights?"
Articles
Articles is my saved reading library.
It preserves source URLs and retained article content, with summaries or learnings where they've been captured. It also has concept pages that connect material on the same topic.
This lets an agent search the material I've saved instead of answering everything from the public web or its training data.
There's an important distinction between saving a link and saving an article's contents. The API supports both, but a link-only save doesn't pretend the system fetched or read the article.
Family
Family holds detailed household and immediate-family knowledge: logistics, documents, health records and household projects.
It has a separate home so those details don't get duplicated across work CRM pages.
Any ongoing project like a house remodel I'm doing lives in a special projects section inside the Family section.
How information gets into it
There's multiple ways to get information in.
Some information comes from explicit saves while I'm using an agent. For example: "Cono recommended me to watch the TV show Slow horses" or "I need to buy XX next time I'm in the US".
If I share a preference, make a decision or ask it to remember something, the agent can retain the useful fact with its source and date.
I have a big automated process called Nightly CRM, which stays inside Hermes.
It collects the completed day's evidence from:
- All my Gmail accounts.
- All connected Slack workspaces.
- WhatsApp.
- Calendar-linked Granola meeting transcripts.
- Eligible conversations I've had with any of the agents (It reads the daily sessions)
The unit of review is a conversation, in chronological order. Reading a single message without its replies is a good way to save a commitment that was already canceled three messages later.
The model extracts concrete meaning: asks, decisions, delivered work, changes in the relationship, personal context and remaining commitments.
Deterministic code then validates the references and applies the updates. The model doesn't get to invent a file path or rewrite arbitrary CRM pages.
If an email or meeting already captured something, a later conversation with my assistant shouldn't create a duplicate version of the same event.
Managing TODOs and tasks
There's also a separate task lane for real obligations as part of the Nightly CRM:
- It can create tasks that I need to do based on something I promised
- It can create tasks that the bots needs to do based on something I promised. These are tasks that don't need my input. For example: Intros on email, asking somebody for something, etc.
- It also keeps a list of things that are "Owed to me" and if they're not delivered it'll automatically follow up with peeps using WhatsApp, Slack or Email.
Saving and search indexing are separate
A fact being saved and a fact being searchable are two different states.
A canonical writer commits the knowledge first. A separate publisher updates the search index.
That means Nightly CRM can finish its actual writes without waiting on search publication. If indexing is delayed, the system should report that delay rather than claim the information is already searchable.
The API also returns a locator for immediate readback after a save, so an agent can verify the retained record while search publication is still pending.
The search is a custom hybrid retrieval system built on SQLite.
It combines several signals rather than relying entirely on embeddings.
Keyword search
SQLite FTS5 provides full-text search, ranked with BM25.
This is useful for exact words, names, distinctive phrases and terms that should not be blurred into "roughly the same meaning."
The system also has local identity and alias matching. Looking up a person by a known email or alias should not depend on whether their profile happens to be semantically similar to the query.
Semantic search
The semantic side uses OpenAI's text-embedding-3-small with 1,536-dimensional vectors.
An embedding represents text as a list of numbers. Text about similar subjects tends to get similar vectors, which helps when my question uses different wording from the saved record.
The system breaks approved text into heading-aware chunks, with a default maximum of 3,200 characters. It carries heading context into continuation chunks and uses limited overlap when splitting longer sections.
When semantic search is permitted for a query, it embeds the query and compares it against stored chunk vectors using cosine similarity. Those comparisons happen locally.
This implementation scores the stored vectors directly. It isn't using a separate hosted vector database or an approximate-nearest-neighbor service.
For each document, its best-matching chunk contributes the semantic score.
Combining the results
Keyword and semantic rankings contribute through reciprocal-rank fusion, using 1 / (60 + rank).
The system adds explicit signals for exact identities, aliases, mentions, relevant context and structured intent. It can distinguish a request about a person from a request for saved reading or unfinished items.
It also applies admission checks to semantic-only results. Being the closest vector doesn't automatically mean a document is relevant; something will always come first, even in a bad result set.
There are local links and relationship-aware signals too.
What gets embedded, and what doesn't
The index supports three embedding policies per record:
- allow: embed a provider-safe version of the body.
- descriptors_only: embed a limited metadata description, not the full body.
- deny: don't create an embedding for that record.
The current corpus uses all three.
Provider-safe text is also different from the original Markdown. The preprocessing strips things such as URLs and filesystem paths from the embedding payload.
I've authorized OpenAI embeddings across sensitivity classes, but permission doesn't mean every existing record is currently configured for full-body embedding. The effective record policy still controls what the pipeline sends.
Query embedding is conditional too. Some locally recognized private or restricted matches suppress the external query call and stay on local retrieval paths.
A source read is different again: reading the original evidence happens locally and doesn't send that body to the embedding provider.
The embedding is a search aid. It isn't the authoritative memory, and it isn't what the agent should quote as evidence.
Search finds candidates; reads recover the evidence
Every result comes with a stable record locator, a source hash and an index generation.
The agent uses those to read the matching source.
That second step matters. Finding a person's name or a promising title doesn't mean the agent has recovered the fact I asked about.
Reads are bound to the returned generation. If the generation no longer matches, the agent has to do a fresh lookup rather than silently reading a different revision.
The runtime also exposes freshness information and verifies source integrity. An unavailable query, a stale index and a genuine "no match" are different outcomes.
For exhaustive questions, there's a separate paginated list operation. Ranked search is not a reliable way to answer "show me everything."
Making it portable with MCP and REST
I exposed the same backend through two interfaces:
- Streamable HTTP MCP for agents that support it.
- REST for agents or tools that can make HTTP requests.
Both offer four operations:
- search
- read
- list
- save
The REST interface also exposes capabilities, and there's an OpenAPI schema plus a getting-started guide.
The service is reachable over HTTPS through Tailscale Funnel, but publicly reachable doesn't mean publicly readable.
Each client gets its own revocable bearer credential. The server stores credential digests rather than the original passwords, and permissions determine which operations and sections a client can access.
Hermes uses the same local API. Grok Bot and Instinct have separate client credentials.
I deliberately kept external CRM writes out of this interface. Other agents can query the CRM, but Nightly CRM remains inside Hermes. External saves route to Notes, Articles, Family or an existing registered project.
That gives me shared memory without letting every agent run its own competing CRM importer.
How I use it across agents
I give each agent operation instructions:
Before answering something about my people, preferences, decisions, projects or past work, search the wiki. Read the matching evidence. If the answer isn't there, check whatever authorized history you actually have before asking me to explain it again.
When new durable information comes up, save it to the right section with its source and original date.
Corrections preserve the previous evidence as history. Suggestions stay labeled as suggestions rather than quietly becoming decisions.
Saves have idempotency keys so a retry after a timeout doesn't create another copy. Updates use source-hash checks so two agents don't blindly overwrite each other's changes.
The service doesn't magically see every conversation I have with every agent. Each agent still has to capture the relevant facts from its own conversations. That's a behavior requirement, not something MCP solves by itself.
The files are portable too
The canonical wiki lives on my server and is synced with my devices in case I want to read the markdown. To be honest, I never do 😂
What I wanted from this
I want to choose an agent because it's good at the task, without paying the context-rebuilding tax every time.
The agents can have different models, tools and conversation histories. They can still consult the same maintained knowledge about me.
I'll keep switching agents. I don't want to keep rebuilding my memory.