Building Alfred part 3: company memory as a governance problem rather than a vector database

Company Memory Is a Governance Problem, Not a Vector Database

Building Alfred, Part 3 of 6. Part 1 was the architecture, Part 2 was grounding. This one is about memory, and it contains the single most contrarian decision in the system.

A new engineer follows a procedure that worked perfectly six months ago.

The document is still searchable. The instructions are clear. The answer looks authoritative. It is also wrong. The provider changed, the approval process moved, and the person who understood the exception has left.

The knowledge was stored. It was never governed.

This is where “company second brain” becomes actively dangerous. A brain that remembers everything without understanding ownership, time or confidence does not create intelligence. It creates very fast institutional confusion.

We shipped company memory with zero embeddings

Let me put the contrarian part first, because it is the thing most people will disagree with.

Alfred’s governed memory has no vector database. No embeddings. No pgvector. No reranker. No BM25 library. A repo-wide grep for embedding|pgvector|voyage|cohere|rerank|bm25|faiss|qdrant|pinecone|weaviate|chroma|text-embedding returns two hits, both in an audio noise analysis function where vector happens to be a local NumPy variable.

Retrieval is Postgres full-text search. That is it.

This was not a shortcut we are embarrassed about. It follows from a claim I believe: similarity is not truth, and the hard part of company memory is not finding the passage.

A vector database does not know that a decision was reversed. It does not know that a customer-specific rule should be hidden from another team. It does not know that an incident note was a hypothesis, that a procedure needs review next quarter, or that a personal preference should never become company policy.

Embed every document and conversation into one pool and you build a system that is excellent at retrieving your organisation’s contradictions.

The right question is not “can the AI find this again.” It is whether the information has earned the right to influence a future answer. That question is answered by review, provenance, visibility and time, none of which are vector operations.

So we spent our complexity budget there instead. Nine tables, one migration, and a retrieval query with three weighted signals.

The schema

Everything is raw psycopg3 SQL with dict_row. There is no ORM model. The only Python type is a frozen dataclass:

@dataclass(frozen=True)
class RetrievedMemory:
    page_id: str
    title: str
    summary: str
    body: str
    page_type: str
    namespace: str
    confidence: float
    updated_at: str
    sources: list
    score: float

The tables:

Table Purpose
alfred_wiki_pages Shared company knowledge, the governed core
alfred_wiki_sources Provenance rows, unique per (page_id, source_uri)
alfred_wiki_revisions Full prior-state snapshots, unique per (page_id, revision_no)
alfred_wiki_links Typed edges between pages
alfred_memory_candidates Proposals awaiting human review
alfred_personal_memories Per-user, unique per (user_email, memory_key)
alfred_episodes Opt-in task summaries, unique per session_id
alfred_memory_feedback Thumbs up/down tied to a page and session
alfred_memory_retrievals Retrieval telemetry, no raw queries

The interesting column on alfred_wiki_pages is the generated search vector, with field weights:

search_document TSVECTOR GENERATED ALWAYS AS (
    setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
    setweight(to_tsvector('english', coalesce(summary, '')), 'B') ||
    setweight(to_tsvector('english', coalesce(body_markdown, '')), 'C')
) STORED

A GIN index on that column, plus a btree on (status, visibility, namespace, updated_at DESC), is the entire retrieval infrastructure.

Constraints do real work here. page_type is a seven-value CHECK. visibility is CHECK-constrained to ('team','operations','tech','admin'). status to ('draft','published','disputed','archived'). confidence is NUMERIC(4,3) with a 0 to 1 CHECK. Putting these in the database rather than in Python means a future code path cannot invent a fifth visibility tier by accident.

Postgres full-text search. No embeddings. Role visibility filtered in SQL before anything reaches the prompt propose_memory agent tool candidate status pending human review approve or reject published page retrievable sources never empty revisions to_jsonb snapshot retrieval score, no threshold lex 0.75 + title 0.20 + confidence 0.05 A floor, not a cutoff: with no lexical match the agent still gets the core pages.
Nothing writes a page directly. Proposal, review and publication are separate states, and every page keeps its sources and revisions.

Memory begins as a candidate, not a fact

The most important thing the system does is refuse to write directly.

An agent that notices something useful cannot create a page. It can only propose:

memory_type ∈ {wiki, personal, procedure}
title      ≤ 200 chars
content    ≤ 12,000 chars
confidence 0..1
sources    ≤ 20 items, each requiring a uri

That lands a pending row in alfred_memory_candidates. A human with the manage_alfred_memory capability approves or rejects it.

Provenance is never empty, which took a small amount of deliberate work. If a proposal arrives with no sources, one is synthesised from the conversation it came from:

{"type": "chat",
 "uri": f"alfred://sessions/{session_id}",
 "label": "Originating Alfred chat"}

And if a human publishes a page directly without citing anything, that is recorded too:

{"type": "human", "uri": f"human://{actor}", ...}

“A person asserted this” is a legitimate source. “Unknown origin” is not, and a knowledge base full of unattributed statements is exactly the failure mode we were trying to avoid.

Revisions, and a trick worth stealing

Publishing a page takes SELECT ... FOR UPDATE, computes the next revision number, writes the prior state into alfred_wiki_revisions, then updates. Standard, except for how the snapshot is captured:

SELECT {PAGE_FIELDS},
       to_jsonb(alfred_wiki_pages) - 'search_document' AS revision_snapshot
FROM alfred_wiki_pages WHERE slug = %s FOR UPDATE

to_jsonb(row) serialises the whole row. The - 'search_document' subtracts the generated tsvector, which is both enormous and derivable. There is no column list to keep in sync, so adding a column next quarter automatically appears in future snapshots, and the JSONB cast handles type conversion.

There is a test that exists purely to pin this behaviour, with a fake cursor whose comment reads: “This is what PostgreSQL to_jsonb returns: JSON-safe strings/numbers, even though the selected row also contains native DB types.”

One honest wart: restore_revision does not write a revision of the state it overwrites. It is the only unrevisioned mutation in the service, and it copies the snapshot’s status back verbatim, which means restoring an archive-time snapshot silently un-archives the page. That is a bug, not a design.

Visibility is enforced in SQL

Role to visibility mapping is a single dict:

WIKI_VISIBILITY_ROLES = {
    "team":       frozenset({"admin", "operations", "tech"}),
    "operations": frozenset({"admin", "operations"}),
    "tech":       frozenset({"admin", "tech"}),
    "admin":      frozenset({"admin"}),
}

allowed_visibilities(roles) inverts that into a tuple passed straight into the query as visibility = ANY(%s). Per-user scoping for personal memories and episodes is likewise in the statement itself, never in a Python filter: every mutation carries AND user_email = %s.

This is more reliable than asking a model to remember which paragraph it should not repeat to a particular role. The rows never reach the prompt.

A small thing that saves real support time:

def _require_uuid(value: str | None, field: str) -> str | None:
    """Reject non-UUID identifiers before they reach a UUID column (else 500)."""

422 instead of a psycopg cast error in a stack trace.

Retrieval is one query and three signals

Here is the core of it:

WITH ranked AS (
    SELECT p.*,
           COALESCE(ts_rank_cd(p.search_document,
                    to_tsquery('english', NULLIF(%s, ''))), 0) AS lexical,
           CASE WHEN p.title <> '' AND strpos(lower(%s), lower(p.title)) > 0
                THEN 1.0 ELSE 0.0 END AS exact_title
    FROM alfred_wiki_pages p
    WHERE p.status = 'published' AND p.visibility = ANY(%s)
      AND (p.expires_at IS NULL OR p.expires_at > NOW())
      AND (p.search_document @@ to_tsquery('english', NULLIF(%s, ''))
           OR (p.title <> '' AND strpos(lower(%s), lower(p.title)) > 0))
)

and the score:

(lexical * 0.75 + exact_title * 0.20 + confidence::float * 0.05) AS score

Top-k is 8 in production, clamped to a maximum of 20. Episodes are 3, clamped to 5.

The tsquery decision

The obvious way to turn a user question into a tsquery is websearch_to_tsquery. We do not, and the docstring explains why:

_TOKEN_RE = re.compile(r"[a-z0-9]+")

def or_tsquery(query: str, max_terms: int = 12) -> str:
    seen: list[str] = []
    for token in _TOKEN_RE.findall(query.lower()):
        if len(token) < 2 or token in seen:
            continue
        seen.append(token)
        if len(seen) >= max_terms:
            break
    return " | ".join(seen)

websearch_to_tsquery ANDs terms, “which collapses recall to near zero on real questions.” A user asking “why does the Airtel trunk drop on transfers after we changed the dialer config” produces a conjunction no page will ever satisfy. So lexemes are OR’d, capped at twelve, de-duplicated, order preserved.

Reducing tokens to alphanumerics is also the injection guard. There is no way to smuggle tsquery operators through it. Empty output is handled on the SQL side with NULLIF(%s, ''), so a query made entirely of stopwords degrades to “matches nothing” instead of raising.

What the numbers actually do

Three things about that scoring formula that only become obvious in production.

ts_rank_cd is unnormalised and typically returns small values, roughly 0.01 to 0.3. So the 0.20 exact_title term dominates ranking whenever it fires. In practice this is a two-tier ranker: pages whose title appears verbatim in the message, then everything else.

exact_title is strpos(lower(user_message), lower(page_title)) > 0. Note the direction. The page title must appear inside the user’s message, not the reverse. It appears in the WHERE clause as an OR branch, which makes it a recall widener as well as a booster. A page titled “Calls” matches nearly every message ever sent. A page with a long descriptive title effectively never fires this signal. Title length is now a ranking parameter, which nobody intends but everybody who does lexical retrieval eventually discovers.

There is no score threshold. None. A single overlapping lexeme qualifies a page for consideration. This was intentional, and it pairs with the next design choice.

A floor, not a threshold

Most retrieval systems add a minimum relevance cutoff. We did the opposite:

if not memories:
    # Retrieval floor: never hand the agent an empty knowledge base.
    memories = await self.core_pages(user_roles)

core_pages returns the three highest-confidence pages of type data-contract, system or glossary, ordered by confidence then recency, with score forced to 0.0 so the model can see they were not matched.

The reasoning: an agent with zero context confabulates from training data. An agent holding your data contracts and glossary at least fails inside your vocabulary. A floor is cheap insurance; a threshold optimises a metric nobody experiences.

Memory rides in the uncached tier, for billing reasons

This is my favourite decision in the codebase because it is a retrieval choice made for cost, not relevance:

# Governed memory retrieved for THIS query rides here, not in the
# cached tier: it is relevance-ranked per turn, so caching it would
# force a cache-write on every single turn and never hit.

Prompt caching charges 1.25x to write and 0.1x to read. Content that changes every turn is the worst possible cache candidate: you pay the premium every time and never collect the discount. So the stable system prompt gets a cache breakpoint and per-query memory deliberately does not.

There is a test asserting the retrieved page is absent from every block carrying cache_control and present in blocks[-1]. Somebody will eventually try to “optimise” this by caching it. The test is there to stop them.

Telemetry without surveillance

Every retrieval logs a row. What it does not log is the question:

query_hash = hashlib.sha256(query.encode("utf-8")).hexdigest()
await self._fetchone(
    """INSERT INTO alfred_memory_retrievals
       (session_id, user_email, query_hash, page_ids, latency_ms)
       VALUES (%s, %s, %s, %s, %s) RETURNING id""", ...)

There is a test that reads the migration file as a string and asserts it contains query_hash TEXT NOT NULL and does not contain query TEXT NOT NULL. A regression guard against a future engineer helpfully adding the plaintext column for debugging.

I want to be precise about what this buys, because “one-way hash” is doing some work in that sentence. It is an unsalted, unkeyed SHA-256 of the raw user message, stored next to user_email. For short predictable questions (“what is our refund policy”) it is dictionary reversible in seconds. It genuinely prevents casual browsing of what colleagues asked. It is not a privacy guarantee against a determined party with database access. An HMAC with a server-held key would cost one line and fix that, and it should.

A second cost worth naming: that insert is awaited in the hot read path, so every turn pays a write round trip to log a read.

Where the design outran the implementation

The published version of this article described capabilities that the schema supports and the code does not yet exercise. Since this series is meant to be readable by someone who later reads the code, here is the accounting.

TTL and expiry are declared but dead. alfred_wiki_pages.expires_at, valid_from, and alfred_personal_memories.expires_at are all filtered on read. Nothing anywhere writes them. There is no API field, no column in any INSERT or UPDATE. So “searches only unexpired pages” is vacuously true, and the expiring_pages metric in the evaluation summary is structurally always zero. There is no decay, no recency half-life, no garbage collection.

The disputed status is never written. Neither is draft; every write path hardcodes status='published'. Neither is superseded on candidates. There is no contradiction detection of any kind.

Conflict resolution is slug collision plus a prompt instruction. Approving a candidate calls slugify(title), so a new candidate whose title slugifies to an existing slug overwrites that page, with a revision recorded. Two contradictory facts on the same topic resolve to whichever was approved last. What handles the rest is a sentence in the prompt: “prefer more recent, higher-confidence pages when they conflict,” backed by the confidence and updated-date annotations attached to every memory entry. Conflict resolution is delegated to the model at read time because the storage layer has no machinery for it. That is a defensible interim position, but it is not the same as governance.

Also: slugify truncates to 100 characters, so two long distinct titles sharing a prefix collide into one page.

Redaction is not implemented. The rule that credentials, tokens, private message contents and customer-specific sensitive data must never become memory exists only as English in the system prompt and the tool description. A repo-wide grep for redact|scrub|pii|anonymi in the application returns one unrelated hit. There is no pattern matching, no entropy check, no denylist, no reviewer-side scanning.

There is no ingestion pipeline. No connector, crawler, browser snapshot, call transcript or document ever writes into these tables. No queue, no worker, no retry, no backfill. The complete list of writers is: the agent’s propose_memory tool, a deprecated record_knowledge tool, three HTTP endpoints, and the migration’s one-time copy from the legacy flat table. Memory grows because a human approved something.

Approval flattens the reviewer’s choices. approve_candidate hardcodes namespace="company", visibility="team", summary=content[:500], and page type to procedure or system. A reviewer cannot set visibility or page type. The operations, tech and admin tiers and the seven-value page type enum are only reachable through a direct admin publish endpoint, which has no UI.

There is no UI for most of it. All fifteen memory endpoints exist. The frontend implements personal memories, personal candidates, shared candidates and a metrics line. No page browser, no revision viewer, no restore button, no archive button.

alfred_wiki_links is unused. The knowledge graph edge table was created, complete with a self-reference CHECK constraint, and is referenced by zero lines of application code.

Episodes are retrieved but barely creatable. They are pulled into every prompt. There is no UI to create one, and propose_memory does not accept episode as a type. Creating one requires hand-crafting an HTTP call. Their search vector also lacks setweight, so every lexeme lands in weight class D and titles do not outrank bodies, unlike wiki pages. And retrieve_episodes still uses websearch_to_tsquery, the exact AND-semantics function whose recall collapse motivated or_tsquery for the wiki path. Episode recall suffers the documented bug that the wiki path was fixed to avoid.

Personal memories bypass relevance entirely. No query, no limit. Every active unexpired personal memory for a user is injected into every turn, in the uncached tier. That is an unbounded prompt growth path per user, and the first person to accumulate two hundred remembered preferences will find it.

Writing that list was uncomfortable. It is also the most useful part of this article, because the gap between a schema that anticipates governance and a system that performs governance is where most “AI memory” projects actually live, including ours.

The recall bug, with its own postmortem in the source

One more thing, because it is a good illustration of how these systems fail quietly:

# OR-match the lexemes (see or_tsquery) and, separately, boost a page whose
# title appears verbatim inside the user's message. The prior code passed the
# whole message through websearch_to_tsquery (AND) and tested title LIKE
# %whole-message%, so both signals almost never fired.

Two signals, both broken, in a system that returned results and never errored. Retrieval quality bugs do not page you. They just make the assistant seem a bit dim, and everyone concludes the model is not good enough.

The organisational lesson

Building this forced us to make implicit company behaviour explicit. We had to define what counts as a procedure, who owns a data contract, how customer-specific knowledge is separated, when an incident lesson expires, and which teams can approve shared facts.

That work made Alfred better. It also exposed where the organisation had been running on oral history. An AI cannot govern knowledge more clearly than the company governs it, and the schema is where that becomes undeniable, because you have to pick a default for visibility and someone has to own that decision.

Next, in Part 4: acting on knowledge, which changes the risk profile completely.

What knowledge inside your company is searchable today but no longer trustworthy?

Next in the series

Graduated agency

Alfred runs on DialNexa, our voice AI platform for businesses. If you are building internal agents and want to compare notes, the comments are open.

Leave a Reply

Your email address will not be published. Required fields are marked *