Everyone Wants a Company Second Brain. Almost Nobody Is Building One.

Building Alfred, Part 1 of 6. This series documents the engineering behind Alfred, the internal agent platform we built at DialNexa. Code references point at the actual implementation.

It is 11:47 at night and a customer call has gone wrong.

The founder wants to know whether the problem is isolated. Operations wants to know what happened on the call. An engineer wants logs, timestamps and the code path. Customer success wants something they can send the customer.

Everyone is asking the same question. The answer is spread across a call record in one Postgres database, service logs in Loki, provider API artifacts in another table, a config change in agent_versions, and a pull request merged three days ago.

That night is why Alfred exists. Not because we wanted a chat box over a document folder, but because the questions that matter inside a company do not respect the boundaries between tools, teams and databases.

The phrase “company second brain” is easy to say and easy to trivialise. Most products described that way are search systems with good prose. You ask, they retrieve, they summarise. That saves time. It is not the same as understanding how a company is operating right now.

This article is the architecture overview. The five that follow go deep on each hard part: grounding, memory governance, agency, production reliability, and the learning loop.

The test that separates search from a second brain

Ask a question that cannot be answered by recall:

Why did this customer call fail, has it happened before, what changed, and what should we do next?

Then watch what the system has to do. It has to locate the correct call from a partial description. Derive a bounded time window. Pull filtered logs rather than a stream dump. Cross-reference provider artifacts. Read the code path. Preserve the permissions of the person asking. Show its evidence. And know whether it may propose a change or only prepare one for review.

The prose quality is the least interesting part. The product is the chain of custody between the question and the outcome.

What Alfred actually is

Alfred is a feature-sliced FastAPI application. The agent lives in app/features/alfred/, split into api/, domain/services/, and domain/services/tools/. Two classes carry most of the weight:

• AgentLoop (app/features/alfred/domain/services/agent_loop.py, 1,889 lines) drives a turn.
• ToolRegistry (.../tools/registry.py, 2,548 lines) owns every capability the model can invoke.

The numbers that define the operating envelope, all from app/shared/config.py:

Setting	Value	What it bounds
max_iterations	30	Model calls per user message
max_output_tokens	64,000	Per model call
compaction_trigger_tokens	150,000	When history gets summarised
keep_recent_turns	4	Turns preserved verbatim through compaction
tool_result_max_chars	30,000	Per tool result entering context
sql_row_limit	200	Rows any query can return
sql_statement_timeout_ms	10,000	Per query
max_parallel_runs	3	Concurrent runs per user
Default monthly budget	$25 USD	Per user, enforced before work starts

There are 45 tool names defined. Forty-four are reachable by the model, and the maximum advertised to a single admin or tech user in one request is 42 client tools, plus Anthropic’s server-side web_search and whatever MCP connector toolsets that user’s roles allow.

The role model is deliberately small. Three roles: admin, operations, tech. Ten capabilities in a single dict at app/features/users/domain/permissions.py.

Six subsystems, and where each one lives

1. Perception

Alfred does not get a connection to the company. It gets narrow typed tools, each with a purpose, a permission boundary and a bounded response. Reads reach call_logs and organizations in nexa_prod; calls, llm_calls, telephony_calls and a dozen latency and quality metric tables in batman; GitHub repositories through a scoped org token; twelve managed business connectors through a gateway we run ourselves; a user’s linked Gmail; and a browser page snapshot the user explicitly shares.

The design rule is that a good tool says “here is the smallest defensible slice,” not “here is everything.” Part 2 covers this in detail.

2. Grounding

Every tool result is persisted as a tool_result block in chat_messages, JSONB, alongside the tool_use block that requested it. That means a conclusion reached three weeks ago can still be traced to the exact evidence that produced it. Governed memory entries arrive in the prompt carrying their provenance:

# agent_loop.py, memory entries rendered for the prompt
f"[memory:{page_id}; confidence={confidence:.2f}; updated={updated}]"

with source URIs appended as a Sources: suffix.

3. Reasoning

Model routing is a cost and latency decision as much as a quality one. Ordinary work runs on Sonnet. Difficult planning escalates to Opus, either through the escalate_to_opus tool or automatically via auto_opus_for_coding. Utility work such as conversation titles and history compaction runs on claude-haiku-4-5 under a separate utility_max_tokens of 2,000, and is billed to its own purpose code so it never hides inside the main response cost.

4. Memory

This is where the second brain metaphor misleads people most. Companies do not need a machine that remembers everything. They need one that knows what deserves to become shared knowledge. Alfred’s memory layer is nine Postgres tables with role-scoped visibility, revision snapshots, candidate review and provenance that is never empty. Part 3 is entirely about it, including the parts that are schema-ready but not yet wired.

5. Action

Read and write are different products. Reads are broad and cheap to allow. Writes are narrow, gated, and in the code path that matters, structurally impossible to reach without passing a check. Part 4 covers graduated agency.

6. Learning

Gaps, memory feedback, retrieval telemetry and per-call cost accounting feed a nightly digest. Part 6 covers the loop, and is honest about which parts of it currently have no consumer.

Anatomy of a single turn

This is the part that most architecture diagrams skip, and it is where the engineering actually lives.

A user message arrives. RunRegistry acquires a per-session lock, Redis SET NX with a 900 second TTL, falling back to a chat_sessions status compare-and-set when Redis is unavailable. A background task is created that is deliberately detached from the HTTP request, so a client disconnect never cancels the work.

Then build_system_blocks assembles the system prompt in two or three blocks, and the ordering is a billing decision:

# prompts.py, on why per-query memory is NOT in the cached tier
# Governed memory retrieved for THIS query rides here, not in the
# cached tier: it is relevance-ranked per turn, so caching it would
# force a cache-write on every single turn and never hit.

Block one is STABLE_SYSTEM_PROMPT, roughly 157 lines, byte-identical on every request, marked cache_control: ephemeral. Block two is a conditional legacy knowledge base, also cached. Block three is the volatile tail: who is asking, today’s UTC date, the memory retrieved for this specific query, connector lines, Gmail account lines, skills, and the current coding phase. It is never cached, on purpose.

The consequence is worth stating plainly: every turn pays full input price for identity, memory and connector context. We accepted that because the alternative, caching a block that changes every turn, means paying the 1.25x cache-write multiplier on every turn and never once getting the 0.1x read.

Then the loop runs:

while iterations < self._settings.max_iterations:
    # stream a Messages API call
    # if stop_reason == "tool_use": execute tools, persist results, continue
    # else: break

Tool execution is parallel. Every tool_use block in one assistant message launches immediately, then results are awaited in request order so the history stays deterministic:

task = asyncio.create_task(
    self._tools.execute(block.name, dict(block.input or {}), tool_context)
)
pending.append((block, task))
...
for block, task in pending:
    result = await task

There is no semaphore here and no per-tool timeout in the loop. Timeouts live inside individual tools: 10 seconds for a SQL statement, 30 for a GitHub API call, 120 for a git subprocess, up to 900 for a configured validation command. That is a real limitation and we know it. A tool with a long internal timeout can hold a turn open.

Every tool returns the same shape, and it is always a string:

@dataclass(frozen=True)
class ToolExecutionResult:
    content: str
    is_error: bool

A tool never raises into the loop. Errors are caught, converted, and handed back to the model as text so it can adjust:

except Exception as exc:  # surfaced to the model so it can adjust
    logger.exception("Tool %s failed", name)
    return ToolExecutionResult(content=f"Tool failed: {exc}", is_error=True)

There are no automatic tool retries. Retry is the model’s job, and the prompt says so: “If a tool call fails, retry it or take another approach immediately, in the same turn.” The only backoff in the system is at the model stream layer, where a retryable status code triggers exponential retry with one hard rule.

_RETRYABLE_STATUS_CODES = frozenset({408, 409, 429, 500, 502, 503, 504, 529})

The rule: if produced_output or not retryable: raise. A turn that has already streamed text to the user is never replayed. Duplicated partial output is worse than a clean error.

Trust has to live in the architecture

The single most useful thing we learned is that trust cannot be added at the end as a paragraph in a system prompt. Prompt instructions are probabilistic. Security boundaries should not be.

Take read-only SQL. The prompt does tell the model that run_sql is read only. That instruction is not what makes it true. Four independent layers do:

First, validation. validate_read_only_query strips block comments, line comments, string literals and dollar-quoted bodies, then requires the first keyword to be select or with, rejects any semicolon before the trailing one, and blocks 24 keywords including insert, copy, call, do, set, lock, prepare and security.

Second, structure. The statement Alfred actually executes is not the model’s statement. It is a wrapper:

return f"SELECT * FROM (\n{inner}\n) AS _agent_result LIMIT {max_rows + 1}"

Postgres refuses data-modifying statements, including data-modifying CTEs, below top level. The + 1 is how truncation gets detected rather than silently hidden.

Third, and this is the load-bearing one, the transaction itself:

await conn.set_read_only(True)   # this is the real enforcement boundary
...
await conn.rollback()            # always, on every path

Fourth, optionally, SELECT-only database credentials.

The comment in sql_tool.py above the keyword denylist is honest about the hierarchy: “Defense-in-depth validation. The real boundary is the read-only transaction.” Layer one exists to give the model a useful error message. Layer three exists to make the mistake impossible.

The same pattern repeats everywhere it matters. The internal API proxy tool is GET only, dispatched in-process over httpx.ASGITransport with the caller’s own bearer token, so per-route RBAC still applies. It carries both an allowlist and a denylist:

ALLOWED_PREFIXES = ("/calls", "/batches", "/logs", "/recordings",
                    "/transcripts", "/billing", "/settings", "/users", "/operations")
BLOCKED_PREFIXES = ("/alfred", "/auth", "/discord-relay")  # recursion / auth surfaces

Repository writes can only ever target a generated branch under batman-agent/, are always non-force, and there is no merge primitive at all. The comment in repo_manager.py states the design intent: “There is deliberately no primitive that writes without opening a PR and no merge primitive.”

And the strongest defense against prompt injection is not a classifier. It is that an injected instruction which convinces the reasoning layer to attempt something inappropriate still hits a server-side role check, an endpoint allowlist or a read-only transaction, and gets refused. The line in agent_loop.py that sets the coding approval requirement from capability rather than from phrasing says it best:

# The heuristic improves routing, but safety cannot depend on phrasing.
coding_preflight_required=repository_write_capable,

The part demos skip

The first Alfred demos were exciting. Ask a question, watch it inspect real evidence, get a useful answer.

The work that made people actually depend on it was less glamorous, and it was all failure handling:

Two requests colliding in one conversation. A user closing the tab mid-investigation. A deployment restarting the process between a tool request and its result, leaving an orphaned tool_use block that makes the next API call structurally invalid. A conversation reaching the context limit. A connector becoming unavailable. Concurrent runs racing past a budget.

So there is repair_unpaired_tool_uses, which injects synthetic error results in memory so a replayed history is valid, with text that tells the model the truth: “This tool call was interrupted by a server restart before it finished.” There is heal_stale_runs() at startup to clear leftover running claims. There is downgrade_unavailable_mcp_blocks for history that references a connector which has since been revoked. Optional subsystems load through asyncio.gather(..., return_exceptions=True) so a missing repository token disables coding rather than breaking chat for everyone.

None of this demos well. It is the difference between a system people try twice and one they use on a Tuesday night when something is broken.

What is not built yet

Writing this series meant reading our own code carefully, which surfaced a set of honest gaps. Naming them is more useful than hiding them.

There are no per-tool timeouts in the agent loop and no bound on tool fan-out. Cancellation cancels the driving task but does not appear to cancel in-flight tool tasks, so a cancel during a git push does not stop the push. Idempotency keys exist in exactly one place, the managed connector write path, and are model-supplied and optional, which means a replayed turn could double-send an email or double-push a branch. There is no structured per-tool-call audit table; auditability depends entirely on the chat transcript in Postgres surviving. The capability model is effectively one bit: manage_repository_code covers eleven tools, and sending email as the user, scheduling autonomous spend and overwriting a team-shared skill require no capability beyond having Alfred enabled.

Each of those is a real item on a real backlog, not a philosophical position.

What building this changed

Before Alfred, it was easy to assume progress would come mostly from stronger models. Stronger models helped. They were not the hard part.

The hard part was organisational. We had to decide who owns a fact and who can approve it. Which actions are reversible. What evidence stays attached to a conclusion. When the system should stop and ask. How much autonomy a specific role gets for a specific tool.

Those decisions sit at the intersection of product, engineering and company culture, which is why a second brain cannot be bought as a model upgrade.

One smaller thing surprised me. The interface matters less than the continuity. Alfred shows up in its own workspace, in a Discord thread, through a scheduled Repeat task, or reading a page a user shared from their browser. The surface changes. Identity, permissions, evidence, memory and task history have to stay coherent across all of them. People do not want another destination to visit. They want intelligence where the work already is, without losing the guardrails.

The series

1. This article. Why a second brain is a system, not a chat box.
2. Grounding. How an agent accesses company reality without drowning in data: tool design, read-only enforcement, connector gateways and the browser bridge.
3. Memory. Why we chose Postgres full-text search over a vector database, and what governed memory actually requires.
4. Agency. Graduated permissions, plan approval gates, and why the pull request is the product boundary.
5. Reliability. Run locks, event replay, history repair, compaction and scheduled autonomy.
6. Learning. Gaps, feedback, cost as a behavioural signal, and the nightly digest.

Next: grounding. Because an AI cannot become your second brain if it cannot see what your first brain is dealing with.

If you are building agents inside your company, I would like to know where trust breaks first for your team: access, accuracy, memory, action or reliability.

---

Alfred runs on DialNexa, our voice AI platform for businesses. If you are building internal agents and want to compare notes, the comments are open.

Written by
Aditya Kamat
|
Published Jul 27, 2026

Co-Founder, DialNexa

Co-Founder of DialNexa. Expert in voice AI, conversational technology, and enterprise telephony. Building the future of AI-powered customer engagement.