A common assumption is that giving an agent a 200K-token context window gives it memory. It doesn't. A context window is working space for a single turn of reasoning; memory is a separate retrieval system where facts carry sources and last-verified dates and the agent can say "I don't know this yet." The distinction decides whether an agent can tell you that your largest customer renewed nine days ago, or whether it has to re-ingest the entire thread to work it out. Context is the scratchpad. Memory is the filing cabinet.
This piece goes under the hood of AI chief of staff architecture. Four layers run the system: memory, connectors, an agent runtime with policy checks, and an approval queue. Each has its own failure modes and its own cost drivers. The goal here is a mental model you can build from, and a framework for sizing spend before any vendor conversation. If you want a product tour instead, start with the Product · Brigham.ai page.
The four layers in one view
An AI chief of staff is not one process. It is four cooperating services. Memory stores what the business knows, with sources attached and last-verified dates so the agent can quote where a claim came from and when. Connectors reach into Gmail, Google Calendar, Drive, the CRM, GitHub, Search Console, Stripe (read-only), and any MCP-compatible server to pull current state. The runtime is the loop: a model proposes a tool call, policy code decides whether to allow it, the call runs, and every result is logged with its cost. The approval queue holds sensitive actions until a named person approves the exact payload. Each layer runs independently, so a connector outage never corrupts memory, and a slow memory lookup never blocks approvals from clearing.
Layer one: persistent memory for AI agents
The memory layer is not a vector database by itself. It is a records store where every fact carries three things: a raw source (an email, a Stripe invoice, a Notion page), the extracted claim, and the timestamp of last verification. When the agent uses a fact, it cites the source. When the fact is older than a threshold, say a billing contact unchecked in 60 days, it is marked stale and either re-verified against the system of record or raised to a person.
This matters because retrieval alone does not solve hallucination. The CRAG benchmark found that even leading industry RAG systems answer only 63% of questions without hallucination. Sources and timestamps give a reviewer a way to spot the other 37%. Three operations keep the layer honest: write (capture a new fact), verify (re-check against the record), and expire (mark stale). A good implementation also records deletions and the reason for them, which turns out to be the single most useful thing in an audit.
Layer two: email ingestion for AI agents and other connectors
Connectors are where ambition meets grim reality. Email ingestion for AI agents is not "install the app and let it run." Google's own Gmail API guidance instructs developers to choose the most narrowly focused scope possible, and warns that storing restricted-scope data on servers triggers a formal security assessment. The practical consequence: a serious connector reads messages at the narrowest scope that fits the job, processes them in memory, and persists only the extracted facts, not a raw mirror of the inbox. The same logic applies to the CRM connector, which reads continuously, deduplicates, and writes back only through reversible, policy-checked calls. More on scopes and residency in Data security controls for an AI chief of staff.
Volume is not small. The average worker receives 117 emails and 153 Teams messages a day, with interruptions arriving roughly every two minutes according to Microsoft's 2025 Work Trend Index. Every connector has three failure modes (rate-limited, unauthorized, stale), and the runtime has to treat each as a recoverable event rather than a silent null.
Layer three: the agent orchestration stack
The runtime is where a model decides what to do and the system decides whether to let it. On every tool call, policy code evaluates the action: which domain is this in, who is the requester, is it a read, a reversible write, or a sensitive action, and does the current approval cover this exact payload. The 2025 OWASP Top 10 for LLM Applications ranks Prompt Injection as the #1 risk, a position it has held since the list was first compiled. A disciplined agent orchestration stack assumes the model will sometimes be manipulated and makes the policy layer, not the model, the thing that refuses.
Model choice sits inside this frame. MIRAGE-Bench, the first unified benchmark for hallucinations in interactive agent scenarios, found that even top proprietary models hallucinate at substantial rates when acting as agents, with GPT-4o scoring 0.339. The runtime treats every model output as a proposal to be checked, never as a decision that has already been made.
Layer four: the approval queue
Any sensitive action, sending email, publishing a post, deploying code, paying an invoice, changing an ad budget, or deleting records, stops in the approval queue until a named person approves the exact payload. Approvals are bound to the content, not to the agent or the job: a yes on "send this email to Jamie" does not authorize a second email with one word changed. The reasoning is worked through in approvals bound to the exact action.
Three design choices keep the queue useful. Unapproved sensitive actions have deadlines and expire rather than lingering indefinitely. Finance is a protected domain granted per person, never by role: an admin title does not confer spending authority. And the queue is fully auditable, with approvals, denials, and timeouts logged with the payload they covered.
Autonomous agent cost: where the money goes
The headline number, model tokens, is rarely the full picture. Anthropic prices Claude Sonnet 4 at $3 per million input tokens and $15 per million output tokens for prompts at or below 200K tokens, with premium rates of $6 and $22.50 applied automatically once input exceeds 200K. Base rates move in both directions depending on how you call.
Cache reads run at a tenth of base input cost. Batch processing takes 50% off. The 200K+ tier doubles input. The practical consequence is architectural: an agent that re-sends a 180K-token system prompt uncached on every run costs far more than one that caches aggressively and keeps individual turns under the 200K line. Beyond models, autonomous agent cost picks up contributions from storage (memory grows), compute (policy checks, embeddings, re-ranking), approval round-trips (latency rather than money, but it shapes design), and the specialist agents stacked on top. The driver most teams underestimate is retry cost: an agent that fails and retries three times costs three times as much per attempt, and silent retry loops are how bills go sideways.
Sizing per-person AI cost before you talk to vendors
A defensible per-person AI cost estimate has three inputs. First, daily ingested volume: emails, CRM events, calendar changes, documents touched. The 117 emails and 153 Teams messages figure from the Work Trend Index is a reasonable starting assumption for a desk worker. Second, average context size per agent call in tokens, and the cache hit rate you expect. Third, the number of specialist agents you plan to run and how often each one calls: a morning briefing, follow-up drafts, a weekly pipeline review, a monthly board pack.
Multiply through on both a low and a high assumption, add roughly 30% for retries and policy overhead, and you have a defensible monthly per-person figure. The exercise is worth more than the output, because it tells you which lever (caching, model tier, retrieval depth) matters most to your bill.
Where to go from here
If you are evaluating vendors, bring the three-input estimate above and ask the vendor to sharpen each number against their own telemetry. A credible answer will name a cache hit rate and a retry rate, not a slogan.
If you are building in-house, start with the memory layer. Connectors and runtimes are solved problems with reference implementations; a records store with sources, last-verified dates, and audit trails is where the hard work actually lives.
If you are already running a pilot, instrument two numbers: the retry rate and the share of calls crossing the 200K-token boundary. Those two alone decide whether per-person cost stays in a reasonable band or quietly climbs. When you are ready for a conversation, Get started with Brigham.ai.
