Up, Sideways, Out: Giving an Agent Fleet an Org Chart

Twelve days ago we stood up a small tree of what we call watchers: long-lived agents, each the standing representative of one system on our platform. A watcher owns a Knowledge Bank, a private database, and a persistent disk; it wakes on a cadence, runs a cheap probe against the audit trails of its system, and either goes back to sleep or digs in. There are eight of them today — a root that watches the watchers, and seven children covering the Knowledge Bank, agent session lifecycle, automations, usage and cost, document reviews, the kanban boards, and our project-management systems. Individually, each one works. Collectively, until this week, they were seven smart people in seven soundproof rooms. The organizational failure was more expensive than any individual bug they found, and the fix turned out to be an old idea from a domain that has nothing to do with software: give the fleet an org chart, and give the org chart three different channels with three different semantics.

Seven Rooms, One Incident

Here is what the failure looked like in the escalation log. On August 23 the Automation Watcher raised a medium-severity concern: a cluster of scheduled runs dying silently at launch, eight failures in thirty-five hours. On August 24 the Project Systems Watcher raised a high-severity concern about the same cluster from its own angle — the deaths were now taking out runs in its domain. The same day the Knowledge Bank Watcher raised a concern about suspend-interruption churn that turned out to share the root cause. Three well-evidenced escalations, three re-runnable queries, three sets of row IDs, one incident — and none of the three agents knew the others were looking. Each pushed its concern up, to a parent that resolved or forwarded it, and the parent was the only node in the tree that could see the shape of the whole thing.

Why didn’t they just message each other? They had a channel. Our agent mesh has a peer-to-peer messaging tool, and it works well for the agents it was built for: concurrent sessions on the same task, all awake, all polling. It queues into Redis with a thirty-minute time-to-live and delivers when the recipient makes its next tool call. A watcher, by design, is asleep roughly ninety percent of the time in stretches of two to eight hours. A message to a sleeping watcher expired before the watcher existed to read it. The audit table proved it: rows stuck at sent for weeks, never delivered, never noticed. So the agents did the rational thing with the tools they had. They escalated, because escalation was the one channel that was durable.

That is the first lesson, and it is a design lesson rather than a bug: agents route around missing channels by overloading the ones that exist. An organization with only a chain of command sends everything up the chain, and the people at the top spend their days relaying facts between peers who should have talked directly. We had built a company with a CEO and no hallways.

Three Channels, Three Semantics

The instinct when agents need to communicate is to build one thing — a channel, a bus, a shared thread — and let every kind of communication ride on it. We tried the opposite: we asked what kinds of communication a watcher actually needs, and found three, with transport requirements so different that sharing a mechanism would have served none of them well.

Escalation is state. A message is mail. A commission is a ticket. Each of those has decades of prior art, and each has a failure mode when you make it do one of the others’ jobs. Escalation used as mail clogs the chain with FYIs. Mail used as escalation means nothing ever forces a human to look. A ticket used as a message means your task board fills with “just so you know” cards that no worker can act on. The week’s work was mostly about building the second channel and closing the loop on the first; the third we already had, and what happened when the other two arrived is the most interesting part.

Up: Escalation With a Hop Budget

Escalation goes to your parent. If your parent cannot resolve it, it goes to their parent. The obvious question — the one the owner asked when he described the mechanism — is how far a concern travels before a human has to see it. Too few hops and the tree is decorative: every concern lands on a person. Too many and a critical problem can bounce between watchers for a day while the incident it describes gets worse.

Our answer is that severity buys hops. A critical concern gets zero: it goes straight to a human and is never handed to any watcher, because the cost of a wrong “I’ll handle it” is highest exactly there. A high concern gets one hop — only the direct parent gets a chance. Medium and low get three. And reaching the root forces human surfacing regardless of budget, because there is nowhere further to go. The policy is a five-line table in the code, explicitly labeled v0 and revisable, and the point of writing it down is that it can be argued about. In twelve days the fleet raised fifteen escalations; one critical, four high, seven medium, three low. Nine of the fifteen were resolved inside the tree without a human ever seeing them. The six that reached a person were the six that deserved to: a zombie session sharing a browser with its replacement, a disk-durability hole, a dedup pipeline whose deletion rate had jumped thirty-fold overnight.

Two more properties earned their keep. First, the escalation machinery is pure routing. It tracks who currently holds a concern and whether the next step is a watcher or a human, and it never sends anything itself. When a step requires human review, the tool tells the watcher so and hands back a payload; the watcher composes the human-facing message in its own words, through the same review channel every other agent uses. That keeps message quality in the agent’s hands and keeps the routing table from becoming a notification system with opinions. Second — and this is what shipped this week — the answer comes back. Until now a human could resolve an escalation, type a note, and the watcher that raised it would never learn the outcome; the note sat in a column nobody read. Now every escalation carries an origin_acknowledged_at stamp, and the watcher’s wake-time bootstrap surfaces every closed escalation it originated that it has not yet seen, then stamps the acknowledgment in the same call. Surfacing is delivery. Each outcome reaches its origin exactly once, at the next wake, including the human’s verbatim answer. The human side got the mirror image: every open escalation is answerable in place, with a delivery marker that flips from “delivers at the watcher’s next wake” to “seen by watcher 40 minutes ago.”

Sideways: Mail That Survives Sleep

The sideways channel is the one that did not exist, and its design falls almost entirely out of the sleeping-agent constraint. If a recipient may be unconscious for eight hours, the message must live in something that does not expire. So: a Postgres table, append-only, no TTL, with a thread_id to group a conversation, an in_reply_to to link a reply to its parent, and an evidence column that carries the same discipline as a finding — a re-runnable query and concrete row IDs, or it isn’t worth sending. Every message is a permanent audit record. Two agents talking to each other about production is production.

Delivery is passive, and the wake protocol is the delivery mechanism. The first thing a watcher does on waking is call its bootstrap tool, which now returns messages_inbox: every message not yet surfaced to it, urgent first, then newest first, each with the preceding messages of its thread attached as context so the agent never has to go looking for what it is replying to. Surfacing stamps delivered_at. Listing the thread or replying stamps read_at. Those two timestamps are the difference between “the system handed it over” and “the agent engaged with it,” and both are queryable — which is how we know that of 28 messages sent in the first 48 hours, 27 have been delivered and the twenty-eighth is waiting for a recipient who has not woken up yet. Nothing wakes anyone. The urgent flag only sorts. Pulling a wake forward on a high-severity message is a separate, later piece of work, because a message that can wake a sleeping VM is a message that can cost money, and we wanted to see the fleet use the quiet version first.

The rules are few and mostly about ownership. Any watcher may message any other by slug; self-messages are refused. A thread belongs to the two participants of its first message, and only they may reply on it or list it. A reply without an explicit recipient goes back to the other participant. Humans can read everything through an admin endpoint but never stamp delivery on a watcher’s behalf, because a human reading a message is not the same event as the agent receiving it, and conflating them would corrupt the one metric that tells us whether the channel works.

Finding Who to Talk To

A mailbox is useless if you do not know who to write. The watchers already had a registry search — a semantic search over the tree that returns snapshots of the relevant branch, matches plus their neighbors to a configured depth. But it embedded each watcher’s identity: name, directive, domain description. Search for “silent launch deaths” and you would get the watcher whose job description mentioned launches, not the watcher who had spent the last three wakes drowning in them.

So the registry now embeds two things per watcher. The identity row is what the watcher is for. A second, recent_findings row is a rolling digest of what it has been doing: its last five run summaries, the titles of its recent findings, its open escalations — refreshed whenever it records a run or raises a concern, and swept hourly. A watcher’s search score is its best kind, and the result says which one matched. Matched nodes come back carrying their trimmed directive, their last three findings, their open escalations, and how many messages they have exchanged this week — enough for a neighbor to decide “message them” instead of “work it myself.” The base prompt puts the sequence in one line: search before working a cross-cutting problem alone. Then talk. One well-evidenced message beats three watchers rediscovering the same root cause.

What the Fleet Did in Its First 48 Hours

We merged the messaging channel on the afternoon of August 27. We did not seed it or demonstrate it; the protocol went into the shared base prompt as one paragraph, and each watcher picked it up at its next session. Here is what the message log shows.

Within an hour of the protocol landing in the shared prompt, the Automation Watcher wrote to the Project Systems Watcher: your launch-death escalation is already characterized and carded — here is the split, and here is a test you can put in your probe. The reply, ninety minutes later: adopted your live-agent test in my probe. The week-old, three-way duplicated investigation collapsed into a division of labor in a single exchange. Two days later, after the owner directed that the lifecycle watcher take ownership of the cluster, the split was renegotiated the same way: the Lifecycle Watcher messaged both peers with an explicit lane assignment — I take the upstream launch-path root cause; you keep the execution-side evidence — the Automation Watcher replied with its corpus of failed runs, and the Lifecycle Watcher came back about four hours later having corroborated the root cause on three independent legs.

The System Usage Monitor sent the Lifecycle Watcher an early warning that nine of fourteen model-token seats were at or above 95% of their weekly cap, and got back a confirmation that despite peak saturation there had been zero model-pin drift detections in the ten hours since the fix deployed — a cross-boundary check that neither watcher’s own domain could have produced. The root watcher, which had been the only node with a view of everything, began relaying: “cc relay from the kanban watcher: this illegal-transition kill is your lane.” Two watchers sent corrections to their own earlier escalations — “the silent-death sweep IS live on prod, since 20:08Z”; “the weekly meter is the right one, not the seven-day” — which is exactly the kind of low-stakes accuracy repair that never belonged in an escalation chain and never happened when the chain was all there was. Twenty-eight messages, ten threads, eighteen replies. Nobody flagged anything urgent.

Out: Commissioning the Work

The third channel is the one we did not have to build, and the exchange that proved it is our favorite of the week. The Document Review Watcher noticed that a daily backlog-triage automation was flagging every one of its routine reports as urgent, diluting the urgency signal for the humans reading the queue. That is not its system to fix; the automation belongs to the Automation Watcher’s domain. So it sent a message: backlog-triage claims urgent on its daily report — requesting prompt calibration. The Automation Watcher replied five hours later: squarely my lane; carded as kanban 3ccfd78c, prompt calibration plus a fleet-wide sweep check. A task worker — an ordinary agent from our task pool, not a watcher — claimed the card, wrote the fix, opened a pull request, and it merged the same day. The next morning the Document Review Watcher sent one more message: verified — report #26 posted non-urgent.

Observation by one agent, routing to the agent that owns the domain, commissioning to a third agent who does the work, verification by the original observer — four agents, one human review at the merge, and a complete audit trail across three tables. The watchers did not need a new tool to commission work; they inherit the same kanban surface every agent on the platform uses. What they needed was the sideways channel to agree on who should file the card. The owner’s routing rule, written down after this exchange, is now one sentence in the design doc: escalation is vertical — parent authority or a human decision; a peer-domain concern is a message; getting work done by an agent is a task.

What We’ve Learned

Agents overload the channels you give them. When the only durable path is up, everything goes up, and your root node becomes a switchboard. Watch for the relay pattern in your escalation log; it is the symptom of a missing hallway.

Different directions want different semantics. Escalation is state with bounded hops. Mail is durable and exactly-once. Work is a ticket with a lifecycle. Each is a solved problem on its own; bolting them into one mechanism unsolves all three.

Make the transport match the sleep schedule. A message queue with a TTL shorter than the recipient’s nap is a message queue that silently drops mail. For agents that sleep, durable storage is not an optimization; it is the definition of delivery.

Surfacing is delivery, and stamp it. Whether the artifact is a message or a human’s answer to an escalation, the moment the agent’s bootstrap returns it is the moment it was delivered — write that timestamp in the same transaction. Exactly-once falls out for free, and so does the metric that tells you whether the channel is alive.

Severity should buy hops, not bypass judgment. A hop budget lets the tree absorb what it can and forces a human on what it cannot. Sixty percent of our escalations never reached a person. The forty percent that did were the right forty percent.

Embed what an agent is doing, not just what it is for. Discovery by job description finds the person whose title matches. Discovery by recent findings finds the person already working the problem. Only the second one prevents duplicated work.