# Canopy Designed and built end to end, KIKI World · 2022 to 2026 · Published October 6, 2026 A multi-tenant agent operating system. Many specialized agents run against live tenant systems under one intelligence contract, one output contract, one review path, and one learned decision layer. ## Thesis Canopy is a runtime that runs many specialized agents against live tenant systems. They share one intelligence contract, one output contract, one review-and-settle path, one learning program, and one decision layer. The unit of work is a discipline: a researched craft, implemented once as a templated worker. A tenant is an activation row, a layered identity, and optional connector credentials. Adding a tenant is a row. Adding a craft is a folder. The operator agent is the same system speaking through a conversation, not a second stack on top of it. Two failures define the contract by contrast. A single model call wrapped around a hardcoded rubric is one. A chat surface that maps a phrase to a webhook and calls the result orchestration is the other. Both put a model where a system should be. The path to keep in mind is concrete. The operator agent takes a confirmation and enqueues a discipline job. The worker perceives live state, selects a discrete plan, reasons, renders under the output contract, and inserts a reviewable. A human settles it. A publisher claims the row before any side-effecting write. The decision layer records the episode, and the next run starts from that record. ## Work classes and admission Interactive streaming and batch consumers once shared one event loop and one Redis. A worker deploy bounced the conversation stream, and fleet ingest starved chat. The failure was contention between classes of work, not tenant count, so the floor that replaced it is a process split. - The **HTTP process** owns the conversation streams over server-sent events, compose and lint for public copy, webhook enqueue, and health. It never starts a queue consumer and never registers a schedule. - The **workers process** owns every BullMQ consumer, a singleton scheduler behind a Redis lock, the dead-letter handler, and the fleet heartbeat. It has no public hostname. Deploys are independent. On shutdown, the HTTP process refuses new interactive work and lets in-flight streams finish, and workers return active jobs to the queue under the same job id, so a second replica continues them. Isolation is admission, enforced by four Redis semaphores: - an **interactive** pool, which refuses with a retry hint rather than start a compose it cannot finish; - a **global batch** pool, which defers a job to delayed rather than hold it active; - a **per-tenant batch** pool, acquired first, so a workspace at its cap never briefly holds a fleet slot; - an **ingest** pool, so a knowledge refresh at its cap neither occupies the audit pool nor refuses chat. Provider quota is a token bucket that delays work and never refuses chat. ```mermaid %% Two processes, one image. Admission isolates the classes of work, not one client per workflow. flowchart TD console["Operator console"] --> http["HTTP process: streams, compose, enqueue"] http -->|"interactive pool"| redis["Redis: queues, locks, semaphores"] workers["Workers process: consumers, scheduler"] -->|"batch, tenant, ingest pools"| redis http --> pg["Postgres"] workers --> pg ``` ## The intelligence contract An agent that perceives a system and produces a reviewable is an agent only when it implements four layers and four faculties, then passes three gates. The layers run from general to particular. The standard is thin and written once. The discipline is a researched craft, specialized in that craft rather than in one tenant's instance of it. The tenant mechanism is reusable and holds no tenant's data. The live instance is one real tenant's knowledge, configuration, and accumulating outcomes. Depth lives in the discipline. A thin discipline that passes the contract is a hollow agent. The faculties are what each worker does on every run. - **Perception** reads the actual state of the system from the most authoritative source available. Cheap signals seed reasoning. They never become findings. - **Cognition** is a bounded hypothesize-then-verify tool loop. Each hypothesis is verified with a tool before it may become a finding. The loop caps model turns and tool calls, and on error returns what it accumulated with a degrade flag. It never throws. - **Living knowledge** retrieves the discipline's playbook by kind, with a code seed so a fresh environment still grounds. New tactics enter as rows, not as a redeploy. - **Outcome** reads the prior run and the real performance of what the agent touched. An unmeasurable outcome stays blended. It is never zeroed. The gates decide whether a reviewable is client-facing: the output contract, schema validation before insert, and an evaluation against a frozen-rubric baseline on a golden set. Discipline reasoning and the operator agent share one orchestrator: plan, act, observe, repeat, inside hard caps with a graceful termination reason. A usage sink fires after every model call, so cost accounting never depends on a caller remembering to meter. ## The operator agent The operator agent is the front door to every discipline. One run function serves every surface, each through its own channel formatter, so there is no second brain. It acts only for its own tenant. The tenant id is closed over in the tool factory and is never a model input. Its charter has six roles. As strategist, it answers a compound ask by composing a coordination graph of agents, grounded against this workspace's installed capabilities, then drives each step confirm-first. As extrapolator, it reasons from this workspace's audits, identity, and memory, never from a generic definition. As analyst, it computes when no curated read answers the question. As creative collaborator, it composes and revises copy inside the conversation. As coordinator, it reads each landed artifact back and builds the next recommendation on it. As memory, it carries goals, preferences, corrections, and open work across sessions. A trigger returns a queued acknowledgment with a link and an estimated duration. It never fabricates findings inline. Its job id is byte-identical to the worker webhook's and the scheduled dispatcher's, so three callers collapse to one run. A per-conversation turn lock, with a heartbeat, makes a second concurrent send return busy. High-stakes writes stage a proposal and then require a typed phrase. A capability is complete only when it can be a graph step, read back, remembered, covered by an evaluation case, and computed over. ## The analysis loop A summarized read leaves an agent able to relay a conclusion and unable to answer the question one step to the left of it. So when a question matches no curated read, the agent acquires the bounded raw dataset that covers it and computes. A dataset primitive fetches rows server-side into a per-conversation store. The model receives a handle, the schema, the row count, the window actually covered, a source label, and a small sample. Rows never enter the model's context in bulk, and a handle resolves only inside the conversation that created it. The model then writes a transform, which runs in a QuickJS sandbox compiled to WebAssembly, with no filesystem, no network, no host references, and hard memory and time caps. The model designs the analysis and the sandbox executes it. A cross-source question becomes one row-level join inside the sandbox, not two summaries stitched together in the model's head. Generated code that is not an analysis over tenant rows runs elsewhere, in separate kernel-isolated environments. The answer carries its provenance: source, window, and caveats. When the data is unreachable, the agent abstains. It names the gap and says what connecting would unlock. It never presents a wider window as if it answered the one asked. ## The output contract and retrieval Every public prompt composes from one tree of written registers, in a locked order: identity, surface rules, the advisor, outbound and reply registers where they apply, channel examples, and a shared anti-style list last, because models weight the latest context most heavily. The shared list removes machine artifacts and does not impose a house voice. Two tenant identities compile through the same composer and look nothing alike. The refine path is to think, compose, critique, revise, and then lint deterministically. A critical lint violation never lands a reviewable. Factual claims recall tenant knowledge first and cite it, so an unverified metric fails lint. Retrieval is hybrid. Vector and keyword search run in parallel and are merged by reciprocal rank fusion, biased toward semantics, while keyword search keeps exact strings from disappearing. Tenant scope and kind are predicates in the query, not a filter applied afterwards. Provenance is ranked, so a delivered intake overrides what was synthesized from a public site, and a tenant's own edit overrides both. ## Review and side-effecting writes Nothing leaves the system without a human settle. A worker validates the artifact, runs the output contract, and inserts a pending row on the review ledger. A human accepts, edits and accepts, or rejects it. Settling fires a trigger that enqueues a publisher job. The publisher contract is ordered: 1. Fetch the full row. 2. Type-gate on kind and sink **before** claiming. A mismatched row is a no-op without a claim. 3. Atomically claim the row. 4. Perform the side-effecting write upstream. 5. Mark it published on success. On failure, release, log, alert, and throw. Claiming before the type check is a known loop: claim, mismatch, release, and the trigger fires again. The defenses are the gate ahead of the claim, a CI scan that fails any publisher that claims without one, and a rate cap per subscriber. Idempotency is mechanical, through job ids derived from the artifact, the hour, or the tenant and date, so re-fires collapse. Autonomous actions run under the same discipline as approved ones: a mandate with a spending limit and an expiry, a kill switch, and the same atomic claim before the write. Disciplines talk to families of external systems through one interface per family and never branch on a provider's name. Tool access over MCP and delegation over A2A carry scoped credentials and recorded authority. Connector credentials are verified with a live call before they are stored, scoped read-only by default, and never held in a table, a config blob, or a file. A demo workspace routes to synthetic connectors whatever its keys, so it cannot leak a live call. ## The decision layer The compounding advantage is a learned policy over plans, not a fine-tuned renderer. The frontier model stays frozen and renders under the three gates. What is learned is which discrete plan to choose for a context. Plans are factored from the discipline's own doctrine, versioned, and legalized by feasibility masks drawn from live context. A shared runtime selects the plan before the worker renders, in one of three modes: observation logs the choice without steering, canary splits arms on real runs, and on steers. ```mermaid %% Train on the richest signal, promote on the most independent one. flowchart TD ctx["Context features"] --> policy["Plan policy"] policy --> plan["Discrete plan"] plan --> render["Frozen renderer, three gates"] render --> ledger["Review ledger"] ledger --> episode["Policy episode"] episode --> train["Offline training: BC, AWR, preferences"] train --> tiers["Independent tiers: verifiers, backtest, replay"] tiers --> canary["Canary arms on real runs"] canary --> policy ``` Organic settles are too scarce to be the only fuel, so fuel is manufactured through the real compose path from sampled contexts, with a spending envelope that stops sampling at a cap. Two lanes score it. The absolute lane uses deterministic rubrics and craft scorers. The pairwise lane uses a judge from a different model family that runs both orderings. A pair whose winner does not survive the swap is stored as a tie and excluded. A Bradley-Terry fit becomes a versioned artifact, which the trainer consumes beside behaviour cloning and advantage-weighted regression. A model-scored rubric dimension earns reward weight only after it clears a chance-corrected agreement floor against human gold. Real episodes are never deleted to make room for synthetic ones. Promotion runs on tiers that did not train the policy: deterministic verifiers, backtests against reserved corpora, replay of historical outcomes, and finally live comparative evidence from canaries. A policy shaped by a judge is not promoted on that judge's word. A margin names its tier, or it is not a margin. ## Cost and discipline Agent runs are metered like a production system, traced with the OpenTelemetry GenAI conventions. The platform's credits and the run's actual cost meet on one run id, carried by the review row, every costed step, and the usage row. Cost is priced at write, so a later change to a price list does not rewrite history, and a missing cost stays null rather than becoming zero. Four pairing rules keep a large fleet coherent. A producer lands with its consumer, so no screen exists for data nothing writes. A new artifact family lands with its operator-agent wiring. Merged is not shipped until the migration is applied and verified. Counts are generated from code, never typed by hand.