Skip to main content

Platform

Reference architecture

Six layers, one boundary, no outbound dependency. Layer contents change with your constraints; the shape rarely does.

On-premises reference architectureSix layers inside the client security boundary. From the bottom: infrastructure (GPU nodes, block and object storage, network boundary); serving (inference engines, model router, KV and prompt cache); data and retrieval (vector index, document store, connectors); agent runtime (planner, tool broker, memory, policy engine, state store); governance (identity, authorisation, audit log, evaluation, observability), which spans the full width; and interfaces (APIs, internal UI, existing line-of-business systems). Nothing crosses the boundary outward.client security boundary, no egressINTERFACESInternal APIsOIDC-protectedOperator consoletraces, approvalsLine-of-business systemsITSM, CRM, EDRMBatch / scheduled jobsAGENT RUNTIMEPlannergraph executionTool brokertyped contracts, MCPPolicy engineleast privilegeMemoryworking / episodicState storecheckpoint + replayDATA & RETRIEVALVector indexhybrid + rerankDocument storeACL-awareFeature storeSource connectorsSERVINGInference enginevLLM / SGLang / NIMModel routersize-to-taskPrompt + KV cacheEmbedding + rerankINFRASTRUCTUREGPU nodes (Kubernetes, MIG-aware)Block + object storageSegmented network, no outbound routeGovernance plane · identity (AD/LDAP, OIDC) · authorisation · immutable audit log · evaluation · OpenTelemetry
Figure 1. The full stack. Weights, traces and logs remain inside the boundary. The governance plane spans every layer.

01Infrastructure

GPU nodes under Kubernetes with the device plugin and topology awareness configured, block and object storage, and a network segment with no outbound route.

Instead, when · On smaller estates we have run single-node serving under systemd with no orchestrator at all. If you have two GPUs and one model, Kubernetes is overhead you will pay for at 03:00.

02Serving

An inference engine (vLLM or SGLang, or NVIDIA NIM where its curated model profiles and validated configurations are worth the packaging), a router that sends each task to the smallest model that handles it, and prompt and KV caching.

Instead, when · Multi-node coordination through Ray Serve when a model genuinely does not fit on one node, or when one cluster must host several models. Not before. It is a distributed system you will have to operate.

03Data & retrieval

Hybrid search with a reranking stage, a document store that carries entitlement metadata, and connectors that record provenance on ingest.

Instead, when · Where the corpus is small and stable, a well-tuned lexical index with no vectors at all. We have replaced vector search with BM25 and a reranker more than once, to the client’s benefit.

04Agent runtime

A planner executing a declared graph over typed state, a tool broker exposing typed contracts over MCP, a policy engine enforcing least privilege, tiered memory, and a state store that checkpoints at node boundaries.

Instead, when · For single-step tasks, no agent runtime at all. A retrieval-grounded call with a good evaluation set is often the correct architecture and is much easier to operate.

05Governance

Identity from your existing directory, authorisation per agent rather than per system, an append-only audit log in a framework-independent format, the evaluation harness, and OpenTelemetry throughout.

Instead, when · Nothing. This layer is not optional and it cannot be added later without rebuilding the ones above it.

06Interfaces

APIs protected by your identity provider, an operator console for traces and approvals, integrations into the systems people already use, and scheduled batch paths.

Instead, when · Where a workflow already exists in an established tool, we integrate into it rather than building another place for people to look.

Inside the agent runtime

The planner executes a graph you can read. Every step that needs a capability goes through the tool broker, which asks the policy engine first. Every transition is written to the audit log, which is also what the evaluation harness replays.

Agent runtime internalsA request enters the planner, which executes a directed graph of steps. Each step that needs a capability calls the tool broker, which checks the policy engine before invoking a typed tool. Working and episodic memory are read and written through the state store. Every transition is written to an append-only audit log, which feeds both the operator console and the evaluation harness. Human approval gates interrupt the graph and resume it from a checkpoint.Requestuser or schedulePlannerexplicit graph, typed stateTool brokerMCP servers, typed I/OPolicy enginescopes, quotas, redactionToolssearch, systems, codeApproval gateinterrupt / resumeMemory + statecheckpointsAppend-only audit log. Every step, tool call and decisionEvaluation harnessreplays tracesObservabilityOpenTelemetry
Figure 2. Agent runtime. The audit tap is a first-class path, not a logging afterthought.

Where the protocols sit

MCP standardises how an agent reaches tools and context. A2A standardises how agents delegate to one another. They are complementary layers rather than competing choices, and most systems need the first without the second.

MCP and A2A operate at different layersTwo agents sit side by side. Each agent uses the Model Context Protocol downwards to reach tools, files and databases in a standard way. The two agents talk to each other sideways using the Agent-to-Agent protocol to discover capabilities and delegate tasks. The protocols are complementary: MCP standardises an agent's access to context and tools; A2A standardises delegation between agents.Agent Aplanner + toolsAgent Bspecialist, other teamA2A, discovery, task delegation, statusMCP, one standard way to reach context and capabilityDocument storeSearch indexTicketing systemData warehouseDifferent layers, not competitors. Most systems that need both end up using both.A single agent with three tools needs MCP and no A2A. Two teams' agents cooperating need A2A on top.
Figure 3. Downwards to tools, sideways to agents.

Where would this break for you?

Every estate has something a reference design does not survive. Hardware you already own, a latency budget you cannot miss, a network with no route out. Tell us what yours is and we will go through it properly.

Talk to an engineer Security model