Stop stuffing the prompt: an agent platform where domain teams own the tools

Most assistants die at the second feature. The shape that survives: the model pulls what it needs through tools designed for it, built by the teams who own the data, reviewed against one contract.

Praveen Kumar

Praveen Kumar

Founder & Technology Leader

13 min read
On this page

The first version of an AI assistant is easy. You fetch the user's data, paste it into the prompt with a page of rules, and the model answers well enough to get a demo approved. The second feature is where it starts to die.

I have watched this happen often enough to predict the sequence. The second feature needs more data, so the handler fetches more. It needs another rule, so the prompt grows. By the fifth feature every question pays for every feature, a single failed fetch breaks every answer, and the prompt has become the place where business logic hides. Nobody can say which of the nineteen rules still matter. The team starts talking about a bigger model.

The fix is not a bigger model. The tools are the product. The model should decide what to fetch, each tool should be designed for a model to call, the teams who own the data should own the tools, and a small review contract should keep quality consistent as the catalogue grows. This article describes that shape, using a patient-portal assistant for a clinic network as the example, and ships a reference implementation you can run.

The second feature is where assistants die

Call the first pattern push. Before the model has read the question, the route handler decides what data it will need, fetches all of it, serialises it, and puts it in front of the model with instructions on how to read it. For a clinic assistant that means every prescription, every upcoming visit, the billing balance and a summary table, on every request, including "what time is my appointment tomorrow?"

Push has four properties that only get worse:

  • Cost scales with features, not with questions. Every new capability adds tokens to every request. Turn two re-sends everything. Nothing is cacheable because the date and the user's data are mixed into the prefix.
  • One failure fails everything. If the pharmacy service is slow, the appointment question gets the generic error too, because the handler fetched both before the model ran.
  • Logic migrates into prose. "Count a prescription as due if fewer than seven days of supply remain" is arithmetic. Written as a prompt rule it is right most of the time, which is the wrong number of times for a clinic.
  • Adding a feature touches five places. A fetch, a slot in the context, a prompt line, a rule, and a test that nobody writes because the behaviour lives in natural language.

Call the alternative pull. The model reads the question first, then asks for what it needs through tools. Here is the same question through both.

The same question through a push flow that fetches everything first, and a pull flow where the model asks one tool

In the pull flow, adding the medications feature meant adding one tool with a schema, a description and a test. Nothing else changed.

Pull, not push

The distinction is not whether your code fetches data. It always will. The distinction is who decides what to fetch, and when.

PushPull
Who decides what data is neededThe handler, the same for every questionThe model, after reading the question
What the model seesOne large blob plus rules for reading itA small snapshot, cached tool schemas, and the results it asked for
Adding a featureA fetch, a context slot, a prompt line, a ruleOne tool with a schema, a description and a test
An upstream outageThe whole answer failsOne tool returns an error; the model answers what it can and says what is unavailable
Cost driverContext size times turns, uncachedCached prefix plus small results; scales with question complexity

Pure pull has one cost: a trivial question like "do I have anything booked?" pays an extra round trip. So keep a snapshot, but make it the only context injected without the model asking, and keep it small. Ours is assembled from one cheap status tool per domain, fetched in parallel, and looks like this:

json
{
  "date": "2026-10-05",
  "appointments": {
    "outcome": "data",
    "data": { "next_appointment": { "clinician": "Dr Rao", "starts_at": "2026-10-08T10:30:00+00:00" }, "days_until_next": 3, "booked_in_next_30_days": 2 }
  },
  "medications": {
    "outcome": "data",
    "data": { "active_prescriptions": 2, "due_for_refill_within_7_days": 1 }
  }
}

A few hundred tokens. It gives the model today's date, the headline facts it needs to interpret any tool result, and nothing else. Adding a field to it is a deliberate decision by the team that owns that domain, not the default.

Seven rules for a tool a model can use well

Your agent is only as good as its tools. These rules came out of reviewing tools that looked fine in a pull request and produced bad answers in production. Each one is enforced by the contract checker in the reference implementation, so a tool that breaks it does not merge.

1. Question-shaped, not table-shaped

A tool called list_transactions that returns rows and leaves the aggregation to the model is a table-shaped tool. It invites three turns of fetching and arithmetic in prose. The question-shaped version is spending_summary(period, group_by): one call, one answer, the maths done in code. For the clinic that means medications_refills_due(period) returning what runs out and when, not list_prescriptions plus "please work out the dates".

2. Periods are named or explicit, resolved on the server, and echoed back

Models are bad at dates and worse at time zones. Accept period="this_month" or an explicit start and end, resolve it against the server's clock, and return the exact range you used so the answer can quote dates rather than "this month":

python
resolve_period("this_week", today=date(2026, 10, 5))
# ResolvedPeriod(period="this_week", start="2026-10-05", end="2026-10-11")

3. Typed, bounded output

Every tool declares an output schema and returns structured content. Every list takes a limit with a small default and a hard maximum. Keys stay in a stable order. The model, the renderer and the tests all read the same shape.

4. Three outcomes, always

This is the rule that matters most. A tool returns exactly one of:

  • data: here is what you asked for.
  • empty: I looked, and there is confirmed nothing, for this reason.
  • error: I could not look, with a code and whether retrying might help.

The model must never infer absence from failure. A pharmacy outage must not become "you have no prescriptions". In the kit that is a single envelope type:

python
class Envelope(BaseModel, Generic[T]):
    outcome: Literal["data", "empty", "error"]
    data: T | None = None
    reason: str | None = None          # only for empty
    error: ToolError | None = None     # only for error
    resolved: ResolvedPeriod | None = None
    as_of: str

and the system prompt says it once: empty means nothing exists, error means you could not check, say what you could not retrieve and answer the rest.

5. Deterministic maths lives in the tool

Days of supply left, months to a goal, safe-to-spend after bills, whatever your domain's arithmetic is: it belongs in the tool, unit-tested, never restated as a prompt rule. The refill tool computes days_of_supply_left from the dispensed quantity, the daily dose and the dispensed date, and the guidance paragraph tells the model the number is authoritative.

6. Descriptions are written for the model

A description is not documentation for engineers. It answers three questions: when to use this tool, when not to, and what the returned numbers mean. Together with one guidance paragraph per domain, good descriptions replace most of the rules that used to live in the prompt.

python
@server.tool(
    name="medications_refills_due",
    title="Refills due",
    description=(
        "Use when: the patient asks what is running out, when, or whether a refill is "
        "needed, for a period such as next_30_days. "
        "Do not use: to request a refill (use medications_request_refill). "
        "Returns: prescriptions that run out inside the period, soonest first, with "
        "server-computed days of supply left."
    ),
    annotations=ToolAnnotations(read_only_hint=True),
    structured_output=True,
)
async def medications_refills_due(
    period: NamedPeriod | None = "next_30_days",
    limit: Annotated[int, Field(ge=1, le=50)] = 10,
    ctx: Context = None,
) -> Envelope[list[RefillOut]]:
    ...

7. Reads and writes are different tools, and writes propose first

A write is never a flag on a read. It is its own tool, marked as not read-only, and it follows a confirmation contract: called with confirmed=false it returns a proposal and changes nothing; called with confirmed=true and the same idempotency_key it executes; called again with the same key it returns the same result instead of booking twice. The model shows the proposal, the patient agrees, then the action happens. Idempotency is what makes retries safe when the network is not.

Who owns a tool

Here is the organisational decision that decides whether the catalogue stays good. Three shapes are possible.

A single tools service owned by the AI team is the obvious first move and the wrong one. The AI team ends up re-implementing every domain's semantics one hop away from the data, drifting as the APIs change, and becoming the bottleneck for every feature. Generating tools automatically from API specifications is the other extreme: zero code, but table-shaped tools by construction, and every API change silently changes what the model can do.

The shape that lasts is the third one. Each domain team builds its tools inside the repository it already owns, next to its API, sharing the same service layer and database access. The AI team owns two things: a shared kit, and the review.

The kit is a small package every tool server is built on. It provides the result envelope, period resolution, identity extraction, telemetry, and the in-memory test harness. Domain teams write only tool bodies and descriptions, which is exactly the part they know better than anyone. One detail from the kit is worth calling out: identity comes from the request, never from a tool parameter. No tool takes a patient_id. A tool cannot be asked about another patient because there is no such argument to pass.

The review is a checklist, and the checklist is code. The reviewing team runs it against the server's published tool list, which is what the model will actually see:

python
async with Client(medications.server) as client:
    tools = (await client.list_tools()).tools
    violations = check_tools(tools, prefix="medications", instructions=client.instructions)
assert violations == []

It fails the build for a tool without a prefix, without a title, without an explicit read-only annotation, with a description missing any of the three sections, with an identity parameter, with an unbounded list, without the three-outcome envelope, or a write without an idempotency key and a confirmation step. A schema snapshot per server turns any change to a tool's shape into a contract test that someone has to approve. The reviewer's job becomes judgement about the model-facing contract, not plumbing.

The platform around the tools

With tools owned by domain teams, the platform the AI team runs is small and stays small.

Clients call an orchestrator that owns the model loop and sees one logical tool endpoint; behind it each domain team runs its own server built on the shared kit

Groups. A group is one domain, one server, one name prefix, one guidance paragraph the server publishes itself, and one cheap status tool that feeds the snapshot. Prefixes make a catalogue of twenty tools scannable: medications_refills_due and appointments_available_slots route themselves. Guidance comes from the server at connection time, so when the pharmacy team changes a definition, the guidance changes in the same pull request as the tool.

One logical endpoint. The orchestrator never knows how many servers exist. A registry connects to each, unions the tool lists, enforces the prefixes, and forwards the user's identity on every call. When a second consumer appears, a staff console or a voice channel, the registry is replaced by a gateway with the same contract and nothing above it changes. Do not add the gateway on day one; add it when the second consumer arrives.

Prompt caching by construction. Order the request as tools, then system prompt plus guidance, then messages, and put the cache breakpoints on the first two. They never change between requests. The date and the snapshot go in the first user message, after the breakpoints. The most common silent mistake is a date embedded in the system prompt, which invalidates the cache on every request. Check cache_read_input_tokens on a repeated question; zero means something per-request has leaked into the prefix.

python
return {
    "model": self.model,
    "max_tokens": 1024,
    "tools": self.registry.anthropic_tools(),   # last tool carries cache_control
    "system": self.system_blocks(),             # prompt + guidance, cache_control
    "messages": messages,                       # date and snapshot live here
}

Graceful degradation. A tool's error outcome goes back to the model as data, and the system prompt tells it to answer with what it has. No retry loop around the whole request. In the reference implementation there is a test where the pharmacy system is down: the answer still reports the next appointment and says plainly that refills could not be checked.

One structured answer, many renderings. The model decides what the answer says; code decides how each surface shows it. The model returns a canonical answer, and deterministic renderers produce Markdown for the app, plain text for voice, or JSON for a widget. Calls to action such as "confirm this booking" are typed data attached to tool results and policy, not phrases to be pattern-matched out of the text.

When this is overkill

If you have one tool, one surface and no second feature on the roadmap, a prompt with the data in it is fine, and faster to ship. If your model provider executes tools for you and you are content with that boundary, you need less of the registry. If your latency budget cannot afford a tool round trip, you are building a different kind of system and should push a curated snapshot instead. And if your tools would be written by one team anyway, the ownership argument is moot; keep the contract, skip the organisational part.

Everything else, including the first assistant that will grow, is better off with the tools as the product from the start.

The checklist

Before the next tool merges:

  • Name carries the group prefix; title and an explicit read-only annotation are set.
  • Description says when to use, when not to, and what the numbers mean.
  • Periods are named or explicit, resolved on the server, echoed back.
  • Output is typed; every list has a bounded limit with a default.
  • Result is one of data, empty with a reason, or error with a code.
  • Any arithmetic the answer depends on is in the tool and unit-tested.
  • Identity comes from the request; no identity parameter exists.
  • Writes are separate tools, propose on confirmed=false, execute on confirmed=true, and are idempotent by key.
  • The server publishes one guidance paragraph and one status tool.
  • The contract checker passes in CI, and at least one evaluation case covers the new tool.

The reference implementation is on GitHub at nexifypro/engineering-notes: the kit, two domain servers, the orchestrator, and the tests that enforce every rule above. Clone it, run the tests, and point the orchestrator at your own first tool.

Praveen Kumar

Written by

Praveen Kumar

Talk to the team

Use this article

Share

Ready to work with us?

Whether you need a dedicated team, a project partner, or strategic technology guidance, we're here to help.