4 min read

Version the Agent Package, Not Just the Prompt

Mehdi Rezaei
Mehdi
Author
Engineering
Software
Technology

Most teams still "version" an agent by editing a prompt in a UI and hoping staging behaves the same next Tuesday. That is how you get silent regressions: same chat, different tools, different model, different permissions, and nobody can say which combination produced last week's good answers.

If the agent ships to customers or sits on a production path, treat it like an API. Version the whole package.

A prompt is not a release unit

Prompt text is one input. Behavior also depends on:

  • which tools exist and what their JSON schemas say
  • which model and provider pin you use
  • temperature / reasoning settings that change tool-calling style
  • permission sets and approval gates
  • retrieval corpora or skill files loaded into context
  • eval cases that define "still good enough to ship"

Change any of those and you changed the agent. Shipping only a prompt diff is like shipping a new HTTP handler while silently swapping the database client and calling it a copy tweak.

I have watched incidents where support swore the prompt was unchanged. They were right. Someone had widened an MCP allowlist. The model started calling a write tool it previously could not see. The transcript looked similar. The side effects were not.

Put a manifest next to the code

Keep an agent definition in the repo. Pin versions. Review it in PRs. Deploy it the same way you deploy services.

ts
1// agents/support-triage/manifest.ts
2export const agentManifest = {
3  name: "support-triage",
4  version: "1.4.0", // semver of the package, not the model
5  model: {
6    provider: "openai",
7    id: "gpt-5.2",
8    pinned: true,
9  },
10  instructions: "./SYSTEM.md",
11  tools: [
12    { name: "searchDocs", schema: "./tools/searchDocs.json", permission: "read" },
13    { name: "createTicketNote", schema: "./tools/createTicketNote.json", permission: "write_approved" },
14  ],
15  permissions: {
16    default: "read",
17    writeRequiresApproval: true,
18    forbidden: ["refund", "deleteCustomer"],
19  },
20  evals: "./evals/*.json",
21} as const

Or the same idea as JSON if your runtime loads config, not TypeScript. The format matters less than the rule: one version string names the whole bundle. Bump it when tools, schemas, instructions, model pin, or permission policy change. Do not bump it for typo-only docs that the agent never loads.

Wire runtime load to that version:

ts
1const agent = await loadAgent("support-triage@1.4.0")
2// logs: agent_version, model_id, tool_hashes, permission_hash

When a bad answer ships, you can replay `1.4.0` instead of archaeology through Slack edits.

Hash the pieces that matter at load time: instruction file hash, each tool schema hash, permission config hash, model id. Log them with the request id. Support tickets become "which package?" instead of "what did we tell it last month?"

Semver for agents, not vibes

Borrow API semver and make it boring:

  • **patch**: instruction wording that does not change tools, permissions, or expected tool traces; typo and clarification only
  • **minor**: new read-only tool, tighter schema, new eval cases, optional capability behind a flag defaulted off
  • **major**: new write tool, removed refusal, wider permissions, model family change, or any change that invalidates prior eval expectations

Model pins belong in the package. "Use the latest" is not a pin. If the provider renames or retires an id, that is a deliberate upgrade PR with evals, not an automatic surprise on Monday morning.

Evals are part of the package or they do not count

A versioned agent without eval cases is a tagged guess. Keep a small set of fixtures next to the manifest: inputs, allowed tool traces, forbidden tool names, and expected citation or refusal shapes. Run them on every manifest PR. Fail the PR when a tool schema change breaks an expected call, or when a permission change suddenly allows `refund`.

You do not need a research benchmark. You need regression guards for the behaviors that already hurt you once: leaking internal ids, inventing policy, calling write tools on read-only tickets, skipping the docs search before answering.

Store golden traces as data, not screenshots of a chat UI. If you cannot run the suite in CI against the same loader production uses, the suite is theater.

Staging is not "a different prompt"

Mirror production package versions in staging. Promote `1.4.0` the way you promote a build artifact. Canary a percentage of traffic on `1.5.0` if the agent is customer-facing. Roll back by version string, not by paste-reverting a dashboard field under pressure.

Dashboard prompt editors are fine for experiments. They are a weak source of truth for anything with credentials attached.

What to do this week

Pick one production agent. Move its system prompt out of the dashboard into a file. List its tools and schemas in the same directory. Pin the model id. Write five eval cases from real failures. Tag the directory `1.0.0` and make the runtime log that version on every request.

Next change that alters tools or permissions gets a minor or major bump, a PR, and an eval run — the same discipline you already apply to public APIs. Prompt-only edits can stay patches. Everything else is a release.

Agents that matter deserve release hygiene. Version the package, or keep rediscovering the same outage with a newer prompt.

Share this article