“The model provider does not retain our prompts” is a useful property. It is not the same thing as “our AI feature does not retain customer data.” Teams confuse those statements because the model call is the visible part of an agent. The data trail around it is where most of the system actually lives.
OpenAI's Zero Data Retention offering makes the provider-side promise much clearer for eligible API customers: prompts and responses are not retained after processing. That changes an important boundary. It does not delete the copy in your request logs, the trace in your observability vendor, the tool arguments in a queue, the failed job in a dead-letter store, or the transcript a support workflow saved for later.
The practical response is not to dismiss ZDR. It is to use it accurately: as one control in a data architecture that you still own.
Start with the actual path, not the chat window
Draw the path for one production request. A customer uploads a document, your backend extracts text, an agent retrieves internal context, calls a model, invokes tools, writes a result, and emits metrics. Each arrow can copy data. Each retry can copy it again.
In a real system the list is usually longer than anyone expects:
- reverse-proxy and application request logs
- serverless platform logs and exception reports
- job queues, retry records, and dead-letter messages
- LLM traces that capture prompts, tool calls, and responses
- vector stores, file stores, and temporary object storage
- analytics events, session replay, and support tickets
- database tables that store conversation state or generated output
ZDR changes one box in that diagram. It can be a very valuable box, especially for sensitive prompts and regulated workloads. But it does not give you permission to stop drawing the rest of the diagram.
The first useful design exercise is classifying what each system needs. A log usually needs a request id, latency, model id, token count, and outcome. It rarely needs the full customer prompt. A trace may need the selected tool and a sanitized failure reason. It often does not need unredacted tool arguments. A product audit record may need the final action and approval identity, not the hidden reasoning that led there.
Once you ask “what is the minimum data this component needs?”, the architecture gets simpler and safer at the same time.
Observability is where privacy promises go to die
Agent observability is useful because failures are hard to reproduce from a status code. Engineers want to inspect a prompt, every retrieved chunk, tool inputs, tool outputs, and the model response. That is a great debugging story and a terrible default retention policy.
Make full-content traces exceptional. Sample them only in a protected development environment, or gate them behind a temporary incident flag with an expiry. In production, prefer structured metadata: trace id, route, model, latency, token counts, cache hit, tool name, tool result class, and an error code. When content is necessary to diagnose an incident, redact it before export and give the trace a short retention window.
This does not mean flying blind. It means choosing observations that help answer the operational questions you actually have. Did the model call time out? Did a tool call fail authorization? Did retrieval return zero results? Did an approval wait too long? Those questions do not require a permanent copy of every customer conversation.
The same discipline applies to errors. Many frameworks capture the request object when an exception bubbles out. A convenient error-reporting integration can therefore export prompts, file names, access tokens, or tool payloads without any agent-specific code. Treat error reporting as part of your data review, not as a neutral utility installed beside it.
Do not smuggle private data through tool calls
The model provider may have a strict retention setting, while a tool it calls writes every argument to its own database. That is normal: tools need state. The mistake is making them accept more state than they need.
Pass stable identifiers instead of raw records where possible. A tool that looks up an order should receive an order id and an authorization context, not an entire customer profile copied into a JSON argument. A document-processing tool should receive a short-lived file reference, not a base64 document inside the agent transcript. A write tool should return the created record id and a result status, not echo the submitted content back into every trace.
This has a reliability benefit too. Smaller tool schemas produce smaller prompts, easier retries, clearer authorization checks, and fewer accidental copies in logs. Privacy and operational quality often point in the same direction when the system carries less incidental state.
Retention needs an owner and a clock
“We only keep data when necessary” is not a policy a database can execute. Each store needs an owner, a reason for existing, and a deletion schedule. If a system cannot say when a record expires, it will quietly become permanent.
Use explicit TTLs for temporary uploads, queue payloads, agent sessions, and debug traces. Run deletion as a normal job and measure it. Keep the minimal audit record that you need for security or customer support separately from the rich execution transcript you need only during debugging. Do not rely on a vendor's default retention period to match your product promise; defaults change and integrations multiply.
There will be exceptions. A customer may ask you to preserve a document, an investigation may require a legal hold, or a regulated workflow may need a longer audit history. Make those exceptions named states with access control and a review date. “Just in case” is not an exception category.
ZDR also changes how you debug
The tradeoff is real: if the provider will not retain a request, you cannot ask them later to retrieve it for you. Teams that rely on vendor-side history must replace that habit with better local diagnostics, not a hidden copy of all prompts.
Build deterministic replay inputs for the scenarios you own: synthetic documents, scrubbed fixtures, and recorded tool contracts with secrets removed. Log a correlation id on every boundary so an incident can be reconstructed from metadata. For high-risk flows, store a customer-approved, redacted reproduction bundle for a short period rather than raw traffic indefinitely.
This is a more disciplined debugging model. It asks engineers to make production behavior observable without treating customer content as a free telemetry feed. The effort pays off even if you never enable ZDR, because it reduces the blast radius of a logging mistake.
A practical rollout
Choose one agent that handles data you would not want in a screenshot. Map its copies end to end. Turn off full prompt logging in production. Add redaction at the boundary before traces and error reports leave your infrastructure. Replace raw tool payloads with ids or minimal fields. Put TTLs on queues, temporary files, and traces. Then enable the provider's eligible retention setting and document exactly what it covers.
Finally, test deletion. Create a known test record, let its retention period pass in a safe environment, and verify it disappears from every store you named. A privacy policy without a deletion test is just a hope that background jobs are configured correctly.
Zero Data Retention is a meaningful new lever for AI systems. Use it. Just do not let one vendor-level setting become a story your application cannot honestly tell. Your customers care about the data path, not which component happened to make the model call.