An AI prototype proves possibility.
A production system proves repeatability.
Those are different engineering goals.
A prototype asks, "Can the model do this?"
Production asks, "Can the entire system do this reliably for real users, with real permissions, changing data, API failures, audit requirements, support expectations, and a team that has to maintain it six months from now?"
The transition is not mainly about adding more servers.
It is about turning implicit assumptions into explicit architecture.
Decision one: define the system boundary
The prototype often looks like one application talking to one model.
Production is usually a network of systems.
Before hardening anything, draw the boundary.
Identify:
- users
- identity provider
- source systems
- model providers
- retrieval layer
- databases
- queues
- file storage
- external APIs
- approval interfaces
- observability
- analytics
- admin tools
Then mark which components are authoritative for each important fact.
The customer record may belong to the CRM. The case state may belong to your application database. The contract belongs to the document system. The final approved action belongs to the operational platform.
If the AI layer becomes an unofficial second source of truth, production gets messy quickly.
Decision two: choose workflow versus agent deliberately
Not every AI feature needs an agent.
If the process has a known sequence, implement the sequence.
For example:
1. receive document 2. extract fields 3. validate fields 4. match account 5. create draft record 6. request approval
That is a workflow.
An agent becomes useful when the system needs to choose among tools or dynamically plan based on changing evidence.
Open-ended autonomy is not automatically more capable. It is often harder to test.
Use the least dynamic architecture that solves the business problem.
Our AI Agent Orchestration work uses agents where planning creates value and deterministic workflows where the path is already known.
Decision three: decide what the model is allowed to know
A prototype may send the model everything because it is convenient.
Production should create explicit context boundaries.
For each model call, ask:
- What information is required?
- What information is prohibited?
- What can be retrieved on demand?
- What should be redacted?
- What should remain in application logic?
Do not put entire customer records into context when three fields are enough.
Do not send confidential documents to a component that only needs a classification label.
Smaller context can improve privacy, cost, latency, and sometimes answer quality.
Decision four: design retrieval around the data
Prototype RAG often means vector search over chunks.
Production retrieval should reflect the shape of the source.
Use exact identifiers when they exist. Use structured queries for structured data. Use metadata filters for version, tenant, jurisdiction, or status. Use lexical search when terminology matters. Use semantic retrieval for conceptual discovery. Use reranking where candidate quality needs improvement.
The retrieval strategy should be chosen per query class rather than globally.
This is why Deterministic Retrieval and RAG Pipeline Design are separate architecture concerns.
Decision five: preserve source identity
A production system should know where important information came from.
If the model says a contract renews on October 1, the application should be able to point to the contract and relevant clause or structured record.
Source identity enables:
- citations
- debugging
- version control
- permission checks
- deletion propagation
- auditability
- user trust
Throwing anonymous text chunks into context removes valuable structure.
Preserve source IDs, versions, timestamps, document hierarchy, and useful metadata through the retrieval pipeline.
Decision six: treat permissions as infrastructure
Prototype permissions often mean "only our team can access the demo."
Production permissions are more complicated.
Users may belong to different:
- organizations
- departments
- legal matters
- customer accounts
- projects
- properties
- regions
- roles
Access checks should happen before sensitive information is retrieved or actions are executed.
Do not use the model to decide whether a user is authorized.
Do not rely on hiding unauthorized citations after generation.
Use actual application authorization.
Decision seven: make state durable
A prototype conversation can live in memory until the browser refreshes.
A production workflow may last hours or days.
Store important state explicitly.
Examples include:
- current workflow step
- record IDs
- completed actions
- pending approvals
- retries
- deadlines
- external transaction references
- failure reason
If the service restarts, the workflow should resume without asking the model to reconstruct history from a conversation transcript.
Durable state is especially important when agents can perform actions.
Decision eight: design tool APIs for a probabilistic caller
An agent tool is not just a function.
It is an interface that a language model may call with incomplete or incorrect assumptions.
Make tools narrow.
Validate every input.
Enforce authorization server-side.
Return structured errors.
Use idempotency for actions that may be retried.
Separate high-risk actions from low-risk actions.
A generic run_sql or call_api tool may be convenient for a prototype and dangerous in production.
Prefer business operations with clear boundaries.
Decision nine: define human approval intentionally
Some teams put humans everywhere because they do not trust the AI.
Others remove humans everywhere because they want to call the system autonomous.
Both can be wrong.
Approval belongs where the cost of a wrong action justifies it.
Common approval points include:
- external communication
- financial actions
- legal or contractual changes
- account closure
- irreversible record changes
- high-value exceptions
The review interface should show evidence, not just the model's recommendation.
Over time, low-risk actions with strong evaluation results may become automatic.
Decision ten: define failure behavior before success behavior
Ask what happens when:
- the model provider is unavailable
- retrieval returns nothing
- sources conflict
- a tool times out
- an API rate-limits you
- a write succeeds but the response is lost
- an ingestion job fails
- the user lacks permission
- the model returns malformed structured output
- an approval sits unanswered
Production systems are defined by how they handle these conditions.
A graceful failure may be a structured escalation, a retry queue, a partial result, or a message that more information is required.
The worst design is silent ambiguity.
Decision eleven: create an evaluation harness
Do not launch based on a collection of successful manual tests.
Build repeatable evaluations from real workflows.
For a knowledge system, test:
- source retrieval
- citations
- answer groundedness
- missing-answer behavior
- permissions
For an agent, test:
- tool selection
- arguments
- action sequence
- approval behavior
- retries
- final workflow outcome
Add difficult cases over time.
Every production incident that reveals a new failure pattern should become a regression test when possible.
This turns operational experience into system quality.
Decision twelve: build observability at useful boundaries
Logging every token is not the goal.
Being able to explain a failure is.
A useful production trace often includes:
- request ID
- user and tenant scope
- model version
- prompt or configuration version
- retrieval strategy
- sources retrieved
- tool calls
- validation results
- latency
- cost
- approval events
- final outcome
Handle sensitive data carefully. Traceability should not create uncontrolled copies of customer data.
See Audit Logs and Traceability for our approach.
Decision thirteen: decide who owns updates
AI systems change even when your code does not.
Models change. source data changes. APIs change. policies change. prompts change. retrieval indexes change.
Production needs ownership.
Who approves model upgrades?
Who monitors ingestion failures?
Who reviews quality metrics?
Who responds when an agent action fails?
Who maintains connectors?
Who owns the evaluation set?
A technically solid system can still decay if nobody owns these jobs.
Decision fourteen: design for model replacement
Do not assume one model provider will remain the best fit forever.
You do not need a giant abstraction layer that pretends all models are identical. You do need clean enough boundaries that changing a provider does not require rewriting the business domain.
Keep model-specific behavior near the model adapter.
Keep permissions, workflow state, source identity, validation, and business rules outside the prompt where possible.
Then evaluate a new model against the same test set before switching.
Decision fifteen: understand cost at the workflow level
Token price is only one part of AI cost.
A workflow may include:
- multiple model calls
- embeddings
- vector search
- reranking
- document parsing
- transcription
- storage
- observability
- retries
- external API calls
Measure the cost per completed business task.
A model that is twice as expensive per token may be cheaper overall if it reduces retries or uses fewer calls. A smaller model may be ideal for classification while a stronger model handles rare complex cases.
Optimize the workflow, not the marketing price of one API.
Decision sixteen: build an operational handoff
A prototype is owned by the people who built it.
Production needs runbooks.
Document:
- how to deploy
- how to roll back
- how to rotate credentials
- how to inspect failed jobs
- how to re-run ingestion
- how to disable an agent action
- how to change a model
- how to access traces
- how to handle incidents
- how to restore from backup
If only one engineer knows how the system works, you have not finished the production transition.
The architecture that matters is mostly not glamorous
Reliable AI systems are built from ordinary engineering disciplines applied to probabilistic components.
Identity. permissions. data modeling. APIs. state. queues. retries. testing. observability. deployment. ownership.
The model adds new capabilities and new failure modes. It does not repeal the old rules.
This is why our Delivery Process moves from workflow mapping to prototype validation to integration and production hardening rather than treating the prototype as a nearly finished product.
A final production-readiness test
Before launch, ask:
- Can we reproduce why an answer or action happened?
- Can we stop the system from accessing unauthorized data?
- Can we change the model without rewriting business rules?
- Can failed actions resume safely?
- Can we test quality before deploying changes?
- Can we trace source data to generated output?
- Do we know what happens when dependencies fail?
- Does a human have a clear escalation path?
- Is the cost per completed workflow measurable?
- Does someone own the system after launch?
If the answer to several of those is no, the project is still closer to a prototype than production.
That is not a criticism. It is a useful diagnosis.
Production is not a model setting. It is an operating condition.
For related services, see AI Architecture Review, AI Integration and Automation, and Secure and Private AI.
Next step: Talk to an engineer about applying this to your stack.