"Can this run privately?" sounds like a deployment question.
It is usually a data-boundary question.
When legal teams, healthcare organizations, financial operations groups, manufacturers, government contractors, or other regulated businesses evaluate AI knowledge systems, they are rarely worried about where a chat window is hosted. They are worried about what data leaves their control, which vendors can process it, where logs are stored, who can retrieve sensitive documents, how access is revoked, and whether they can explain what happened after the fact.
That means private AI architecture starts with data flows, not model branding.
An on-prem model can still sit on top of a badly governed data pipeline. A cloud model can sometimes operate inside a carefully constrained private architecture. The correct design depends on the organization's security posture, contractual obligations, data classification, latency needs, infrastructure, and tolerance for external dependencies.
This article is an engineering overview, not legal or compliance advice. Your legal, privacy, and security teams should define the obligations that the system must satisfy.
Private AI is not one deployment pattern
Teams often use "private AI" to describe several different architectures.
They are not equivalent.
Managed cloud with enterprise controls
The application and model are hosted by external providers, but data handling, retention, access, and contractual controls are configured for enterprise use.
Private network or VPC deployment
The application, retrieval layer, databases, and sometimes model endpoints run inside a private cloud network controlled by the customer.
Hybrid architecture
Sensitive retrieval and business data stay inside the customer's environment while selected model requests are sent to an approved external endpoint under strict controls.
On-prem deployment
Core services run on infrastructure physically or logically controlled by the organization, including local model inference where required.
Air-gapped or highly isolated deployment
The system operates with little or no external connectivity and uses approved local models, local indexes, and controlled update processes.
The right choice is not automatically the most isolated option. It is the architecture that satisfies actual requirements without creating unnecessary operational burden.
See Secure and Private AI and On-Prem LLM Deployment for related patterns.
Start with a data-flow diagram
Before arguing about cloud versus on-prem, draw the system.
For a private knowledge base, map:
- where source documents live
- how documents are ingested
- where extracted text is stored
- where indexes are stored
- where embeddings are generated, if used
- where prompts are assembled
- where model inference occurs
- where traces and logs are stored
- where user feedback is stored
- where backups are stored
- which external services receive any part of the data
Then classify the data at each step.
A surprising number of privacy problems appear outside the main model call. A team may carefully choose a private model endpoint while sending raw documents to an external parsing service, copying prompts into analytics, or storing complete traces in a third-party observability platform.
Private architecture is only as private as the least controlled data path.
Keep authorization inside retrieval
A company knowledge base should not become one giant pool of text just because it is internally hosted.
Employees already have different access rights in systems such as document stores, legal matter repositories, HR platforms, ticketing systems, and customer databases.
The AI layer should respect those boundaries.
A robust design links the user's identity to retrieval scope before context reaches the model.
Depending on the organization, scope may be based on:
- tenant
- legal matter
- customer account
- department
- project
- office
- security group
- document ACL
- role
- geography
- data classification
The interface should not be the only enforcement point.
If a user is not allowed to access a document, that document should not be retrieved into the model context and merely hidden later.
This is why Role-Based Access Control is part of knowledge-system architecture rather than a UI feature.
Decide whether the model itself must be local
Some organizations need local inference. Others do not.
If a policy requires that sensitive prompts cannot leave a controlled environment, local or private-hosted inference may be mandatory.
If approved external model endpoints can be used under defined contracts and technical controls, a hybrid architecture may provide better model quality or lower operational overhead.
The engineering team should not make this decision by instinct.
Ask:
- What data is sent to the model?
- Is that data personal, confidential, privileged, regulated, or export-controlled?
- Can sensitive fields be removed before inference?
- Is external processing contractually permitted?
- Are retention settings acceptable?
- What are the availability requirements?
- What latency is acceptable?
- Can the organization operate model infrastructure reliably?
- Are the required model capabilities available in the private environment?
Sometimes the best architecture uses a small local model for classification and routing, deterministic retrieval for sensitive records, and an approved larger model only for sanitized synthesis.
There is no requirement that every task use the same model.
Retrieval can be more sensitive than generation
In a knowledge system, the retrieval layer often touches more sensitive data than the model ever sees.
The index may contain references to every document. Metadata may reveal customer names, matter numbers, employee information, internal projects, or document classifications.
If embeddings are used, organizations may also need to decide whether embeddings are treated as sensitive derived data under their policies.
That means the index deserves the same architecture review as the source database.
Questions include:
- Where is the index stored?
- Is it encrypted at rest and in transit?
- Can one tenant query another tenant's vectors?
- How are deletes propagated?
- How are access changes reflected?
- Who can administer the index?
- Are backups covered by retention policies?
This is one reason we prefer retrieval architectures that preserve source identity and explicit metadata rather than treating the index as an opaque AI asset.
On-prem does not eliminate operations work
Running locally gives control. It also creates responsibility.
Someone has to own:
- model serving
- hardware capacity
- patching
- dependency updates
- vulnerability management
- monitoring
- backups
- failover
- index rebuilds
- model upgrades
- performance tuning
- deployment automation
If the organization already operates private infrastructure at this level, that may be routine.
If it does not, on-prem AI can create a new platform team by accident.
A realistic architecture should include total operating cost, not only inference cost.
Sometimes a private cloud environment with strict boundaries is operationally safer than an on-prem stack that nobody has time to maintain.
Design the knowledge lifecycle
A regulated knowledge base needs to know not only how data gets in, but how it changes and leaves.
You should be able to answer:
- How is a new document published?
- How quickly is it searchable?
- What happens when a document is replaced?
- Are old versions retained?
- Which version is authoritative?
- How is a deleted source removed from the index?
- What happens when a user's access is revoked?
- How long are prompts and traces retained?
- How are backups expired?
This is where many AI projects inherit risk from ordinary data engineering.
If your deletion process removes a file from the source drive but leaves its extracted text, vector representation, cached answer, and debug trace indefinitely, the AI layer is not aligned with the source lifecycle.
Auditability should be designed before launch
In sensitive workflows, you may need to reconstruct how an answer was produced.
A useful audit record can include:
- authenticated user
- timestamp
- user request
- retrieval scope
- source records retrieved
- policy filters applied
- model and prompt version
- tools called
- citations returned
- approvals
- final action or response
Not every system should retain every field forever. Logging itself must follow privacy and security rules.
The point is to make traceability intentional.
Our Audit Logs and Traceability work focuses on this operational layer.
Build for source-grounded answers
Private deployment does not automatically make answers reliable.
A locally hosted model can hallucinate just as confidently as a cloud-hosted one.
For knowledge workflows, reliability usually comes from limiting the answer to approved evidence.
Useful patterns include:
- retrieve only authorized sources
- prefer current versions
- include source citations
- instruct the model to abstain when evidence is insufficient
- expose the source passage to the user
- evaluate citation accuracy
- separate facts retrieved from the source from generated explanation
A private model with weak retrieval is still a weak knowledge system.
Consider deterministic retrieval for sensitive workflows
Highly regulated workflows often contain exact identifiers, structured records, strict scope, and version requirements.
Those are strong signals for deterministic retrieval.
If a lawyer asks about a specific matter, resolve the matter first. If an operator asks about a known account, retrieve that account. If a policy has jurisdiction and effective-date rules, filter by those rules before semantic ranking.
Vector search can still help inside the authorized candidate set.
This combination improves both accuracy and explainability.
See Deterministic Retrieval for a deeper explanation.
Human approval belongs around high-impact actions
Many private knowledge bases evolve into agents.
The system begins by answering questions, then teams want it to draft responses, update records, create tickets, send notices, or start workflows.
That is where action governance becomes critical.
A useful pattern is to keep retrieval and drafting automated while requiring approval for high-impact external or irreversible actions.
Examples include:
- sending legal correspondence
- changing account status
- approving a claim
- posting a financial transaction
- altering a regulated record
- making an employment decision
The approval interface should show the evidence the system used, not only the generated recommendation.
A practical private AI architecture
For many regulated businesses, a strong baseline looks like this:
1. Sources remain in approved systems. 2. Connectors ingest only required content and metadata. 3. Data is normalized inside an approved boundary. 4. Authorization metadata stays attached to each source. 5. Indexes are stored in a controlled environment. 6. Retrieval applies identity and scope before ranking. 7. Model inference uses the minimum necessary context. 8. Answers include source references. 9. High-risk actions require validation and, where required, human approval. 10. Logs and evaluations are stored according to a defined retention policy.
The exact technologies can change. These boundaries matter more than the logo on the model endpoint.
Private should mean controllable
The goal of private AI is not to make the system isolated for the sake of isolation.
The goal is to make data use understandable and controllable.
You should know:
- what data the system can reach
- who can reach it through the AI layer
- where that data is processed
- how long derived data is retained
- what the model is allowed to do
- how bad outcomes are detected
- how access is revoked
- how the system is operated after launch
That is the difference between "we host an LLM" and "we operate a private AI system."
For implementation options, see Private LLM Architecture, Internal Knowledge and RAG, and RAG Knowledge Systems.
Next step: Talk to an engineer about applying this to your stack.