Most guidance on self-hosting AI collapses two very different workloads into one recommendation, and that conflation is why so many teams buy the wrong infrastructure. Running a large language model and running a retrieval-augmented generation pipeline place almost opposite demands on hardware. One is bound by GPU memory and parallel compute. The other is bound by RAM, disk speed, and network latency, which are exactly the resources a well-specified virtual server delivers cheaply.
Understanding that split is the difference between a private AI deployment that works on a $40 monthly server and one that stalls on a $2,000 GPU instance you did not need. Providers such as HIP.Hosting, running KVM virtualization with NVMe storage and memory-optimized tiers reaching 64GB of RAM, sit squarely in that first category. This guide covers what a VPS or VDS genuinely handles well in a private AI stack, where the honest limits sit, and how the hybrid architecture most successful teams land on actually distributes the work.
The Two Layers That Get Confused
Retrieval-augmented generation, introduced in the foundational RAG paper by Lewis and colleagues at NeurIPS 2020, combines what the authors describe as parametric memory, meaning knowledge encoded in model weights, with non-parametric memory, meaning an external index of documents retrieved at query time. That architecture is now the standard approach for grounding AI systems in private organizational data.
The practical consequence is a pipeline with distinct stages. Documents are chunked and converted into vector embeddings. Those vectors are stored and indexed in a database. At query time, the user’s question is embedded, similar vectors are retrieved through nearest-neighbor search, and the retrieved text is passed to a language model that generates the final answer.
Only that last stage is GPU-hungry. Everything before it, the chunking, the embedding generation, the vector storage, the similarity search, runs comfortably on CPU with sufficient RAM. This is the single most important fact for anyone sizing infrastructure, and it is routinely buried under content that treats self-hosted AI as one monolithic hardware problem. A KVM virtual server with NVMe storage and generous memory allocation is configured almost exactly for the retrieval half of this workload, which is where most of the sensitive data actually lives.
Why the Retrieval Layer Belongs on a Virtual Server
Vector databases are memory-bound systems, and this maps unusually well onto what VPS and VDS products offer.
The arithmetic is straightforward enough to plan around. A common embedding model producing 768-dimensional vectors at float32 precision consumes roughly 3KB per vector. One million document chunks therefore occupy around 3GB of raw vector data, and the HNSW index structures that make similarity search fast typically add fifty to one hundred percent overhead on top. Call it 5GB for a million-chunk corpus, which is a genuinely large internal knowledge base for most organizations.
That fits comfortably on a 16GB virtual server with room for the operating system, the application layer, and caching. Scale to 64GB and you are handling corpora in the range of ten million chunks, which covers the document estate of most mid-sized companies entirely. Quantizing vectors to int8 cuts memory requirements by roughly four times, extending that ceiling substantially further at a small accuracy cost.
Disk speed matters here more than raw CPU count. Vector databases like Qdrant, Weaviate, Milvus, and the pgvector extension for PostgreSQL all benefit substantially from NVMe storage during index building, during persistence operations, and when portions of the index spill from memory. A server with fast NVMe and generous RAM will outperform one with more cores and slower disk for this specific workload almost every time.
Embedding generation is the other retrieval-side task, and it is far lighter than people expect. Compact models such as all-MiniLM or BGE-base run acceptably on CPU, processing documents in the range of tens per second on a multi-core virtual server. For a corpus that gets indexed once and updated incrementally, that throughput is entirely sufficient. There is no reason to rent GPU capacity for a job that runs overnight and then idles.
Where the Honest Limits Are
Being direct about this matters, because the alternative is teams discovering it after they have committed budget.
A CPU-only virtual server cannot run a large language model at interactive speed. A 70 billion parameter model at four-bit quantization needs roughly 35GB of memory and, critically, the memory bandwidth of a GPU to move weights fast enough for real-time generation. On CPU, that same model produces output slowly enough to be unusable for conversational interfaces, often in the range of one to three tokens per second.
Smaller models are a genuinely different story. A 7 billion or 9 billion parameter model at four-bit quantization occupies around 4 to 5GB and runs on CPU at perhaps five to fifteen tokens per second depending on the processor. That is too slow for a chat interface where someone is watching the cursor, but perfectly workable for asynchronous tasks: classifying support tickets overnight, summarizing documents in batch, extracting structured data from forms, or generating draft responses that a human reviews later.
The distinction that actually matters is not model size in isolation but whether a human is waiting. Interactive use demands GPU. Batch and asynchronous use does not, and a substantial share of real business AI workloads are batch workloads that got designed as interactive ones out of habit.
Matching Workload to Infrastructure
The table below maps the components of a private AI stack against what they actually require.
| Component | Primary Constraint | VPS/VDS Suitable | Typical Requirement |
|---|---|---|---|
| Document chunking and preprocessing | CPU, brief | Yes | 2 to 4 vCPU |
| Embedding generation (batch) | CPU, sustained | Yes | 4 to 8 vCPU, runs offline |
| Vector database (1M chunks) | RAM | Yes | 8 to 16GB RAM, NVMe |
| Vector database (10M chunks) | RAM | Yes | 48 to 64GB RAM, NVMe |
| Similarity search at query time | RAM, low latency | Yes | Same server as vector DB |
| Small LLM, batch or async | RAM, CPU | Yes | 8 to 16GB RAM, 7B at 4-bit |
| Small LLM, interactive chat | GPU memory bandwidth | Marginal | GPU strongly preferred |
| Large LLM (70B), any use | GPU VRAM | No | 35GB+ VRAM required |
| Orchestration and API layer | Low, mostly I/O | Yes | 2 vCPU, 4GB RAM |
Reading down that table, the pattern is clear. Nearly every component of a private RAG system runs well on a virtual server. The single exception is interactive generation with a large model, and that exception has a well-established workaround.
The Hybrid Architecture Most Teams Actually Build
Here is the arrangement that solves the practical problem, and it is worth understanding because it delivers most of the privacy benefit without GPU costs.
Your documents, your embeddings, your vector index, and your retrieval logic all live on your own virtual server. That is where the sensitive material sits, and it never leaves your infrastructure. When a user asks a question, the query is embedded locally, similarity search runs locally, and the relevant chunks are retrieved locally. Only at the final generation step does anything external happen, and what gets sent is the user’s question plus the specific retrieved passages, not your document corpus.
This is a meaningfully different privacy posture than sending your entire knowledge base to a third party for indexing. Your proprietary data stays under your control, your access logs are yours, and switching generation providers becomes trivial because the expensive and sensitive part of the system is infrastructure you own.
For organizations where even sending retrieved passages externally is unacceptable, whether for regulatory reasons or contractual ones, the small-model path on the same server keeps everything internal at the cost of generation quality and speed. Many teams run both, routing sensitive queries to the local model and general queries to an external API, which is a routing decision your orchestration layer can make automatically based on document classification.
Building this kind of multi-stage pipeline has become considerably more approachable as the tooling has matured. Frameworks and libraries covered across the modern AI programming stack handle most of the connective work that previously required custom engineering, and the orchestration patterns behind how AI agents coordinate multi-step workflows apply directly to routing decisions inside a RAG pipeline.
The Governance Argument for Owning Your Retrieval Layer
Privacy is the usual reason cited for self-hosting, but the governance case is broader and often more compelling to the people who approve budgets.
When your vector store runs on infrastructure you control, you determine retention policy rather than accepting someone else’s. You can audit exactly which documents were retrieved for which query, which matters enormously when an AI system produces a wrong answer and someone needs to establish why. You control geographic placement, which is not a preference but a legal requirement under data residency rules in many jurisdictions. European hosting locations exist precisely because organizations subject to GDPR need their processing to occur within specific boundaries, and a provider offering data centers across the Netherlands, Germany, Finland, and other European locations solves a compliance problem rather than merely a latency one.
Structuring this responsibly benefits from an established framework rather than improvisation. The NIST AI Risk Management Framework organizes AI governance around four functions covering governance, risk mapping, measurement, and management across the full lifecycle, and it is increasingly referenced by sector regulators as a practical companion for compliance work. Self-hosting your retrieval layer makes several of its requirements substantially easier to satisfy, because you can actually inspect and log what the system did.
It is worth being honest that self-hosting shifts security responsibility rather than eliminating it. An exposed vector database endpoint without authentication is a genuine liability, and a poorly secured server holding your entire document corpus is a worse outcome than a well-secured third-party service. The infrastructure choice creates the opportunity for better governance. It does not deliver it automatically.
Sizing Your Server Realistically
Working through these questions in order prevents the most common overspending and undersizing mistakes.
- Count your document chunks before choosing a plan. Multiply chunks by embedding dimensions by four bytes, then add fifty to one hundred percent for index overhead, and size RAM from that figure rather than guessing.
- Prioritize RAM over vCPU count for the vector database. Similarity search is memory-bound, and a memory-optimized plan tier will outperform a balanced one with the same price at this specific job.
- Insist on NVMe rather than SATA SSD. Index building, persistence, and any memory spill all hit disk hard, and the difference is measurable in real workloads.
- Choose your server location for compliance first and latency second. Data residency obligations are not negotiable in the way that a hundred milliseconds of latency usually is.
- Decide early whether generation happens locally or via API. This single decision determines whether you need GPU capacity at all, and it is far cheaper to answer before provisioning than after.
- Plan for incremental reindexing, not one-time indexing. Document corpora grow, embedding models improve, and a server sized exactly for today’s corpus will be uncomfortable within a year.
VPS and VDS for Private AI: Common Questions
Small quantized models can run on CPU-only virtual servers at usable speeds for batch and asynchronous work. A 7 billion parameter model at four-bit quantization occupies roughly 4 to 5GB of RAM and generates perhaps five to fifteen tokens per second on a multi-core server, which is workable for overnight document processing, classification, or summarization. It is too slow for interactive chat where a person is waiting. Large models in the 70 billion parameter range require GPU memory bandwidth and will not run acceptably on CPU regardless of how much RAM the server has.
For the retrieval half of a RAG pipeline, a well-specified VPS or VDS is genuinely well-matched rather than a compromise. Vector databases are memory-bound, embedding generation runs acceptably on CPU, and similarity search benefits more from RAM and fast NVMe storage than from GPU compute. A 16GB server handles roughly a million document chunks comfortably, and 64GB extends that to around ten million. The generation step is the only component that may require different infrastructure.
Estimate by multiplying your document chunk count by your embedding dimensions by four bytes for float32 precision, then adding fifty to one hundred percent for index overhead. A million chunks at 768 dimensions works out to roughly 3GB of raw vectors and around 5GB with index structures included. Quantizing vectors to int8 reduces memory requirements by approximately four times with a modest accuracy tradeoff, which meaningfully extends what a given server can hold.
For most organizations, a hybrid arrangement delivers the best balance. Your documents, embeddings, vector index, and retrieval logic run on your own virtual server, keeping sensitive material entirely under your control. Only the final generation step calls an external API, and what gets transmitted is the user’s question plus specific retrieved passages rather than your document corpus. This preserves the substantial majority of the privacy benefit while avoiding GPU infrastructure costs, and it makes switching generation providers straightforward since the valuable part of the system is infrastructure you own.
It meaningfully changes your exposure by keeping your document corpus, embeddings, and query logs on infrastructure you control rather than a third party’s. That matters for regulated data, contractual confidentiality obligations, and jurisdictions with data residency requirements. However, self-hosting transfers security responsibility to your organization rather than automatically improving your posture. An unauthenticated vector database endpoint or an unpatched server holding your entire knowledge base is genuinely worse than a well-secured managed service, so the infrastructure choice creates the opportunity for stronger governance rather than guaranteeing it.
RAM first, since vector search is memory-bound and insufficient memory forces index spilling that degrades query latency severely. NVMe storage second, because index building, persistence, and any memory overflow all depend heavily on disk throughput. Server location third, particularly where data residency rules apply, since compliance requirements are not negotiable. vCPU count matters least for retrieval workloads specifically, though it becomes relevant if you plan to run a small language model on the same machine for generation.
