Vector Databases Explained: When You Actually Need One

Vector databases got a wave of hype starting in 2023 when retrieval-augmented generation became the standard pattern for AI applications. Pinecone, Weaviate, Qdrant, Chroma, and a dozen others raced to define the category. Every AI architecture diagram has one in the middle.

Most projects do not need a dedicated one. Here is the actual decision tree.

What a Vector Database Actually Does

When you embed a piece of text using a model like OpenAI's text-embedding-3 or Cohere's embed-english-v3, you get back a list of numbers (a "vector") that represents the semantic meaning of that text. Similar texts produce similar vectors. The math of "similar vectors" is what powers semantic search.

A vector database stores these vectors and lets you query "find the N most similar vectors to this one" efficiently. That is genuinely useful. It is also a problem that traditional databases can solve, with a small extension.

Why pgvector Is the Default Answer

PostgreSQL with the pgvector extension can store embeddings, index them with HNSW or IVFFlat for fast similarity search, and return results in milliseconds for datasets up to several million vectors. It runs on your existing PostgreSQL infrastructure. You can join vector queries with relational queries (filter by user, by date, by category) using normal SQL.

For the vast majority of business RAG applications (a few thousand to a few million chunks of content), this is genuinely all you need. No additional service, no additional vendor, no additional bill. Your existing PostgreSQL just gets a new capability.

When You Outgrow pgvector

You should consider a dedicated vector database when:

• You have tens of millions of vectors and queries are getting slow
• You need sub-50ms query latency at high concurrency
• You want hybrid search that combines vector similarity with keyword matching out of the box
• You need advanced features like multi-vector storage, sparse vectors, or quantization for memory efficiency
• You are running a large multi-tenant system where vector workloads need to be isolated from your transactional database

If none of these apply, pgvector is fine. If several apply, look at Qdrant (great open source option, can self-host or use cloud), Pinecone (managed service with a great developer experience and clear pricing), or Weaviate (powerful but heavier).

The Hidden Cost: Embedding Generation

The vector database is rarely the bottleneck in a RAG system. The cost and latency that surprise teams are usually:

• Embedding generation costs (every chunk you index, every query you embed)
• The LLM call to generate the final answer
• The complexity of keeping the vector store in sync with the source content

Picking the right embedding model matters more than picking the right vector database. OpenAI's embeddings are cheap and good. Cohere's are excellent for many languages. Google's are competitive on price. Open source models like BGE and E5 can be self-hosted if cost is a constraint.

The Real Performance Problems

When a RAG system performs poorly, the vector database is almost never the cause. The actual culprits are usually:

• Bad chunking. Splitting documents at arbitrary character counts loses semantic meaning. Use sentence-aware or section-aware chunking.
• Wrong embedding model. Using a small or outdated model produces noisy results. Use a current, appropriately-sized model.
• No re-ranking. The top 10 vector matches are rarely the most useful 10 results. A re-ranker pass dramatically improves quality.
• Missing metadata filters. Searching across all content when you could filter to a single product, customer, or date range wastes the model's attention.
• Poor prompt engineering. Even with perfect retrieval, a vague prompt produces vague answers.

If your RAG system feels slow or unhelpful, fix these before you swap your vector database.

A Practical Starting Stack

For a typical business RAG project, the stack we reach for first looks like this:

• PostgreSQL with pgvector for storage and search
• OpenAI text-embedding-3 or Cohere embed-english-v3 for embeddings
• Sentence-aware chunking with about 500 tokens per chunk and 50-token overlap
• Cohere Rerank for re-ranking after initial retrieval
• Claude or GPT-4 for the final generation step
• Metadata fields to filter by source, date, and access permissions

This stack will handle the first several million vectors and serve queries in well under a second. Migrate components when you have a specific reason to, not preemptively.

The Bottom Line

Vector databases are a category. They are also marketing. The right vector database for most projects is the one you already have (PostgreSQL) with a free extension (pgvector). The dedicated vector databases are excellent products. They are also excellent at solving problems most projects do not have yet.

At Stunzer Digital, we have built RAG systems on pgvector, Pinecone, Qdrant, and others. The right choice depends on your scale, your existing infrastructure, and your team. If you want a no-vendor-pitch recommendation for your specific situation, that is a 30-minute conversation.