Benchmarking results for vector databases - Redis

🌈 Abstract

The article presents a comprehensive benchmark comparison of Redis 7.4 with various vector database providers, general-purpose databases with vector capabilities, and Redis imitators on cloud service providers. It highlights the performance advantages of Redis in terms of querying throughput, latency, and indexing times across different datasets and use cases.

🙋 Q&A

[01] Let's start with what matters most, how fast Redis is.

1. What are the key metrics used to benchmark the performance of the vector databases?

• The two key metrics used are:
  • Throughput: Indicates a system's capability to process numerous queries or large datasets in a short amount of time
  • Latency: Measures how fast individual similarity searches return results

2. How did Redis perform compared to other vector database providers in terms of querying throughput and latency?

• Redis achieved up to 3.4 times higher queries per second (QPS) than Qdrant, 3.3 times higher QPS than Milvus, and 1.7 times higher QPS than Weaviate for the same recall levels.
• On latency, Redis achieved up to 4 times less latency than Qdrant, 4.67 times less latency than Milvus, and 1.71 times faster than Weaviate for the same recall levels.

3. How did Redis perform compared to other vector database providers in terms of ingestion and indexing?

• Qdrant is the fastest in ingestion and indexing due to its multiple segments index design, but Redis excels in fast querying.
• Redis showed up to 2.8 times lower indexing time than Milvus and up to 3.2 times lower indexing time than Weaviate.

[02] Comparison with general-purpose databases with vector capabilities

1. How did Redis perform compared to general-purpose databases with vector capabilities?

• Redis significantly outperformed competitors like Amazon Aurora PostgreSQL, MongoDB Atlas, and Amazon OpenSearch in terms of querying throughput and latency.
• Redis achieved up to 9.5 times higher queries per second (QPS) and up to 9.7 times lower latencies than Amazon Aurora PostgreSQL for the same recall.
• Against MongoDB Atlas, Redis demonstrated up to 11 times higher QPS and up to 14.2 times lower latencies.
• Against Amazon OpenSearch, Redis demonstrated up to 53 times higher QPS and up to 53 times lower latencies.

2. How did Redis perform compared to general-purpose databases in terms of ingestion and indexing?

• Redis showed a substantial advantage over Amazon Aurora PostgreSQL, with indexing times ranging from 5.5 to 19 times lower.

[03] Comparison with Redis imitators on cloud service providers

1. How did Redis perform compared to Redis imitators like Amazon MemoryDB and Google Cloud MemoryStore for Redis?

• Against Amazon MemoryDB, Redis achieved up to 3.9 times higher queries per second (QPS) and up to 4.1 times lower latencies for the same recall.
• Compared to GCP MemoryStore for Redis v7.2, Redis demonstrated up to 2.5 times higher QPS and up to 4.8 times lower latencies.

2. How did Redis perform compared to Redis imitators in terms of ingestion and indexing?

• Redis had an advantage over Amazon MemoryDB with indexing times ranging from 1.39 to 3.89 times lower.
• Against GCP MemoryStore for Redis v7.2, Redis showed an even greater indexing advantage, with times ranging from 4.9 to 10.34 times lower.

[04] Enhancements to the Redis Query Engine

1. What were the key challenges that the Redis engineering team had to overcome to achieve the showcased performance results?

• Redis' single-thread architecture and the assumption that most commands are short and independent needed to be addressed for long-running queries like vector searches.
• Scaling out with sharding doesn't significantly decrease the total query time, as the compute-heavy nature of vector comparisons holds the main thread longer than regular Redis workloads.

2. How did the Redis engineering team address these challenges?

• They introduced a new enhancement to the Redis Query Engine, enabling concurrent access to the index through a producer-consumer pattern.
• Multiple queries are executed concurrently on separate threads, while the main thread handles incoming requests and prepares new queries.
• This approach allows for efficient resource utilization and significant performance improvements.

3. What is the "scaling factor" concept introduced in the article?

• The scaling factor is a ratio that demonstrates how effectively the added resources (threads and vCPUs) can increase the achievable throughput for vector workloads.
• The article confirms that the theoretical scaling factor aligns closely with the empirical results, both for vector and full-text search use cases.