Zilliz rolled out Milvus v3.0, its open-source vector database, adding lake-native data access and an upgraded retrieval engine for production AI developers.
Lake-native architecture keeps raw data in open formats on cloud object storage instead of duplicating it into standalone databases. Zilliz unveiled its Vector Lakebase concept back in June’s public preview; CEO Charles Xie described it as a unified data foundation enabling real-time queries, data exploration and multi-petabyte AI training pipelines without data copies, migration or parallel storage stacks. The unified object storage layer supports both low-latency online vector search and large-scale analytics on identical source files.
Zilliz Co-founder & CTO James Luan commented: “Milvus 3.0 runs production retrieval directly atop existing data stores and delivers a more powerful engine for modern AI workloads. This same core powers Zilliz Vector Lakebase, enhanced with elastic scaling, simplified operations and enterprise-grade tooling for large-scale AI data management.”
Vector databases store numerical vector embeddings for text, images and other content to power similarity search. Zilliz states Milvus is the world’s most widely used open-source vector database, adopted by over 10,000 firms for RAG, recommendation engines and AI agents, with more than 100 million Docker pulls recorded.
Traditional AI pipelines maintain separate data copies for real-time retrieval and offline analytics, bringing extra storage costs, lengthy data exports, synchronization pipelines and heavy operational overhead.
Core Milvus 3.0 Capabilities
1.External Collections: Create Milvus collections over lake files in Lance, Iceberg, Parquet or Vortex. The system builds vector, full-text, JSON and scalar indexes on source data without duplication, exposing them via standard native APIs. Incremental sync automatically updates indexes as lake data changes.
2.Loon Storage Engine: A new manifest-based engine cutting read amplification for low-latency object storage lookups, using open Arrow-compatible Vortex as default columnar format.
3.Point-in-Time Snapshots: Lightweight read-only collection snapshots enable offline deduplication, evaluation and validation jobs to run on stable datasets without disrupting production writes.
4.Spark Connector & Batch Operators: A Spark DataSource V2 connector integrates Milvus with Spark, Databricks and EMR batch workflows, alongside native vector batch functions for deduplication and clustering.
Standard Top-K vector search only fetches the most similar vectors by distance calculation, suffering poor scaling with high vector counts and dimensions. Milvus 3.0 expands retrieval far beyond basic nearest-neighbor search:
1.Server-side sorting, aggregation and faceted search combine semantic similarity with metadata filters (timestamp, price, category, tenant etc.), eliminating client-side post-processing.
2.StructList multi-vector retrieval supports entities with multiple embeddings (document chunks, image patches, multimodal product data) and works with late-interaction models like ColBERT and ColPali.
3.Optimized sparse index cuts storage footprint by ~70% while matching original recall performance, paired with SINDI learned sparse vectors, built-in MinHash, nullable vector fields, custom full-text dictionaries and broader Faiss index compatibility.
Luan added that production AI workloads require ranking, grouping and multi-vector lookup natively within the database, and v3.0 reworked storage to deliver fast object storage access alongside richer in-engine processing.
This unified core powers both self-hosted open-source Milvus and fully managed Zilliz Cloud. Zilliz Cloud extends the base engine into a complete Vector Lakebase with elastic compute, discovery tools, batch analytics, unified indexing, enterprise governance and automation on a single source of AI data.
Industry Competitive Context
The update raises pressure on rivals Pinecone, Qdrant and Weaviate, all racing to cut search latency: Qdrant launched faster indexing, high-availability clusters and compliance audit logs in April; Pinecone rolled out precomputed vector sets and composable retrievers in May to accelerate agent workflows.
Pure vector database vendors compete against multi-model platforms including Regatta, SingleStore and SurrealDB. While they integrate with lakehouses such as Databricks and Snowflake, deeper cross-database partnerships face architectural and storage engine barriers. The market may see vector players add native structured, graph, document or time-series support over time.
Availability
Milvus 3.0 remains an LF AI & Data graduated project under Apache 2.0 licensing, deployable via Docker or Kubernetes including air-gapped environments, with compatibility for S3, GCS and Azure Blob object storage. Initial SDK coverage includes Python, Go and Node.js, with Java support upcoming. Source code, release notes and quickstart guides are hosted on GitHub, alongside community office hours for developer support.
Managed Zilliz Cloud embeds Milvus 3.0 as its core indexing layer, adding enterprise elasticity, security and unified operations across a single shared data copy.
Footnote
External Collections implement zero-copy lake indexing without data migration; Loon resolves the read amplification bottleneck that previously limited object storage for low-latency vector queries.
Beijing Qianxing Jietong Technology Co., Ltd.
Sandy Yang/Global Strategy Director
WhatsApp / WeChat: +86 13426366826
Email: yangyd@qianxingdata.com
Website: www.qianxingdata.com/www.storagesserver.com
Business Focus:
ICT Product Distribution/System Integration & Services/Infrastructure Solutions
With 20+ years of IT distribution experience, we partner with leading global brands to deliver reliable products and professional services.
“Using Technology to Build an Intelligent World”Your Trusted ICT Product Service Provider!
Twoja wiadomość musi mieć od 20 do 3000 znaków!
Polish
