Mark nmslib references for vector search as deprecated #9107

kotwanikunal · 2025-01-23T18:31:34Z

Description

Marks nmslib engine references for vector search as deprecated
Codebase PR: Deprecate nmslib engine k-NN#2427

Issues Resolved

N/A

Version

List the OpenSearch version to which this PR applies, e.g. 2.14, 2.12--2.14, or all.

= 2.19.0

Frontend features

N/A

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Kunal Kotwani <[email protected]>

github-actions · 2025-01-23T18:31:53Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

kolchfa-aws

Thank you, @kotwanikunal! A couple of styling suggestions. Also, should we remove some of the mentions of NMSLIB in the text and just keep it in the tables?

kolchfa-aws · 2025-01-24T15:07:34Z

_field-types/supported-field-types/knn-vector.md

-|:---|:---|:---|
-| `in_memory` (Default) | `nmslib`       | Prioritizes low-latency search. This mode uses the `nmslib` engine without any quantization applied. It is configured with the default parameter values for vector search in OpenSearch.                                                            |
+| Mode    | Default engine | Description                                                                                                                                                                                                                                             |
+|:---|:---------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|


Suggested change

|:---|:---------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

|:---|:---|:---|

kolchfa-aws · 2025-01-24T15:08:05Z

_field-types/supported-field-types/knn-vector.md

-| `32x`             | `faiss`                        |
+| Compression level | Supported engines                            |
+|:------------------|:---------------------------------------------|
+| `1x`              | `faiss`, `lucene`, and `nmslib (Deprecated)` |


kolchfa-aws · 2025-01-24T15:09:07Z

_search-plugins/knn/api.md

@@ -173,11 +173,11 @@ For the warmup operation to function properly, follow these best practices:
 Introduced 2.14
 {: .label .label-purple }

-During approximate k-NN search or warmup operations, the native library indexes (`nmslib` and `faiss` engines) are loaded into native memory. Currently, you can evict an index from cache or native memory by either deleting the index or setting the k-NN cluster settings `knn.cache.item.expiry.enabled` and `knn.cache.item.expiry.minutes`, which removes the index from the cache if it is idle for a given period of time. However, you cannot evict an index from the cache without deleting the index. To solve this problem, you can use the k-NN clear cache API operation, which clears a given set of indexes from the cache.
+During approximate k-NN search or warmup operations, the native library indexes (`nmslib` (Deprecated) and `faiss` engines) are loaded into native memory. Currently, you can evict an index from cache or native memory by either deleting the index or setting the k-NN cluster settings `knn.cache.item.expiry.enabled` and `knn.cache.item.expiry.minutes`, which removes the index from the cache if it is idle for a given period of time. However, you cannot evict an index from the cache without deleting the index. To solve this problem, you can use the k-NN clear cache API operation, which clears a given set of indexes from the cache.


Suggested change

During approximate k-NN search or warmup operations, the native library indexes (`nmslib` (Deprecated) and `faiss` engines) are loaded into native memory. Currently, you can evict an index from cache or native memory by either deleting the index or setting the k-NN cluster settings `knn.cache.item.expiry.enabled` and `knn.cache.item.expiry.minutes`, which removes the index from the cache if it is idle for a given period of time. However, you cannot evict an index from the cache without deleting the index. To solve this problem, you can use the k-NN clear cache API operation, which clears a given set of indexes from the cache.

During approximate k-NN search or warmup operations, the native library indexes (for the `faiss` and `nmslib` [deprecated] engines) are loaded into native memory. Currently, you can evict an index from cache or native memory by either deleting the index or setting the k-NN cluster settings `knn.cache.item.expiry.enabled` and `knn.cache.item.expiry.minutes`, which removes the index from the cache if it is idle for a given period of time. However, you cannot evict an index from the cache without deleting the index. To solve this problem, you can use the k-NN clear cache API operation, which clears a given set of indexes from the cache.

kolchfa-aws · 2025-01-24T15:10:03Z

_search-plugins/knn/api.md


 The k-NN clear cache API evicts all native library files for all shards (primaries and replicas) of all indexes specified in the request. Similarly to how the [warmup operation](#warmup-operation) behaves, the k-NN clear cache API is idempotent, meaning that if you try to clear the cache for an index that has already been evicted from the cache, it does not have any additional effect.

-This API operation only works with indexes created using the `nmslib` and `faiss` engines. It has no effect on indexes created using the `lucene` engine.
+This API operation only works with indexes created using the `nmslib` (Deprecated) and `faiss` engines. It has no effect on indexes created using the `lucene` engine.


Suggested change

This API operation only works with indexes created using the `nmslib` (Deprecated) and `faiss` engines. It has no effect on indexes created using the `lucene` engine.

This API operation only works with indexes created using the `faiss` and `nmslib` (deprecated) engines. It has no effect on indexes created using the `lucene` engine.

kolchfa-aws · 2025-01-24T15:10:53Z

_search-plugins/knn/api.md

@@ -236,7 +236,7 @@ Response field |  Description
 `error` | An error message explaining why the model is in a failed state.
 `space_type` | The space type for which this model is trained, for example, Euclidean or cosine. Note - this value can be set in the top-level of the request as well
 `dimension` | The dimensionality of the vector space for which this model is designed.
-`engine` | The native library used to create the model, either `faiss` or `nmslib`. 
+`engine` | The native library used to create the model, either `faiss` or `nmslib` (Deprecated). 


Suggested change

`engine` | The native library used to create the model, either `faiss` or `nmslib` (Deprecated).

`engine` | The native library used to create the model, either `faiss` or `nmslib` (deprecated).

kolchfa-aws · 2025-01-24T15:12:27Z

_search-plugins/knn/knn-index.md

@@ -92,7 +92,7 @@ Mapping parameter | Required | Default | Updatable | Description
 :--- | :--- | :--- | :--- | :---
 `name` | true | n/a | false | The identifier for the nearest neighbor method.
 `space_type` | false | l2 | false | The vector space used to calculate the distance between vectors. Note: This value can also be specified at the top level of the mapping.
-`engine` | false | faiss  | false | The approximate k-NN library to use for indexing and search. The available libraries are `faiss`, `nmslib`, and `lucene`.
+`engine` | false | faiss  | false | The approximate k-NN library to use for indexing and search. The available libraries are `faiss`, `nmslib` (Deprecated), and `lucene`.


kolchfa-aws · 2025-01-24T15:12:39Z

_search-plugins/knn/knn-index.md

@@ -108,7 +108,7 @@ Parameter name | Required | Default | Updatable | Description
 `ef_construction` | false | 100 | false | The size of the dynamic list used during k-NN graph creation. Higher values result in a more accurate graph but slower indexing speed.
 `m` | false | 16 | false | The number of bidirectional links that the plugin creates for each new element. Increasing and decreasing this value can have a large impact on memory consumption. Keep this value between 2 and 100.

-For nmslib, *ef_search* is set in the [index settings](#index-settings).
+For nmslib (Deprecated), *ef_search* is set in the [index settings](#index-settings).


Suggested change

For nmslib (Deprecated), *ef_search* is set in the [index settings](#index-settings).

For nmslib (deprecated), *ef_search* is set in the [index settings](#index-settings).

kolchfa-aws · 2025-01-24T15:13:13Z

_search-plugins/knn/knn-index.md

@@ -367,7 +367,7 @@ At the moment, several parameters defined in the settings are in the deprecation
 Setting | Default | Updatable | Description
 :--- | :--- | :--- | :---
 `index.knn` | false | false | Whether the index should build native library indexes for the `knn_vector` fields. If set to false, the `knn_vector` fields will be stored in doc values, but approximate k-NN search functionality will be disabled.
-`index.knn.algo_param.ef_search` | 100 | true | The size of the dynamic list used during k-NN searches. Higher values result in more accurate but slower searches. Only available for NMSLIB.
+`index.knn.algo_param.ef_search` | 100 | true | The size of the dynamic list used during k-NN searches. Higher values result in more accurate but slower searches. Only available for NMSLIB (Deprecated).


Suggested change

`index.knn.algo_param.ef_search` | 100 | true | The size of the dynamic list used during k-NN searches. Higher values result in more accurate but slower searches. Only available for NMSLIB (Deprecated).

`index.knn.algo_param.ef_search` (Deprecated) | 100 | true | The size of the dynamic list used during k-NN searches. Higher values result in more accurate but slower searches. Only available for NMSLIB.

kolchfa-aws · 2025-01-24T15:13:44Z

_search-plugins/knn/knn-vector-quantization.md

@@ -9,7 +9,7 @@ has_math: true

 # k-NN vector quantization

-By default, the k-NN plugin supports the indexing and querying of vectors of type `float`, where each dimension of the vector occupies 4 bytes of memory. For use cases that require ingestion on a large scale, keeping `float` vectors can be expensive because OpenSearch needs to construct, load, save, and search graphs (for native `nmslib` and `faiss` engines). To reduce the memory footprint, you can use vector quantization.
+By default, the k-NN plugin supports the indexing and querying of vectors of type `float`, where each dimension of the vector occupies 4 bytes of memory. For use cases that require ingestion on a large scale, keeping `float` vectors can be expensive because OpenSearch needs to construct, load, save, and search graphs (for native `nmslib` (Deprecated) and `faiss` engines). To reduce the memory footprint, you can use vector quantization.


Suggested change

By default, the k-NN plugin supports the indexing and querying of vectors of type `float`, where each dimension of the vector occupies 4 bytes of memory. For use cases that require ingestion on a large scale, keeping `float` vectors can be expensive because OpenSearch needs to construct, load, save, and search graphs (for native `nmslib` (Deprecated) and `faiss` engines). To reduce the memory footprint, you can use vector quantization.

By default, the k-NN plugin supports the indexing and querying of vectors of type `float`, where each dimension of the vector occupies 4 bytes of memory. For use cases that require ingestion on a large scale, keeping `float` vectors can be expensive because OpenSearch needs to construct, load, save, and search graphs (for native `faiss` and `nmslib` [deprecated] engines). To reduce the memory footprint, you can use vector quantization.

kolchfa-aws · 2025-01-24T15:14:14Z

_search-plugins/vector-search.md

@@ -88,12 +88,12 @@ The following table lists the combinations of search methods and libraries suppo

 Method | Engine
 :--- | :---
-HNSW | NMSLIB, Faiss, Lucene
+HNSW | NMSLIB (Deprecated), Faiss, Lucene


Suggested change

HNSW | NMSLIB (Deprecated), Faiss, Lucene

HNSW | Faiss, Lucene, NMSLIB (deprecated)

Mark nmslib references for vector search as deprecated

09dd2de

Signed-off-by: Kunal Kotwani <[email protected]>

kotwanikunal mentioned this pull request Jan 23, 2025

Deprecate nmslib engine opensearch-project/k-NN#2427

Open

5 tasks

github-actions bot assigned kolchfa-aws Jan 23, 2025

kotwanikunal marked this pull request as ready for review January 23, 2025 18:49

kotwanikunal requested review from kolchfa-aws, Naarcha-AWS, AMoo-Miki, natebower, dlvenable and epugh as code owners January 23, 2025 18:49

kolchfa-aws approved these changes Jan 24, 2025

View reviewed changes

kolchfa-aws added release-notes PR: Include this PR in the automated release notes v2.19.0 labels Jan 24, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mark nmslib references for vector search as deprecated #9107

Mark nmslib references for vector search as deprecated #9107

kotwanikunal commented Jan 23, 2025

github-actions bot commented Jan 23, 2025

kolchfa-aws left a comment

kolchfa-aws Jan 24, 2025

kolchfa-aws Jan 24, 2025

kolchfa-aws Jan 24, 2025

kolchfa-aws Jan 24, 2025

kolchfa-aws Jan 24, 2025

kolchfa-aws Jan 24, 2025

kolchfa-aws Jan 24, 2025

kolchfa-aws Jan 24, 2025

kolchfa-aws Jan 24, 2025

kolchfa-aws Jan 24, 2025

	\|:---\|:---------------\|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\|
	\|:---\|:---\|:---\|

	\| `1x` \| `faiss`, `lucene`, and `nmslib (Deprecated)` \|
	\| `1x` \| `faiss`, `lucene`, and `nmslib` (deprecated) \|

	This API operation only works with indexes created using the `nmslib` (Deprecated) and `faiss` engines. It has no effect on indexes created using the `lucene` engine.
	This API operation only works with indexes created using the `faiss` and `nmslib` (deprecated) engines. It has no effect on indexes created using the `lucene` engine.

	`engine` \| The native library used to create the model, either `faiss` or `nmslib` (Deprecated).
	`engine` \| The native library used to create the model, either `faiss` or `nmslib` (deprecated).

	`engine` \| false \| faiss \| false \| The approximate k-NN library to use for indexing and search. The available libraries are `faiss`, `nmslib` (Deprecated), and `lucene`.
	`engine` \| false \| faiss \| false \| The approximate k-NN library to use for indexing and search. The available libraries are `faiss`, `lucene`, and `nmslib` (deprecated).

	For nmslib (Deprecated), ef_search is set in the [index settings](#index-settings).
	For nmslib (deprecated), ef_search is set in the [index settings](#index-settings).

	`index.knn.algo_param.ef_search` \| 100 \| true \| The size of the dynamic list used during k-NN searches. Higher values result in more accurate but slower searches. Only available for NMSLIB (Deprecated).
	`index.knn.algo_param.ef_search` (Deprecated) \| 100 \| true \| The size of the dynamic list used during k-NN searches. Higher values result in more accurate but slower searches. Only available for NMSLIB.

	HNSW \| NMSLIB (Deprecated), Faiss, Lucene
	HNSW \| Faiss, Lucene, NMSLIB (deprecated)

Mark nmslib references for vector search as deprecated #9107

Are you sure you want to change the base?

Mark nmslib references for vector search as deprecated #9107

Conversation

kotwanikunal commented Jan 23, 2025

Description

Issues Resolved

Version

Frontend features

Checklist

github-actions bot commented Jan 23, 2025

kolchfa-aws left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment