[Blogpost] Configurable Automation for OpenSearch ML Use Cases #2698

owaiskazi19 · 2024-04-02T18:32:54Z

Description

Configurable Automation for OpenSearch ML Use Cases for 2.13

Issues Resolved

#2697

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

minalsha

suggestion to lets add all the team members to the blogpost.

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

Signed-off-by: owaiskazi19 <[email protected]>

Signed-off-by: Fanit Kolchina <[email protected]>

Signed-off-by: owaiskazi19 <[email protected]>

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

Signed-off-by: owaiskazi19 <[email protected]>

natebower

@owaiskazi19 @kolchfa-aws Editorial review complete. Please see my comments and changes and let me know if you have any questions. Thanks!

_community_members/amitgalitz.md

_community_members/hnyng.md

_community_members/jpalis.md

_community_members/ohltyler.md

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

natebower · 2024-04-04T13:51:24Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

+}
+```
+
+With Flow Framework, we've simplified this complex setup process, enabling you to focus on your tasks without the burden of navigating complex APIs. Our goal is for you to use OpenSearch seamlessly, unlocking new possibilities in your projects.


"realizing" or "uncovering" instead of "unlocking"?

natebower · 2024-04-04T13:52:30Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

+
+## Additional default use cases
+
+You can explore more default use cases by viewing [substitution templates](https://github.com/opensearch-project/flow-framework/tree/2.13/src/main/resources/substitutionTemplates) with their corresponding [defaults](https://github.com/opensearch-project/flow-framework/tree/2.13/src/main/resources/defaults).


"and" their corresponding defaults?

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>

kolchfa-aws · 2024-04-04T14:34:23Z

@natebower Thank you for the review. I addressed your comments and accepted your suggestions.

dbwiddis

LGTM, a few suggestions.

dbwiddis · 2024-04-04T15:09:11Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

+1. Create a connector for a remote model, specifying pre- and post-processing functions.
+1. Register an embedding model using the connector ID obtained in the previous step.
+1. Configure an ingest pipeline to generate vector embeddings using the model ID of the registered model.
+1. Create a k-NN index and add the pipeline created in the previous step.


It's not clear to the reader how complex this is, particularly since the same steps are essentially repeated on lines 38-40. We need to highlight in the sentence above (line 25) that these require 4 separate API calls, perhaps adding the words "copy and paste" when refer to "using the X ID"

dbwiddis · 2024-04-04T15:09:27Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

+1. Configure an ingest pipeline to generate vector embeddings using the model ID of the registered model.
+1. Create a k-NN index and add the pipeline created in the previous step.
+
+This complex setup typically required you to be familiar with the OpenSearch ML Commons APIs. However, we are simplifying this experience through the Flow Framework plugin. Let's demonstrate how the plugin simplifies this process using the preceding semantic search example.


Don't think we need "typically"

dbwiddis · 2024-04-04T15:11:33Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

+
+```json
+{
+  "create_index.name": "my-nlp-index"


This is confusing having this separated from the line 47 API, particularly with the response JSON in between. I think this goes in the block under line 49, but even I'm not sure. Make it clear, perhaps including it as a line 50 after describing that it's optional, or perhaps repeating the whole REST call with both lines.

I will remove the json completely

dbwiddis · 2024-04-04T15:14:40Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

+Once the workflow is provisioned, you can ingest documents into the index created by the workflow:
+
+```json
+PUT /my-nlp-index/_doc/1


We should probably use the path without the document ID included. Having the 1 here and the s1 in the "id" field is confusing. (We can leave out the "id" field as well since it's not the same as the "_id".)

dbwiddis · 2024-04-04T15:15:00Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

+    "neural": {
+      "passage_embedding": {
+        "query_text": "Hi world",
+        "k": 100


Maybe a smaller k?

It's a lowercase k only

I meant a number less than 100, heh. Like "k": 10

Signed-off-by: owaiskazi19 <[email protected]>

dbwiddis

Looks really good. Some minor suggestions.

dbwiddis · 2024-04-05T17:57:15Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

+meta_description: Explore the simplicity of integrating Machine Learning capabilities within OpenSearch through an innovative and groundbreaking framework designed to simplify complex setup tasks.
+---
+
+In OpenSearch, to use machine learning (ML) offerings, such as semantic, hybrid, and multimodal search, you often have to grapple with complex setup and preprocessing tasks. Additionally, you must write verbose queries, which can be a time-consuming and error-prone process.


This sentence has a lot of commas. I tried to rewrite it to make it better but couldn't really do much better. So I guess it's fine as is! :|

@kolchfa-aws any inputs here?

dbwiddis · 2024-04-05T17:58:17Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

+
+In OpenSearch, to use machine learning (ML) offerings, such as semantic, hybrid, and multimodal search, you often have to grapple with complex setup and preprocessing tasks. Additionally, you must write verbose queries, which can be a time-consuming and error-prone process.
+
+In this blog post, we introduce the OpenSearch Flow Framework plugin, [released in version 2.13](https://opensearch.org/blog/2.13-is-ready-for-download/) and designed to streamline this cumbersome process. By using this plugin, you can simplify complex setups with just one click. We've provided automated templates, enabling you to create connectors, register models, deploy them, and register agents and tools through a single API call. This eliminates the complexity of calling multiple APIs and orchestrating setups based on the responses.


Nit: "click" is kind of GUI centric and we're still API. Can we maybe say "one simple API call?"

dbwiddis · 2024-04-05T17:58:48Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

+
+## Before the Flow Framework plugin
+
+Previously, setting up semantic search involves *4 separate API* calls outlined in the [semantic search documentation](https://opensearch.org/docs/latest/search-plugins/semantic-search/):


Put the closing * after "calls"

dbwiddis · 2024-04-05T17:59:31Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

+    "neural": {
+      "passage_embedding": {
+        "query_text": "Hi world",
+        "k": 100


I meant a number less than 100, heh. Like "k": 10

Signed-off-by: owaiskazi19 <[email protected]>

natebower

@kolchfa-aws Final edits

natebower · 2024-04-08T10:50:19Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

@@ -22,14 +22,14 @@ In this blog post, we introduce the OpenSearch Flow Framework plugin, [released

 ## Before the Flow Framework plugin

-Previously, setting up semantic search involved the steps outlined in the [semantic search documentation](https://opensearch.org/docs/latest/search-plugins/semantic-search/):
+Previously, setting up semantic search involves *4 separate API* calls outlined in the [semantic search documentation](https://opensearch.org/docs/latest/search-plugins/semantic-search/):


"involved four". Comma after "calls".

natebower · 2024-04-08T10:51:18Z

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md

-  "create_index.name": "my-nlp-index"
-}
-```
+Note: The workflow in the previous step creates a default k-NN index. The default index name is `my-nlp-index`


Add terminating period.

Signed-off-by: Fanit Kolchina <[email protected]>

kolchfa-aws · 2024-04-08T13:13:15Z

Thank you for reviewing the edits, @natebower! I've addressed your final comments.

Signed-off-by: Fanit Kolchina <[email protected]>

…ct-website into flow-framework

pajuric · 2024-04-08T19:00:58Z

@nateynateynate @krisfreedain - Blog is ready to publish today.

nateynateynate

Let's push this live.

owaiskazi19 · 2024-04-08T22:14:28Z

@nateynateynate thanks for pushing it to live. I don't see @dbwiddis and @jackiehanyang as the author though I have added them in the PR. Any idea?

owaiskazi19 requested review from elfisher, AMoo-Miki, nknize, krisfreedain, peterzhuamazon, CEHENKLE, kolchfa-aws and nateynateynate as code owners April 2, 2024 18:32

owaiskazi19 force-pushed the flow-framework branch 3 times, most recently from 5853a50 to e270e65 Compare April 2, 2024 18:41

minalsha reviewed Apr 2, 2024

View reviewed changes

amitgalitz reviewed Apr 2, 2024

View reviewed changes

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md Outdated Show resolved Hide resolved

Added blogpost for Flow Framework

43ddfae

Signed-off-by: owaiskazi19 <[email protected]>

owaiskazi19 force-pushed the flow-framework branch from e270e65 to 43ddfae Compare April 2, 2024 22:51

kolchfa-aws and others added 2 commits April 3, 2024 11:54

Blog rewrites

830a1a1

Signed-off-by: Fanit Kolchina <[email protected]>

Added additional team members

8c9810a

Signed-off-by: owaiskazi19 <[email protected]>

owaiskazi19 force-pushed the flow-framework branch from 44d2993 to 8c9810a Compare April 3, 2024 16:13

pajuric reviewed Apr 3, 2024

View reviewed changes

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md Outdated Show resolved Hide resolved

Added release link

ed4c93d

Signed-off-by: owaiskazi19 <[email protected]>

nateynateynate added blog-under-review new blog New blog post labels Apr 3, 2024

natebower reviewed Apr 4, 2024

View reviewed changes

kolchfa-aws reviewed Apr 4, 2024

View reviewed changes

_posts/2024-04-05-Configurable-Automation-for-OpenSearch-ML-Use-Cases.md Outdated Show resolved Hide resolved

Apply suggestions from code review

f935dd9

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>

dbwiddis reviewed Apr 4, 2024

View reviewed changes

Addressed PR comments

cae7323

Signed-off-by: owaiskazi19 <[email protected]>

dbwiddis approved these changes Apr 5, 2024

View reviewed changes

Minor comments

296bc92

Signed-off-by: owaiskazi19 <[email protected]>

natebower reviewed Apr 8, 2024

View reviewed changes

Final editorial comments

5dad660

Signed-off-by: Fanit Kolchina <[email protected]>

owaiskazi19 and others added 3 commits April 8, 2024 10:05

Merge branch 'opensearch-project:main' into flow-framework

939f428

Change date

bcd469b

Signed-off-by: Fanit Kolchina <[email protected]>

Merge branch 'flow-framework' of https://github.com/owaiskazi19/proje…

0de3cff

…ct-website into flow-framework

nateynateynate approved these changes Apr 8, 2024

View reviewed changes

nateynateynate merged commit c9fd1a2 into opensearch-project:main Apr 8, 2024
5 checks passed

owaiskazi19 mentioned this pull request Apr 8, 2024

Added missing authors for Flow Framework's Blogpost #2728

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Blogpost] Configurable Automation for OpenSearch ML Use Cases #2698

[Blogpost] Configurable Automation for OpenSearch ML Use Cases #2698

owaiskazi19 commented Apr 2, 2024

minalsha left a comment

natebower left a comment

natebower Apr 4, 2024

natebower Apr 4, 2024

kolchfa-aws commented Apr 4, 2024

dbwiddis left a comment

dbwiddis Apr 4, 2024 •

edited

Loading

dbwiddis Apr 4, 2024

dbwiddis Apr 4, 2024 •

edited

Loading

owaiskazi19 Apr 5, 2024

dbwiddis Apr 4, 2024

dbwiddis Apr 4, 2024

owaiskazi19 Apr 5, 2024

dbwiddis Apr 5, 2024

owaiskazi19 Apr 5, 2024

dbwiddis left a comment

dbwiddis Apr 5, 2024

owaiskazi19 Apr 5, 2024

dbwiddis Apr 5, 2024

dbwiddis Apr 5, 2024

dbwiddis Apr 5, 2024

natebower left a comment

natebower Apr 8, 2024

natebower Apr 8, 2024

kolchfa-aws commented Apr 8, 2024

pajuric commented Apr 8, 2024

nateynateynate left a comment

owaiskazi19 commented Apr 8, 2024


		## Additional default use cases

		You can explore more default use cases by viewing [substitution templates](https://github.com/opensearch-project/flow-framework/tree/2.13/src/main/resources/substitutionTemplates) with their corresponding [defaults](https://github.com/opensearch-project/flow-framework/tree/2.13/src/main/resources/defaults).


		In OpenSearch, to use machine learning (ML) offerings, such as semantic, hybrid, and multimodal search, you often have to grapple with complex setup and preprocessing tasks. Additionally, you must write verbose queries, which can be a time-consuming and error-prone process.

		In this blog post, we introduce the OpenSearch Flow Framework plugin, [released in version 2.13](https://opensearch.org/blog/2.13-is-ready-for-download/) and designed to streamline this cumbersome process. By using this plugin, you can simplify complex setups with just one click. We've provided automated templates, enabling you to create connectors, register models, deploy them, and register agents and tools through a single API call. This eliminates the complexity of calling multiple APIs and orchestrating setups based on the responses.


		## Before the Flow Framework plugin

		Previously, setting up semantic search involves 4 separate API calls outlined in the [semantic search documentation](https://opensearch.org/docs/latest/search-plugins/semantic-search/):

[Blogpost] Configurable Automation for OpenSearch ML Use Cases #2698

[Blogpost] Configurable Automation for OpenSearch ML Use Cases #2698

Conversation

owaiskazi19 commented Apr 2, 2024

Description

Issues Resolved

Check List

minalsha left a comment

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kolchfa-aws commented Apr 4, 2024

dbwiddis left a comment

Choose a reason for hiding this comment

dbwiddis Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbwiddis Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbwiddis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kolchfa-aws commented Apr 8, 2024

pajuric commented Apr 8, 2024

nateynateynate left a comment

Choose a reason for hiding this comment

owaiskazi19 commented Apr 8, 2024

dbwiddis Apr 4, 2024 •

edited

Loading

dbwiddis Apr 4, 2024 •

edited

Loading