Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BLOG] From Chaos to Clarity: Revolutionizing OpenSearch Clients and Documentation with a Unified API Specification #3570

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
+134 −3
Open

[BLOG] From Chaos to Clarity: Revolutionizing OpenSearch Clients and Documentation with a Unified API Specification #3570

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
51f7e9d
[BLOG] From Chaos to Clarity: Revolutionizing OpenSearch Clients and …
nhtruong Jan 23, 2025
dc360da
# nbower -> naarcha
nhtruong Jan 24, 2025
22a352d
Doc review
kolchfa-aws Jan 24, 2025
c736483
One more change
kolchfa-aws Jan 24, 2025
76c9011
Resolve merge conflicts
kolchfa-aws Jan 24, 2025
7db6142
Review comments
kolchfa-aws Jan 24, 2025
3135e3e
Apply suggestions from code review
kolchfa-aws Jan 24, 2025
1c05bf9
Apply suggestions from code review
kolchfa-aws Jan 27, 2025
eeb55bc
Apply suggestions from code review
kolchfa-aws Jan 27, 2025
8e0d911
Update _community_members/saimedhi.md
kolchfa-aws Jan 27, 2025
9655690
Update _community_members/wbeckler.md
kolchfa-aws Jan 27, 2025
d6aba68
Corrected William Beckler's profile.
nhtruong Feb 3, 2025
4384dcd
Corrected wbeckler.md
nhtruong Feb 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions _community_members/saimedhi.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
name: Sai Medhini Reddy Maryada
short_name: saimedhi
photo: '/assets/media/community/members/saimedhi.jpg'
title: 'OpenSearch Community Member: Sai Medhini Reddy Maryada'
primary_title: Sai Medhini Reddy Maryada
breadcrumbs:
icon: community
items:
- title: Community
url: /community/index.html
- title: Members
url: /community/members/index.html
- title: 'Sai Medhini Reddy Maryada's Profile'
url: '/community/members/sai-medhini-reddy-maryada.html'
github: saimedhi
linkedin: saimedhinimaryada
job_title_and_company: 'Software Engineer at AWS working on OpenSearch'
personas:
- author
permalink: '/community/members/sai-medhini-reddy-maryada.html'
redirect_from: '/authors/saimedhi/'
---

**Sai Medhini Reddy Maryada** is a Software Engineer at AWS working on OpenSearch. She retrofitted the legacy OpenSearch Python and PHP API generators to work with the modern OpenSearch API specification. These generators also acted as proof of concept for the API Code Generator project, which were used to make the spec more friendly for code generation purposes.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 1 addition & 1 deletion _community_members/theotr.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ permalink: '/community/members/theo-truong.html'
redirect_from: '/authors/theotr/'
---

**Theo Truong** is a Software Engineer at AWS working on OpenSearch clients and API specification.
**Theo Truong** is a Software Engineer at AWS working on OpenSearch clients and API specification. He set the foundation for the OpenSearch API specification and overhauled the OpenSearch Ruby and OpenSearch JavaScript clients using API code generators. He is also helping the OpenSearch documentation team with tooling and automation.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 1 addition & 1 deletion _community_members/vachshah.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,4 @@
redirect_from: '/authors/vachshah/'
---

**Vacha Shah** is a Software Engineer at AWS working on the OpenSearch project.
**Vacha Shah** is a Software Engineer at AWS and a former team lead of the OpenSearch clients team.

Check failure on line 24 in _community_members/vachshah.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Vacha. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Vacha. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_community_members/vachshah.md", "range": {"start": {"line": 24, "column": 3}}}, "severity": "ERROR"}
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
25 changes: 25 additions & 0 deletions _community_members/wbeckler.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
name: William Beckler
short_name: wbeckler
photo: '/assets/media/community/members/wbeckler.jpg'
title: 'OpenSearch Community Member: William Beckler'
primary_title: William Beckler
breadcrumbs:
icon: community
items:
- title: Community
url: /community/index.html
- title: Members
url: /community/members/index.html
- title: 'William Beckler's Profile'
url: '/community/members/william-beckler.html'
github: wbeckler
linkedin: thegeneral
job_title_and_company: 'Former Software Development Manager at AWS working on OpenSearch'
personas:
- author
permalink: '/community/members/william-beckler.html'
redirect_from: '/authors/wbeckler/'
---

**William Beckler** is a former Software Development Manager at AWS working on OpenSearch. He helped lay the groundwork for AWS OpenSearch to transfer to the Linux Foundation. William also spearheaded the Client API Generation project.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
3 changes: 2 additions & 1 deletion _community_members/xtansia.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,12 @@ breadcrumbs:
url: '/community/members/thomas-farr.html'
github: xtansia
twitter: xtansia
linkedin: xtansia
job_title_and_company: 'Software Development Engineer at AWS'
personas:
- author
permalink: '/community/members/thomas-farr.html'
redirect_from: '/authors/xtansia/'
---

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the first file, we use "API Code Generator project", but here we use "API Code Generation project". Also, Bill's bio refers to "the Client API Generation project". Does this refer to the same or a different project?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They all refer the same project: API Code Generation for OS Clients (Generating the API implementation code in each OpenSearch client, which accounts for 99% of the client code). Sometimes we use Client CodeGen for short. I do see that this can lead to confusions.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the name of the project to be uniform.

**Thomas Farr** is a Software Development Engineer at AWS working on the OpenSearch Clients team, with a primary focus on the .NET and Rust clients.
**Thomas Farr** is a Software Development Engineer at AWS working on the OpenSearch Clients team, maintaining the Java, .NET, and Rust clients. Thomas plays a pivotal role in the development of the OpenSearch Java client, API specification, and the API Code Generation project.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
80 changes: 80 additions & 0 deletions ...nizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
---
layout: post
title: "From chaos to clarity: Revolutionizing OpenSearch clients and documentation using a unified API specification"
authors:
- xtansia
- dblock
- theotr
- wbeckler
- saimedhi
- vachshah
- kolchfa
- naarcha
date: 2025-01-24
categories:
- community
meta_keywords: API Specification, OpenSearch Clients, OpenSearch Documentation, Code Generation, Automation
meta_description: The OpenSearch API Specification has become the authoritative reference, bridging the gaps between clients, documentation, and the server. By adopting and extending the OpenAPI Specification, we've achieved modularity, testability, and automation, transforming how APIs are described, tested, and implemented. This unified approach has revolutionized client development and documentation, ensuring consistency, accuracy, and rapid adaptability across OpenSearch.
has_math: false
has_science_table: false
---

In February 2024, we decided to revamp the OpenSearch API Specification repository by adopting the OpenAPI Specification. While this change offered significant benefits, we encountered some challenges during implementation. Most notably, although OpenAPI Specification supports multiple files, its built-in modularity features weren't sufficient for effectively organizing OpenSearch's extensive API documentation into smaller, manageable sections. Despite these initial obstacles, we successfully addressed these challenges over the past year:
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

- The OpenSearch API comprises 1,021 routes, making it impractical to specify within a single OpenAPI YAML file. To address this, we adopted a modular structure that divides the API into _namespaces_ and _schema_ files. Each namespace represents a domain of the OpenSearch APIs, making the specification easier to manage and update. This approach, while not standard in OpenAPI Specification, led us to develop a merger tool that generates a unified OpenAPI Specification file compatible with most existing OpenAPI Specification tools.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
- The OpenSearch API was developed using several custom conventions that don't conform to OpenAPI Specification standards. To address this, we created extensions such as `x-operation-group`, which groups operations performing similar tasks into a single client API function. To ensure compliance with both OpenAPI Specification and our custom extensions, we built a custom OpenAPI Specification linter that checks the specification for consistency.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

- Testing the specification at scale---required to ensure generation of valid clients---posed a significant challenge. Existing OpenAPI Specification testing frameworks were often language-specific and lacked the required flexibility. To overcome this, we developed a data-driven test framework that uses tests written in YAML, making it accessible to contributors regardless of their programming expertise or programming language knowledge. With this framework, we can confidently claim the accuracy of our specification across multiple OpenSearch versions.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

## Key achievements

These efforts have resulted in a modular, maintainable, and testable OpenSearch API specification. With help of OpenSearch contributors, we have reached the following important milestones over the past year:
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

- **Complete coverage** of all core OpenSearch APIs.
- **60% coverage** of plugin APIs, including Index Management, Security, and ML Commons, with plans to reach full coverage by the end of 2025.
- **100% test coverage** for all APIs in the specification, covering OpenSearch 1.x and 2.x.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

## Using the specification to improve OpenSearch clients

Historically, maintaining OpenSearch clients was a tedious and error-prone process. In the past two years, more than half of the GitHub issues created in the OpenSearch client repositories were related to missing or outdated API functions. The rapid addition of new features and APIs to OpenSearch exceeded our maintainers' capacity to add them to the specification. This constant need to update the specification also prevented us from focusing on other important aspects of client development, such as performance improvements.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

With the unified OpenSearch API specification, we've begun generating API functions directly from the specification. This approach eliminates manual updates, improves consistency, and ensures all clients stay synchronized with the latest server capabilities.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

Here are the key milestones:

- We updated legacy API generators for [PHP](https://github.com/opensearch-project/opensearch-php/pull/203), [.NET](https://github.com/opensearch-project/opensearch-net/pull/228), and [Python](https://github.com/opensearch-project/opensearch-py/pull/721) clients to read and parse the OpenAPI specification.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
- In December, we released a new major version of the JavaScript client. This JavaScript client version's API functions aregenerated entirely from the specification. The client also includes a complete set of request and response types for each API function, which was the most requested feature from our TypeScript users. For more information, see [this blog post](https://opensearch.org/blog/Introducing-OpenSearch-JS-Client-3.0/).

Check failure on line 47 in _posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.SpacingWords] There should be one space between words in 'functions aregenerated'.

Raw Output:
{"message": "[OpenSearch.SpacingWords] There should be one space between words in 'functions aregenerated'.", "location": {"path": "_posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md", "range": {"start": {"line": 47, "column": 111}}}, "severity": "ERROR"}
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
- We [overhauled the Ruby client](https://github.com/opensearch-project/opensearch-ruby/pull/261) to use the OpenSearch API specification as the source for its API implementation. The newly generated API functions provide greater consistency and predictability compared to their handwritten predecessors. For example, all input arguments are now cloned into a new hash before any operations are performed on them---the step that was missing during manually writing the functions. The new major version of the Ruby client, featuring these generated API functions, is currently in beta testing and will be released in the coming months.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
- We completely revamped the Java client, our most verbose and feature-rich high-level client with strongly typed request and response objects. We replaced [hundreds of thousands lines of legacy code](https://github.com/opensearch-project/opensearch-java/pulls?q=sort%3Aupdated-desc+is%3Apr+is%3Amerged+%5EGenerate+author%3AXtansia) in this client. Additionally, while working on an API generator for this client, we uncovered many errors in the spec that we have since fixed.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

The implementation of API generators has transformed our client development process. Instead of manually updating API functions, we can now focus on enhancing client performance, stability, and usability. Using the OpenSearch API specification as the single reference ensures all clients remain synchronized with each other. When developers add a new API to OpenSearch or its plugins, they simply update the specification, and the functionality becomes immediately available across all OpenSearch clients. Similarly, when an API bug is discovered in one client, fixing it in the specification automatically resolves the issue across all clients. The generated API functions also provide much greater consistency in features and behavior compared to their previous handwritten counterparts.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

Looking ahead, having a complete specification will enable us to rapidly develop new clients in additional programming languages!

## Generating documentation from the specification

Like the clients, OpenSearch documentation faced challenges keeping pace with the rapid development of OpenSearch server and plugins. The process of manually writing and updating descriptions for each API's parameters, requests, and responses was time-consuming and error-prone. Documentation would occasionally become inconsistent with both the clients and the server, causing user confusion. These inconsistencies created a poor experience for our users made it more difficult for new applications to adopt OpenSearch.

Check failure on line 57 in _posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: API's. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: API's. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md", "range": {"start": {"line": 57, "column": 202}}}, "severity": "ERROR"}
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line 57: The first sentence references "the OpenSearch server and plugins", but the penultimate sentence references "the clients and the server". Line 67 references "the server and clients". Should the phrase in the first sentence be "the OpenSearch server and clients", or is "plugins" correct?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plugins is correct. Clients are generated off of OpenSearch core (server) and plugins.

We've begun addressing this by generating documentation components directly from the OpenSearch API specification. For example, [path and query parameter tables are now generated from the spec](https://github.com/opensearch-project/documentation-website/pull/8692), replacing handwritten tables. This process has also helped uncover and correct spec errors.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

In the near future, the following components will also be generated from the spec:
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

- Request and response body fields and their descriptions.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
- Examples of requests and responses, derived from tests in the spec repository.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
- Usage examples for each client language.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

These changes ensure the documentation remains accurate and consistent with the server and clients, creating a better experience for developers and reducing barriers to OpenSearch adoption.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

## Thank you

Over the past year, it has been rewarding to witness the evolution and improvement of the OpenSearch API specification. The specification now serves as the authoritative reference for OpenSearch clients, documentation, and server, ensuring consistency across all components. We look forward to seeing how the OpenSearch project will continue to grow with the OpenSearch API specification as its foundation.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

We couldn't have achieved this without the dedication of our contributors. Their feedback, suggestions, and contributions have been invaluable to the project. The Amazon OpenSearch team would like to give a special thanks to the following contributors for their outstanding work on the OpenSearch API specification:
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
* [Niyazbek Torekeldi](https://github.com/Tokesh): For backfilling the OpenSearch API specification with missing APIs and fixing numerous spec errors. Tokesh also helped us reach 100% test coverage for the spec.

Check failure on line 75 in _posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Niyazbek. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Niyazbek. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md", "range": {"start": {"line": 75, "column": 4}}}, "severity": "ERROR"}

Check failure on line 75 in _posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Torekeldi. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Torekeldi. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md", "range": {"start": {"line": 75, "column": 13}}}, "severity": "ERROR"}

Check failure on line 75 in _posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: backfilling. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: backfilling. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md", "range": {"start": {"line": 75, "column": 56}}}, "severity": "ERROR"}

Check failure on line 75 in _posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Tokesh. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Tokesh. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md", "range": {"start": {"line": 75, "column": 152}}}, "severity": "ERROR"}
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
* [Andriy Redko](https://github.com/reta): For his many contributions to the OpenSearch project, especially for sharing his expertise during many discussions about the spec repository.

Check failure on line 76 in _posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Andriy. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Andriy. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md", "range": {"start": {"line": 76, "column": 4}}}, "severity": "ERROR"}

Check failure on line 76 in _posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Redko. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Redko. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_posts/2025-02-20-revolutionizing-opensearch-clients-and-documentation-with-a-unified-api-specification.md", "range": {"start": {"line": 76, "column": 11}}}, "severity": "ERROR"}
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

## Looking ahead

We look forward to continuing this journey with you. If you have feedback or suggestions for the OpenSearch API specification, please open an issue in the [OpenSearch API Specification repository](https://github.com/opensearch-project/opensearch-api-specification/issues). Your contributions help shape the future of OpenSearch.
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
Binary file added assets/media/community/members/saimedhi.jpg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/media/community/members/wbeckler.jpg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading