diff --git a/data-prepper-plugins/opensearch/README.md b/data-prepper-plugins/opensearch/README.md index fb6c5e329d..7fd0fa9de0 100644 --- a/data-prepper-plugins/opensearch/README.md +++ b/data-prepper-plugins/opensearch/README.md @@ -200,6 +200,42 @@ the flush timeout and instead flush whatever is present at the end of each batch - `routing` (optional): A string which is used as hash for generating sharding id for the document when it is stored in the OpenSearch. Each incoming record is searched for this field and if it is present, it is used as the routing field for the document, if it is not present, default routing mechanism used by the OpenSearch when storing the document. Standard Data Prepper Json pointer syntax is used for retrieving the value. If the field has "/" in it then the incoming record is searched in the json sub-objects instead of just in the root of the json object. For example, if the field is specified as `info/id`, then the root of the event is searched for `info` and if it is found, then `id` is searched inside it. The value specified for `id` is used as the routing id. * This routing string can also be a formatted string, such as `routing-${rid}-name`. When OpenSearchSink is sending data to OpenSearch, the format portion "${rid}" will be replaced by it's value in the event that is being processed. The format may also be like "${rid1/rid2/rid3}" in which case the field "rid1/rid2/rid3" is searched in the event and replaced by its value. * Additionally, the formatted string can include expressions to evaluate to format the routing string. For example, `my-${rid}-${getMetadata(\"some_metadata_key\")}-name` will inject both the `rid` value from the Event, as well as the value of `some_metadata_key` from the Event metadata to construct the routing string. + Examples: + 1. Routing config with simple strings +``` + sink: + opensearch: + hosts: ["https://your-amazon-opensearch-service-endpoint"] + aws_sigv4: true + cert: path/to/cert + insecure: false + routing: "test_routing_string" + bulk_size: 4 +``` + + 2. Routing config with keys from event +``` + sink: + opensearch: + hosts: ["https://your-amazon-opensearch-service-endpoint"] + aws_sigv4: true + cert: path/to/cert + insecure: false + routing: "${/info/id}" + bulk_size: 4 +``` + + 3. Routing config with more complex expressions +``` + sink: + opensearch: + hosts: ["https://your-amazon-opensearch-service-endpoint"] + aws_sigv4: true + cert: path/to/cert + insecure: false + routing: '${/info/id}-test-${getMetadata("metadata_key")}' + bulk_size: 4 +``` - `ism_policy_file` (optional): A String of absolute file path or AWS S3 URI for an ISM (Index State Management) policy JSON file. This policy file is effective only when there is no built-in policy file for the index type. For example, `custom` index type is currently the only one without a built-in policy file, thus it would use the policy file here if it's provided through this parameter. OpenSearch documentation has more about [ISM policies.](https://opensearch.org/docs/latest/im-plugin/ism/policies/)