Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Tokenizer - Letter #8498

Merged
merged 4 commits into from
Jan 3, 2025
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions _analyzers/tokenizers/letter-tokenizers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
layout: default
title: Letter Tokenizer
parent: Tokenizers
nav_order: 60
---

# Letter tokenizer

The `letter` tokenizer splits text into words when it finds any character that isn't a letter. It works well for many European languages but struggles with some Asian languages where words aren’t separated by spaces.


## Example

Let's use the `letter` tokenizer to process text by breaking it into individual terms whenever it encounters non-letter characters.


```json
POST _analyze
{
"tokenizer": "letter",
"text": "Cats 4EVER love chasing butterflies and rainbows!"
}

```
{% include copy-curl.html %}

Analyzing the text "Cats 4EVER love chasing butterflies and rainbows!" with the `letter` tokenizer produces the output:

```
"Cats", "EVER", "love", "chasing", "butterflies", "and", "rainbows"
```

## Configuration

The letter tokenizer does not have any customizable settings.
Loading