You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Not necessarily part of this project, but I was thinking, to make Cloudflare D1 viable for CJK projects, I would want to have some way to search for CJK text too. As SQLite official FTS does not support tokenizing Japanese well, it may be interesting to implement my own tokenizer in JavaScript to support this. (Can Intl.Segmenter be used in CF Workers?) or alternatively, perhaps we would just need to do a prepass on the Japanese text before passing to SQLite FTS.
The text was updated successfully, but these errors were encountered:
Not necessarily part of this project, but I was thinking, to make Cloudflare D1 viable for CJK projects, I would want to have some way to search for CJK text too. As SQLite official FTS does not support tokenizing Japanese well, it may be interesting to implement my own tokenizer in JavaScript to support this. (Can Intl.Segmenter be used in CF Workers?) or alternatively, perhaps we would just need to do a prepass on the Japanese text before passing to SQLite FTS.
The text was updated successfully, but these errors were encountered: