Build Fails Due to Unexpected Character in tinyld

[alexzhovnir]

I'm transitioning my project from Create React App (CRA) to Vite. The project runs without any issues using npm run dev, but fails to build with npm run build. The error is as follows:

[alexzhovnir]

Steps to Reproduce

1. Install tinyld in a Vite project.
2. Run npm run build.

Expected Behavior

The build should complete successfully without errors.

Actual Behavior

The build fails with the above-mentioned error due to the unexpected character in tinyld.

Additional Context

The issue seems to be related to the handling of non-JavaScript files or unexpected characters in the tinyld module during the build process with Rollup.

Environment

• Vite version: 5.2.13
• Node version: v18.20.3
• tinyld version: v1.3.4

Vite config

`error during build:
node_modules/tinyld/dist/tinyld.normal.node.mjs (1:528704): Unexpected character '\u{200c}' (Note that you need plugins to import files that are not JavaScript)
file: /Users/zhovnir/Documents/GitHub/vite-diglot-ai/node_modules/tinyld/dist/tinyld.normal.node.mjs:1:528704

1: var N={TINYLD_CONFIG:"normal"};var y=/[,.。，、#%&/\+*¡!¿?[]！？;:…„“«»”"“_–—~]/gi,p=/[0-9]/g,S=/[\uFF10-\uFF19]/g,v=/\s\s+/g,j=/’/gi,H=/[\u0300-\u036f]/g;function X(a){return typeof a=="string"||a instanceof String}function V(a){return a.toLowerCase().replace(j,"'").replace(y," ").replace(S,i=>String.fromCharCode(i.charCodeAt(0)-65248)).replace(p,"").replace(v," ").trim()}function U(a){return a.normalize("NFD").replace(H,"")}var P={light:{TRAINING_UNIQUE_GRAMS:... and lines more`

Thank you for your assistance!

[kefniark]

Hello, yes I remember a similar issue like 2 years ago. At the time it was a special UTF-8 character not supported by esbuild minification. And after reporting it to esbuild, it was fixed under 48h

Unexpected character '\u{200c}

One of the temporary workaround could be to use vite.rollup.external settings, to exclude this library from processing.
I don't think we can help much here, it's totally outside of our scope.

It's one of the downside of a library like tinyld, the dictionary cover many languages and includes a lot of utf8 characters that are not so usual in codebase.

[mrowles]

Did you find a work around to this @alexzhovnir ?