input is not valid Modified UTF-8: illegal continuation byte 0

[iprovalo]

Latest master, I am seeing this new exception:

 java_vm_ext.cc:598] JNI DETECTED ERROR IN APPLICATION: input is not valid Modified UTF-8: illegal continuation byte 0
              java_vm_ext.cc:598]     string: ' ?'
              java_vm_ext.cc:598]     input: '0x20 0xc4'
              java_vm_ext.cc:598]     in call to NewStringUTF
              java_vm_ext.cc:598]     from java.lang.Object[] com.k2fsa.sherpa.onnx.OfflineRecognizer.getResult(long)

Steps to reproduce:

ASR with whisper small model, pass the language as cs (Czech), utterance - "check check". Reproduces almost 100%. This is a regression introduced in the last 2-4 weeks.

[iprovalo]

@csukuangfj I tested some fixes locally, like this:

std::string text_str = isValidUtf8(result.text) ? result.text : sanitizeUtf8(result.text);
jstring text = env->NewStringUTF(text_str.c_str());

and added my own implementation of the new methods (isValidUtf8 and sanitizeUtf8). By I think all the normalization is supposed to happen in the text-utils.cc

[csukuangfj]

We have already done

sherpa-onnx/sherpa-onnx/csrc/offline-recognizer-impl.cc

Lines 498 to 500 in 4801094

	std::string OfflineRecognizerImpl::ApplyInverseTextNormalization(
	std::string text) const {
	text = RemoveInvalidUtf8Sequences(text);

sherpa-onnx/sherpa-onnx/csrc/offline-recognizer-whisper-impl.h

Line 159 in 4801094

r.text = ApplyInverseTextNormalization(std::move(r.text));

Can you output the byte sequence of the invalid utf8 string?

[iprovalo]

We have already done

sherpa-onnx/sherpa-onnx/csrc/offline-recognizer-impl.cc

Lines 498 to 500 in 4801094

std::string OfflineRecognizerImpl::ApplyInverseTextNormalization(
std::string text) const {
text = RemoveInvalidUtf8Sequences(text);
sherpa-onnx/sherpa-onnx/csrc/offline-recognizer-whisper-impl.h

Line 159 in 4801094

r.text = ApplyInverseTextNormalization(std::move(r.text));
Can you output the byte sequence of the invalid utf8 string?

This is what I get in the log: input: '0x20 0xc4'

[csukuangfj]

So the string has only two bytes 0x20 0xc4?

We need to update our RemoveInvalidUtf8Sequences() to handle that. Would you mind creating a PR to fix RemoveInvalidUtf8Sequences() for your case?

[iprovalo]

So the string has only two bytes 0x20 0xc4?

We need to update our RemoveInvalidUtf8Sequences() to handle that. Would you mind creating a PR to fix RemoveInvalidUtf8Sequences() for your case?

No problem, I can do that. How do I run the unit-tests for this project?

[csukuangfj]

No problem, I can do that.

Thanks!

---

How do I run the unit-tests for this project?

Please use

git clone https://github.com/k2-fsa/sherpa-onnx
cd sherpa-onnx
mkdir build
cd build
cmake -DSHERPA_ONNX_ENABLE_TESTS=ON ..
make test-utils-test

./bin/test-utils-test

Please also update text-utils-test.cc to include your case.

[iprovalo]

No problem, I can do that.

Thanks!

How do I run the unit-tests for this project?

Please use

git clone https://github.com/k2-fsa/sherpa-onnx
cd sherpa-onnx
mkdir build
cd build
cmake -DSHERPA_ONNX_ENABLE_TESTS=ON ..
make test-utils-test

./bin/test-utils-test
Please also update text-utils-test.cc to include your case.

After I successfully ran cmake -DSHERPA_ONNX_ENABLE_TESTS=ON ..

Then I try to run this ./bin/test-utils-test

I get this error:
zsh: no such file or directory: ./bin/test-utils-test

[iprovalo]

I got it:

make -j4 
./bin/test-utils-test

[iprovalo]

@csukuangfj here is the PR with a fix: #1904

[csukuangfj]

Closing by #1904