Model selection by doing translation and evaluation on the fly #55
Comments
|
I ran some experiments to check if computing BLEU on the fly (#61) and naive unknown word replacement (#60) work fine or not. It seems like the BLEU score calculation works as intended though there's a possibility of bugs. If we try to translate the following German sentence to English taken from
the output of the model is
You may see that the translated sentence contains two German words, i.e., Due to the limit of vocabulary size, both German words are out of vocabulary, meaning that the input sentence is fed into the model as follows.
Note that the original output from the model before UNK replacement would be the following.
As you can see, this sort of naive UNK replacement may work pretty well if UNK tokens are proper noun and have same in both source and target languages. |
After the discussion in #52, we ended up with the need of model selection based on evaluation metrics such as BLEU on the fly, which allows us to save the disk space as well as to merge the evaluation process into the training script.
I'm looking at the way of non-blocking computation of evaluation scores using
Timerwhere an evaluation script is executed regularly with a certain time interval.Here is the sample output using a toy script.
Also, I'm going to implement unknown word replacement by using alignment weights during translation.
Let me know if you have a better idea or an easier way to achieve such things.
The text was updated successfully, but these errors were encountered: