Fix bug in calculating score when output from LLM is broken #317
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks @Yongtae723 , This makes sense. I'll check this while I modify faithfulness prompts + demonstrations as mentioned here. |
|
alright, Make sense. Forcing json output might solve this issue. I am looking forward to hear your experiment result. And if possible, I would like to check it works as our expectation. So if possible let me know your PR! |
jjmachan
pushed a commit
that referenced
this pull request
Nov 24, 2023
|
Thanks! and sorry for my late responce I checked prompt and tested. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If generated answer is too stupid, evaluation of faithfull will be broken.
For example, if you change fake result into "I love christmass" ,which is completely nun sense answer on cell 10 on this notebook, the result of faithfullness will be 1. This is not good. You can try it out.
I debug the reason and showed like if the result is too stupid,
1st and 2nd output from LLM will be
There is no relevant statement that can be created from the given answer.but ragas implementation is based the assumption there are "verdict: yes" or "verdict: no".
The way to solve this issue is not only one. for example, slitly modify template prompt for stupid answer is one example, but it can be affect the result.
So I make pr for my suggestion.
Maybe this PR will conflict my PR #307, If so I will fix conflict!
Thanks