Fixed metric adaptation issue with the prompt factory #1187
Conversation
dosubot
bot
added
the
size:XXL
| long_form_answer_prompt: Prompt = field( | ||
| default_factory=lambda: LONG_FORM_ANSWER_PROMPT | ||
| default_factory=long_form_answer_prompt_factory |
There was a problem hiding this comment.
what if we make a factory inside Prompt object and call it init() or factory().
There was a problem hiding this comment.
I think init goes a bit against the scope of __init__ and is misleading. The cleanest way would be to create a PromptFactory class, but that would add more complexity, so I'd say having a factory method is a good middle-way.
ssoima
force-pushed
the
feature/fix-language-adaptation-prompt-factory-issues
branch
from
8dc528c to
79c7683
Compare
dosubot
bot
added
size:M
|
@jjmachan , the failing lint check is not related to my code changes |
|
@jjmachan are you still following this? |
|
hey @ssoima with the new Prompt object this has been fixed closing this for now but I'm really sorry we couldn't merge it 🙁 but at the same time thanks a million for taking the time to raise this, really grateful too and do checkout this form https://docs.google.com/forms/d/e/1FAIpQLSdM9FrrZrnpByG4XxuTbcAB-zn-Z7i_a7CsMkgBVOWQjRJckg/viewform - our way of saying thank you 🙂 |
Issue:
Runtime metric language adaptation (e.g. adapting to italian and then to german in the same run) does not work.
The reason is that the @DataClass factories in the metrics use singleton objects (which only get initialized when the module is loaded) => when the prompt is instantiated a second time, the same object gets returned. Check out this loom for a debug preview: https://www.loom.com/share/2a500a4d83064489936886674e8641f9?sid=682f87fc-4052-4a33-89e4-5e4d1a6663dd
Solution:
Changed the prompts to factory methods that returns a new prompt at every call