Prompt Error, TimeOut error

[giambascientist86]

2024-11-20 16:06:08,086 - ERROR - Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
2024-11-20 16:06:08,093 - ERROR - Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
2024-11-20 16:06:08,094 - ERROR - Prompt fix_output_format failed to parse output: The output parser failed to parse the output including retries.
2024-11-20 16:06:08,094 - ERROR - Prompt context_recall_classific[ ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
This two errors are continuously appearing when evaluating on HuggingFace models and HF Datasets

Ragas version:latest
Python version:3.12

Code to Reproduce

Error trace

Expected behaviour:
I expect no output parser errors!!

Additional context
Add any other context about the problem here.

[giambascientist86]

Hello guys ; any update on this issue? I ma still facing it blocking my pipeline..:)

[NoamDetournay]

Hi @giambascientist86, I encountered the same issue and managed to resolve it by using a callback to capture the format of the prompt and the model's output. I actually came across a great solution in #1729, which worked really well.

Here's how I did it:

from langchain_core.callbacks import BaseCallbackHandler
from ragas import evaluate,RunConfig

class TestCallback(BaseCallbackHandler):

    def on_llm_start(self, serialized, prompts, **kwargs):
        print(f"**********Prompts*********:\n {prompts[0]}\n\n")

    def on_llm_end(self, response, **kwargs):
        print(f"**********Response**********:\n {response}\n\n")

score = evaluate(dataset,metrics=[faithfulness],
                 llm=llm,
                 embeddings=azure_embeddings,
                 raise_exceptions=True,
                 callbacks=[TestCallback()],
                 run_config=RunConfig(timeout=10,max_retries=1,max_wait=60,max_workers=1)
                )
score.to_pandas()

Hope it helps!

Also I use ragas == 0.2.6