Faithfulness prompt update to avoid having single quotes in response

[michaelromagne]

The error described in this comment is not resolved when computing Faithfulness.

After digging in judge LLMs responses, there is a JSON parse error happening when parsing the output of the judge. For instance, it happens for the below answer:

{
  "statements": [
    {
      "statement": "The Norwegian Dawn cruise ship was denied access to Mauritius.",
      "reason": "The context explicitly states that local authorities denied permission for the Norwegian Dawn ship to access the Mauritius capital of Port Louis.",
      "verdict": 1
    },
    {
      "statement": "The denial of access was due to potential health risks.",
      "reason": "The context directly mentions that the ship was denied access \"citing \\\"potential health risks.\\\"\"",
      "verdict": 1
    },
    {
      "statement": "The specific health risk was a potential cholera outbreak on the Norwegian Dawn cruise ship.",
      "reason": "While the context mentions fears of a potential cholera outbreak in the title, it does not explicitly state that cholera was the specific health risk on the Norwegian Dawn. The context only mentions \'stomach-related illness\' without specifying cholera.",
      "verdict": 0
    }
  ]
}

The error is that the generated context has single quotes as you can see in the last reason. This is not allowed in JSON, and it happens frequently with the current prompt for Faithfulness as it often cite elements from the retrieved context to explain its verdict.

I tried to change the PydanticOutputParser logic in Langchain Core, the one that Ragas uses to parse JSON. I tried to replace single quotes by double quotes with simple string replace but it did not work.

Thus, an immediate solution that worked for me was to specifically ask the judge LLM to only output double quotes, not single quotes, and the error disappears.

[jjmachan]

hey @michaelromagne thanks you so much for addressing this but I was wondering if we could add this to the main pydantic object here instead

ragas/src/ragas/prompt/pydantic_prompt.py

Lines 44 to 49 in f5bc2b5

	def _generate_output_signature(self, indent: int = 4) -> str:
	return (
	f"Please return the output in a JSON format that complies with the "
	f"following schema as specified in JSON Schema:\n"
	f"{self.output_model.model_json_schema()}"
	)

that way other metrics will also get the benefit - what do you think?

[michaelromagne]

Yes indeed, that's perfect. I'll change it tomorrow, thanks 😄

[jjmachan]

that would be awesome - hopefully we can get this out in the next release (tues morning PT)

[michaelromagne]

✅ The change works like a charm, thanks for the recommendation

[michaelromagne]

@jjmachan is it good for you ?

[jjmachan]

sweet, this works great! thank you so much 🙂