Answer relevancy metric is worse in languages other than English

[icejean]

I just try ragas to evaluate my GraphRAG app in Chinese, and find that the metric answer relevancy is worse for every question. And I find out that the cause is the question generated is in English， so the embedding of the original question and the generated question is quite different. To address this issue, I modify the function in ~/ragas/prompt/pydantic_prompt.py to demand LLM to output the generated question in Chinese, and it does work.

    def _generate_output_signature(self, indent: int = 4) -> str:
        return (
            f"Please return the output in a JSON format that complies with the "
            f"following schema as specified in JSON Schema and the generated question in Chinese:\n"
            f"{self.output_model.model_json_schema()}"
        )

But I know the function is called not only for this metric, and a solution is needed to support all languages, so I write down the issue here.
Best regards
Jean from China

[jjmachan]

hey there Jean 👋🏽

Have you tried adapting the metrics into chineese? is it still doing down for that?
https://docs.ragas.io/en/stable/howtos/customizations/metrics/_metrics_language_adaptation/

[icejean]

hey there Jean 👋🏽

Have you tried adapting the metrics into chineese? is it still doing down for that? https://docs.ragas.io/en/stable/howtos/customizations/metrics/_metrics_language_adaptation/

Not yet, I'm new to ragas and don't know about the feature, I'll have a try soon, thanks for your reply. 8-)