V0.2.7 Context Recall only returning 1 or 0 values

[timelesshc]

[ ] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question
After upgrading the version to 0.2.7, the Context Recall metrics only return 1 or 0 values. Is this intentional? The previous version would return values between 0 and 1.

Code Examples
This community speaks code. Share your code snippets to help us understand your question better.

Additional context
Anything else you want to share with us?

[sahusiddharth]

Hi @timelesshc,

Are you still facing this issue?

I tested with a simple example, I received fractional values. The results you're seeing might be related to your data. I've included the code snippet I used below for reference:

from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import LLMContextRecall

sample = SingleTurnSample(
    user_input="Tell me about the Great Wall of China.",
    response="The Great Wall of China is in northern China and was built to protect against invasions.",
    reference="The Great Wall of China is in northern China, spans thousands of miles, and was built to protect against invasions.",
    retrieved_contexts=[
        "The Great Wall of China is a historic structure located in northern China.",
        "The wall was built over centuries to protect against invasions from nomadic tribes."
    ],
)

context_recall = LLMContextRecall(llm=evaluator_llm)
await context_recall.single_turn_ascore(sample)

Output

0.6666666666666666

In the above example:

1. The claim "The Great Wall of China is in northern China" can be derived from the first retrieved context.
2. The claim "The wall was built to protect against invasions" can be derived from the second retrieved context.
3. The claim "spans thousands of miles" is not directly supported by the retrieved contexts.