Inconsistency in the Value of EVALMODE_TO_COLUMNS[EvaluationMode.ca]

[jucgonzalezes]

Hi,

It seems there might be an inconsistency in the value of the dictionary EVALMODE_TO_COLUMNS[EvaluationMode.ca] within src/ragas/validation.py. Currently, the value is set to ["context", "summary"]

ragas/src/ragas/validation.py

Lines 62 to 71 in d5b60bb

	EVALMODE_TO_COLUMNS = {
	EvaluationMode.qac: ["question", "answer", "contexts"],
	EvaluationMode.qa: ["question", "answer"],
	EvaluationMode.qc: ["question", "contexts"],
	EvaluationMode.gc: ["ground_truth", "contexts"],
	EvaluationMode.ga: ["ground_truth", "answer"],
	EvaluationMode.qga: ["question", "ground_truth", "answer"],
	EvaluationMode.qcg: ["question", "contexts", "ground_truth"],
	EvaluationMode.ca: ["contexts", "summary"],
	}

but I believe it should be ["context", "answer"] as in src/ragas/metrics/base.py:get_required_columns():

ragas/src/ragas/metrics/base.py

Lines 38 to 59 in d5b60bb

	def get_required_columns(
	eval_mod: EvaluationMode, ignore_columns: t.Optional[t.List[str]] = None
	) -> t.List[str]:
	if eval_mod == EvaluationMode.qac:
	keys = ["question", "answer", "contexts"]
	elif eval_mod == EvaluationMode.qa:
	keys = ["question", "answer"]
	elif eval_mod == EvaluationMode.qc:
	keys = ["question", "contexts"]
	elif eval_mod == EvaluationMode.gc:
	keys = ["contexts", "ground_truth"]
	elif eval_mod == EvaluationMode.ga:
	keys = ["answer", "ground_truth"]
	elif eval_mod == EvaluationMode.qga:
	keys = ["question", "contexts", "answer", "ground_truth"]
	elif eval_mod == EvaluationMode.qcg:
	keys = ["question", "contexts", "ground_truth"]
	elif eval_mod == EvaluationMode.ca:
	keys = ["contexts", "answer"]
	ignore_columns = ignore_columns or []
	return [k for k in keys if k not in ignore_columns]

Is this the expected behavior? If not, could you please take look into it?

Thanks!

Best regards,
Juan

[dosubot]

Hey @jucgonzalezes!

I'm here to help you out with any bugs, questions, or contributions you have in mind. Let's tackle this together!

The value of EVALMODE_TO_COLUMNS[EvaluationMode.ca] in src/ragas/validation.py should indeed be ["contexts", "answer"] as indicated in the get_required_columns() function from src/ragas/metrics/base.py ^[1][2]. The current value ["contexts", "summary"] appears to be incorrect and should be updated to maintain consistency across the codebase.

To continue talking to Dosu, mention @dosu.