Enhance langgraph integration to preserve metadata

[i-w-a]

Enhance langgraph integration to preserve AI metadata

Description

This PR updates the langgraph.py integration to ensure that metadata are preserved. This enhancement is crucial for multi-agent scenarios where identifying the source AI is important for evaluation.

Changes

• Updated langgraph.py to ensure AI names and metadata are preserved.

Motivation and Context

In the current implementation, metadata such as the name assigned to an AI is not saved. In the era of multi-agent systems, it is essential to have information about which AI made a particular statement for accurate evaluation. This update addresses this issue by preserving the necessary metadata.

Example Code and Output

import json
from typing import List, Union

from langchain_core.messages import AIMessage, HumanMessage, SystemMessage, ToolMessage

import ragas.messages as r
from ragas.integrations.langgraph import convert_message_with_metadata

def test_convert_message_with_metadata():
    from langchain_core.messages import HumanMessage, AIMessage

    human_message = HumanMessage(content="Hello", name="me", additional_kwargs={"key1": "value1"})
    ai_message = AIMessage(content="Hi", name="ai_1", additional_kwargs={"tool_calls": [{"function": {"name": "tool1", "arguments": '{"arg1": "val1"}'}}]})

    converted_messages = convert_message_with_metadata([human_message, ai_message])

    for msg in converted_messages:
        print(f"Content: {msg.content}, Metadata: {msg.metadata}")

if __name__ == "__main__":
    test_convert_message_with_metadata()

Output
Content: Hello, Metadata: {'additional_kwargs': {'key1': 'value1'}, 'response_metadata': {}, 'type': 'human', 'name': 'me', 'id': None, 'example': False}
Content: Hi, Metadata: {'additional_kwargs': {'tool_calls': [{'function': {'name': 'tool1', 'arguments': '{"arg1": "val1"}'}}]}, 'response_metadata': {}, 'type': 'ai', 'name': 'ai_1', 'id': None, 'example': False, 'tool_calls': [{'name': 'tool1', 'args': {'arg1': 'val1'}, 'id': None, 'type': 'tool_call'}], 'invalid_tool_calls': [], 'usage_metadata': None}

[jjmachan]

thanks a lot @i-w-a for contributing this 🙂
@sahusiddharth could you take a look and merge it in? would love to have it in before the next release

[sahusiddharth]

I think making a completely new function for handling metadata might not be the most efficient approach. It could be more effective to enhance the existing convert_to_ragas_messages function by incorporating metadata extraction into it by default.

This way, we could avoid redundancy.

[jjmachan]

what about deleting the old one and replacing it with this @sahusiddharth - would that be easier? or will we loose functionality?

[sahusiddharth]

Sure, we can do that. There won’t be any change in functionality.

[i-w-a]

Thank you for your comment!
Would I rewrite the existing implementation to replace convert_to_ragas_message?

[sahusiddharth]

@i-w-a, Yes, please keep the function name as convert_to_ragas_message.

[i-w-a]

Hi team,

I'm facing test failures and would appreciate your help.

1. Dataset Schema Tests: The tests tests/unit/test_dataset_schema.py::test_single_type_evaluation_dataset[eval_sample0] and [eval_sample1] are failing with the error "All samples must be of the same type". This indicates a mix of SingleTurnSample and MultiTurnSample in the test data. I'll investigate the test data generation, but any insights would be welcome.

Any guidance on these issues would be greatly appreciated.

Thanks,

[sahusiddharth]

@i-w-a, It seems that the test cases are failing because of a previous PR that modified the output of the error message triggered by the assertion error. I have just raised a PR that will fix it.

[i-w-a]

@sahusiddharth
Thank you for the comment!
I am waiting for your PR to be merged!