Building chat bots: How to save a Conversation for later use?

[onerinas]

I'm using Langchain::Conversation to build a chatbot. Since we need the conversation memory, I'm saving the class in the db and later using it to continue the Conversation, as shown below.

chat = Langchain::Conversation.new(llm: llm)
chat.message("Hello")

# encode to save in the database
encoded_chat = Base64.encode64(Marshal.dump(chat))

# Retrieve for later use
chat = Marshal.load(Base64.decode64(encoded_chat))
chat.message("Tell me about future technologies")

The problem here is that it breaks after updating the gem, which changes this class.

A better solution would be to save not the encoded class but just the history in the DB, which can be passed to a new instance of Conversation whenever a new message comes in from the user. I don't see a way to set the conversation history. There is add_examples, but I believe that serves a different purpose?

Curious to know how others handle such scenarios.

[andreibondarev]

@onerinas Take a look at the Assistants that we'll be rolling out soon. That PR will deprecate the Langchain::Conversation class.

When using Assistants, you'll be able to save the hashes. For example for OpenAI you'll be able to call something like:

assistant.thread.messages.map &: to_openai_format

We'll probably have similar methods for other LLMs, but basically you'll need to figure out how to best store the Langchain::Message representation.

[onerinas]

Does our implementation of Assistants use OpenAI's Assistants API? I haven't looked into the code yet and I hope that's not the case. If yes, does not make sense to deprecate Langchain::Conversation as it is a different offering and has a different pricing as well.

[andreibondarev]

@onerinas It doesn't use the OpenAI Assistants API but it does use the OpenAI functions (which has been deprecated and now called tools:).

[onerinas]

Good to know its not the direct map of the OpenAI's Assistant feature.

So with Assistant, how do we save the messages for later use? Assuming we can access the messages like this

1. assistant.thread.messages.map &: to_openai_format
2. and save them in our database

How do we then use it for later use? Can I pass the history (messages saved in our DB) while creating the assistant or set the history after instantiating it?

I'm trying to figure out how we can keep the conversation history while using the Assistant as chatbot.

[andreibondarev]

@onerinas You'd instantiate a Langchain::Thread and pass the messages to it here: https://github.com/andreibondarev/langchainrb/pull/422/files#diff-6ec65bec0259e981b815da287e8340a8d0241372cd5efd973887db9786d8453eR8

thread = Langchain::Thread.new messages: old_messages
assistant = Langchain::Assistant.new thread: thread, ...

Keep in mind that the thread.messages also contain tool information and tool calls and outputs from tools. All of this will be needed to rebuild the full conversation history.

Would that work for you?

[onerinas]

Yes, it might solve the problem. Thank you for the pointer :)

[mattlindsey]

The question mentions saving the messages to a db, so if it's a rails app is it best to do it yourself? Or does langchainrb_rails offer any anything?

P.S. Just found rails generate langchainrb_rails:assistant, so that's probably it!
P.P.S. Actually that doesn't save the messages, does it? Saving/persisting the messages is a different thing. Right?

[mattlindsey]

Looks like you are already enhanching the Langchain::Assistant in langchainrb_rails to do stuff like that. Looks great!

[andreibondarev]

@mattlindsey Did you get a chance to try it out?

[mattlindsey]

I will try tomorrow and report back! :)

[mattlindsey]

I'm trying it, but i'm not sure of the usage. I ran the generator and in my rails app I do following to create a new Assistant and save it, but the tool doesn't get saved and I can't seem to do things like assistant.add_message_and_run("hello"):

    @assistant = Assistant.create!
       instructions: "You are a Story writer assistant.",
       tools: [
         Langchain::Tool::Wikipedia.new
       ]
    )

[mattlindsey]

Oh. I forgot that I should be using the code in the pull request where you add the UI with views, controller, routes, etc. Let me try that. I think I can point my Gemfile to that PR and try again.