Updated testset generation question prompts to use JSON formatting instructions + prompt tweaks for smaller LLMs

[fschuh]

-> This PR is mostly targeted at making the Ragas testset generation prompts work with smaller LLMs that can be run locally.

Some of the testset generation prompts are currently not making use of the JSON format instructions that were added to the eval prompts not too long ago. This PR fixes that.

Also, a few other notable fixes in this PR:

• Improved testset generation question rewriting prompts to reduce verbosity so they work with smaller and chattier LLMs
• Improved JSON input rewrite prompt to reduce verbosity so it works with smaller LLMs
• Fixed incorrect index in one of the examples of find_relevant_contexts prompt (this will make it more explicit that the index is 1-based).

[jjmachan]

@fschuh thanks a lot for pushing in this PR but we are reworking the testset generation #1321

would love to run the updated version by you and work to get these fixes applied their too if your interested

[fschuh]

Sure thing, I can take a look to see if I can apply these fixes on the new experimental testset generation, if they are still relevant of course.
Would it make sense to apply the fixes on both? I still find the old testset generation quite good, despite being a bit slower than I'd like but it still generates good questions.

[jjmachan]

@shahules786 can you take a look?

[shahules786]

Hey @fschuh thanks for pushing the PR. I see the changes in prompts, but the changes in prompts are mostly always specific to the model you're using. We understand it was hard to change prompts of Ragas metrics, and testgen components. From v0.2 we will introduce set_prompts and get_prompts interfaces for all the components so that you can modify prompts without installing ragas from the source and then raising a PR. Prompts are fundamentally just like hyperparameters for these algorithms.

[KylinMountain]

I think we can try to augment json parser which is able to parse the json in markdown or anything else.

[shahules786]

@KylinMountain that's a good idea. I also have been thinking about it. The Prompt object should ideally be able to convert and parse prompts to any format like markdown, XML,etc
This could be too much for 0.2 release, so I'll add this to discussions for 0.3.

[KylinMountain]

@KylinMountain that's a good idea. I also have been thinking about it. The Prompt object should ideally be able to convert and parse prompts to any format like markdown, XML,etc This could be too much for 0.2 release, so I'll add this to discussions for 0.3.

glad to hear that.

[jjmachan]

@shahules786 are we merging this in or should we create a new issue to track this problem?

[shahules786]

@jjmachan No.
@fschuh If you're are happy with the explanation of why this is not mergeable, please considering closing the PR.

Hey @fschuh thanks for pushing the PR. I see the changes in prompts, but the changes in prompts are mostly always specific to the model you're using. We understand it was hard to change prompts of Ragas metrics, and testgen components. From v0.2 we will introduce set_prompts and get_prompts interfaces for all the components so that you can modify prompts without installing ragas from the source and then raising a PR. Prompts are fundamentally just like hyperparameters for these algorithms.

[jjmachan]

closing this since we have merged get and set prompt #1391

@fschuh would love to get your feedback on they new way and if solves the original problem you were fixing with this PR.

are you on discord?

[fschuh]

@jjmachan this mixin solution seems to allow full control of the prompts without module reload hacks or modifying the Ragas code, which is great.
I see this was only merged in experimental though. Are there plans for bringing this out of experimental status any time soon? Perhaps on 0.2?

[jjmachan]

yes @fschuh will be out on monday - just adding the final touches 🙂