3_raft_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Model evaluation \n",
    "\n",
    "Did our fine tuning actually improve the model's performance on our RAG task? Let's find out!\n",
    "\n",
    "In order to measure the efficacy of RAFT, we'll compare a base gpt-4o-mini model with our fine tuned model.\n",
    "In the first notebook, we generated a test set that's never been used to train the model, we will use the test set to compare our 2 models.\n",
    "\n",
    "We will go through the following steps:\n",
    "1. Load the test set\n",
    "2. Perform inference with both models on the test set\n",
    "3. Clean up the models answers to remove the Chain of Thought and only keep the final answer\n",
    "3. Define evaluation metrics\n",
    "4. Run evaluation for both models\n",
    "5. Plot results\n",
    "\n",
    "\n",
    "#### 1. Loading the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "test_df = pd.read_json('./data/training_data/banking_test.jsonl', lines=True)\n",
    "test_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Run inference on the test set with both models\n",
    "\n",
    "Make sure your `.env` file contains endpoint, api key and deployment name for the baseline model and the fine tuned model. Here we compare the fine tuned gpt-4o-mini with gpt-4o-mini base but you could switch the baseline model to any model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv\n",
    "import os\n",
    "from openai import AzureOpenAI\n",
    "\n",
    "\n",
    "load_dotenv()\n",
    "# run the base and finetuned models through the dataset\n",
    "BASELINE_OPENAI_DEPLOYMENT = os.getenv(\"BASELINE_OPENAI_DEPLOYMENT\")\n",
    "BASELINE_OPENAI_ENDPOINT= os.getenv(\"BASELINE_OPENAI_ENDPOINT\")\n",
    "BASELINE_OPENAI_KEY= os.getenv(\"BASELINE_OPENAI_KEY\")\n",
    "\n",
    "FINETUNED_OPENAI_DEPLOYMENT = os.getenv(\"FINETUNED_OPENAI_DEPLOYMENT\")\n",
    "FINETUNED_OPENAI_ENDPOINT = os.getenv(\"FINETUNED_OPENAI_ENDPOINT\")\n",
    "FINETUNED_OPENAI_KEY = os.getenv(\"FINETUNED_OPENAI_KEY\")\n",
    "\n",
    "baseline_client = AzureOpenAI(\n",
    "    azure_endpoint=BASELINE_OPENAI_ENDPOINT, \n",
    "    api_key=BASELINE_OPENAI_KEY,\n",
    "    api_version=\"2024-02-01\"\n",
    "    )\n",
    "\n",
    "finetuned_client = AzureOpenAI(\n",
    "    azure_endpoint=FINETUNED_OPENAI_ENDPOINT, \n",
    "    api_key=FINETUNED_OPENAI_KEY,\n",
    "    api_version=\"2024-02-01\"\n",
    "\n",
    "    )\n",
    "\n",
    "# get the predictions\n",
    "def get_model_completions(client, prompt, deployment):\n",
    "    \"\"\"\n",
    "    This function generates a model completion from a given prompt using the OpenAI API.\n",
    "\n",
    "    Parameters:\n",
    "    client (openai.Client): The AzureOpenAI client being used.\n",
    "    prompt (str): The prompt to be sent to the model for completion.\n",
    "    deployment (str): The identifier of the model deployment to be used for completion.\n",
    "\n",
    "    Returns:\n",
    "    str: The completed message content from the model. If an exception occurs during the process, it returns None and prints the exception.\n",
    "    \"\"\"\n",
    "\n",
    "\n",
    "    messages = [\n",
    "        {'role':'user','content':prompt}\n",
    "        ]\n",
    "    try:\n",
    "        response = client.chat.completions.create(\n",
    "        messages=messages,\n",
    "        model=deployment,\n",
    "        temperature=0.3,\n",
    "    )\n",
    "    \n",
    "        return response.choices[0].message.content\n",
    "\n",
    "    except Exception as e:\n",
    "        print(e)\n",
    "        return None\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "tqdm.pandas()\n",
    "\n",
    "test_df['baseline_model_response'] = test_df.progress_apply(lambda x: get_model_completions(baseline_client, x.instruction, BASELINE_OPENAI_DEPLOYMENT), axis=1)\n",
    "test_df['finetuned_model_response'] = test_df.progress_apply(lambda x: get_model_completions(finetuned_client, x.instruction, FINETUNED_OPENAI_DEPLOYMENT), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Clean up the model answers\n",
    "\n",
    "Because our fine tuned model has been trained with Chain of Thought answers, we need to clean up the answers to extract the final answer and match the format of the baseline model answers.\n",
    "\n",
    "Similarly, to run evaluation on RAG, we'll need to extract a clean context string with the content of the retrieved documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_final_answer(cot_answer: str) -> str:\n",
    "    \"\"\"\n",
    "    Extracts the final answer from the cot_answer field\n",
    "    \"\"\"\n",
    "    if cot_answer:\n",
    "        return cot_answer.split(\"<ANSWER>: \")[-1]\n",
    "    return None\n",
    "\n",
    "def extract_context(instruction: str) -> str:\n",
    "    \"\"\"\n",
    "    Extracts the context from the instruction field.\n",
    "    Keeps all <DOCUMENTS/> and removes the last line with the question.\n",
    "    \"\"\"\n",
    "    return \"\\n\".join(instruction.split(\"\\n\")[:-1])\n",
    "\n",
    "test_df['gold_final_answer'] = test_df.cot_answer.apply(extract_final_answer)\n",
    "test_df.rename(columns={'context':'context_docs'}, inplace=True)\n",
    "test_df['context'] = test_df.instruction.apply(extract_context)\n",
    "test_df['baseline_final_answer'] = test_df.baseline_model_response.apply(extract_final_answer)\n",
    "test_df['finetuned_final_answer'] = test_df.finetuned_model_response.apply(extract_final_answer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Define evaluation metrics\n",
    "\n",
    "We'll use RAGAS to evaluate the performance of the models on this RAG task. Ragas is a framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines. Ragas offers metrics tailored for evaluating each component of your RAG pipeline.\n",
    "\n",
    "For the scope of this workshop, we are only interested in evaluation the generation part of the pipeline.\n",
    "\n",
    "Ragas provides a few out of the box metrics we can compute, these metrics require either an LLM as a judge or an embedding model:\n",
    "- Answer relevancy: assesses how pertinent the generated answer is to the given prompt. A lower score is assigned to answers that are incomplete or contain redundant information and higher scores indicate better relevancy\n",
    "- Faithfulness: This measures the factual consistency of the generated answer against the given context\n",
    "- Answer similarity: semantic resemblance between the generated answer and the ground truth.\n",
    "- Answer correctness: Answer correctness encompasses two critical aspects: semantic similarity between the generated answer and the ground truth, as well as factual similarity. These aspects are combined using a weighted scheme to formulate the answer correctness score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.metrics import (\n",
    "    answer_relevancy,\n",
    "    faithfulness,\n",
    "    answer_similarity,\n",
    "    answer_correctness\n",
    ")\n",
    "from ragas.metrics.critique import harmfulness\n",
    "\n",
    "# list of metrics we're going to use\n",
    "metrics = [\n",
    "    faithfulness,\n",
    "    answer_relevancy,\n",
    "    answer_similarity,\n",
    "    answer_correctness\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai.chat_models import AzureChatOpenAI\n",
    "from langchain_openai.embeddings import AzureOpenAIEmbeddings\n",
    "from ragas import evaluate\n",
    "from dotenv import load_dotenv\n",
    "import os\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "judge_model_endpoint = os.getenv(\"JUDGE_OPENAI_ENDPOINT\")\n",
    "judge_model_api_key = os.getenv(\"JUDGE_OPENAI_API_KEY\")\n",
    "judge_model_deployment = os.getenv(\"JUDGE_OPENAI_DEPLOYMENT\")\n",
    "embedding_model_deployment= os.getenv(\"EMBEDDING_OPENAI_DEPLOYMENT\")\n",
    "\n",
    "azure_model = AzureChatOpenAI(\n",
    "    openai_api_version=\"2024-02-01\",\n",
    "    azure_endpoint=judge_model_endpoint,\n",
    "    azure_deployment=judge_model_deployment,\n",
    "    validate_base_url=False,\n",
    "    api_key=judge_model_api_key,\n",
    ")\n",
    "\n",
    "# init the embeddings for answer_relevancy, answer_correctness and answer_similarity\n",
    "azure_embeddings = AzureOpenAIEmbeddings(\n",
    "    openai_api_version=\"2024-02-01\",\n",
    "    azure_endpoint=judge_model_endpoint,\n",
    "    azure_deployment=embedding_model_deployment,\n",
    "    api_key=judge_model_api_key,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import Dataset\n",
    "\n",
    "baseline_df = test_df[['baseline_final_answer',\n",
    "                      'context',\n",
    "                      'gold_final_answer',\n",
    "                      'question']]\n",
    "\n",
    "baseline_df.rename(columns={'baseline_final_answer':'answer', \n",
    "                            'gold_final_answer':'ground_truth',\n",
    "                            'context':'contexts'}, inplace=True)\n",
    "#baseline_df['ground_truth'] = baseline_df['ground_truth'].apply(lambda x: [x] if x else [])\n",
    "baseline_df['contexts'] = baseline_df['contexts'].apply(lambda x: [x] if x else [])\n",
    "\n",
    "dataset = Dataset.from_pandas(baseline_df)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Computing the evaluation metrics for both models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_result = evaluate(\n",
    "    dataset, metrics=metrics, llm=azure_model, embeddings=azure_embeddings\n",
    ")\n",
    "\n",
    "baseline_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "finetuned_df = test_df[['finetuned_final_answer',\n",
    "                      'context',\n",
    "                      'gold_final_answer',\n",
    "                      'question']]\n",
    "\n",
    "finetuned_df.rename(columns={'finetuned_final_answer':'answer', \n",
    "                            'gold_final_answer':'ground_truth',\n",
    "                            'context':'contexts'}, inplace=True)\n",
    "#baseline_df['ground_truth'] = baseline_df['ground_truth'].apply(lambda x: [x] if x else [])\n",
    "finetuned_df['contexts'] = finetuned_df['contexts'].apply(lambda x: [x] if x else [])\n",
    "\n",
    "ft_dataset = Dataset.from_pandas(finetuned_df)\n",
    "\n",
    "ft_result = evaluate(\n",
    "    ft_dataset, metrics=metrics, llm=azure_model, embeddings=azure_embeddings\n",
    ")\n",
    "\n",
    "ft_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "baseline_dict = dict(baseline_result)\n",
    "ft_dict = dict(ft_result)\n",
    "\n",
    "ft_dict['model']=os.getenv(\"FINETUNED_OPENAI_DEPLOYMENT\")\n",
    "baseline_dict['model']=os.getenv(\"BASELINE_OPENAI_DEPLOYMENT\")\n",
    "\n",
    "results_df = pd.DataFrame([baseline_dict, ft_dict])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Plotting the side-by-side comparison of the models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Assuming you have your results_df DataFrame\n",
    "\n",
    "# Reshape the DataFrame\n",
    "melted_df = results_df.melt(id_vars='model', var_name='metric', value_name='value')\n",
    "melted_df['value'] = melted_df['value'].round(2)\n",
    "\n",
    "# Create the bar plot\n",
    "pivoted_data = melted_df.pivot_table(index='metric', columns='model', values='value')\n",
    "ax = pivoted_data.plot(kind='bar', figsize=(10, 6))\n",
    "\n",
    "# Add value labels on top of the bars\n",
    "for container in ax.containers:\n",
    "    ax.bar_label(container)\n",
    "\n",
    "plt.ylabel('Metric Value')\n",
    "plt.title('Model Comparison by Metric')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}