2_raft_finetuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. RAFT GPT-4o-mini fine tuning\n",
    "\n",
    "Now that we've created labelled training data, we can fine tune our model using the Supervised Fine Tuning technique. Azure OpenAI uses LoRA to fine tune models efficiently. **LoRA (Low-Rank Adaptation)** finetuning of a Large Language Model is a technique used to adapt pre-trained language models to specific tasks efficiently and with fewer computational resources.  \n",
    "\n",
    " Instead of adjusting all the model parameters, LoRA introduces a small number of additional parameters (low-rank matrices) that modify the model's behavior. These new parameters are trained while keeping the original model's parameters mostly unchanged. This way, the model can learn the new task without the need for extensive computational resources or time.\n",
    "\n",
    " Azure OpenAI lets developers customize OpenAI models with their own data and easily deploy their custom model using an easy to use and affordable managed service.\n",
    "\n",
    " While Fine Tuning can be a complex process, Azure OpenAI abstracts away a lot of the complexity to make fine tuning accessible to any developer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0. How much will this cost?\n",
    "\n",
    "Fine tuning pricing on Azure OpenAI makes fine tuning experiment cost very predictable. Training pricing is based on the number of tokens you're training your model on. Therefore, it is very easy to predict and manage the cost of your finetuning experiments.\n",
    "\n",
    "For GPT-4o mini, training price is $0.003300 per 1K tokens.\n",
    "\n",
    "So to estimate the cost of our fine tuning job we can use the following formula\n",
    "\n",
    "`(Training cost per 1K input tokens / 1K) * number of tokens in input file * number of epochs trained`\n",
    "\n",
    "**epoch:** a complete iteration through a dataset during the training process of a process, \n",
    "\n",
    "1. If the number of epochs is too low: Your model might be underfitted, which means it could perform poorly because it hasn't learned enough from the training data. In essence, it may not have had enough iterations to effectively learn and adjust its parameters (e.g., weights and biases).\n",
    "\n",
    "2. If the number of epochs is too high: There's a risk of overfitting, where the model becomes too specialized in the training data and performs poorly on unseen data (examples that weren’t in your training dataset).\n",
    "\n",
    "the number of epochs is a parameter of the fine tuning job, usually 3 epochs is a reasonable number\n",
    "\n",
    "**Let's explore our dataset and estimate our fine tuning costs.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv\n",
    "import os\n",
    "import tiktoken\n",
    "import json\n",
    "from openai import AzureOpenAI\n",
    "\n",
    "# loading environment variables\n",
    "load_dotenv()\n",
    "aoai_endpoint = os.getenv(\"AOAI_FINETUNING_ENDPOINT\")\n",
    "aoai_api_key = os.getenv(\"AOAI_FINETUNING_API_KEY\")\n",
    "\n",
    "training_file_path = './data/training_data/banking_train.jsonl'\n",
    "encoding = tiktoken.encoding_for_model(\"gpt-4o-mini\")\n",
    "\n",
    "def num_tokens_from_messages(messages, tokens_per_message=3, tokens_per_name=1):\n",
    "    num_tokens = 0\n",
    "    for message in messages:\n",
    "        num_tokens += tokens_per_message\n",
    "        for key, value in message.items():\n",
    "            num_tokens += len(encoding.encode(value))\n",
    "            if key == \"role\":\n",
    "                num_tokens += tokens_per_name\n",
    "    num_tokens += 3\n",
    "    return num_tokens\n",
    "\n",
    "with open(training_file_path, 'r', encoding='utf-8') as f:\n",
    "    num_tokens=0\n",
    "    dataset = [json.loads(line) for line in f]\n",
    "    messages = [ d.get('messages') for d in dataset]\n",
    "    for message in messages:\n",
    "        \n",
    "        num_tokens += num_tokens_from_messages(message)\n",
    "    \n",
    "print(f\"Number of tokens in training data: {num_tokens}\")\n",
    "\n",
    "training_cost_per_token = 0.003300 / 1000\n",
    "num_epochs = 3\n",
    "total_cost = num_tokens * training_cost_per_token * num_epochs\n",
    "\n",
    "print(f\"Total estimated cost for training: {total_cost:.2f} USD\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Uploading the training and validation data to Azure OpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "client = AzureOpenAI(\n",
    "  azure_endpoint = aoai_endpoint,\n",
    "  api_key = aoai_api_key,\n",
    "  api_version = \"2024-05-01-preview\"  # This API version or later is required to access seed/events/checkpoint features\n",
    ")\n",
    "\n",
    "validation_file_path = './data/training_data/banking_validation.jsonl'\n",
    "\n",
    "# Upload the training and validation dataset files to Azure OpenAI with the SDK.\n",
    "training_response = client.files.create(\n",
    "    file = open(training_file_path, \"rb\"), purpose=\"fine-tune\"\n",
    ")\n",
    "training_file_id = training_response.id\n",
    "\n",
    "validation_response = client.files.create(\n",
    "    file = open(validation_file_path, \"rb\"), purpose=\"fine-tune\"\n",
    ")\n",
    "validation_file_id = validation_response.id\n",
    "\n",
    "print(\"Training file ID:\", training_file_id)\n",
    "print(\"Validation file ID:\", validation_file_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Creating the fine tuning job\n",
    "\n",
    "For each fine tuning job, you can specify the following hyperparameters. \n",
    "\n",
    "- epochs: An \"epoch\" is a term used to describe one complete pass through the entire training dataset\n",
    "- learning rate multiplier: this will be used as the learning rate for the fine tuning job, as a multiple of the model's original learning rate. We recommend experimenting with values in the range 0.02 to 0.2 to see what produces the best results\n",
    "- batch size:  how many training examples you use at one time during training, common choices are (32, 64, 128, 256). This value is to be tuned based on the size of your data and available compute.\n",
    "\n",
    "The general recommendation is to initially train without specifying any of these, Azure OpenAI will pick a default for you based on dataset size, then adjusting based on results to find the ideal combination"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Submit fine-tuning training job\n",
    "\n",
    "response = client.fine_tuning.jobs.create(\n",
    "    training_file = training_file_id,\n",
    "    validation_file = validation_file_id,\n",
    "    model = \"gpt-4o-mini\", # Enter base model name. Note that in Azure OpenAI the model name contains dashes and cannot contain dot/period characters.\n",
    "    seed = 105 # seed parameter controls reproducibility of the fine-tuning job. If no seed is specified one will be generated automatically.\n",
    ")\n",
    "\n",
    "job_id = response.id\n",
    "\n",
    "# You can use the job ID to monitor the status of the fine-tuning job.\n",
    "# The fine-tuning job will take some time to start and complete.\n",
    "\n",
    "print(\"Job ID:\", response.id)\n",
    "print(\"Status:\", response.status)\n",
    "print(response.model_dump_json(indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Monitor the fine tuning job\n",
    "\n",
    "The training time depends on the number of tokens and number of epochs, typically you can expect a job this size to run for a little over an hour and a half. We have already fine-tuned and deployed a model so you can use it directly without waiting for your fine tuning job to complete.\n",
    "\n",
    "[Fine tuned model](https://oai.azure.com/resource/finetune/ftjob-6d1293138cd844e7bab02a141a60c697/details?wsid=/subscriptions/3c791225-4905-4a40-860b-0a0c9cd2af91/resourceGroups/RG-FineTuning-AIGBBWorkshop/providers/Microsoft.CognitiveServices/accounts/aoai-raft-gbb-workshop&tid=604b58b3-fa4e-4a57-b566-cac3f88a3ae8)\n",
    "\n",
    "[Fine tuned model deployment](https://oai.azure.com/resource/deployments/%2Fsubscriptions%2F3c791225-4905-4a40-860b-0a0c9cd2af91%2FresourceGroups%2FRG-FineTuning-AIGBBWorkshop%2Fproviders%2FMicrosoft.CognitiveServices%2Faccounts%2Faoai-raft-gbb-workshop%2Fdeployments%2Fgpt-4o-mini-ft-raft-banking?wsid=/subscriptions/3c791225-4905-4a40-860b-0a0c9cd2af91/resourceGroups/RG-FineTuning-AIGBBWorkshop/providers/Microsoft.CognitiveServices/accounts/aoai-raft-gbb-workshop&tid=604b58b3-fa4e-4a57-b566-cac3f88a3ae8)\n",
    "\n",
    "You can monitor your fine tuning job from this notebook or in the Azure OpenAI's new studio.\n",
    "\n",
    "Go to Tools > Fine-tuning > Click on your job \n",
    "\n",
    "![alt text](./static/ft_monitor.png \"Azure OpenAI Studio Fine tuning job\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**We can also monitor the job from this notebook**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "import time\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "# Get the status of our fine-tuning job.\n",
    "response = client.fine_tuning.jobs.retrieve(job_id)\n",
    "\n",
    "status = response.status\n",
    "\n",
    "# If the job isn't done yet, poll it every 10 seconds.\n",
    "while status not in [\"succeeded\", \"failed\"]:\n",
    "    time.sleep(10)\n",
    "\n",
    "    response = client.fine_tuning.jobs.retrieve(job_id)\n",
    "    print(response.model_dump_json(indent=2))\n",
    "    print(\"Elapsed time: {} minutes {} seconds\".format(int((time.time() - start_time) // 60), int((time.time() - start_time) % 60)))\n",
    "    status = response.status\n",
    "    print(f'Status: {status}')\n",
    "    clear_output(wait=True)\n",
    "\n",
    "print(f'Fine-tuning job {job_id} finished with status: {status}')\n",
    "\n",
    "# List all fine-tuning jobs for this resource.\n",
    "print('Checking other fine-tune jobs for this resource.')\n",
    "response = client.fine_tuning.jobs.list()\n",
    "print(f'Found {len(response.data)} fine-tune jobs.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve fine_tuned_model name\n",
    "response = client.fine_tuning.jobs.retrieve(job_id)\n",
    "fine_tuned_model = response.fine_tuned_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Analyze the fine tuned model in Azure OpenAI Studio\n",
    "\n",
    "Head here for a fine tuned model in the shared AI GBB tenant:\n",
    "[Fine tuning job](https://oai.azure.com/resource/finetune/ftjob-6d1293138cd844e7bab02a141a60c697/details?wsid=/subscriptions/3c791225-4905-4a40-860b-0a0c9cd2af91/resourceGroups/RG-FineTuning-AIGBBWorkshop/providers/Microsoft.CognitiveServices/accounts/aoai-raft-gbb-workshop&tid=604b58b3-fa4e-4a57-b566-cac3f88a3ae8)\n",
    "\n",
    "##### 4.a Training plots\n",
    "\n",
    "When the model is done training, head to your Azure OpenAI Studio to analyze your model training metrics.\n",
    "\n",
    "Two charts are available to analyze your fine tuning job and sanity check that the training went smoothly:\n",
    "- Loss curve: Value of the loss function (how wrong the model is) over time during training process --> this curve should go down over time as the model weights converge towards the optimum. \n",
    "- Token Accuracy: Shows the accuracy of the model's predictions at the token level (e.g., words or subwords) over time during training. A higher token accuracy suggests that the model is better able to capture the nuances of the language and generate more accurate text.\n",
    "\n",
    "Each of these charts has the metrics computed both on the training data and on the validation set. \n",
    "\n",
    "To analyze these plots, one should look for the following:\n",
    "\n",
    "- A smooth curve: A smooth curve indicates that the model is learning consistently. Sharp changes or spikes in the curve could indicate issues with the learning rate or data preprocessing.\n",
    "- Plateau: A plateau in the curve indicates that the model has stopped improving and further training may not be necessary.\n",
    "- Overfitting: If the training loss continues to decrease but the validation loss starts to increase, it could be a sign of overfitting. This means that the model is not generalizing well to new data and may perform poorly on unseen data.\n",
    "- Underfitting: If both the training and validation loss remain high, it could be a sign of underfitting. This means that the model is not learning the patterns in the data well enough and may need a more complex - architecture or more training data.\n",
    "- Optimal stopping point: By analyzing the loss curve and token accuracy plot, one can determine the optimal stopping point for training, where the model has reached its best performance without overfitting.\n",
    "\n",
    "Now head to the studio and ensure your curves look roughly like the below\n",
    "\n",
    "\n",
    "\n",
    "![Alt text](./static/ft_metrics.png \"AOAI training plots\")\n",
    "\n",
    "##### 4.b Model Checkpoints\n",
    "\n",
    "In the Studio, go to the checkpoints tab, you'll see a model checkpoint corresponding to each completed epoch. A checkpoint is a fully functional version of a model which can both be deployed and used as the target model for subsequent fine-tuning jobs. Checkpoints can be particularly useful, as they can provide a snapshot of your model prior to overfitting having occurred. \n",
    "\n",
    "![Alt text](./static/ft_checkpoints.png \"AOAI training plots\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Create a new deployment with the fine tuned model\n",
    "\n",
    "When the fine-tuning job succeeds, the value of the fine_tuned_model variable in the response body is set to the name of your customized model. Your model is now also available for discovery from the list Models API. However, you can't issue completion calls to your customized model until your customized model is deployed. You must deploy your customized model to make it available for use with completion calls\n",
    "\n",
    "#### 5.a From the notebook\n",
    "To create a new deployment from a notebook, you'll need an access token from Azure, \n",
    "Open a terminal and run:\n",
    "\n",
    "`az login`\n",
    "\n",
    "`az account get-access-token`\n",
    "\n",
    "paste the token in the next cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deploy fine-tuned model\n",
    "import requests\n",
    "\n",
    "token = \"YOUR TOKEN\"\n",
    "subscription = os.getenv(\"AZURE_SUBSCRIPTION_ID\")\n",
    "resource_group = os.getenv(\"AZURE_RESOURCE_GROUP_NAME\")\n",
    "resource_name = aoai_endpoint.split(\"https://\")[1].split(\".\")[0]\n",
    "model_deployment_name = \"gpt-4o-mini-ft-raft-banking\"\n",
    "\n",
    "deploy_params = {'api-version': \"2023-05-01\"}\n",
    "deploy_headers = {'Authorization': 'Bearer {}'.format(token), 'Content-Type': 'application/json'}\n",
    "\n",
    "deploy_data = {\n",
    "    \"sku\": {\"name\": \"standard\", \"capacity\": 50},\n",
    "    \"properties\": {\n",
    "        \"model\": {\n",
    "            \"format\": \"OpenAI\",\n",
    "            \"name\": fine_tuned_model, #retrieve this value from the previous call, it will look like gpt-35-turbo-0613.ft-b044a9d3cf9c4228b5d393567f693b83\n",
    "            \"version\": \"1\"\n",
    "        }\n",
    "    }\n",
    "}\n",
    "deploy_data = json.dumps(deploy_data)\n",
    "\n",
    "request_url = f'https://management.azure.com/subscriptions/{subscription}/resourceGroups/{resource_group}/providers/Microsoft.CognitiveServices/accounts/{resource_name}/deployments/{model_deployment_name}'\n",
    "\n",
    "print('Creating a new deployment...')\n",
    "\n",
    "r = requests.put(request_url, params=deploy_params, headers=deploy_headers, data=deploy_data)\n",
    "\n",
    "print(r)\n",
    "print(r.reason)\n",
    "print(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.b From the studio\n",
    "\n",
    "On the fine tuning job page, click 'Deploy'\n",
    "\n",
    "![alt text](./static/ft_deploy.png \"AOAI Deploy model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}