examples/int4_finetuning/LLaMA_lora_int4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# [xTuring](https://github.com/stochasticai/xturing) - LLaMA INT4 efficient fine-tuning tutorial\n",
    "\n",
    "This tutorial aims to show how easy it is to perform fine-tuning with xTuring. This notebook shows how to fine-tune LLaMA 7B model on GPU which has limited memory, it requires only 6GB VRAM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 1. Install the `xTuring` library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "!pip install xturing --upgrade\n",
    "!pip install xturing[int4] --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 2. Load model and dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "from xturing.datasets.instruction_dataset import InstructionDataset\n",
    "from xturing.models import BaseModel\n",
    "from pytorch_lightning.loggers import WandbLogger\n",
    "\n",
    "# Initializes WandB integration \n",
    "wandb_logger = WandbLogger()\n",
    "\n",
    "instruction_dataset = InstructionDataset(\"../llama/alpaca_data\")\n",
    "# Initializes the model\n",
    "model = BaseModel.create(\"llama_lora_int4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 3. Start the finetuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Finetuned the model\n",
    "model.finetune(dataset=instruction_dataset, logger=wandb_logger)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 4. Generate an output text with the fine-tuned model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Once the model has been finetuned, you can start doing inferences\n",
    "output = model.generate(texts=[\"Why LLM models are becoming so important?\"])\n",
    "print(\"Generated output by the model: {}\".format(output))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}