fine_tune/9_fine_tune_video_deduplication_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "Video Deduplication, also known as Video Copy Detection or Video Identification by Fingerprinting, means that given a query video, you need to find or retrieve the videos with the same content as query video.\n",
    "The ISC model has been proven to perform well in Video Deduplication [link](https://github.com/ant-research/VCSL). In this tutorial, we show how to fine-tune the ISC operator for better image embedding, which does great help to Video Deduplication in your custom data domain."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "### Requirements\n",
    "\n",
    "If you want to fine tune this operator, make sure your timm version is 0.4.12, and the higher version may cause model collapse during training."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "source": [
    "! python -m pip install tqdm augly timm==0.4.12 pytorch-metric-learning==0.9.99"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "### Download dataset\n",
    "ISC is trained using [contrastive learning](https://lilianweng.github.io/posts/2021-05-31-contrastive/), which is a type of self-supervised training. The training images do not require any labels. We only need to prepare a folder `./training_images`, under which a large number of diverse training images can be stored.\n",
    "\n",
    "In the original training of [ISC21-Descriptor-Track-1st](https://github.com/lyakaap/ISC21-Descriptor-Track-1st), the training dataset is a huge dataset which takes more than 165G space. And it uses [multi-steps training strategy](https://arxiv.org/abs/2104.00298).\n",
    "\n",
    "In our fine-tune example, to simplification, we prepare a small dataset to run, and you can replace it with your own custom dataset."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "! curl -L https://github.com/towhee-io/examples/releases/download/data/isc_training_image_examples.zip -o ./training_images.zip\n",
    "! unzip -q -o ./training_images.zip"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "### Get started to fine-tune\n",
    "Just call method op.train() and pass in your args."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "source": [
    "import towhee\n",
    "\n",
    "op = towhee.ops.image_embedding.isc().get_op()\n",
    "op.train(training_args={\n",
    "        'train_data_dir': './training_images',\n",
    "        'distributed': False,\n",
    "        'output_dir': './output',\n",
    "        'gpu': 0,\n",
    "        'epochs': 2,\n",
    "        'batch_size': 8,\n",
    "        'init_lr': 0.1\n",
    "    })"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "These are each arg infos in training_args:\n",
    "- output_dir\n",
    "  - default: './output'\n",
    "  - metadata_dict: {'help': 'output checkpoint saving dir.'}\n",
    "\n",
    "- distributed\n",
    "  - default: False\n",
    "  - metadata_dict: {'help': 'If true, use all gpu in your machine, else use only one gpu.'}\n",
    "\n",
    "- gpu\n",
    "  - default: 0\n",
    "  - metadata_dict: {'help': 'When distributed is False, use this gpu No. in your machine.'}\n",
    "\n",
    "- start_epoch\n",
    "  - default: 0\n",
    "  - metadata_dict: {'help': 'Start epoch number.'}\n",
    "\n",
    "- epochs\n",
    "  - default: 6\n",
    "  - metadata_dict: {'help': 'End epoch number.'}\n",
    "\n",
    "- batch_size\n",
    "  - default: 128\n",
    "  - metadata_dict: {'help': 'Total batch size in all gpu.'}\n",
    "\n",
    "- init_lr\n",
    "  - default: 0.1\n",
    "  - metadata_dict: {'help': 'init learning rate in SGD.'}\n",
    "\n",
    "- train_data_dir\n",
    "  - default: None\n",
    "  - metadata_dict: {'help': 'The dir containing all training data image files.'}\n"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "### Load trained model"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "source": [
    "new_op = towhee.ops.image_embedding.isc(checkpoint_path='./output/checkpoint_epoch0001.pth.tar').get_op()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "### Your custom training\n",
    "Your can change [training script](https://towhee.io/image-embedding/isc/src/branch/main/train_isc.py) in your custom way.\n",
    "Or your can refer to the [original repo](https://github.com/lyakaap/ISC21-Descriptor-Track-1st) and [paper](https://arxiv.org/abs/2112.04323) to learn more about contrastive learning and image instance retrieval."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}