main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\Huge G \\cup n \\; \\sqrt{-1} \\; \\ell \\; e \\; \\emptyset\n",
    "$$\n",
    "\n",
    "<p style=\"text-align: center\">A lipsync project, made by Nil Atabey, Leonardo Biason and Günak Yüzak</p>\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 align=\"center\"><b>Table of Contents</b></h2>\n",
    "\n",
    "1. [Code structure](#1-code-structure)\n",
    "2. [Import of the Packages](#2-import-of-the-packages)\n",
    "3. [Data Loading](#3-data-loading)\n",
    "4. [Model Settings](#4-model-settings)\n",
    "5. [Training + Testing](#5-training--testing)\n",
    "\n",
    "$$\n",
    "\\newcommand{\\goto}{\\; \\longrightarrow \\;}\n",
    "\\newcommand{\\tdconv}{\\text{2D Convolution} }\n",
    "\\newcommand{\\relu}{\\text{ReLU} }\n",
    "$$\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Code Structure\n",
    "\n",
    "The code structure is the following:\n",
    "\n",
    "```python\n",
    "project\n",
    " ├ assets\n",
    " │  ├ cnn.py\n",
    " │  └ dataloader.py\n",
    " └ data\n",
    "```\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Import of the Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Standard packages needed that can be installed with either `conda` or `pip`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pytorch imports\n",
    "import torch\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import datasets\n",
    "from torchvision.transforms import ToTensor\n",
    "from torch import nn\n",
    "import torchmetrics\n",
    "import torchinfo\n",
    "\n",
    "# Utils imports\n",
    "import numpy as np\n",
    "import os\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Custom imports from our libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from assets.gnldataloader import GNLDataLoader\n",
    "from assets.cnn import LabialCNN\n",
    "from assets.loops import train_loop, test_loop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Data Loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to clean the code and make sure that there were the same number of videos and labels, the following code was performed:\n",
    "\n",
    "```python\n",
    "a, b = sorted(os.listdir(\"data/matching/fronts\")), sorted(os.listdir(\"data/matching/labels\"))\n",
    "a_new, b_new = set([item[:-4] for item in a]), set([item[:-5] for item in b])\n",
    "tot = a_new.intersection(b_new)\n",
    "print(tot)\n",
    "\n",
    "for item in b:\n",
    "    if item[:-5] in tot:\n",
    "        prev_path = os.path.join(path_labels, item)\n",
    "        os.rename(prev_path, os.path.join(\"data/matching/labels\", item))\n",
    "\n",
    "print(len(tot))\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[DEBUG] The data dir has been recognized\n",
      "[DEBUG] The label dir has been recognized\n",
      "[DEBUG] Items in the data folder: 5129\n",
      "[DEBUG] Items in the labels folder: 5129\n",
      "[DEBUG] Index of the dataloader: slice(0, 128, None)\n",
      "[DEBUG] Data folder: ['s10_l_bbat9p.mov', 's10_l_bbay5n.mov', 's10_l_bbbg8s.mov', 's10_l_bbbg9p.mov', 's10_l_bgin7n.mov', 's10_l_braczp.mov', 's10_l_brin2n.mov', 's10_l_bwbc1s.mov', 's10_l_bwbi7s.mov', 's10_l_bwbp9a.mov', 's10_l_bwiu3s.mov', 's10_l_bwwoza.mov', 's10_l_lbbt4a.mov', 's10_l_lgif6n.mov', 's10_l_lgwp6p.mov', 's10_l_lgwz8a.mov', 's10_l_lral6p.mov', 's10_l_lwix7n.mov', 's10_l_pbbj5n.mov', 's10_l_pbbq3s.mov', 's10_l_pbwf9p.mov', 's10_l_pbwm2p.mov', 's10_l_pbwq2a.mov', 's10_l_pbwz2n.mov', 's10_l_pgae7s.mov', 's10_l_pgba6s.mov', 's10_l_pgil3n.mov', 's10_l_pgir2a.mov', 's10_l_pgwe9p.mov', 's10_l_prax4n.mov', 's10_l_prayzp.mov', 's10_l_prws1p.mov', 's10_l_pwwh1p.mov', 's10_l_pwws8n.mov', 's10_l_sbbh6a.mov', 's10_l_sbib5a.mov', 's10_l_sgad4s.mov', 's10_l_sgbj5p.mov', 's10_l_sgiv8p.mov', 's10_l_srai8s.mov', 's10_l_sravzp.mov', 's10_l_srid1a.mov', 's10_l_sriu1p.mov', 's10_l_swar4p.mov', 's10_l_swib5n.mov', 's10_p_bbat9p.mov', 's10_p_bbay5n.mov', 's10_p_bbbg8s.mov', 's10_p_bbbg9p.mov', 's10_p_bgin7n.mov', 's10_p_braczp.mov', 's10_p_brin2n.mov', 's10_p_bwbc1s.mov', 's10_p_bwbi7s.mov', 's10_p_bwbp9a.mov', 's10_p_bwiu3s.mov', 's10_p_bwwoza.mov', 's10_p_lbbt4a.mov', 's10_p_lgif6n.mov', 's10_p_lgwp6p.mov', 's10_p_lgwz8a.mov', 's10_p_lral6p.mov', 's10_p_lwaa3p.mov', 's10_p_lwaozp.mov', 's10_p_lwix7n.mov', 's10_p_pbbj5n.mov', 's10_p_pbbq3s.mov', 's10_p_pbwf9p.mov', 's10_p_pbwm2p.mov', 's10_p_pbwq2a.mov', 's10_p_pbwz2n.mov', 's10_p_pgae7s.mov', 's10_p_pgba6s.mov', 's10_p_pgil3n.mov', 's10_p_pgir2a.mov', 's10_p_pgwe9p.mov', 's10_p_prax4n.mov', 's10_p_prayzp.mov', 's10_p_prbm4p.mov', 's10_p_prws1p.mov', 's10_p_pwwh1p.mov', 's10_p_pwwk7n.mov', 's10_p_pwws8n.mov', 's10_p_sbbh6a.mov', 's10_p_sbib5a.mov', 's10_p_sgad4s.mov', 's10_p_sgbj5p.mov', 's10_p_sgiv8p.mov', 's10_p_srai8s.mov', 's10_p_sravzp.mov', 's10_p_srbk3s.mov', 's10_p_srid1a.mov', 's10_p_sriu1p.mov', 's10_p_swar4p.mov', 's10_p_swib5n.mov', 's11_l_bbab2n.mov', 's11_l_bbbg8p.mov', 's11_l_bbin9p.mov', 's11_l_bbwi4a.mov', 's11_l_bgia7s.mov', 's11_l_bgic3n.mov', 's11_l_bgiu3s.mov', 's11_l_brwv1s.mov', 's11_l_lbbk5n.mov', 's11_l_lbirzp.mov', 's11_l_lbiy7n.mov', 's11_l_lgia1n.mov', 's11_l_lrby1p.mov', 's11_l_lrbz8p.mov', 's11_l_lrwg9p.mov', 's11_l_lwbb8s.mov', 's11_l_lwbhzs.mov', 's11_l_lwbtzn.mov', 's11_l_lwbz6n.mov', 's11_l_lwis2s.mov', 's11_l_lwwv2n.mov', 's11_l_pbir4a.mov', 's11_l_pgbl7p.mov', 's11_l_pgbm6s.mov', 's11_l_pgbm8n.mov', 's11_l_prae2s.mov', 's11_l_prae3n.mov', 's11_l_prbl5n.mov', 's11_l_prid6a.mov', 's11_l_prii7n.mov', 's11_l_prwtzn.mov', 's11_l_pwwf9s.mov', 's11_l_sbbo4s.mov']\n",
      "[DEBUG] Labels folder: ['s10_l_bbat9p.json', 's10_l_bbay5n.json', 's10_l_bbbg8s.json', 's10_l_bbbg9p.json', 's10_l_bgin7n.json', 's10_l_braczp.json', 's10_l_brin2n.json', 's10_l_bwbc1s.json', 's10_l_bwbi7s.json', 's10_l_bwbp9a.json', 's10_l_bwiu3s.json', 's10_l_bwwoza.json', 's10_l_lbbt4a.json', 's10_l_lgif6n.json', 's10_l_lgwp6p.json', 's10_l_lgwz8a.json', 's10_l_lral6p.json', 's10_l_lwix7n.json', 's10_l_pbbj5n.json', 's10_l_pbbq3s.json', 's10_l_pbwf9p.json', 's10_l_pbwm2p.json', 's10_l_pbwq2a.json', 's10_l_pbwz2n.json', 's10_l_pgae7s.json', 's10_l_pgba6s.json', 's10_l_pgil3n.json', 's10_l_pgir2a.json', 's10_l_pgwe9p.json', 's10_l_prax4n.json', 's10_l_prayzp.json', 's10_l_prws1p.json', 's10_l_pwwh1p.json', 's10_l_pwws8n.json', 's10_l_sbbh6a.json', 's10_l_sbib5a.json', 's10_l_sgad4s.json', 's10_l_sgbj5p.json', 's10_l_sgiv8p.json', 's10_l_srai8s.json', 's10_l_sravzp.json', 's10_l_srid1a.json', 's10_l_sriu1p.json', 's10_l_swar4p.json', 's10_l_swib5n.json', 's10_p_bbat9p.json', 's10_p_bbay5n.json', 's10_p_bbbg8s.json', 's10_p_bbbg9p.json', 's10_p_bgin7n.json', 's10_p_braczp.json', 's10_p_brin2n.json', 's10_p_bwbc1s.json', 's10_p_bwbi7s.json', 's10_p_bwbp9a.json', 's10_p_bwiu3s.json', 's10_p_bwwoza.json', 's10_p_lbbt4a.json', 's10_p_lgif6n.json', 's10_p_lgwp6p.json', 's10_p_lgwz8a.json', 's10_p_lral6p.json', 's10_p_lwaa3p.json', 's10_p_lwaozp.json', 's10_p_lwix7n.json', 's10_p_pbbj5n.json', 's10_p_pbbq3s.json', 's10_p_pbwf9p.json', 's10_p_pbwm2p.json', 's10_p_pbwq2a.json', 's10_p_pbwz2n.json', 's10_p_pgae7s.json', 's10_p_pgba6s.json', 's10_p_pgil3n.json', 's10_p_pgir2a.json', 's10_p_pgwe9p.json', 's10_p_prax4n.json', 's10_p_prayzp.json', 's10_p_prbm4p.json', 's10_p_prws1p.json', 's10_p_pwwh1p.json', 's10_p_pwwk7n.json', 's10_p_pwws8n.json', 's10_p_sbbh6a.json', 's10_p_sbib5a.json', 's10_p_sgad4s.json', 's10_p_sgbj5p.json', 's10_p_sgiv8p.json', 's10_p_srai8s.json', 's10_p_sravzp.json', 's10_p_srbk3s.json', 's10_p_srid1a.json', 's10_p_sriu1p.json', 's10_p_swar4p.json', 's10_p_swib5n.json', 's11_l_bbab2n.json', 's11_l_bbbg8p.json', 's11_l_bbin9p.json', 's11_l_bbwi4a.json', 's11_l_bgia7s.json', 's11_l_bgic3n.json', 's11_l_bgiu3s.json', 's11_l_brwv1s.json', 's11_l_lbbk5n.json', 's11_l_lbirzp.json', 's11_l_lbiy7n.json', 's11_l_lgia1n.json', 's11_l_lrby1p.json', 's11_l_lrbz8p.json', 's11_l_lrwg9p.json', 's11_l_lwbb8s.json', 's11_l_lwbhzs.json', 's11_l_lwbtzn.json', 's11_l_lwbz6n.json', 's11_l_lwis2s.json', 's11_l_lwwv2n.json', 's11_l_pbir4a.json', 's11_l_pgbl7p.json', 's11_l_pgbm6s.json', 's11_l_pgbm8n.json', 's11_l_prae2s.json', 's11_l_prae3n.json', 's11_l_prbl5n.json', 's11_l_prid6a.json', 's11_l_prii7n.json', 's11_l_prwtzn.json', 's11_l_pwwf9s.json', 's11_l_sbbo4s.json']\n",
      "[DEBUG] Trying to open the video at path data/matching/fronts\\s10_l_bbat9p.mov\n",
      "[DEBUG] Video data/matching/fronts\\s10_l_bbat9p.mov opened\n",
      "[DEBUG] Shape of video: (75, 100, 150)\n",
      "[DEBUG] Label: tensor([37.,  2.,  9., 14., 37.,  2., 12., 21.,  5., 37.,  1., 20., 37., 20.,\n",
      "        37., 14.,  9., 14.,  5., 37., 16., 12.,  5.,  1., 19.,  5., 37., 37.,\n",
      "        37., 37., 37., 37., 37., 37., 37., 37., 37.])\n",
      "[DEBUG] Sentence: [' ', 'b', 'i', 'n', ' ', 'b', 'l', 'u', 'e', ' ', 'a', 't', ' ', 't', ' ', 'n', 'i', 'n', 'e', ' ', 'p', 'l', 'e', 'a', 's', 'e', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ']\n",
      "[DEBUG] Length: 37\n",
      "\n",
      "[DEBUG] Trying to open the video at path data/matching/fronts\\s10_l_bbay5n.mov\n",
      "[DEBUG] Video data/matching/fronts\\s10_l_bbay5n.mov opened\n",
      "[DEBUG] Shape of video: (75, 100, 150)\n",
      "[DEBUG] Label: tensor([37.,  2.,  9., 14., 37.,  2., 12., 21.,  5., 37.,  1., 20., 37., 25.,\n",
      "        37.,  6.,  9., 22.,  5., 37., 14., 15., 23., 37., 37., 37., 37., 37.,\n",
      "        37., 37., 37., 37., 37., 37., 37., 37., 37.])\n",
      "[DEBUG] Sentence: [' ', 'b', 'i', 'n', ' ', 'b', 'l', 'u', 'e', ' ', 'a', 't', ' ', 'y', ' ', 'f', 'i', 'v', 'e', ' ', 'n', 'o', 'w', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ']\n",
      "[DEBUG] Length: 37\n",
      "\n",
      "[DEBUG] Trying to open the video at path data/matching/fronts\\s10_l_bbbg8s.mov\n",
      "[DEBUG] Video data/matching/fronts\\s10_l_bbbg8s.mov opened\n",
      "[DEBUG] Shape of video: (75, 100, 150)\n",
      "[DEBUG] Label: tensor([37.,  2.,  9., 14., 37.,  2., 12., 21.,  5., 37.,  2., 25., 37.,  7.,\n",
      "        37.,  5.,  9.,  7.,  8., 20., 37., 19., 15., 15., 14., 37., 37., 37.,\n",
      "        37., 37., 37., 37., 37., 37., 37., 37., 37.])\n",
      "[DEBUG] Sentence: [' ', 'b', 'i', 'n', ' ', 'b', 'l', 'u', 'e', ' ', 'b', 'y', ' ', 'g', ' ', 'e', 'i', 'g', 'h', 't', ' ', 's', 'o', 'o', 'n', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ']\n",
      "[DEBUG] Length: 37\n",
      "\n",
      "[DEBUG] Trying to open the video at path data/matching/fronts\\s10_l_bbbg9p.mov\n",
      "[DEBUG] Video data/matching/fronts\\s10_l_bbbg9p.mov opened\n",
      "[DEBUG] Shape of video: (75, 100, 150)\n",
      "[DEBUG] Label: tensor([37.,  2.,  9., 14., 37.,  2., 12., 21.,  5., 37.,  2., 25., 37.,  7.,\n",
      "        37., 14.,  9., 14.,  5., 37., 16., 12.,  5.,  1., 19.,  5., 37., 37.,\n",
      "        37., 37., 37., 37., 37., 37., 37., 37., 37.])\n",
      "[DEBUG] Sentence: [' ', 'b', 'i', 'n', ' ', 'b', 'l', 'u', 'e', ' ', 'b', 'y', ' ', 'g', ' ', 'n', 'i', 'n', 'e', ' ', 'p', 'l', 'e', 'a', 's', 'e', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ']\n",
      "[DEBUG] Length: 37\n",
      "\n",
      "[DEBUG] Trying to open the video at path data/matching/fronts\\s10_l_bgin7n.mov\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[7], line 16\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[38;5;28mprint\u001b[39m(\n\u001b[0;32m      9\u001b[0m     \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m[DEBUG] Items in the data folder: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mlen\u001b[39m(\u001b[38;5;28msorted\u001b[39m(os\u001b[38;5;241m.\u001b[39mlistdir(path_data)))\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m,\n\u001b[0;32m     10\u001b[0m     \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m[DEBUG] Items in the labels folder: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mlen\u001b[39m(\u001b[38;5;28msorted\u001b[39m(os\u001b[38;5;241m.\u001b[39mlistdir(path_labels)))\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m,\n\u001b[0;32m     11\u001b[0m     sep\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m     12\u001b[0m )\n\u001b[0;32m     13\u001b[0m \u001b[38;5;66;03m#print(dataset[0][1][0])\u001b[39;00m\n\u001b[0;32m     14\u001b[0m \u001b[38;5;66;03m# print(dataset[0][1][0].shape)\u001b[39;00m\n\u001b[1;32m---> 16\u001b[0m dset_train \u001b[38;5;241m=\u001b[39m \u001b[43mdataset\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m:\u001b[49m\u001b[38;5;241;43m128\u001b[39;49m\u001b[43m]\u001b[49m\n\u001b[0;32m     18\u001b[0m dataloader_train \u001b[38;5;241m=\u001b[39m DataLoader(dset_train, batch_size\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m32\u001b[39m, shuffle\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m, num_workers\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m128\u001b[39m)\n\u001b[0;32m     19\u001b[0m dataloader_test \u001b[38;5;241m=\u001b[39m DataLoader(dataset[\u001b[38;5;241m128\u001b[39m:\u001b[38;5;241m192\u001b[39m], batch_size\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m32\u001b[39m, shuffle\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m, num_workers\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m128\u001b[39m)\n",
      "File \u001b[1;32mc:\\Users\\Leonardo\\Documents\\University\\AILabProject\\assets\\gnldataloader.py:73\u001b[0m, in \u001b[0;36mGNLDataLoader.__getitem__\u001b[1;34m(self, index, straight)\u001b[0m\n\u001b[0;32m     70\u001b[0m to_return \u001b[38;5;241m=\u001b[39m []\n\u001b[0;32m     72\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m ind, item \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28menumerate\u001b[39m(datas):\n\u001b[1;32m---> 73\u001b[0m     to_return\u001b[38;5;241m.\u001b[39mappend((\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__load_video__\u001b[49m\u001b[43m(\u001b[49m\u001b[43mitem\u001b[49m\u001b[43m)\u001b[49m, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m__load_label__(labels[ind])))\n\u001b[0;32m     76\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m'''return (\u001b[39;00m\n\u001b[0;32m     77\u001b[0m \u001b[38;5;124;03m    [self.__load_video__(data_piece) for data_piece in datas],\u001b[39;00m\n\u001b[0;32m     78\u001b[0m \u001b[38;5;124;03m    [self.__load_label__(label_piece) for label_piece in labels]\u001b[39;00m\n\u001b[0;32m     79\u001b[0m \u001b[38;5;124;03m)'''\u001b[39;00m\n\u001b[0;32m     81\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mtuple\u001b[39m(to_return)\n",
      "File \u001b[1;32mc:\\Users\\Leonardo\\Documents\\University\\AILabProject\\assets\\gnldataloader.py:120\u001b[0m, in \u001b[0;36mGNLDataLoader.__load_video__\u001b[1;34m(self, video_path)\u001b[0m\n\u001b[0;32m    106\u001b[0m gframe \u001b[38;5;241m=\u001b[39m cv2\u001b[38;5;241m.\u001b[39mcvtColor(frame, cv2\u001b[38;5;241m.\u001b[39mCOLOR_BGR2GRAY)\u001b[38;5;66;03m# .astype('uint8')  # Format to 8-bit image. 'int8' doesn't seem to do the job either\u001b[39;00m\n\u001b[0;32m    108\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m'''if self.debug:\u001b[39;00m\n\u001b[0;32m    109\u001b[0m \u001b[38;5;124;03m    \u001b[39;00m\n\u001b[0;32m    110\u001b[0m \u001b[38;5;124;03m    cv2.imshow(\"Frame\", gframe)\u001b[39;00m\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    116\u001b[0m \u001b[38;5;124;03m    homog = False if prev_frame != gframe.shape else True\u001b[39;00m\n\u001b[0;32m    117\u001b[0m \u001b[38;5;124;03m    print(gframe.shape, homog)'''\u001b[39;00m\n\u001b[1;32m--> 120\u001b[0m facedetect \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mface_detector\u001b[49m\u001b[43m(\u001b[49m\u001b[43mgframe\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    122\u001b[0m \u001b[38;5;66;03m#HAVE A CHECK IF THE FACE IS FOUND OR NOT\u001b[39;00m\n\u001b[0;32m    126\u001b[0m face_landmarks \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mlandmark(gframe, facedetect[\u001b[38;5;241m0\u001b[39m])\n",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Create the dataloaders of our project\n",
    "path_data = \"data/matching/fronts\" # \"data/lombardgrid_front/lombardgrid/front\"\n",
    "path_labels = \"data/matching/labels\" # \"data/lombardgrid_alignment/lombardgrid/alignment\"\n",
    "\n",
    "dataset = GNLDataLoader(path_labels, path_data, transform=None, debug=True)\n",
    "\n",
    "# Test\n",
    "print(\n",
    "    f\"[DEBUG] Items in the data folder: {len(sorted(os.listdir(path_data)))}\",\n",
    "    f\"[DEBUG] Items in the labels folder: {len(sorted(os.listdir(path_labels)))}\",\n",
    "    sep=\"\\n\"\n",
    ")\n",
    "#print(dataset[0][1][0])\n",
    "# print(dataset[0][1][0].shape)\n",
    "\n",
    "dset_train = dataset[0:128]\n",
    "\n",
    "dataloader_train = DataLoader(dset_train, batch_size=32, shuffle=True, num_workers=128)\n",
    "dataloader_test = DataLoader(dataset[128:192], batch_size=32, shuffle=True, num_workers=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4) Model Settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following settings are applied:\n",
    "> `device`: specifies where the model must be trained. If an Nvidia GPU is detected, then CUDA will be used;<br>\n",
    "> `epochs`: the number of epochs;<br>\n",
    "> `batch_size`: the size of each singular batch of analysed images;<br>\n",
    "> `learning_rate`: `N/A`;<br>\n",
    "> `loss_fn`: the loss function of the model;<br>\n",
    "> `optimizer`: the optimizer of the model. For now it's `AdamW`, which is more performant than `SGD`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model has the following layers:\n",
    "\n",
    "$$\n",
    "\\underbrace{x}_{\\text{input}} \\goto \\underbrace{st_0(3, \\; 5, \\; 5)}_{\\text{ST CNN}} \\goto \\underbrace{p_0(1, \\; 2, \\; 2)}_{\\text{Normalization Pool}} \\goto \\underbrace{st_1(3, \\; 5, \\; 5)}_{\\text{ST CNN}} \\goto \\underbrace{p_1(1, \\; 2, \\; 2)}_{\\text{Normalization Pool}} \\goto\n",
    "$$\n",
    "$$\n",
    "\\goto \\underbrace{st_2(3, \\; 5, \\; 5)}_{\\text{ST CNN}} \\goto \\underbrace{p_2(1, \\; 2, \\; 2)}_{\\text{Normalization Pool}} \\goto \\underbrace{y}_{\\text{Output}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=====================================================================================================================================================================\n",
      "Layer (type:depth-idx)                   Input Shape               Output Shape              Param #                   Kernel Shape              Mult-Adds\n",
      "=====================================================================================================================================================================\n",
      "LabialCNN                                [1, 75, 100, 150]         [75, 37]                  --                        --                        --\n",
      "├─Sequential: 1-1                        [1, 75, 100, 150]         [32, 75, 6, 9]            --                        --                        --\n",
      "│    └─Conv3d: 2-1                       [1, 75, 100, 150]         [8, 75, 50, 75]           608                       [3, 5, 5]                 18,240,000\n",
      "│    └─ReLU: 2-2                         [8, 75, 50, 75]           [8, 75, 50, 75]           --                        --                        --\n",
      "│    └─MaxPool3d: 2-3                    [8, 75, 50, 75]           [8, 75, 25, 37]           --                        [1, 2, 2]                 --\n",
      "│    └─Conv3d: 2-4                       [8, 75, 25, 37]           [16, 75, 25, 37]          9,616                     [3, 5, 5]                 142,316,800\n",
      "│    └─ReLU: 2-5                         [16, 75, 25, 37]          [16, 75, 25, 37]          --                        --                        --\n",
      "│    └─MaxPool3d: 2-6                    [16, 75, 25, 37]          [16, 75, 12, 18]          --                        [1, 2, 2]                 --\n",
      "│    └─Conv3d: 2-7                       [16, 75, 12, 18]          [32, 75, 12, 18]          38,432                    [3, 5, 5]                 265,641,984\n",
      "│    └─ReLU: 2-8                         [32, 75, 12, 18]          [32, 75, 12, 18]          --                        --                        --\n",
      "│    └─MaxPool3d: 2-9                    [32, 75, 12, 18]          [32, 75, 6, 9]            --                        [1, 2, 2]                 --\n",
      "├─Sequential: 1-2                        [75, 1728]                [75, 37]                  --                        --                        --\n",
      "│    └─GRU: 2-10                         [75, 1728]                [75, 512]                 4,233,216                 --                        162,555,494,400\n",
      "│    └─SelectItem: 2-11                  [75, 512]                 [75, 512]                 --                        --                        --\n",
      "│    └─Linear: 2-12                      [75, 512]                 [75, 37]                  18,981                    --                        1,423,575\n",
      "│    └─Softmax: 2-13                     [75, 37]                  [75, 37]                  --                        --                        --\n",
      "=====================================================================================================================================================================\n",
      "Total params: 4,300,853\n",
      "Trainable params: 4,300,853\n",
      "Non-trainable params: 0\n",
      "Total mult-adds (Units.GIGABYTES): 162.98\n",
      "=====================================================================================================================================================================\n",
      "Input size (MB): 4.50\n",
      "Forward/backward pass size (MB): 31.36\n",
      "Params size (MB): 17.20\n",
      "Estimated Total Size (MB): 53.06\n",
      "=====================================================================================================================================================================\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\Leonardo\\miniconda3\\Lib\\site-packages\\torch\\nn\\modules\\module.py:1532: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  return self._call_impl(*args, **kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "=====================================================================================================================================================================\n",
       "Layer (type:depth-idx)                   Input Shape               Output Shape              Param #                   Kernel Shape              Mult-Adds\n",
       "=====================================================================================================================================================================\n",
       "LabialCNN                                [1, 75, 100, 150]         [75, 37]                  --                        --                        --\n",
       "├─Sequential: 1-1                        [1, 75, 100, 150]         [32, 75, 6, 9]            --                        --                        --\n",
       "│    └─Conv3d: 2-1                       [1, 75, 100, 150]         [8, 75, 50, 75]           608                       [3, 5, 5]                 18,240,000\n",
       "│    └─ReLU: 2-2                         [8, 75, 50, 75]           [8, 75, 50, 75]           --                        --                        --\n",
       "│    └─MaxPool3d: 2-3                    [8, 75, 50, 75]           [8, 75, 25, 37]           --                        [1, 2, 2]                 --\n",
       "│    └─Conv3d: 2-4                       [8, 75, 25, 37]           [16, 75, 25, 37]          9,616                     [3, 5, 5]                 142,316,800\n",
       "│    └─ReLU: 2-5                         [16, 75, 25, 37]          [16, 75, 25, 37]          --                        --                        --\n",
       "│    └─MaxPool3d: 2-6                    [16, 75, 25, 37]          [16, 75, 12, 18]          --                        [1, 2, 2]                 --\n",
       "│    └─Conv3d: 2-7                       [16, 75, 12, 18]          [32, 75, 12, 18]          38,432                    [3, 5, 5]                 265,641,984\n",
       "│    └─ReLU: 2-8                         [32, 75, 12, 18]          [32, 75, 12, 18]          --                        --                        --\n",
       "│    └─MaxPool3d: 2-9                    [32, 75, 12, 18]          [32, 75, 6, 9]            --                        [1, 2, 2]                 --\n",
       "├─Sequential: 1-2                        [75, 1728]                [75, 37]                  --                        --                        --\n",
       "│    └─GRU: 2-10                         [75, 1728]                [75, 512]                 4,233,216                 --                        162,555,494,400\n",
       "│    └─SelectItem: 2-11                  [75, 512]                 [75, 512]                 --                        --                        --\n",
       "│    └─Linear: 2-12                      [75, 512]                 [75, 37]                  18,981                    --                        1,423,575\n",
       "│    └─Softmax: 2-13                     [75, 37]                  [75, 37]                  --                        --                        --\n",
       "=====================================================================================================================================================================\n",
       "Total params: 4,300,853\n",
       "Trainable params: 4,300,853\n",
       "Non-trainable params: 0\n",
       "Total mult-adds (Units.GIGABYTES): 162.98\n",
       "=====================================================================================================================================================================\n",
       "Input size (MB): 4.50\n",
       "Forward/backward pass size (MB): 31.36\n",
       "Params size (MB): 17.20\n",
       "Estimated Total Size (MB): 53.06\n",
       "====================================================================================================================================================================="
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "\n",
    "model = LabialCNN(debug=False).to(device)\n",
    "\n",
    "# Print the summary of the model\n",
    "torchinfo.summary(model, (1,75, 100, 150), col_names = (\"input_size\", \"output_size\", \"num_params\", \"kernel_size\", \"mult_adds\"), verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the hyperparameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 2\n",
    "batch_size = 32\n",
    "learning_rate = 10 ** (-4)\n",
    "dropout = 0.5\n",
    "\n",
    "loss_fn = nn.CTCLoss(reduction=\"mean\")\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5) Training + Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "→ Loss: -1.6055443286895752 [Batch 1/4, Epoch 1/2]\n",
      "→ Loss: -1.599102258682251 [Batch 2/4, Epoch 1/2]\n",
      "→ Loss: -1.5805745124816895 [Batch 3/4, Epoch 1/2]\n",
      "→ Loss: -1.5423518419265747 [Batch 4/4, Epoch 1/2]\n",
      "===     The epoch 1/2 has finished training     ===\n",
      "===        The testing loop has finished        ===\n",
      "→ Loss: -1.5854508876800537 [Batch 1/4, Epoch 2/2]\n",
      "→ Loss: -1.576768159866333 [Batch 2/4, Epoch 2/2]\n",
      "→ Loss: -1.534742832183838 [Batch 3/4, Epoch 2/2]\n",
      "→ Loss: -1.568716049194336 [Batch 4/4, Epoch 2/2]\n",
      "===     The epoch 2/2 has finished training     ===\n",
      "===        The testing loop has finished        ===\n",
      "===          The training has finished          ===\n"
     ]
    }
   ],
   "source": [
    "for epoch_ind in range(epochs):\n",
    "    train_loop(device, dataloader_train, model, loss_fn, optimizer, epochs, epoch_ind, debug=True)\n",
    "    test_loop(device, dataloader_test, model, loss_fn, debug=True)\n",
    "\n",
    "print(\"===          The training has finished          ===\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}