Fair_ML_Presentation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Fair-ML-Presentation.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyNOCzTZfz8D6mKYhefaQoHc",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/valeriu-balaban/fair-ml-presentation/blob/main/Fair_ML_Presentation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NIUKw_LT2Mjg"
      },
      "source": [
        "# Learning Fair ML Models\n",
        "\n",
        " - Fairness will play a major role in automated systems such as ML\n",
        " - For now we do not always have many answers, just a lot of questions\n",
        " - Let's make it interactive. Think and answer questions \n",
        " - There is no right of wrong answer\n",
        " - Presented methods can be easily integrated in any project\n",
        " - Code is in vanila Pytorch and can be ported to Keras, Tensorflow, JAX, etc.\n",
        "\n",
        "**Outline:**\n",
        "1. When things go wrong, examples\n",
        "1. Datasets description\n",
        "1. Definition and fairness metrics\n",
        "1. Conditional Value at Risk method (simplified)\n",
        "1. Moments penalization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Qecy1h8JqIb"
      },
      "source": [
        "## 1. Problem with $``the\\ algorithm\"$ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hquMcFeXF0yT"
      },
      "source": [
        "\n",
        "  - Tiktok's algorithm prioritizes white people:\n",
        "    ![tiktok](https://raw.githubusercontent.com/valeriu-balaban/fair-ml-presentation/main/Screenshot%202021-07-19%20at%2011-31-32%20tiktok%20race%20problem%20at%20DuckDuckGo.png)\n",
        "\n",
        "  - Youtube's algorithm has other issues ([source](https://www.wsj.com/articles/youtubes-search-algorithm-directs-viewers-to-false-and-sexualized-videos-study-finds-11625644803)):\n",
        "  ![youtube](https://raw.githubusercontent.com/valeriu-balaban/fair-ml-presentation/main/Screenshot%202021-07-19%20at%2011-30-24%20YouTube%E2%80%99s%20Search%20Algorithm%20Directs%20Viewers%20to%20False%20and%20Sexualized%20Videos%2C%20Study%20Finds.png)\n",
        "\n",
        "  - Facebook's algorithm has ome other problems ([source](https://www.technologyreview.com/2019/04/05/1175/facebook-algorithm-discriminates-ai-bias/)):\n",
        "  ![facebook](https://raw.githubusercontent.com/valeriu-balaban/fair-ml-presentation/main/Screenshot%202021-07-19%20at%2011-35-14%20Facebook%E2%80%99s%20ad-serving%20algorithm%20discriminates%20by%20gender%20and%20race.png)\n",
        "\n",
        "  - And the same problem can be posed for self-driving cars ([source](https://www.technologyreview.com/2018/10/24/139313/a-global-ethics-study-aims-to-help-ai-solve-the-self-driving-trolley-problem/))\n",
        "  ![self-driving-cars](https://raw.githubusercontent.com/valeriu-balaban/fair-ml-presentation/main/Screenshot%202021-07-19%20at%2012-01-08%20Should%20a%20self-driving%20car%20kill%20the%20baby%20or%20the%20grandma%20Depends%20on%20where%20you%E2%80%99re%20from%20.png)\n",
        "\n",
        "  - Evaluating the danger of an offender during (pre)trial ([source](https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm))\n",
        "    ![propublica](https://static.propublica.org/projects/algorithmic-bias/assets/img/generated/methodology-risk-of-recidivism-scores-by-race-900*363-482d1c.png)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s8nMdiUKb9Hb"
      },
      "source": [
        "None of these companies designed the ML algorithm to be biased. Yet it ended up biased on multiple occasions. \n",
        "\n",
        "**Why?**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R0EEk_70cMTq"
      },
      "source": [
        "### Answer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1n2aj4fFcO-E"
      },
      "source": [
        "*Learning from data without suppervision can (usually does) lead to issues because humans that generate the data discriminate by race, skin color, gender, age, etc.*\n",
        "\n",
        "https://www.youtube.com/embed/NaWJhlDb6sE\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ms5nTNaFHxYG"
      },
      "source": [
        "## 2. Toy dataset "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Degpfl9Td9et"
      },
      "source": [
        "import torch\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hek2c1_Md9LI"
      },
      "source": [
        "def generate_toy_data(n_samples, n_samples_low, n_dimensions, seed=0):\n",
        "  \"\"\"\n",
        "  Dataset generation code from Michele Donini, et al. \"Empirical Risk\n",
        "  Minimization under Fairness Constraints. https://github.com/jmikko/fair_ERM\n",
        "  \"\"\"\n",
        "  np.random.seed(seed)\n",
        "  varA = 0.8\n",
        "  aveApos = [-1.0] * n_dimensions\n",
        "  aveAneg = [1.0] * n_dimensions\n",
        "  varB = 0.5\n",
        "  aveBpos = [0.5] * int(n_dimensions / 2) + [-0.5] * int(n_dimensions / 2 + n_dimensions % 2)\n",
        "  aveBneg = [0.5] * n_dimensions\n",
        "  X = np.random.multivariate_normal(aveApos, np.diag([varA] * n_dimensions), n_samples)\n",
        "  X = np.vstack([X, np.random.multivariate_normal(aveAneg, np.diag([varA] * n_dimensions), n_samples)])\n",
        "  X = np.vstack([X, np.random.multivariate_normal(aveBpos, np.diag([varB] * n_dimensions), n_samples_low)])\n",
        "  X = np.vstack([X, np.random.multivariate_normal(aveBneg, np.diag([varB] * n_dimensions), n_samples)])\n",
        "  sensible_feature = [1] * (n_samples * 2) + [-1] * (n_samples + n_samples_low)\n",
        "  sensible_feature = np.array(sensible_feature)\n",
        "  sensible_feature.shape = (len(sensible_feature), 1)\n",
        "  X = np.hstack([X, sensible_feature])\n",
        "  y = [1] * n_samples + [-1] * n_samples + [1] * n_samples_low + [-1] * n_samples\n",
        "  y = np.array(y)\n",
        "  sensible_feature_id = len(X[1, :]) - 1\n",
        "  idx_A = list(range(0, n_samples * 2))\n",
        "  idx_B = list(range(n_samples * 2, n_samples * 3 + n_samples_low))\n",
        "\n",
        "  return X, y, sensible_feature_id, idx_A, idx_B"
      ],
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UsbFm12YI3kc"
      },
      "source": [
        "# Parameters\n",
        "n_samples = 100\n",
        "n_samples_low = 20\n",
        "\n",
        "# Prepare the dataset\n",
        "X, y, _,_,_ = generate_toy_data(n_samples, n_samples_low, n_dimensions=2, seed=3)"
      ],
      "execution_count": 51,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mA3ahx0aJEGk",
        "outputId": "76836ea0-dbce-4649-95d3-0e828e30ac28"
      },
      "source": [
        "X[0, :], y"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([ 0.59979794, -0.60957372,  1.        ]),\n",
              " array([ 1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
              "         1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
              "         1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
              "         1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
              "         1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
              "         1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1, -1, -1,\n",
              "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
              "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
              "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
              "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
              "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
              "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  1,  1,  1,  1,\n",
              "         1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1, -1,\n",
              "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
              "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
              "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
              "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
              "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
              "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YeLFbSpafBZb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 483
        },
        "outputId": "baad547b-724f-48b0-e889-7d5ecc0e9424"
      },
      "source": [
        "# Plot data\n",
        "s1 = (X[:, 2] < 0)\n",
        "s2 = (X[:, 2] > 0)\n",
        "\n",
        "c1s1 = (y[s1] == -1)\n",
        "c1s2 = (y[s2] == -1)\n",
        "c2s1 = (y[s1] ==  1)\n",
        "c2s2 = (y[s2] ==  1)\n",
        "\n",
        "\n",
        "plt.figure(figsize=(8, 8))\n",
        "\n",
        "plt.scatter(X[s1, 0][c1s1], X[s1, 1][c1s1], marker='x', color='b', label=\"class 1, subgroup a\")\n",
        "plt.scatter(X[s2, 0][c1s2], X[s2, 1][c1s2], marker='o', color='b', label=\"class 1, subgroup b\")\n",
        "\n",
        "plt.scatter(X[s1, 0][c2s1], X[s1, 1][c2s1], marker='x', color='r', label=\"class 2, subgroup a\")\n",
        "plt.scatter(X[s2, 0][c2s2], X[s2, 1][c2s2], marker='o', color='r', label=\"class 2, subgroup b\")\n",
        "\n",
        "plt.legend()\n",
        "sns.despine()"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeIAAAHSCAYAAAAwk8gOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dfZScVZ0v+u/u7sSktU0ksASBrsZ1soRDSDomJASvaPQ4YWBGDoyyJpYOHA4nV7jaRbMGrtqSbiLtOs64aJozx/GGK8dIt170zmEJLhw8ToajjOGlHRLkRQZx8nZ1XZKGDnCzwNC17x+7n+6nnnr2U8/73s9T389atTpV3f3Ufqor9Xv23r+9f0JKCSIiIjKjw3QDiIiI2hkDMRERkUEMxERERAYxEBMRERnEQExERGQQAzEREZFBXSae9OKLL5Z///d/b+KpiYiITBC6bxjpER89etTE0xIREVmHQ9NEREQGMRATEREZxEBMRERkkJFkLT8nTpzA4cOH8cYbb5huCllgyZIlOOOMM7Bo0SLTTSEiypQ1gfjw4cPo6elBX18fhNAml1EbkFJienoahw8fxllnnWW6OUREmbJmaPqNN97AihUrGIQJQgisWLGCoyNE1BasCcQAGIRpHt8LRNQurArENhoZGcHXv/71TI49NDSEM888E+94xzsyOf6HP/xhTE1NZXJsIiJKR2EDsZTB94vgT//0T/H444+bboYvKSXq9brpZhARlV4hA/HICDA4uBB8pVT3R0aSHfc73/kOVq9ejTVr1uAzn/lM0/fvuusunH/++VizZg3+7M/+DMePHwcA/OAHP8CqVauwZs0aXHTRRQCAZ555Bhs2bEB/fz9Wr16NF154oel4F1xwAU477bTQ7fM75v79+7Fq1ar5n/n617+OEdcLcc8996C/vx+rVq2aD/pHjhzBxz72MZx77rm49tprUalUcPToUezfvx/ve9/78Bd/8RdYtWoVDh06hJtuugmrVq3Ceeedh3vvvRcA8PDDD+NP/uRP5p/jc5/7HL797W8DAPr6+nDzzTfjvPPOw4YNG/Cb3/ym6Twef/xxbNq0CWvXrsWFF16I559/PvRrQERUNoULxFICMzPA+PhCMB4cVPdnZuL3jJ955hncdttt2L17N/bt24fx8fGmn7niiivwxBNPYN++fTjnnHPwrW99CwCwY8cOPPTQQ9i3bx/uv/9+AMA3v/lN1Go17N27F1NTUzjjjDNin7MjzjGPHz+OvXv34hvf+AauueYaAMCtt96Kj3zkI3jmmWfwiU98AgcPHpz/+RdeeAHXX389nnnmGUxNTWHv3r3Yt28ffvrTn+Kmm27C73//+5bPuWzZMvzqV7/C5z73Odxwww1N3z/77LPx85//HE8++SR27NiBL33pSxFeBSKicrFm+VJYQgBjY+rf4+PqBgC1mno8bo7P7t278clPfhInn3wyAOCkk05q+pmnn34aX/7ylzEzM4PXX38dW7ZsAQB84AMfwNVXX40rr7wSV1xxBQBg06ZNGB0dxeHDh3HFFVdg5cqV8RrmEueYW7duBQBcdNFFePXVVzEzM4NHHnkE9913HwDg4osvxrve9a75n69UKrjgggsAAI888gi2bt2Kzs5OvPvd78aHPvQhPPHEE3jnO98Z6jm3bt2KwcHBpu8fO3YMV111FV544QUIIXDixIlwLwARUQkVrkcMNAZjR5IgHNbVV1+Nv/mbv8GvfvUrDA8Pzy+v+eY3v4nbbrsNhw4dwrp16zA9PY1PfepTuP/++7F06VJccskl2L17d+Ln9ztmV1dXw1yud8mPN/u4VTby29/+9pbtiPKcfs93yy23YPPmzXj66afxwAMPcJkSEbW1QgZiZzjazT1nHMdHPvIR/OAHP8D09DQA4OWXX276mddeew2nnXYaTpw4gcnJyfnHX3zxRWzcuBE7duzAKaecgkOHDuG3v/0t3vve92JgYACXXXYZnnrqqdBtue+++/DFL36x6XG/Y7773e/GSy+9hOnpabz55pv40Y9+1PA7zrzuI488gmXLlmHZsmX4wAc+gO9///sAgJ/85Cd45ZVXfNvxwQ9+EPfeey9mZ2dx5MgR/OxnP8OGDRtQqVTw7LPP4s0338TMzAz+4R/+wfc57733XmzatKnpuMeOHcPpp58OAPNzy0RE7apwQ9PuOWFnONq5D8TvGZ977rkYGhrChz70IXR2dmLt2rVNQeIrX/kKNm7ciFNOOQUbN27Ea6+9BgC46aab8MILL0BKiY9+9KNYs2YNvva1r+Gee+7BokWLcOqpp/rOg95888347ne/i+PHj+OMM87Atddei5GREbz44ou+w7/f//73m465aNEibN++HRs2bMDpp5+Os88+u+F3lixZgrVr1+LEiRO4++67AQDDw8PYunUr7rnnHmzatAmnnnoqenp68Prrrzf87uWXX449e/ZgzZo1EELgr/7qr3DqqacCAK688kqsWrUKZ511FtauXdvwe6+88gpWr16Nt73tbfje977ne95XXXUVbrvtNlx66aUt/jJEROUmpIF1P+vXr5fe9a3PPfcczjnnnFC/PzKiErOcoOsE5+XLk2dO2+DTn/40xsbGcMopp2Ry/DfffBOdnZ3o6urCnj17cN1112Hv3r2pHLuvrw9TU1Pzc+1JRHlPUHuYnASGhoCDB4HeXmB0FKhWTbeKKBRtF7FwPWJABVspF3q+zpxxWTZjmpiYyPT4Bw8exJVXXol6vY7FixfjrrvuyvT5iNIwOQls2wbMrRrEgQPqPsBgTMVWyB4xtQe+J8itr08FX69KBdi/P+/WEEWm7SoWMlmLiNqPa7l7qMeJioKBmIgKobc32uNERcFATESFMDoKdHc3PtbdrR4nKjIGYiIqhGoV2LlTzQkLob7u3MlELSo+BuIWWAYxfVm+plRu1apKzKrX1VcGYSoDBmKDWAaRiMpiclJltnd0qK+uzQephcIG4iz+6CyDmE8ZRADYt28fNm3ahJUrV3IdM1HBOWu8DxxQezw4a7wZjMMpZCDO4o/OMohKHmUQAeCpp57C7t27sWfPHuzYsQO/+93vQr4KRGSboaGFjVYcx4+rx6m1xIFYCLFECPG4EGKfEOIZIcStaTQsSBZ/9LBlED/4wQ/ivPPOw+TkJJ555hkAC2UQ77rrLszOzgJQJQu/+tWv4mtf+xoOHDiApUuXxm/cnDjH1JVB/PM//3MA8coghn3OrVu3Ys+ePb4/c9lll2Hp0qU4+eSTsXnzZmuH6ImoNa7xTiaNHvGbAD4ipVwDoB/AxUKIC1I4rpapPzrLICpJyyDGaRcR2YtrvJNJHIil4pTtWTR3y3TfzCz+6CyD2CyrMogA8MMf/hBvvPEGpqen8fDDD+P8888P/foQkV24xjuZVIo+CCE6AfwSwL8B8F+llI+lcVyd0dHGzd+B5H90lkHMrwwiAKxevRqbN2/G0aNHccstt+A973lPwF+HiGzmLCNjZax4Ui36IIRYDuA+AJ+XUj7t+d42ANsAoLe3d90Bz+7tUTf4L3M5NJZBVFj0gYhKJJ8yiFLKGSHEPwK4GMDTnu/tBLATUNWXkj5XtVqewOvFMohERO0jcSAWQpwC4MRcEF4K4GMAvpa4ZZSZlStX4sknn8zk2PtZj46IKJI0esSnAdg1N0/cAeD7UsoftfgdIiIiQgqBWEr5FIC1LX+QiIiImhRyZy0iIqKyYCAmIkqJbg/8FBenUAkxELeQVcm+48eP49JLL8XZZ5+Nc889F1/4whdSfw5byyASlVHQHviDg4CrFgtRg+IGYu8lZgEvOf/yL/8Sv/71r/Hkk0/in/7pn/DjH//YdJPmsQwiUTS6PfCvuw4YHwdmZgr5MUU5KGYgHhlRl5jOu1rKVC458yyD2N3djc2bNwMAFi9ejPe///04fPhwYPvKUgbx8ccfx6ZNm7B27VpceOGFeP755wPPm6gIdHvdv/YaUKsBY2MAt1QnP8ULxFKqS8vx8YVgPDiY+JLTZBnEmZkZPPDAA/joRz8a2MaylEE8++yz8fOf/xxPPvkkduzY4bv9J1HRBO11zyBMQYoXiIVQ7+paTQXfjg71NeElp6kyiG+99Ra2bt2KgYEBvPe97w1sY1nKIB47dgyf/OQnsWrVKgwODs6/jtQ+dElNReZX+MDhHsAj8ipeIAYWgrFbDpecWZRB3LZtG1auXOnbc/QqSxnEW265BZs3b8bTTz+NBx54oOn3qdyCkppMS3KBUK0CO3cCPT3qfk8PMDGx0GdgMCadYgZiZzjaLeG73EQZxC9/+cs4duwY7rjjjobHy14G8dixYzj99NMBoKnCFZWfLqlpaMhMexxpXCBUq8CNN6rge+yYuu8M4C1fzuFp8pdq0YdcuOeEneFo5z4Qu2ecdxnEw4cPY3R0FGeffTbe//73A1BJT9dee23pyyDefPPNuOqqq3Dbbbfh0ksvbfWnoZLRJTXpHs9L0AVClAIzIyPqY8r5GHIG8BiESUtKmftt3bp10uvZZ59tekxreFjKWk3Kel3dr9fV/eHh8MewWLValS+99FJmx3/jjTfkiRMnpJRS/uIXv5Br1qxJ7diVSkUeOXIklWNFek+QlSYmpKxUpBRCfXXuq1DVeKtUzLZVCP92CWG2XVQa2phYvB4xUPpLTpZBpDJwhnqdXqYz1HvVVcCuXY29z+5ulexkUm+vaqPf40RZKuYcMdAcdEsShPPglEHct28fnnjiCZx//vmpHXv//v3zmefU3nRDvQ8+qJKaKhX137ZSUfdN1xf3y3q24QLBTxmzzttZMXvERGS9oLngatV84PVy2jM0pNrY26uCsG3t1I00APa1lcIpbo+YiKymG9K1eai3WgX27wfqdfXVxsBma9Y5xcdATESZKNJQb5HYmnVO8TEQE1EmnA0ubJsLLroijjRQMAbiFlgGMX1ZvaZknyIM9RYNRxrKh4HYIJZBJKKoONJQPsUNxBnk77MMYj5lEAFg37592LRpE1auXMl1zEQRcaShXIoZiDPYNZ5lEJU8yiACwFNPPYXdu3djz5492LFjB373u9+1PC4RURkVMxBnkL/PMohKHmUQAeCyyy7D0qVLcfLJJ2Pz5s3zvXWiVriZBZVNMQOxofx9lkFUkpZBjNMuIsDuEopEcRUzEGeQv88yiM2yKoMIAD/84Q/xxhtvYHp6Gg8//HCq22xSeXEzCyqjYm5xOTrauMcbkDh/n2UQ8yuDCACrV6/G5s2bcfToUdxyyy14z3veE/TnIQLAzSyonISUMvcnXb9+vfSub33uuedwzjnnhD/I5KT9m8LG9OlPfxpjY2M45ZRTMjn+m2++ic7OTnR1dWHPnj247rrrsHfv3lSO3dfXh6mpqVQKP0R+T1Dp9fX5V0iqVFT2MJHFtPNvxRyaBkqdvz8xMZFZEAZUGURnGdbAwACXD1FqJieBk09W61uFUP9Oc/42780smBhGeSjm0DQl4pRBzMJ+dkva1uQkcM01wB/+sPDY9DTwH/6D+nca18p5VkhilSPKS3F7xERklaGhxiDsOHEi3WSqvAbD8koMY6+brOoRSym5jIUAqPcCFUtQwlQRk6nySAxjr5sAi3rES5YswfT0ND+ACVJKTE9PY8mSJaabQhEErR4sYmWgPKoccTkWARb1iM844wwcPnwYR44cMd0UssCSJUtCbeFJ9hgdbZ4jBoBFi4pZGSiDVZJNuByLAIsC8aJFi3DWWWeZbgYRxeQMpdZqKkkLAFasAMbHiznMmkdiWG+v/3KsIo4gUHzWDE0TUfFVq8DRo2r7SSnVv/MOwmkmP2WdGMbawgQwEBNRiRRtL2rWFibAop21iIiS4s5bZLES7qxFROTB5CcqIgZiIiqNPJYcEaWNgZiIMmFixygmP1ERMRATUepMJU0x+YmKiIGYiFJncsco2wuzcW9p8rJmQw8iKg8mTfnj3tLkhz1iIkodk6b8cW9p8sNATESpM5E0VYQhX44UkB8GYiJKXd5JU0XZUYsjBeSHO2sRUeEVZUct7xwxoEYKmNndFrizFhGVV1pDvlkPb3N5Fflhj5iICi+NHjF7q5Qx9oiJqLzSSA5jRjOZwkBMRIWXxpAvM5rJFG7oQUS5k1IFTN39OKrV+EPIk5NqXnh2tvl7zGimrLFHTES5GhkBBgdV8AXU18FB9bgJztywXxBmwQjKAwMxEeVGSmBmBhgfXwjGg4Pq/szMQnDOk9/cMAB0djJRi/KROGtaCHEmgO8AeDcACWCnlHI86HeYNU3UvtzB11GrAWNjyYen4+jo8L8AEEIVjiBKifbdnUYgPg3AaVLKfxZC9AD4JYB/L6V8Vvc7DMRE7U1KFQAd9bqZIAwUZzMQKrzsli9JKX8vpfznuX+/BuA5AKcnPS4RlZPTI3ZzzxlnIWijDhP7YhO5pTpHLIToA7AWwGNpHpeIysE9LF2rqZ5wrdY4Z5y2VvtQc7crMi21nbWEEO8A8D8BjEop/7vP97cB2AYAvb296w74jQURUemNjKjELGdO2AnOy5dnkznNoWeyRHZzxAAghFgE4EcAHpJS3t7q5zlHTNTeslhHrMNkLLJEdnPEQggB4FsAngsThImIvEE3y0Qtlh4k26UxR/wBAJ8B8BEhxN652yUpHJeIKDEmY5HtEm9xKaV8BAFdbiIik5ykq6EhtW90b68KwkzGIluwDCIREVH2WAaRiIjIRgzERNRWgjb3IDKBgZioRLwzTVFmnpL8blG02tzDZLt4cdC+GIiJSiJJeUGbShNmGZT8Ki0dP64eN8WGiwNeCJjFQExUAknKC9pUmlAXlK6/Pp1AcfBgtMfzYPriwIYLgXbHrGmikkhSXtCW0oS67SidrTAd3d3x9oO2cbtL0zt/2fialFS2W1xGxUBMlI0k5QVtKE2oC0p+4gQKp/fn7oHGDeppMR0ITV8ItBEuXyIquyTlBU2UJvQTZdvJOMPJNlZaMr3zF7cAtYCUMvfbunXrJBGlp16XslaTElBf/e57f97974GB8L+bpYkJKbu71XM7NyEa7zu3SiW/dmVtYkKdjxDq68REvs/tfc27u/NtQ5vQxsTEW1wSkXlCqDKC7nndsTH1veXLG4eYvWUIAeCxx4CNG1v/btb8tqO85BJg167m4eQy7RVdrZrrlXMLUAsERemsbuwRE2UjqOfr3Nf1nAcGmnvKtkjaYzTZ4ySao42JTNYiKpjJyWS9F1sypPNiY4IWtSVmTROVQVpBxYYM6byYzkommsOsaaIySGPzB1sypKNIsv2mjZt4ELkxEBMVSNKg4h6WrtVUT7hWa9xVyzZJt9/ManlOkosDIjcGYqICSRpUdNnVtVr+GdJhpLH9ZhbrdG3am5tKICiTK6sbs6aJ4klrzWer7GqbuDO7nVvU9c1pZk1HXbNNzdo0i51Z00RlkTRruohsSy5rt8zzNLVxFjuTtYjKolpV2b71uvpa8g8vK5PL3JueOPIIwk65QiGAri71tWhlC01Xm3KzpfwjAzERWcvW5DITFwfucoUAMDurvhatbKEtWew2lX9kICYia9mYXBb34iBplrVfT9JhqkcZhy1FJmzqmXOOmIisJ2Vj0PXez5t3v24nOC9f7p85HfXn/bQqEVmUsoW2zBEbKP+ofcey6AMRWc8bdE0nRI2MNF4MOD11v3a5l2AB6ufcPeqwFxW9vf47hLm/XwS2FJnQvZ4mXkf2iIkKyLYeIgVLI8varyfpaJOs41QZ6Jkza5qoLLiZRPGkkWVdraogUamo+52d6mulwiAch/v1FMLs68hATFQgaew0RY3y2KoyrSxrZ+malMBbb6mvWS5hs2V5T1ZsWQrIQExUIO6s4fFx9QHpzDVyMwklSvDIY3TB1iVYrdi0vKfsGIiJCibvzSSKVNwgSvDIa3TBxiVYYdi0vKfsmKxFVDCtEn/STNxKY9lNnqLWHvZ7LXt6gL/92/SHKYuWYGdgeU/ZMVmLqAy8w5zbtwP9/Qu9uno9vaFVG+ajo85Rttq1ya93f/75jY+99lo2Q7C2LcFqxZaNN9oBAzFRgbiHOW+/HTh2DNi7VwXjZcuAG29ML1DGnY9Oayg7zhxlUPDwzgcPDwPr1gHXXNP88xyCzaZ8JGkElWbK6sYyiETJOKX20igRGOa53McPOvbwcOPzO+0bHo7+vJVK4/M6t0pF/ztBZSLdpQpnZ6Xs7/c/vnMTInqby6ZNyxVmRRsTGYiJCi5KoIxz7LCBPmmdXu+HftwAqQsefudy6qn65wkK+EQxMBATlVGWPeKogbVej98ev56sENECZJjem/eiZXbW/zmcXjRRirQxkXPERAUlU16f6k2M+u53wy+7ceZfgXhLq/yWykifrGLdHGWY+WTn9XJbt675WL293KmKchYUpbO6sUdMlI605mSD5lb9er7e+04veGBA3dzHGRho3SPW9X6dHnCrOcpW88ne3rx7jri/X92PMoxO8bXxvLM2JrL6ElGBRakCFCRo8wZvz9BvGc7YmGrHnXcuPD4woL7eeWfrdukq4ejW/3q1Wrbkt6nGxz+uvvfxj6tRAKcnb/MmG0XnLbTgjFwA7T0CwQ09iCiVzRukVMdxOL/ntwHI5GRjGbxLLgF27YpfCSfsRh7e4e56vbHNfsPhlJ6oG66UDDf0ICK9pJs3+M2/uueMvUHYO5+7axdw1VXxK+GEXfPqDbIdHcHfp3S1GrloVwzERCXm7eXqBsBGR4FFixofW7Qo3OYNrZLGvvvdxiSwWs1/GPzBB9Xz9faqD+ahofC7W9lU0o70irBbl4mKUwzERCUVtbJQ3C0Yg4oaHDrU3PudnvY/jjNfGLfaT5iSdmUv62c723frMlZxKiiTK6sbs6aJshV1DXCcXaz8ntN7P2hjDu+tszN5G4IEZYZTfmzOmk7j/0EAbUxkshZRSbmHjB3OHtXeudGsKu3ojuvV3d08XB2mDd7kqqBkqzZPFKIQMq44xWQtonYjhCoE4Xb77aowhHd4Oqu5O93vr1jRPJ9bqUQ7RtShdyYKUSum5rAZiIlKql4H7r+/8bF16/yrM2U1d6c77vh483xuqza4A66UwCuvRCvRmOeHbNgkuag4x50tY3PYQePWWd04R0yUDb+qTN4qQ85OUl5Zzd1FOe7EhJQrViy0delSKbdskXL79oUdsWo1dX9gQMqNGxvPLWhXrLzmiNOsQOXGOe58ZDiHrY2JDMREJeENANu3q6B7yy2NH97btxttZiC/YANI2dvbeFHhfPVupxm1wlPaQSxpBaogGScSUfa0MZFbXBKVgJRqSNZJzBobA44dA/bubf7ZY8fs3UHKb6tNYGEe1zmfvXsXttB0GxwM3kqzWs12bbGzdAtQfwvn7+Fe2hUX57jLi3PERBnIey7PvXZ3fFw97/g40N+vglYa1ZnyEDWo3HmnfefmDsaOpEEYKMZmGBQPAzFRykxtCuAXAD7+8cbe2O23+5cxtEWUoPLYY6pX3KpEY96cxDG3ZcuSX5TZvhkGJRA0bp3VjXPEVGZx5/JalRtsxT0f6U7Meuutxu8nnSNO2s4gceaI3c9vunyhd054YkLKrq70Eqxs3gyDWtLGRPaIiVIWZy4v6ppYL+fnnf2eZ2cXhqXXr1dDt873nTniOJK2sxXvntHLlgFbtgBXX63O65e/VF+dnr7p3q+Xd7vPoSHgrbcaf8YpLxlHmG08qYCConRWN/aIKQnbewVRe8RpZdp6s6ZnZ5uXLiXJ3M0yI7jV8/p9dZY0pb1MKA1Om4Twfy8IEf/Ytr//SUsbE1MJrADuBvASgKfD/DwDMcVVhLWUcdroN6wcJ7h5f352tvGYSYNlWu1Myt0OZ3jae98GaS85KsL7n7QyD8QXAXg/AzFlzYa1lGHmSOP0Wur1YgTNtNsZlt9r6t3QY2BA3WzoFUsZ/6JMd9+G9z/Fpo2JqcwRSyl/BuDlNI5FFMT0Wsqwc6RR5/Kc47glWYbjHE9XIzjqcd3Lsbz7V+exXEiXib58efPP3nmnfpvLvEWtk9zq/aV7nx84wO0uCy0oSke5AehDQI8YwDYAUwCment7c7j4oDIy2SPIao40q+OmtdWiX6+uq0s9nsccsZT6v3tPT/NjSYamTc6/hnkfBJWV5BC19bIdmpYhArH7xqFpisv0HFlWw71Z7U+cxlKjoIufrBKkvO3UJT05gTeNQDwxIeWiRY3HWrTIXDD2e3/plndxiLoQGIipPFr2WjLu1mQ1R5rl+twkWmX+xmln0Ln6XZT49XydHrE3EG/cGK9N7mIT7tuKFdGPlUSr99fEhD4QJ8nGpsxpYyLXEVPhBM6/ZrytlUx5LtfNux5Wtz7W+1xpPHeQVlsrRl3HGzQPKuXCntnu8oavvQZ0eXbG7+4GLrywcZvLgQG141acv8n0dLTHsxDm/VWt6ms3n3RSdm2jDAVF6bA3AN8D8HsAJwAcBvAfg36ePWLKTIaTyKbW0bplNYQdJM3pgDCvoW541m+gI83XQ9fLBKIfK44o76+JCSkXL25uZ95D6RSJPoYGfTOrGwMxZSaLHRRcTARCh8kLgYkJtc2kOwimkZymm2ePMvyf1pC+DUPTUd5fuvZynthaDMTUJnJIqzY5l5vnhhruHuiyZVJu2dIYILzrdaO0wS/QeoNPHufo5tfLXLw4/x5m2PdXxteclD5tTOQcMZVLDiVqws7lZiGrEnte3qn2Y8eAhx4C/viP1f1Nm9Tc7CuvLISAsHtOS6mO47ZyJXDDDY17Yvf3A9u351fesFoF7r67cc3v3Xfnv59z2PcXyyKWBwMxlUvUHRQKxgl4blkEqKEhVZzA66GH1KYejz3W3Ibx8dYbaThB+KGHGh9/8UUV2AcH1YYhTsGKY8fyLd1YpKIKLItYIkHd5axuHJqmsspy2DrPOeKgdbvOzbtsKGwbli3zP553WNg7V0rNWACiUDg0TZS1rEsEekvsOcPUWfQW4wxvhh0if/VV/8dPnNAfL4/hf/c2nn19xdgyskg9eNJjICZKgdSsfw0zXBvFyEhzgBobSy/YO/yGPQFgyRJgYkKt173zzsbvhR0i1wX5d7wj3vHS0Gr5eRGDNBVIUHc5qxuHpqmMMqLMv2sAACAASURBVM32NTAGOTHhv0Smu1vKP/oj9W9nF6soQ+S6vavTGHKPW/UqKNne9LaqVBramChkXpecLuvXr5dTU1O5Py9R1qRUvSZHvZ7CsKrTXXNnT3V355KE1teneodeixerhKpHH1Xn54wALF8e3DuXUv385KRKCDt4UPWQ160Dzjxzobcf9nhucV6mkRE1YnHnnf69byFU+/xeg0pFDQcThaT/JAiK0lnd2COmMgrsESfp0RosOdWq2IJ3E44grTarSJroFvVlcv+9dHtZO3+yVut1mTRFIWhjIgMxtY8MPy2DMpp3bZmQ9SRjmwZ3bggqPxh1A4+sM77jvEx+F0/eP1GrAF/EoWteOBjBQExtLodPS12P75VllWhdNS+DPWJd2b2k+0wnmUPX9ZzjvEzOjl7ewO0OTq3eOlGe14YKW0W8cCgJBmJqczkFM98P2qQ92oBPzjw+yCcmmodu4/Zik5aQDBrejhpghofV8HqYMopBPciwf16T+5S7Gbyua3cMxNTmTG7Mm8Yn38SEfGVZRdYhZL23Mh+Es/4gT3NIOWmPOExbwg65OntlO+3wBmTv/HeQMH9eGyp3ObhHtTEMxNTmTHYDUhgLNPlBnkZPLq32BwXzqMO+9brq/bqP5QRk77kFBfiwf15TxSy82CM2hoGY2pzpibEUsmNMfpCnMbeZ1tCs3/B23GMHVYFyhHnrROmJJxmaT4Pp/wptjIGYyiN2TPP5RVPJM3GfN+oHedLzS/v1SaM93osR97ByUG/b77nDXNi02uwj7HvRlh6xlMyaNoSBmMohzat5U8kzSXpvUT7IneeZnW38/e3bs21nVoKGt/2Srvr7m8/dvWZZd6wtW6Ts7V0IUn5B2P3eC/NetGmOmIxhIKZyiD2/5V4Q2tkp64Cc7qnIrZjI9YMx7gdy1N9zf98JSO77rYKxbYHD/efr6VkIdu4A6x0taNV2vwuNLVukXLSo8Ri65KbOzmjvRdsubCh3DMRUDrEyPnWLYQH5Zle33IqJXIcK4w5RRv0gn51VQdfbSwz7fLYMpbYaBXHmdb1t9Z677oLFrbdX//4K6gmHfS/asI6YjGEgpnKI1SNuMb74r6jM381zjtjdjChzxEH3vWZnm085apZynHamOQfZ6m8e1HuP2vagLT2958PsY4pIGxNZBpEKxa88X3e3elzr4MHAY/Zi4ft5lN6TUj2PW9jn9RaQCCooISVw443Nj99+e7hCFHHb2aqkYFS6P5/zuF+d5ttvV0UporZdV6LRKfDgrvsb671I5CcoSmd1Y4+Ykojc22rRI57uqVg/RxzI84LUXZt9+A3RupOYnDZ52+hNhPK7r5Okp+jXlrDH8w7Zx3mNoyYDMvuYItDGxC7TFwJEUVWrEav/jY4218eb84eubrzrb0chhOpNAap3lbh0oYZf7y3R83pr/x04gBNXb8MiAMuWVdHfD+zdC2zcCPziF6rc4N69qqc8NgbceqsqA+hXfnD5cvV7jrEx9f3HHlO/pytP2KoHq+OUJPS2Zd064MiR5vKG3p6n89oleY2d95W7ROPoqP79Fvm9SOQnKEpndWOPmHLnyZp2ulR1TxfG9nXETTTdRaeXv2FDYy/WyZ52soyDeo6zs83rc733IzSpZfGFoLbE6XlmnRjF3jBFpI2JDMRElogVODTZRbMQ83eD6ga3yox2L3sKm+wVd623LVnaYXB3KopBGxOFlBlnpvhYv369nJqayv15iWylG5Zdvlw/BAwA6OtT2VAe+1HBWdgPQCUYtUrq6nClbc7OqvtSAjfcoNqwY8fC91sdD1Aj5mGHd4PaEua5TNC87PNJXUQ+tO9kZk0TGSalCsLj4wuZvYOD6v7MTItMX5/U3T90deNLWJhADcoWdp7Lbd06FQA3bQLuvBO4//7G72/a1Dr7uFptzjJuxa8taWWxT06q4NnRob7GzeB2xJ0HJ/IV1F3O6sahaaJGiYZl5yYr60JE2i3Mb07Ym2HtHt6OWyYw6vmnvZNXFsPIXENMMXBomsh2aQzLRh3i9v58vQ50di58f8MG4PHHF+4PDKiv73qXfsg87rB07OH5FiINI0vZ+KJ778/xJqsDamBi505mUZOW9n8zAzGRBdzD0Q738puoxwoRS5q+79eGgQE1PO2o19VX3fGSBihvW+v1xosT5+Mqyvk5891ezoXHvAhXAiMjwKOPAs89Bxw6BJx5JnDOOcAFFyS7aKBS4xwxka3cAbBWU8GhVmucM44iyu5bzvf92uANwkDzHK7X0FDzcu3jx9XjUds+MqLWOzvnL6Wan3bPUTvtDgp+ut2yGh6PMFHv/OhDDwGXX66S2y6/XN1vOadP5Cdo3DqrG+eIiRrZUJnH3Qb3muGNG8PP2cYqyiGbj+deNuVdw+yeo9a2yXVnYkLK7qX11nPEESbqi7TUiqzBOWIi20UdUs66DSMjwCuvAHfcEX7ONs6yHt2I8LJlwLFjzUPlQGNPvWkI3+eAv/7jQfz9o8tx46sj/vPW7vH5kBP1aczpU1vRvzuConRWN/aIiYoh6iYjUTOUw+zu5T6W01v3Phb6gH4n4AwFRNi9hD1iikEbExmIidpIHvVwo279qAtq7i02dcuotMPSYaOkX3UM71fP72a51IpKjYGYqN3ZMA+t4+3luoOwM0cdeY5Y2232eXLdlYDmBbL5tSRraWMiqy8RtQHpSgoG1PSpO0taGpiPdrfthhsaHxscVMuDgIUKUHfcoSo/Of/WVlWS0n+LLt1aMOdA7slod9kmn98ZGWl8zQJ+NDTv38Dk34RyFhSls7qxR5wjloihOTbOa3rrHHuHnT//+eYiFYHD60HjxrrqFwZfGOcp3NPU7iaxh10qHJpuS7rMmeuuY3BuU1FGbPPiXTaVuH1+48YbNy6McTuP1WpSbt+e+oRv2Gtfvxyx/n7/JlEpMBC3Jd2GuN7Fnqzf1hZs7BG72xY0VRvrgO5/ewspuyPdXOSrzyaf8A2bNR5mn29b/jaUGgbitqTbXcHvxt3qS83mTF9vEF6zxr+XmPhJPFH+4f6F4FufrTfG3pgvSJRiEH4XHq1GAzjTVGjamMgtLstMt7efnyzrt6Vdg46imZyEOKsPY+MdmO7pw9j6yfnkolrNk+iUs5GRhbyqZcuANWuAffuACy8Ebr8d6O8H9u5VG3tImeCJ3Jldcz68dwyDNwpICQzeKBp3s9QUemj1No5SHtGnSQ2825s6+3gfOKAeP3BA3ed/pxIIitJZ3dgjzonfOJmul5xVjziLGnQU3hVXSNnV1fj6d3Wpx6X5nnDQNpapDtFqesRAPdTzhH0bJ+0R9/f7b+8Z9dhkJW1MZCAuO+9Y1nXX5RsY+elhTr0uZU+P/+vf02PFBKRfMPIG41SDsGdcfgwLwTjoecK+jePOEW/f3rh/iN8S5rj7eJM1GIjJJc+JJn56mFWA19+bKd1y56w4PJnU9dm6fLi/JocxHOp5oryMUbOmnef0Bl9vW3hNW3gMxGQIPz3Msvz11yUstdw5K6KJCSkrvXUphJS9vVJu2SIlUA+duJbVyxhly1HO8hSeNiYyWYuyNTqqKsO7dXerxyl7o6NAl2cDva6u3F9/KZvvS9lcA9nZRcuRRkLZfJLTQZWYdfAgsHs3sGWLaNhAK+h5snobR6kdXa0CO3eqSlZCqK87d3qqSFExBUXprG7sEbeZMqy5KOI5uLt6PT2q7c6ccY5rloL2Zfb73sBA49xo0mbqerO9vY0/F6ayVNHeAmQVbUxkPWKiVpwu1fHjC491dxvrjkjpvyex7+O3jvgX+w0qKpxyW929Xu8e187yHSEWTmT+PLwnFFNHR3OP3HnOej3x4YnC0r6ZGYiJWolT7T4jPjXvMTio1tr292vi7bAmcs+ZnASGhtSQbW+vGm51X1/oAn9Y7mA89whqtYVhYUgJ3Hqr/oJh5crgBrZg0Z+P2pv+f01QdzmrG4emqVAsyTwO2h3LWzo3bJJTqwSguOX+/IZxASmHMSzH4NlOcmBA7QM91+D6rOsEtmyR9YQZSkxyIktoYyIDMVEruknGFStyb0qr0rlRl/0EZQPH3RbTL/CpPUXqcgzqAPPbSwaVXqrVZL23V9/AgNfIe5/zu2QBbUzk0DRRK5OTwDXXAH/4Q+PjixYB/+2/QX6qmusWkVKqeU9Hvb4wmuv3eJBW86eyaVgZGBhYqAfstMf9PLqh4J4e4NiMxM/WDeJDe10HnJsslhIQnQsnIGfrQGcnBMJP8OqG7nOaEicKov3fmMryJSHExUKI54UQvxFCfCGNYxJZo1pVUcTrxAnILw1hcDC/D3knsLgNDqqY5Pd4U5D1bJj8uZP8Nyp2tilvtR+y0x73+ev2W379dUB0CFz0S88B555A3Nh4AuOdgziAM/0P5rOPupQqCI+PL5y7cxExv4c0kY2CusthbgA6AbwI4L0AFgPYB+DfBv0Oh6apcDTzxHWI3FYDJZ4j9hkzPrG4W169aEI7fxpnw43AzS90e1q6yhTWZxeGsB9EtDlim0s9piXKJiBkFX0cDfpmmBuATQAect3/IoAvBv0OAzEVjia6/CsquX7Q65KnPvQh9dU9F9rTM1/bIfAcXltR8Z0/jVuUQZscdU/AlcTGjVIODMyXI3Tmk4cxLHdtmZD1Xp8Gani3zCxToIqbPEdWyDQQfwLA/+m6/xkAf+Pzc9sATAGY6vWupKfyKkuWjE90eR3dcism5j8U8zpVXY+oZXZwjOxv3YYbrQKd9rUIiCQLQXjuR7z3QwTUVj3iqH8jm96+NteUplDMB2L3jT3iNlG2dSMTqmdWh5D/iorcion5D0EbTrXVDlL1Xs0PtNgw2f0B7xeIIweBgLHVMD0+3a+3ClRR/0Z5/k3DDje3w9B7iWUaiDk0Tf4sLzgQVdAHva7aYJ6nquvwOr3WXVsm5OuIH1ny6pEFBaVWgTro+1Hfjnm9faMONxdh6N2mkQSLZBqIuwD8FsBZrmStc4N+h4G4TViyEUaadB+augCY56nqAof7FnW+1cvkHGXYCwFdII/6dszj7Rv14qYIPWIbRocslV0glioYXwLgX+ayp4da/TwDcZuwtUec8HLd78PRhlPVfQCm3XsymbWbJBDZ2iMOe05FmSO24f+CpbINxFFvDMRtwsZL44zaZMupuq8xFuru2tt7iiPu0Kztc8RhzqkIWdMlHAhLCwMxGWLbZFGGl+umT9X9Ie1Oqtq40d7eU1RJh2ZtzJqOek62ryNmj1iLgZhISlnay3W/QOvUUXA237Ct9xQ1oKQ1NGv6gsmtKMPNUdgyOmQhbUzsymCzLiJ79fb6b4Tss2Vikbi3ohwfX9gb2rsv9HzpQcPi7AkthPq+U8fYfc7Ll4c7L29p6QMH1H3ASGnpVM7JNs7rmKByZfsJitJZ3dgjJmNKfrneaq7Rht5g0l5gkqHZVtWm4h43KduHmykV2piYStEHKjhPIQBM+hcCSP13TahWgZ07VVV4IdTXnTtLcbnu9Crd3IUfnN7ggQPqMac3mORP5hxbd9+P0+ur1VTPvaNDfXX3Cv2exzm204N23w9LV5DiwIHG18p5LfMq5uE9hyL2hCmBoCid1Y09Yosk6SGWvHdZBGF3lYq1xKpFNy1pBm+UTOG5ragbEtE2bow+3617DZwNWZLM07JXSy1oYyIDcbtLkuJoKj3ShvFVC3gD4fbtqgrT9u3qfn1iQk73qC05ZaUiP4UJ3z+Xb55aiyibxvBy2LWz7i013YWa3IloYQVdOybJxi7CsiIyjoGYNJJkEZvIQGYvXEoZIhD6vE7/n+ie3x878LopZJSNu5Qozm5SflWfogZhh+46TtdDb9XTLUPmM69tc8FATBpF6xFzkeK8wECoeZ0OiEq4a5iQUTbu5hpRepDOcLT3dNIMcLrT3b49XDuLsPWkDq9tc8NATBpFmyMu6TrguLSBUPM61SHC93xaRNmkwSfMnOrwsOr5fv7zzacTt0fs1w5dj7a/3/9xXc89qwuFLPHaNjcMxBQgybhU3mNa/NSYF6dHHPp1ahFl8xiO9RuSdgJj2sFY10N3esStLjZS6xEbyPjitW1uGIipJDiOJqWMN0cc+nUKGWXzSFBy7xDmDr6f/7x63ElMcyS5LtTFwFY93dQuSgxlfPHaNjcMxFQizCyRUob43E7yOnkOPnFPXX6rpyZHMNyU4OSWRQfOLxDW683ztxMTUnZ1pXuNFranmziGup7ouS01WemtyzuwcD/LnjGvbXPDQEwWShpQGZCzDYROEJ7/oK7n/kGtC4Szs829TmctcFo9uzjZ3UH3wzzhc1saT3YMNdm9tJ75a83/SrlgICbLJL0M52V8bmIPXSaMTK0CoTsYB93cc51Rg2Peo8WV3sbuv3PxE/diggHWKtqYKKSUue/mtX79ejk1NZX785JF+vr8iy9UKsD+/dn/fgakbNya0Hu/qDo61Ll4CQHU65pf0lV1OHQI+OUvQ1cDaFUcQkrVviBOnQ/v74aV299VSox3DKKG8fmH7kANgxiDEEL/Wmt4C1wAQHd3aXZ0LSL9uyYoSmd1Y4+YEqdqWpbqGbrnVMAuSqytMXVd2RiTuEFJVN4esffwgJRbthRgkw1XA8dQk0BdjmHhfqU3eoOZhGUdbUxkICYzkn5KWPQpE3ousaDD6bGa7RclU5zEDXrNe3rUdU5vrwrCrRKtrDE8LJ/bouaEVXtVML6tazjWW8Sya1ViIC65AvayyjZHHCq7tsB7c8c6hDfdOeXIEGYUonCbbNTrja91b/xELYuuVUlhIC4tywJSJCXLmm75od9Oe3Nn3CN2P43ufpRNNgzso5G5In80lBQDcWnxstcK1vaINc95qLMSvXcbdN/7vRTniOOIsvSozJWTLLtWbXcMxKXFiSDjrJ4j1rw/ZiHCNyFOpNL9zhVX5BYZwg5dF71yEhUGA3FpsUecTEpdBmuzpjXvj39FJdzbJUmksmC8N0wTUtsnmigYA3FpcSIovpRfu8zjTpwn8DnH19FclzhwAKUNIlWopC4LLiyo0LQxscVSeLJetapW6FcqapeBSoUr9sMaGmrc7QBQ94eGYh3Ou8lDqps+jIyo3SikVPelVPdb7UzheX8c7qzgP2EnvofG90dv78Jh3aSEOpGxscZvODtslIDzUrq5X2oA8V9/ohAYiMugWlW7SdXr6iuDcDgHD0Z73BQp1fZS4+MLwWBwUN2fmWmOnl6u98f/3LUfP+xufH90d6sNrrSxZjhMpCom90tZq6n/QrVa40ud+PUnaiWou5zVjUPTZIUiza/rlgPFmGv2m6bWTwXX5cP95c5mCjW/3wbD85Q5bUzkXtNk3uSkGg4Ouf9wqs9bpM14pdRvrCwE8NnPAt/4RqLDOx09R60GjC0bgTgWsOFzCUgZYj9p7+tfr5dmeJ5ywb2myVKmk82KstDSr0fmudWFSNx+bdJSuycqsUdMyTFZiyyVcsJUZEWYX/dOZGp6YUJKzPxvQ/4JVxGexm1+njSLTLTJSVVFq6NDfZ2cTH7MhHybFGoimSg+BmIyy7aEKQuDA4RQw8C1GjA2Bnlmr/ZH33nsIG64IXpyb+6xxpkWcOoTHjig7ht8vbVN+m7j6z+fRV6rqcc5PE1JBXWXs7pxaJrmrVjhP9S6YkX+bTE9TN6KMww6MaGGoX1et+meSuy8qly3erQwUa5lk9p9eJ6SYrIWWerkk4Hp6ebHV6wAjh7Nty19faob5FWpqGFrm1x/PeQ3vwnh+v8r5xLNBp+oNidchVz2K8MkLaWho8O/my2E6o4bYGGTqFy0/5M4NE3R6YZv4wzrvvxytMezFGWYPO5EbErkf/0G7vmje7AfFdQhsB8V3PPBncCnqon23sh0UxK3Xs3wuu7xHNjQJBtnRigHQd3lrG4cmi4w3fDtddfFG9a1aYgybFsMl+sJ2v55YEDdrE/utXAawHSTTD8/ZU4bExmIKRpdsOrsjBdQbfr0CdMWS8r1+F0LDAxIuXGj8aaFZ+HSMZNNsumalDLBQEwp0ZVd1N3ClGO06QM5TFuyXlMa8vXwyx0qc23dsmNF09LTxkQma1E0uoSmzk5gdrb5cRsTndIgZfhdlmSEDKgUdvuK8nRkjyLlClIsTNailIyOqsDg1t2tgoff46Oj+bUtjjjZMdJn54tly/x/N2rVnhQ2OMkt4YpSpfuvZft/IUpBUHc5qxuHpgvOPXS6YoW6ef9teog5jDjz0+5h6a6uxt/t6ko+n2zx+CSX0WbPplkaSh3niMkjjf/xNiVaReGcu25eu1V2zPCwqnwU5nejzidbmrGjm3v+v69g5CAKiYGYXNIKoJYGjUB+5x6n9xml56qtpODjuuv8j3vddfHPOSFdx34rJuSbXQW8ECMyg4GYXNIKoBYPo2oF9YSjvA5hX8OS9Ij9TmO6J1xbOdxKJKUMiIlM1mpHaRVasGEroqhanWPY7JgwmTVSRq+kYFsRjDlOnQO3d73euq0W1nYwz/t393sfUFthIG5HaQXQIqZ5Bp1jpRJ+mVC1qn62UlFRyu93PVWTQlXtsfTixrmmcHvlHa3barrKpXWiZtFTewjqLmd149C0YWkmWaWV9JXX2KWJBLMo6cYmE+A0f4ckc8Q2zl4Yy/62ZFc2MkYbExmI25UtE3cmAo8t565jon0t/g5xs6Ztm/I2vvNY1ruykc20MZE7a5FZ3E7IDiH+DlJG37ErhY3CUuOdsh8ba76fy+YnUobflY3KhDtrkaUsTU5qOyH+DnF27AozlZ4X9xT9+LiKhUaCsHeyXZe4R22DgZiyEXbrSEuTk9pOhn+HalV1qut19dVEEHb4ZX/nHoSjZNFTW2AgJn9JKpTr1qxcf33zMYuYeV1GYZdjuRUwcBjtkMbJoqf2EDSBnNWNyVqWS5pApcvQ8abQOse0PXmqXQT9HYxnOSVnTdIyN+1uV0zWogiSJlB1dITvYjApyx66bCxrspySGxkBZmYWmuyc2vLlXMpLmdP+J0kUiIUQnwQwAuAcABuklKGiKwOx5XSBVAg1r9WKLpD7CXtMylarCOUOxo4CBeHJSbWJyMGDwJlnAl/96sJcdZjsb6IUZJY1/TSAKwD8LOFxyCZJE3dGR8N/sjEpyzwpVRB2Jw05QXdmZiFSGctySsabsnDwYOM2mwU4BSq5RIFYSvmclPL5tBpDlkiaQFWthhuaZlKWHcKs6ynwshtus0m2Y9Y0NUtj8Wel4v94Z+fCMa+6Sn0axsnMpgZJktwBBPd4LVx2E+V8uVSdrBeUyTU3f/xTqCFo7+0y1888DGB9i+NsAzAFYKq3tzefHDUyp1Xmtck9lUsmlZey1daLFmVNRz1f27bZpLalj49B3wx7CxOI3TcuX2oTQcth+OmYmsQvZdh1PZYsu4l6vqle83GpHcWnjYld+fa/qa1Uq/rhbI4XpibxS6nbaAJo3Ggizh6XGYh6vs5b0Mma7u1VqQmRd/jybpztbFTjfhKiGJIuX7ocwH8BcAqAGQB7pZRbWv0ely9RqLXK7jUnJ52kHnv55QSfpOWUWt0M3TpiyxirE8ICJQAa/1vyv2Ik2SxfklLeJ6U8Q0r5Ninlu8MEYSqAxJk/IbTKzPauOZmeVjfp2jKTyV0AYiS56/6+lvR4WzG2K6pubXzYNfMloNu9lv8VEwoat87qxjlii+WZRBVnDrmo88lh5xZjzkGG/rWSJMnlPlVbr0vZ0+P/PuzpaZttKpnakYg2JnKLS2pky/BbmG0yi7IrV9iivHkU77Xl71tEk5PA1VcDb7218FhXF/Dtb7fN2GzSTffaXDZbXMbFQGwxW/6nhdkmsyjBI2zwyyNI2vL3LarJSeDTn164PzHRNkEY4HVcQpltcUllo9tysqMj34kgv4lAt8WLi7MrV9g03zwyyVn/OT4pgSeeaHzsiScKsbtYWli1NBsMxNRIFwBnZ9PLygiTDObs7tXZ6X+Mnp7i9ETCBr88giQ/SeOR9u0uZkIam+6Rj6AJ5KxuTNay3MSElJ2d2WRlRE0W8tYwdtc2Loqw55xXIhU3pYjHot3FqJC0MZGBOA1l/GDLKgBGTbssS5pmxlnTlBNLdhejQmLWdGbyyHQ1IausjKjJQmV9fYmo3TBZKzNlrbGW1Vxi1HlQTkoRUckxECdV1j2TswqAcQJ8tap64fW6+sogTEQlwkCcVJmXg2QRANnDJSJqwECcFJeDRJdHDzeP/bKJiFLAQJwUe3j24c70RFQgzJqm8uE+fERkH2ZNUxtJI4GOQ9tElBMGYiqfpAl0HNomohwxEFP5JE2gK+vacCKyEgMxlU/SBLqyrg0nIisxEFM5RV0i5Z4T7tD8t8hibTjnoonaXpfpBhAZ593Pena2+WeyWBvufV5nLhrg8jeiNsIeMdkpz56i35wwoGohZ7k2PO5cNHvRRKXCHjHZJ6+e4uSkCnp+a44BNaztVxEqLXHmotmLJiod9ojJPmF6ikl7he4lSjpZ7xceZ5kVM7qJSoeBmOzTqqeYxjpf3XC0I4/9wnXLrC65RH+RwYxuotJhICb7nHSS/+NOTzGNXmFQ4Mprv3C/ZVZXXQXs2qW/yChztS+iNsVATHaZnARefbX58cWLF3qoafQKdYHL2Y86r/lW7zKrBx8MvshgtS+i0mEgpkamM3KHhoATJ5of7+lZCI5p9Ar9ApoQaljYpFYXGWlU+/IWejFQ+IWIFjAQ0wIb9ljWBaKXX174dxq9wmpVDQMLV0EUKdWwsMnlQGEuMpLUcx4ZAQYHF4KvlOr+yEi89hJRYgzEtMCGjNywgSiNGtAPPtjcGzSdgRz1IiPKCIaUwMwMMD6+EIwHB9X9mRn2jIlMkVLmflu3bp0kCwkhpfo4brwJkV8bJiak7O5ufP7ubvV42mw4Xz8TE1JWKqodlYr+3OO8VvW6lLVa4+/UaupxIsqSNiYKaeAqEU9b+AAAC7tJREFUeP369XJqair356UW+vr819U6CUx5cTbaOHhQ9YRHR7NJnrLlfOOK234pG/fTrtcbh+iJKAva/2QcmqYFceZes0juSjIHGkXRM5DjZI87w9Fu7jljIsodAzEtiDr3mldyV1aZ3GnNNZsSNXvcPSdcq6kLnVqtcc6YiHLHoWmKL+2hXb8haaBxb2VA9VqLFDCz4t13Gmj92oyMqMSssTF18eEE5+XLmTntktfsCLUV7dA0AzHF19Hh34sSInqxBF1QWboUmJ5u/vmizONmLU7EkLJ52RbniOfFub4hCoGBmDKQZo9YdyydOMHeNHazCqHoOXxkLSZrUQbSTHaKWrSgaHsr27BZCoXCuhqUNwZiii/NZCddYF2xotiZzQ4bNkuhUFhXg/LGQEzJpLXUSNe7Hh8vdmazI0w3y/Q+3wSg+KvaqHgYiMkOQb3rvNYVZ6lVN8tv6Pozn1GvBYNyroq+qo2Kh8laRHlolYrbKlmNabtERcdkLaKWshwabtXNapUJxPlkotJij5gIML94NMzyrSIu2SIiB3vERIFMZzX7ZQh5MW2XqJQYiIkA84tH3UPXQPNOV0zbJSotBuK8cYmKnWxYPOpkh0sJ3HMP03aJ2kSxA3HRghp3V7KXbYtHy7Bki4hCKW4gLmJQMz0PSXpcPEpEhhQ3a7qIO7OnWa3IZixuQETkVcKsadPJNXHYMA+ZtSKOVBARGVTcQFzEoGbbPGQWOPxORBRJcQNxEYNaO8xDFnGkIoqiJQgSkfWKO0cMcC7SRkWcuw/L9O5bRFRk2cwRCyH+WgjxayHEU0KI+4QQy5McLzIu8bBPEUcqwirSsDt77kSFkXRo+n8AWCWlXA3gXwB8MXmTqNDKPPxelGH3PBPmGPCJEkttaFoIcTmAT0gpW37isugDFVJRht3zaieH6omiyGX50jUAfpzi8YjsUpRh97x67kUaqieyWMtALIT4qRDiaZ/bZa6fGQLwFgDtuJQQYpsQYkoIMXXkyJF0Wk+Up6IMu+e1tK8oQ/VElks8NC2EuBrA/wrgo1LK4y1+HACHpokyldeQcVGG6onskFnW9MUAbgbw8bBBmIgyllfPvShD9USWS9QjFkL8BsDbAEzPPfSolPKzrX6PPWKikuBafqKwtD3iYm/oQUREVAwlLPpARERUAgzEREXGDTWICo+BmMqr7EGKJSeJSoFzxFRO7bDrE5cPERUJk7WozbRDkOroUD1hLyFUIRQisgmTtajNtMOuT3ntoEVEmWIgpnJqhyDFDTWISoGBmMqpHYJUUfa+JqJAnCOm8uKuT0RkDyZrERERGcRkLSIiIhsxEBMRERnEQExERGQQAzEREZFBDMREREQGMRATEREZxEBMdnIqJwkBdHWpr2WsoEREba/LdAOImngrJ83Oqq9OmT+AG3MQUWmwR0z2GRpqLF/odvy4+j4RUUkwEJN9WlVIKlMFJSJqewzEZJ9WFZKSVlBy5p87OjjvTETGMRCTffwqJzmSVlBy5p8PHACkXJh3ZjAmIkMYiMk+7vJ+ANDZqb6mUebPb/6Z885EZBADMdmpWgX271e91l27VBA+eFAFzCS9V938MuedicgQBmKyW9pDybr55aTzzkREMTEQk93SHkr2m39OOu9MRJQAAzHZLexQcthMaPf8sxDpzDsTESXAQEz5i7J8KMxQctTha2f+uV5XXxmEicggBmLKV9SgGWYomZnQRFRgDMSUr6hBM8xQMjOhiajAhJQy9yddv369nJqayv15yQIdHaon7CWEGiqOo69P9ay9KhU19ExEZJ7QfYM9YspXFsuHmAlNRAXGQEz5yiJoMhOaiAqMgZjylVXQzDIT2tYiEba2i4gi4RwxURAny9udYNbdbb7HbWu7iEhHO0fMQEwUxNZEMFvbRUQ6TNaiArFpyNXWpVG2touIImMgJrvYVi/Y1iIRtraLiCJjICa72LZLlq1Lo2xtFxFFxkBMdrFtyNXWpVG2touIImOyFtmFSUhEVE5M1qKC4JArEbUZBmKyC4dciajNcGiaiIgoexyaJiIishEDMRERkUEMxERERAYxEBMRERnEQExERGQQAzEREZFBDMREREQGMRATEREZxEBMRERkUKJALIT4ihDiKSHEXiHET4QQ70mrYURERO0gaY/4r6WUq6WU/QB+BGB7Cm0iIiJqG4kCsZTyVdfdtwPIf+NqIiKiAutKegAhxCiAvwBwDMDmxC0iIiJqIy2rLwkhfgrgVJ9vDUkpf+j6uS8CWCKlHNYcZxuAbQDQ29u77oBf8XciIqJy0lZfSq0MohCiF8CDUspVrX6WZRCJiKjNZFMGUQix0nX3MgC/TnI8IiKidpN0jvg/CyHeB6AO4ACAzyZvEhERUftIFIillH+WVkOIiIjaEXfWIiIiMoiBmIiIyCAGYsrP5CTQ1wd0dKivk5OmW0REZFziDT2IQpmcBLZtA44fV/cPHFD3AaBaNdcuIiLD2COmfAwNLQRhx/Hj6nEiojbGQEz5OHgw2uNERG2CgZjy0dsb7XEiojbBQEz5GB0FursbH+vuVo8TEbUxBmLKR7UK7NwJVCqAEOrrzp1M1CKitpda0YcoWPSBiIjaTDZFH4iIiCgZBmIiIiKDGIiJiIgMYiAmIiIyiIGYiIjIIAZiIiIigxiIiYiIDGIgJiIiMoiBmIiIyCAGYiIiIoMYiImIiAxiICYiIjKIgZiIiMggBmIiIiKDGIiJiIgMMlKPWAhxBMCB3J84HycDOGq6ERnjOZYDz7EceI7FcFRKebHfN4wE4jITQkxJKdebbkeWeI7lwHMsB55j8XFomoiIyCAGYiIiIoMYiNO303QDcsBzLAeeYznwHAuOc8REREQGsUdMRERkEANxBoQQXxFCPCWE2CuE+IkQ4j2m25Q2IcRfCyF+PXee9wkhlptuU9qEEJ8UQjwjhKgLIUqTsSmEuFgI8bwQ4jdCiC+Ybk8WhBB3CyFeEkI8bbotWRBCnCmE+EchxLNz79Ga6TalTQixRAjxuBBi39w53mq6TVnh0HQGhBDvlFK+OvfvAQD/Vkr5WcPNSpUQ4o8A7JZSviWE+BoASCn/d8PNSpUQ4hwAdQD/B4C/lFJOGW5SYkKITgD/AuBjAA4DeALAVinls0YbljIhxEUAXgfwHSnlKtPtSZsQ4jQAp0kp/1kI0QPglwD+fZn+jkIIAeDtUsrXhRCLADwCoCalfNRw01LHHnEGnCA85+0ASne1I6X8iZTyrbm7jwI4w2R7siClfE5K+bzpdqRsA4DfSCl/K6X8A4D/C8BlhtuUOinlzwC8bLodWZFS/l5K+c9z/34NwHMATjfbqnRJ5fW5u4vmbqX7LAUYiDMjhBgVQhwCUAWw3XR7MnYNgB+bbgSFcjqAQ677h1GyD/B2I4ToA7AWwGNmW5I+IUSnEGIvgJcA/A8pZenOEWAgjk0I8VMhxNM+t8sAQEo5JKU8E8AkgM+ZbW08rc5x7meGALwFdZ6FE+YciWwlhHgHgL8DcINnJK4UpJSzUsp+qBG3DUKI0k0zAECX6QYUlZTy34X80UkADwIYzrA5mWh1jkKIqwH8CYCPyoImG0T4O5bF/wPgTNf9M+Yeo4KZmzf9OwCTUsr/bro9WZJSzggh/hHAxQBKl4DHHnEGhBArXXcvA/BrU23JihDiYgA3A/i4lPK46fZQaE8AWCmEOEsIsRjAnwO433CbKKK5RKZvAXhOSnm76fZkQQhxirMaQwixFCrBsHSfpQCzpjMhhPg7AO+Dyrg9AOCzUspS9TqEEL8B8DYA03MPPVrCzPDLAfwXAKcAmAGwV0q5xWyrkhNCXALgDgCdAO6WUo4ablLqhBDfA/BhqKo9/y+AYSnlt4w2KkVCiP8FwM8B/ArqcwYAviSlfNBcq9IlhFgNYBfU+7QDwPellDvMtiobDMREREQGcWiaiIjIIAZiIiIigxiIiYiIDGIgJiIiMoiBmIiIyCAGYiIiIoMYiImIiAxiICYiIjLo/wd991VJHcGyZQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 576x576 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UYYUmfoNIFzT"
      },
      "source": [
        "**Questions:**\n",
        "1. What differentiates subgroup a from subgroup b?\n",
        "1. How a fair classification boundary should look?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P82_grNMJ6Rm"
      },
      "source": [
        "### Train a vanila pytorch model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sj5WRgqUJQTh"
      },
      "source": [
        "# data parameters\n",
        "n_samples = 1000\n",
        "n_samples_low = 200\n",
        "\n",
        "# Prepare the dataset\n",
        "X, y, _,_,_ = generate_toy_data(n_samples, n_samples_low, n_dimensions=2, seed=0)\n",
        "X_test, y_test, _,_,_ = generate_toy_data(n_samples, n_samples_low, n_dimensions=2, seed=1)\n",
        "\n",
        "X = torch.tensor(X).float()\n",
        "X_test = torch.tensor(X_test).float()\n",
        "\n",
        "# Change labels from -1, 1 to 0, 1\n",
        "y = torch.tensor(y)\n",
        "y[y == -1] = 0\n",
        "\n",
        "y_test = torch.tensor(y_test)\n",
        "y_test[y_test == -1] = 0\n",
        "\n",
        "\n",
        "data_train = torch.utils.data.TensorDataset(X, y)\n",
        "loss       = torch.nn.CrossEntropyLoss(reduction='none')"
      ],
      "execution_count": 82,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yZMFj5c0KJ1Y",
        "outputId": "97533a68-d200-432e-dd4b-bde40978c07a"
      },
      "source": [
        "# Params\n",
        "num_epochs = 10\n",
        "lr = 0.02\n",
        "batch_size = 256\n",
        "\n",
        "model = torch.nn.Sequential(\n",
        "  torch.nn.Linear(3, 8),\n",
        "  torch.nn.Sigmoid(),\n",
        "  torch.nn.Linear(8, 2),\n",
        ")\n",
        "\n",
        "opt  = torch.optim.Adam(model.parameters(), lr=lr)\n",
        "dl   = torch.utils.data.DataLoader(data_train, batch_size=batch_size, shuffle=True, drop_last=True)\n",
        "\n",
        "\n",
        "for i in range(num_epochs):\n",
        "  print(\"Training epoch\", i)\n",
        "\n",
        "  for x, y_t in dl:\n",
        "    opt.zero_grad()\n",
        "\n",
        "    y_hat  = model(x)\n",
        "    losses = loss(y_hat, y_t)\n",
        "\n",
        "\n",
        "    # Backpropagate and optimize    \n",
        "    losses.mean().backward()\n",
        "    opt.step()"
      ],
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training epoch 0\n",
            "Training epoch 1\n",
            "Training epoch 2\n",
            "Training epoch 3\n",
            "Training epoch 4\n",
            "Training epoch 5\n",
            "Training epoch 6\n",
            "Training epoch 7\n",
            "Training epoch 8\n",
            "Training epoch 9\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T9VItZiqLNj2",
        "outputId": "511a7a04-2d84-40ce-9965-9d8cfb328cc6"
      },
      "source": [
        "# Evaluate model\n",
        "\n",
        "with torch.no_grad():\n",
        "  y_hat = model(X_test)\n",
        "\n",
        "y_class = y_hat.argmax(dim=1)\n",
        "\n",
        "\n",
        "print(\"Number of missclassifications\", (y_class != y_test).sum().item(), \"out of\", y_test.shape[0], \"or\", (y_class != y_test).sum().item() /  y_test.shape[0])"
      ],
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of missclassifications 303 out of 3200 or 0.0946875\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8b0A682FLg-j",
        "outputId": "d707392f-bb38-4abd-91c6-1a9541fdb553"
      },
      "source": [
        "y_mask  = (y_class != y_test)\n",
        "x_group = X_test[:, 2][y_mask]\n",
        "\n",
        "print(\"Number of missclassifications for subgroup a\", (x_group < 0).sum().item(), \"out of\", y_test.shape[0], \"or\", (x_group < 0).sum().item() /  y_test.shape[0])\n",
        "print(\"Number of missclassifications for subgroup b\", (x_group > 0).sum().item(), \"out of\", y_test.shape[0], \"or\", (x_group > 0).sum().item() /  y_test.shape[0])"
      ],
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of missclassifications for subgroup a 181 out of 3200 or 0.0565625\n",
            "Number of missclassifications for subgroup b 122 out of 3200 or 0.038125\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pcw59o8SR6wl"
      },
      "source": [
        "**Questions:**\n",
        "1. The subgroup attribute is part of the training data. Would the results change if we remove it?\n",
        "1. How can we fix this and make the model more fair?\n",
        "1. In this case we have two subgroups and want to be fair to both. But what if we want to be fair to each sample, how can we do that? "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y9K7SNumLJj3"
      },
      "source": [
        "## Definition and fainess metrics\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yAWOUZCkLPlY"
      },
      "source": [
        "### Notation:\n",
        "\n",
        "To simplify things we will focus only on classification. Although, the methods work for both, regression and classification.\n",
        "\n",
        "\n",
        "- $X$ - input ($x$ one input sample)\n",
        "- $Y$ - labels ($y$ one label sample)\n",
        "- $S$ - sensitive attributes which are part of X (e.g., gender, age, race)\n",
        "- $f$ - model we train, i.e., $f(x; \\theta)$ returns the probability of each labels\n",
        "- $\\theta$ - model parameters, e.g., the weights and the biases for NN\n",
        "- $\\ell$ - loss function, here we will always use cross-entropy ($-log f(x; \\theta) [y]$)\n",
        "- $\\mathbb{E}[\\ell(f, X, Y)]$ - loss value, i.e., average loss\n",
        "- $\\min\\mathbb{E}[\\ell(f, X, Y)]$ - optimization objective"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OYegnLkBP7hY"
      },
      "source": [
        "### Definitions of perfect fairness\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q2VCxV6DVkma"
      },
      "source": [
        "\n",
        "- *Demografic parity:* $S \\perp f(X; \\theta)$\n",
        "- *Equaized odds:* $S \\perp f(X; \\theta) \\mid Y$\n",
        "- *Lack of disparate mistreatment:* $\\mathbb{P}(f(X; \\theta) \\neq Y \\mid S = s_a) = \\mathbb{P}(f(X; \\theta) \\neq Y \\mid S = s_b) , \\forall s_a, s_b \\in S$\n",
        "\n",
        "**Questions:**\n",
        "1. What is the difference between the first two from the perspective of application?\n",
        "1. What is the problem with these definitions?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qP5413u4VoIY"
      },
      "source": [
        "### Approximate fairness"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U1ujakf4hVsp"
      },
      "source": [
        "- *Mean difference score:* $\\max \\mid \\mathbb{P}(f(X; \\theta) = y \\mid S = s_a) - \\mathbb{P}(f(X; \\theta) = y \\mid S = s_b) \\mid, \\forall s_a, s_b \\in S, \\forall y \\in Y$\n",
        "\n",
        "- *Covariance:* $\\mathbb{E}[f(X; \\theta) \\cdot S] - \\mathbb{E}[f(X; \\theta)] \\cdot \\mathbb{E}[S]$\n",
        "\n",
        "**Questions:**\n",
        "1. Why these two metrics are approximate\n",
        "1. Why we use the approximate metrics instead of perfect metrics of fairness?\n",
        "\n",
        "[Correlation Wikipedia](https://en.wikipedia.org/wiki/Correlation)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wIA5eMy33r0E"
      },
      "source": [
        "## Conditional Value at Risk method (simplified)\n",
        "\n",
        "**Idea:** what if we minimize each subgroup separately to match the accuracy of the other subgroups.\n",
        "\n",
        "**Question:** How to do that?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C0MQwDSl4Sb0"
      },
      "source": [
        "Let:\n",
        "- $L$ be $\\mathbb{E}[\\ell(f, X, Y)]$ be the average loss for each subgroup.\n",
        "- $L_{s_a}$ be $\\mathbb{E}[\\ell(f, X, Y) \\mid S = s_a]$ be the average loss for subgroup $s_a$.\n",
        "\n",
        "**Then at each step optimize only subgroups that have $L_{s} > L$**\n",
        "\n",
        "Let's see how we can code that."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pRgdKY1zv0zM",
        "outputId": "3e75d830-dacc-431b-d8ae-421e606273b9"
      },
      "source": [
        "# Params\n",
        "num_epochs = 10\n",
        "lr = 0.02\n",
        "batch_size = 256\n",
        "\n",
        "sensitive_idx=2\n",
        "\n",
        "model = torch.nn.Sequential(\n",
        "  torch.nn.Linear(3, 8),\n",
        "  torch.nn.Sigmoid(),\n",
        "  torch.nn.Linear(8, 2),\n",
        ")\n",
        "\n",
        "\n",
        "opt  = torch.optim.Adam(model.parameters(), lr=lr)\n",
        "dl   = torch.utils.data.DataLoader(data_train, batch_size=batch_size, shuffle=True, drop_last=True)\n",
        "\n",
        "\n",
        "for i in range(num_epochs):\n",
        "  print(\"Training epoch\", i)\n",
        "\n",
        "  for x, y_t in dl:\n",
        "    opt.zero_grad()\n",
        "\n",
        "    y_hat  = model(x)\n",
        "    losses = loss(y_hat, y_t)\n",
        "    l_mean = losses.mean().detach()\n",
        "\n",
        "    groups = x[:,sensitive_idx].unique()\n",
        "    sum    = 0\n",
        "    for c in groups:\n",
        "      mask = (x[:,sensitive_idx] == c)\n",
        "      sum += (losses[mask].mean() - l_mean).clip(0.0)\n",
        "\n",
        "    (sum / groups.shape[0]).backward()\n",
        "\n",
        "    opt.step()"
      ],
      "execution_count": 243,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training epoch 0\n",
            "Training epoch 1\n",
            "Training epoch 2\n",
            "Training epoch 3\n",
            "Training epoch 4\n",
            "Training epoch 5\n",
            "Training epoch 6\n",
            "Training epoch 7\n",
            "Training epoch 8\n",
            "Training epoch 9\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n8NCqMhOV8__",
        "outputId": "3da6f43d-99e4-4182-adee-7aa825349b81"
      },
      "source": [
        "# Evaluate model\n",
        "\n",
        "with torch.no_grad():\n",
        "  y_hat = model(X_test)\n",
        "\n",
        "y_class = y_hat.argmax(dim=1)\n",
        "\n",
        "\n",
        "print(\"Number of missclassifications\", (y_class != y_test).sum().item(), \"out of\", y_test.shape[0], \"or\", (y_class != y_test).sum().item() /  y_test.shape[0])"
      ],
      "execution_count": 244,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of missclassifications 365 out of 3200 or 0.1140625\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3u_uovdoWG1m",
        "outputId": "e4a67068-a286-4364-b6a8-827be8b64e1d"
      },
      "source": [
        "y_mask  = (y_class != y_test)\n",
        "x_group = X_test[:, 2][y_mask]\n",
        "\n",
        "print(\"Number of missclassifications for subgroup a\", (x_group < 0).sum().item(), \"out of\", y_test.shape[0], \"or\", (x_group < 0).sum().item() /  y_test.shape[0])\n",
        "print(\"Number of missclassifications for subgroup b\", (x_group > 0).sum().item(), \"out of\", y_test.shape[0], \"or\", (x_group > 0).sum().item() /  y_test.shape[0])"
      ],
      "execution_count": 245,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of missclassifications for subgroup a 173 out of 3200 or 0.0540625\n",
            "Number of missclassifications for subgroup b 192 out of 3200 or 0.06\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kGOOxXRW5zEr"
      },
      "source": [
        "**Questions:**\n",
        "- What do we observe?\n",
        "- Why do we call `.detach()` on `l_mean`?\n",
        "- Why do we clip to `0.0`?\n",
        "- Can we control the fairness, i.e., put more weight on fairness property?\n",
        "- By looking at the accuracy for each subgroup can you tell which one is which?\n",
        "- Can we characterize more precisly the fairness amongst subgroups?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "QvF4-_fVWJ5R",
        "outputId": "de620a6c-af99-40a6-dfe9-58c6c686e13f"
      },
      "source": [
        "# Select predicted prob by subgroup\n",
        "pred_prob   = torch.tensor([y[idx].item() for y, idx in zip(y_hat.softmax(dim=1), y_test)])\n",
        "pred_prob_a = pred_prob[y_mask][x_group < 0]\n",
        "pred_prob_b = pred_prob[y_mask][x_group > 0]\n",
        "\n",
        "plt.hist(pred_prob_a.numpy(), alpha=0.5, bins=30, density=True, label=\"subgroup a\")\n",
        "plt.hist(pred_prob_b.numpy(), alpha=0.5, bins=30, density=True, label=\"subgroup b\")\n",
        "plt.legend()\n",
        "sns.despine()"
      ],
      "execution_count": 248,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAARXElEQVR4nO3df3BVdXrH8c8DZBsVtE6MIAYMdPhZgoVeUhydlbuKwxoEHfHXYKeM7jDUdbrdrpTqymjtH9LWXduZ7ujidsft1CIFRKzO0lKXqKgYQwpYwbWAaCMdSNIRNnRYhTz9I9cIGLwn5Pz45t73aybDvdxzz3m+91w+c/iec56YuwsAEK5BWRcAAPhqBDUABI6gBoDAEdQAEDiCGgACNySJlc6ZM8c3btyYxKoBoFTZmV5I5Ii6vb09idUCQFli6gMAAhdp6sPM9kv6laQTko67ey7JogAAX+jLHHXe3ZnTAICUJXIysTefffaZWltbdezYsbQ2WbYqKytVU1OjioqKrEsBEIOoQe2S/s3MXNKP3X3l6QuY2WJJiyVp9OjRX1pBa2urhg0bptraWpmd8eQm+snd1dHRodbWVo0ZMybrcgDEIOrJxKvcfbqkb0r6tpl9/fQF3H2lu+fcPVddXf2lFRw7dkxVVVWEdMLMTFVVVfzPBSghkYLa3T8u/HlI0npJ9WezMUI6HXzOQGkpGtRmdp6ZDfv8saTrJP1n0oUBALpFmaMeLml94ShtiKR/cvd+33b4+Kb3+7uKU3x39vhY1/fwww9r6NChuu+++2JdLwD0VdGgdvd9ki5PoZaScuLECQ0ePDjrMgD01+ZHoy+bvz+REsrmzsSjR4+qoaFBl19+uaZMmaLVq1dLkmpra3tueW9ubtasWbN63rNjxw5dccUVGjdunJ566ilJUldXl+655x5NnDhRs2fP1vXXX6+1a9f2rGvZsmWaPn261qxZo1WrVqmurk5TpkzRsmXLetY7dOjQnsdr167VokWLJEmLFi3SkiVLlMvlNH78eL344otfGkdnZ6euueYaTZ8+XXV1ddqwYUOsnxOA8KR2HXXWNm7cqJEjR+qll16SJB0+fLjoe3bu3KmtW7fq6NGjmjZtmhoaGvTGG29o//792rVrlw4dOqRJkybprrvu6nlPVVWVWlpadODAAc2cOVPbtm3ThRdeqOuuu07PP/+8brzxxq/c5v79+9XU1KS9e/cqn89rz549qqys7Hm9srJS69ev1/nnn6/29nbNnDlT8+bN4wQiUMLK5oi6rq5OmzZt0rJly/Taa6/pggsuKPqe+fPn65xzztFFF12kfD6vpqYmbdmyRbfccosGDRqkESNGKJ/Pn/Ke2267TZL09ttva9asWaqurtaQIUO0cOFCvfrqq0W3eeutt2rQoEEaN26cxo4dq/fee++U191dDzzwgKZOnaprr71WH3/8sQ4ePNiHTwLAQFM2QT1+/Hi1tLSorq5ODz74oB555BFJ0pAhQ9TV1SVJX7r2+PSj1ChHreedd17RZU5eT1+3+cwzz6itrU3btm3T9u3bNXz4cK6ZBkpc2QT1gQMHdO655+rOO+/U0qVL1dLSIql7Xnnbtm2SpHXr1p3yng0bNujYsWPq6OhQY2OjZsyYoSuvvFLr1q1TV1eXDh48qMbGxl63V19fr1deeUXt7e06ceKEVq1apauvvlqSNHz4cO3evVtdXV1av379Ke9bs2aNurq6tHfvXu3bt08TJkw45fXDhw/r4osvVkVFhTZv3qwPP/wwjo8HQMAym6OO+3K6Yt555x0tXbpUgwYNUkVFhZ544glJ0kMPPaS7775by5cvP+VEoiRNnTpV+Xxe7e3tWr58uUaOHKmbb75ZL7/8siZPnqxRo0Zp+vTpvU6jXHLJJVqxYoXy+bzcXQ0NDZo/f74kacWKFZo7d66qq6uVy+XU2dnZ877Ro0ervr5eR44c0ZNPPnnK/LQkLVy4UDfccIPq6uqUy+U0ceLEmD8pAKExd499pblczpubm0/5u927d2vSpEmxbysLnZ2dGjp0qDo6OlRfX6/XX39dI0aM6Pd6Fy1apLlz52rBggX9Xlcpfd5AptK7PO+Mc6tlc9VHnObOnatPPvlEn376qZYvXx5LSAPAmRDUZ+FM89L99fTTTyeyXgADW9mcTASAgYqgBoDAEdQAEDiCGgACl93JxL5c8hJFzF2rQm1z2tjYqMcee6zXhk1ASYuaGQl1sMsSR9QJOXHiRNYlACgRZRPUpdLmVJKOHDmihoYGTZgwQUuWLOnpVQKgNJXNddSl0uZUkpqamrRr1y5ddtllmjNnjp577rlY7mYEEKayOaIulTanUnfDp7Fjx2rw4MG64447tGXLloifAoCBqGyCulTanJ5tXQAGrrIJ6lJpcyp1T3188MEH6urq0urVq3XVVVf19+MBELDs5qhTvoSmVNqcStKMGTN07733as+ePcrn87rpppti/KQAhIY2p2eBNqdABrK6jpo2pwMTbU4BpImgPgu0OQWQplRPJiYxzYIv43MGSktqQV1ZWamOjg5CJGHuro6Ojl5PQgIYmFKb+qipqVFra6va2trS2mTZqqysVE1NTdZlAIhJakFdUVGhMWPGpLU5ACgZZXPDCwAMVAQ1AASOoAaAwBHUABA4ghoAAhc5qM1ssJn9h5nxy/oAIEV9OaL+jqTdSRUCAOhdpKA2sxpJDZJ+kmw5AIDTRT2i/htJfyrpjL9F1cwWm1mzmTVz9yEAxKdoUJvZXEmH3H3bVy3n7ivdPefuuerq6tgKBIByF+WI+kpJ88xsv6RnJX3DzP4x0aoAAD2KBrW73+/uNe5eK+l2Sb9w9zsTrwwAIIlfHACUvqx+hRVi06egdvdGSY2JVAIA6BV3JgJA4AhqAAgcQQ0AgSOoASBwBDUABI6gBoDAEdQAEDiCGgACR1ADQOAIagAIHEENAIEjqAEgcAO2e97jm96PtNx3Z49PuBKgNJTdv6moXQUDwBE1AASOoAaAwBHUABA4ghoAAkdQA0DgCGoACBxBDQCBI6gBIHAENQAEjqAGgMAR1AAQOIIaAAJHUANA4AZs9zwEKmpHsvz9ydaB7PFdiE3JB3UptW4spbEAiI6pDwAIHEENAIEjqAEgcAQ1AASOoAaAwBUNajOrNLMmM9thZu+a2Z+nURgAoFuUy/N+Lekb7t5pZhWStpjZz919a8K1AQAUIajd3SV1Fp5WFH48yaIAAF+INEdtZoPNbLukQ5I2uftbyZYFAPhcpKB29xPu/juSaiTVm9mU05cxs8Vm1mxmzW1tbXHXCQBlq09Xfbj7J5I2S5rTy2sr3T3n7rnq6uq46gOAshflqo9qM/vNwuNzJM2W9F7ShQEAukW56uMSST8zs8HqDvZ/dvcXky0LCADd3xCIKFd97JQ0LYVaAAC9KPk2p1nKqi0p7VCB0sIt5AAQOIIaAAJHUANA4AhqAAgcQQ0AgSOoASBwBDUABI6gBoDAEdQAEDiCGgACxy3kCFsSjZGirhMDUwnuX46oASBwBDUABI6gBoDAMUddQGtQIJrI/1YSSJc393VEWu6KsVXZbDcf62Z7cEQNAIEjqAEgcAQ1AASOoAaAwBHUABA4ghoAAkdQA0DgCGoACBxBDQCB485ERBN6R7LQ65MGRo0IEkfUABA4ghoAAkdQA0DgCGoACBxBDQCB46qPAETt7xu6qD17peT69qbtzX0d2no82v6b+VE2vZSzEvX7EPfnV4o4ogaAwBHUABC4okFtZqPMbLOZ7TKzd83sO2kUBgDoFmWO+rik77l7i5kNk7TNzDa5+66EawMAKMIRtbv/j7u3FB7/StJuSZcmXRgAoFuf5qjNrFbSNElv9fLaYjNrNrPmtra2eKoDAEQPajMbKmmdpD929yOnv+7uK9095+656urqOGsEgLIW6TpqM6tQd0g/4+7PJVsS4jDzo5XFF9pcJeXvT74YDAiRvjOSto5enHAlOF2Uqz5M0t9L2u3uP0y+JADAyaJMfVwp6fclfcPMthd+rk+4LgBAQdGpD3ffIslSqAUA0AvuTASAwBHUABA4ghoAAkeb0z4qlZakffH4pvfLrsVkX1q2ZrXtqO1QsxxLVuL+DLPGETUABI6gBoDAEdQAEDiCGgACR1ADQOAIagAIHJfnDTBRO5xFtvnRCNtM4PKuCNtFmGL/DqIojqgBIHAENQAEjqAGgMAR1AAQOIIaAAJHUANA4AhqAAhccNdRl2MbUZxZ3C06B0pbS6RjoLSA5YgaAAJHUANA4AhqAAgcQQ0AgSOoASBwBDUABI6gBoDAEdQAEDiCGgACR1ADQOAIagAIHEENAIEjqAEgcAQ1AASuaJtTM/uppLmSDrn7lORLQloGSotH9I79Vz6iHFE/LWlOwnUAAM6gaFC7+6uS/jeFWgAAvYhtjtrMFptZs5k1t7W1xbVaACh7sf0qLndfKWmlJOVyOY9rvXGZ+dHKSMttHb046PUhPOy7/uHzK46rPgAgcAQ1AASuaFCb2SpJb0qaYGatZnZ38mUBAD5XdI7a3e9IoxAAQO+Y+gCAwBHUABA4ghoAAkdQA0DgCGoACBxBDQCBi+0WcqAvaNEJRMcRNQAEjqAGgMAFO/VBRy0A6MYRNQAEjqAGgMAR1AAQOIIaAAJHUANA4AhqAAgcQQ0AgSOoASBwBDUABI6gBoDAEdQAEDiCGgACF2xTJiAJ9MHGQERQJ4wugAD6i6kPAAgcQQ0AgSOoASBwBDUABI6gBoDAEdQAEDiCGgACR1ADQOAIagAIHEENAIGLFNRmNsfMfmlme8zsz5IuCgDwhaJBbWaDJf1I0jclTZZ0h5lNTrowAEC3KEfU9ZL2uPs+d/9U0rOS5idbFgDgc1G6510q6b9Pet4q6fdOX8jMFktaXHjaaWa/PG2RiyS1n02R6fpB3CscIONOBGMvT+U79m/9oD9j3+juc3p7IbY2p+6+UtIZe3qaWbO75+La3kBRruOWGDtjLz9JjT3K1MfHkkad9Lym8HcAgBRECeq3JY0zszFm9jVJt0t6IdmyAACfKzr14e7HzexeSf8qabCkn7r7u2exrXL9VSflOm6JsZcrxh4zc/ck1gsAiAl3JgJA4AhqAAhcrEFd7FZzM/sNM1tdeP0tM6uNc/tZijD2r5tZi5kdN7MFWdSYlAhj/xMz22VmO83sZTO7LIs6kxBh7EvM7B0z225mW0rprt6orSXM7GYzczMrmUv2Iuz3RWbWVtjv283sW/3aoLvH8qPuE417JY2V9DVJOyRNPm2ZeyQ9WXh8u6TVcW0/y5+IY6+VNFXSP0hakHXNKY89L+ncwuM/LLP9fv5Jj+ep+6aGzGtPY+yF5YZJelXSVkm5rOtOcb8vkvR3cW0zziPqKLeaz5f0s8LjtZKuMTOLsYasFB27u+93952SurIoMEFRxr7Z3f+v8HSruq/FLwVRxn7kpKfnSSqVs/dRW0v8haS/lHQszeISlnpbjTiDurdbzS890zLuflzSYUlVMdaQlShjL1V9Hfvdkn6eaEXpiTR2M/u2me2V9FeS/iil2pJWdOxmNl3SKHd/Kc3CUhD1O39zYbpvrZmN6uX1yDiZiNSY2Z2ScpL+Outa0uTuP3L335K0TNKDWdeTBjMbJOmHkr6XdS0Z+RdJte4+VdImfTGTcFbiDOoot5r3LGNmQyRdIKkjxhqyUs632Ucau5ldK+n7kua5+69Tqi1pfd3vz0q6MdGK0lNs7MMkTZHUaGb7Jc2U9EKJnFAsut/dveOk7/lPJP1ufzYYZ1BHudX8BUl/UHi8QNIvvDDzPsCV8232RcduZtMk/VjdIX0ogxqTEmXs40562iDpv1KsL0lfOXZ3P+zuF7l7rbvXqvvcxDx3b86m3FhF2e+XnPR0nqTd/dpizGdDr5f0vrrPiH6/8HePqHsHSVKlpDWS9khqkjQ26zO4KY59hrrnso6q+38R72Zdc4pj/3dJByVtL/y8kHXNKY79byW9Wxj3Zkm/nXXNaY39tGUbVSJXfUTc748W9vuOwn6f2J/tcQs5AASOk4kAEDiCGgACR1ADQOAIagAIHEENAIEjqAEgcAQ1AATu/wEOk/JV1MJB0AAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "yWmITtU0X3to",
        "outputId": "0b8fd4c2-46a2-4229-8c98-4e74b1fb450c"
      },
      "source": [
        "# Select predicted prob by subgroup\n",
        "y_mask  = (y_class == y_test)\n",
        "x_group = X_test[:, 2][y_mask]\n",
        "\n",
        "\n",
        "pred_prob   = torch.tensor([y[idx].item() for y, idx in zip(y_hat.softmax(dim=1), y_test)])\n",
        "pred_prob_a = pred_prob[y_mask][x_group < 0]\n",
        "pred_prob_b = pred_prob[y_mask][x_group > 0]\n",
        "\n",
        "plt.hist(pred_prob_a.numpy(), alpha=0.5, bins=30, density=True, label=\"subgroup a\")\n",
        "plt.hist(pred_prob_b.numpy(), alpha=0.5, bins=30, density=True, label=\"subgroup b\")\n",
        "plt.legend()\n",
        "sns.despine()"
      ],
      "execution_count": 249,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAATJUlEQVR4nO3dfZBV9X3H8feXh8yq+FTER0RwBlAGzEhXijUT2fgwVIjEaIyOtiHaMjY1SZ1oiCaMjv/oTJ3adpLRWRNrmlpixcealoQxEKNRccFnMAYfQlaMLJuqxQxV2V//2BsKK3DP7j177/3tvl8zDPeee+4539+93A+/+zvn/G6klJAk5WdEowuQJA2MAS5JmTLAJSlTBrgkZcoAl6RMjarnzubOnZuWL19ez11K0lAQu1tY1x74li1b6rk7SRrSHEKRpEwZ4JKUKQNckjJV14OYu/PBBx/Q2dnJtm3bGl3KkNfS0sL48eMZPXp0o0uRVIKGB3hnZyf7778/EydOJGK3B1pVgpQS3d3ddHZ2MmnSpEaXI6kEDR9C2bZtG2PHjjW8B1lEMHbsWL/pSENIwwMcMLzrxNdZGlqaIsAlSf3X8DHwvm5e8XKp27vijCmlbu+6665jzJgxXHnllaVuV5L6q+kCfKjbvn07I0eObHQZkkpQtMNZdkfyD4b9EMp7773HvHnz+PjHP8706dO56667AJg4ceKOS/87OjqYM2fOjuc8++yznHzyyUyePJnbbrsNgJ6eHr70pS9x3HHHccYZZ3DWWWexbNmyHdtavHgxM2fO5O6772bp0qXMmDGD6dOns3jx4h3bHTNmzI7by5YtY+HChQAsXLiQyy67jNbWVqZMmcJDDz30kXZs3bqV0047jZkzZzJjxgweeOCBUl8nSc1n2PfAly9fzpFHHsmPfvQjAN55552qz3nuued44okneO+99zjxxBOZN28ev/jFL3j99ddZt24dmzdv5vjjj+eSSy7Z8ZyxY8eydu1aNm3axOzZs1mzZg0HH3wwZ555Jvfffz+f+cxn9rrP119/ndWrV/PKK6/Q1tbGhg0baGlp2fF4S0sL9913HwcccABbtmxh9uzZnH322R64lIawqj3wiLg9IjZHxAs7Lfu7iHgpIp6LiPsi4qDBLXPwzJgxgxUrVrB48WJ+/vOfc+CBB1Z9zoIFC9hnn3045JBDaGtrY/Xq1Tz66KN87nOfY8SIERx++OG0tbXt8pzPf/7zADz11FPMmTOHcePGMWrUKC666CIeeeSRqvs8//zzGTFiBJMnT+bYY4/lpZde2uXxlBLXXHMNJ5xwAqeffjpvvPEGb731Vj9eCUm5KTKEcgcwt8+yFcD0lNIJwMvA1SXXVTdTpkxh7dq1zJgxg29961tcf/31AIwaNYqenh6Aj5w73bdXW6SXu99++1VdZ+ft9Hefd955J11dXaxZs4ZnnnmGww47zHO+pSGuaoCnlB4Bftdn2U9SSh9W7j4BjB+E2upi06ZN7Lvvvlx88cVcddVVrF27Fugdt16zZg0A99xzzy7PeeCBB9i2bRvd3d2sWrWKk046iVNOOYV77rmHnp4e3nrrLVatWrXb/c2aNYuf/exnbNmyhe3bt7N06VJOPfVUAA477DDWr19PT08P99133y7Pu/vuu+np6eGVV17h1VdfZerUqbs8/s4773DooYcyevRoVq5cya9//esyXh5JTayMMfBLgLv29GBELAIWAUyYMKHqxgbraO2ePP/881x11VWMGDGC0aNHc8sttwBw7bXXcumll7JkyZJdDmACnHDCCbS1tbFlyxaWLFnCkUceybnnnsvDDz/MtGnTOProo5k5c+Zuh2OOOOIIbrzxRtra2kgpMW/ePBYsWADAjTfeyPz58xk3bhytra1s3bp1x/MmTJjArFmzePfdd7n11lt3Gf8GuOiii/j0pz/NjBkzaG1t5bjjjiv5lZLUbCKlVH2liInAQyml6X2WfxNoBT6bCmyotbU1dXR07LJs/fr1HH/88f0ouXlt3bqVMWPG0N3dzaxZs3jsscc4/PDDa97uwoULmT9/Puedd17N2xpKr7fUaHU8jXC347QD7oFHxEJgPnBakfAeDubPn8/bb7/N+++/z5IlS0oJb0nakwEFeETMBb4OnJpS+n25JeVrT+PetbrjjjsGZbuS8lbkNMKlwOPA1IjojIhLgW8D+wMrIuKZiLh1kOuUJPVRtQeeUrpwN4u/Nwi1SJL6YdhfSi9JuTLAJSlTzTcXysobyt1eW7kXiTbrdLKrVq3ipptu2u1EV5KGJnvgdbZ9+/ZGlyBpiBj2AT5UppMFePfdd5k3bx5Tp07lsssu2zGXi6ShqfmGUOpsqEwnC7B69WrWrVvHMcccw9y5c7n33ntLuXpTUnMa9j3woTKdLPROlHXssccycuRILrzwQh599NGCr4KkHA37AB8q08kOtC5J+Rr2AT5UppOF3iGU1157jZ6eHu666y4+8YlP1PrySGpizTcGXvJpf9UMlelkAU466SQuv/xyNmzYQFtbG+ecc06Jr5SkZlNoOtmyOJ3swDidrNScsp1OVh/ldLKS6skAL5HTyUqqp6Y4iOnvQdSHr7M0tDQ8wFtaWuju7jZcBllKie7u7t0e/JSUp4YPoYwfP57Ozk66uroaXcqQ19LSwvjx4xtdhqSSNDzAR48ezaRJkxpdhiRlp+FDKJKkgTHAJSlTBrgkZcoAl6RMGeCSlCkDXJIyZYBLUqYMcEnKVNUAj4jbI2JzRLyw07I/iogVEfGryt8HD26ZkqS+ivTA7wDm9ln2DeDhlNJk4OHKfUlSHVUN8JTSI8Dv+ixeAHy/cvv7wN5/Ul2SVLqBjoEfllJ6s3L7t8Bhe1oxIhZFREdEdDhhlSSVp+bJrFJKKSL2OBdsSqkdaIfen1SrdX+SNJiK/kxaMxhoD/ytiDgCoPL35vJKkiQVMdAAfxD4QuX2F4AHyilHklRUkdMIlwKPA1MjojMiLgVuBM6IiF8Bp1fuS5LqqOoYeErpwj08dFrJtUiS+sErMSUpUwa4JGXKAJekTBngkpQpA1ySMmWAS1KmDHBJypQBLkmZMsAlKVMGuCRlygCXpEwZ4JKUKQNckjJlgEtSpgxwScqUAS5JmTLAJSlTBrgkZcoAl6RMGeCSlCkDXJIyZYBLUqYMcEnKlAEuSZmqKcAj4oqIeDEiXoiIpRHRUlZhkqS9G3CAR8RRwFeA1pTSdGAkcEFZhUmS9m5UCc/fJyI+APYFNtVekiSV7+YVLze6hNINuAeeUnoDuAnYCLwJvJNS+knf9SJiUUR0RERHV1fXwCuVJO2iliGUg4EFwCTgSGC/iLi473oppfaUUmtKqXXcuHEDr1SStItaDmKeDryWUupKKX0A3Av8aTllSZKqqSXANwKzI2LfiAjgNGB9OWVJkqqpZQz8SWAZsBZ4vrKt9pLqkiRVUdNZKCmla4FrS6pFkrIye2PBPuvKsdB2den790pMScqUAS5JmTLAJSlTBrgkZcoAl6RMGeCSlCkDXJIyZYBLUqYMcEnKlAEuSZkywCUpUwa4JGXKAJekTBngkpQpA1ySMmWAS1KmDHBJypQBLkmZMsAlKVMGuCRlygCXpEwZ4JKUKQNckjJlgEtSpmoK8Ig4KCKWRcRLEbE+Ik4uqzBJ0t6NqvH5/wgsTymdFxEfA/YtoSZJUgEDDvCIOBD4JLAQIKX0PvB+OWVJErDyhmLrtV09uHU0qVqGUCYBXcA/R8TTEfHdiNivpLokSVXUEuCjgJnALSmlE4H3gG/0XSkiFkVER0R0dHV11bA7SdLOagnwTqAzpfRk5f4yegN9Fyml9pRSa0qpddy4cTXsTpK0swEHeErpt8BvImJqZdFpwLpSqpIkVVXrWShfBu6snIHyKvDF2kuSpH4qcLBz9sZunpiwqA7F1E9NAZ5SegZoLakWSVI/eCWmJGXKAJekTBngkpQpA1ySMmWAS1Kmaj2NUJL6r+gcJ9ore+CSlCl74JLUx+yN7Y0uoRB74JKUKQNckjJlgEtSpgxwScqUAS5JmTLAJSlTBrgkZcoAl6RMGeCSlCkDXJIyZYBLUqYMcEnKlAEuSZkywCUpUwa4JGXKAJekTBngkpSpmn+RJyJGAh3AGyml+bWXJEnFPf5qd6NLaJgyeuBfBdaXsB1JUj/UFOARMR6YB3y3nHIkSUXV2gP/B+DrQM+eVoiIRRHREREdXV1dNe5OkvQHAx4Dj4j5wOaU0pqImLOn9VJK7UA7QGtraxro/iRlYOUNja5gWKmlB34KcHZEvA78EPhURPxrKVVJkqoacICnlK5OKY1PKU0ELgB+mlK6uLTKJEl75XngkpSpms8DB0gprQJWlbEtSVIx9sAlKVMGuCRlygCXpEwZ4JKUKQNckjJlgEtSpgxwScqUAS5JmSrlQh5JQ1iDJqgazj/UUJQ9cEnKlAEuSZkywCUpUwa4JGXKg5iS6sqDk+UxwCUNG7M3tje6hFI5hCJJmTLAJSlTDqFIw5W/IJ89e+CSlCkDXJIyZYBLUqYcA5dUCs/vrj974JKUKQNckjI14ACPiKMjYmVErIuIFyPiq2UWJknau1rGwD8EvpZSWhsR+wNrImJFSmldSbVJkvZiwAGeUnoTeLNy+38iYj1wFGCAS43kBTrDRilnoUTEROBE4MndPLYIWAQwYcKEMnYnqY48u6R51XwQMyLGAPcAf5tSerfv4yml9pRSa0qpddy4cbXuTpJUUVOAR8RoesP7zpTSveWUJEkqYsBDKBERwPeA9Smlvy+vJGmYKTpm3Xb14Nah7NTSAz8F+HPgUxHxTOXPWSXVJUmqopazUB4FosRaJEn94JWYkpQpJ7OShilPD8yfPXBJypQBLkmZcghF6o/+XKZe8ml/j3/vykLrnXzs2FL3q+ZlD1ySMmUPXBosTiqlQWYPXJIyZYBLUqYcQpEGSdHzrMs+6Oj53cOHPXBJypQ9cKmf7OGqWRjgGvJuXvFy1XVmb2xv2PnT/oeggTLA1VwaODe2QarcGOBDQdmh18CrDSUVZ4A3QoN/gaXakMLsjb090UJDCoPQliI94Sc+fJkrzphSeJvSUGSAU2yMFDAwmsjsje2wstiY9R/+Q5KGGgNce9So85glFWOAFzB7Y3vvjWo9vmYfD64Md5TdIy0U9K8Wm0lPUnEGeD9UDapKSNkjlVQPQzbAqx+oa///24NdzEA5m91eedqfhrt8ArzBZ270h3NgSKqHfAJ8CDJwJdUiuwAvOg7dtMMiklQSZyOUpEzVFOARMTcifhkRGyLiG2UVJUmqbsABHhEjge8AfwZMAy6MiGllFSZJ2rtaeuCzgA0ppVdTSu8DPwQWlFOWJKmaWg5iHgX8Zqf7ncCf9F0pIhYBiyp3t0bELwe4v0OALQN8bs5s9/Biu4esa3a3sGi7l6eU5vZdOOhnoaSU2oH2qitWEREdKaXWEkrKiu0eXmz38FJru2sZQnkDOHqn++MryyRJdVBLgD8FTI6ISRHxMeAC4MFyypIkVTPgIZSU0ocRcTnwY2AkcHtK6cXSKvuomodhMmW7hxfbPbzU1O5IKZVViCSpjrwSU5IyZYBLUqaaLsCrXZ4fEQsjoisinqn8+ctG1Fm2ItMSRMT5EbEuIl6MiH+rd42DocD7ffNO7/XLEfF2I+osW4F2T4iIlRHxdEQ8FxFnNaLOshVo9zER8XClzasiYnwj6ixTRNweEZsj4oU9PB4R8U+V1+S5iJhZeOMppab5Q+/B0FeAY4GPAc8C0/qssxD4dqNrbUC7JwNPAwdX7h/a6Lrr0e4+63+Z3oPlDa+9Du93O/DXldvTgNcbXXed2n038IXK7U8BP2h03SW0+5PATOCFPTx+FvBfQNA7keqTRbfdbD3w4Xp5fpF2/xXwnZTSfwOklDbXucbB0N/3+0JgaV0qG1xF2p2AAyq3DwQ21bG+wVKk3dOAn1Zur9zN49lJKT0C/G4vqywA/iX1egI4KCKOKLLtZgvw3V2ef9Ru1ju38lVjWUQcvZvHc1Ok3VOAKRHxWEQ8EREfuaw2Q0XfbyLiGGAS///hzlmRdl8HXBwRncB/0vvtI3dF2v0s8NnK7XOA/SNiqP/IbOHPQV/NFuBF/AcwMaV0ArAC+H6D66mXUfQOo8yhtyd6W0Qc1NCK6usCYFlKaXujC6mTC4E7Ukrj6f2K/YOIyPHz2l9XAqdGxNPAqfRe3T1c3vN+a7Z/EFUvz08pdaeU/rdy97vAH9eptsFUZFqCTuDBlNIHKaXXgJfpDfSc9Wc6hgsYGsMnUKzdlwL/DpBSehxooXfio5wV+XxvSil9NqV0IvDNyrIhceB6LwY8LUmzBXjVy/P7jA2dDayvY32Dpci0BPfT2/smIg6hd0jl1XoWOQgKTccQEccBBwOP17m+wVKk3RuB0wAi4nh6A7yrrlWWr8jn+5CdvmlcDdxe5xob4UHgLypno8wG3kkpvVnkiU31m5hpD5fnR8T1QEdK6UHgKxFxNvAhvQcGFjas4JIUbPePgTMjYh29XymvSill/avIBdsNvR/0H6bKIfvcFWz31+gdJruC3gOaC3Nvf8F2zwFuiIgEPAL8TcMKLklELKW3XYdUjmlcC4wGSCndSu8xjrOADcDvgS8W3nbm/yYkadhqtiEUSVJBBrgkZcoAl6RMGeCSlCkDXJIyZYBLUqYMcEnK1P8BRM0Uk35N2JQAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bAj1hrHJ9BgJ"
      },
      "source": [
        "**Question:**\n",
        "- Knowing the limitations, how can we further improve the method? (hint: from \"on-off\" to variable weight)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c3DRcxlShHTe"
      },
      "source": [
        "## Moments penalization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vehzoCq79iWM"
      },
      "source": [
        "**What if at each step we set a weight to each subgroup based on how far $L_{s}$ is from $L$**\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6sTkxfiqYDCN"
      },
      "source": [
        "# Params\n",
        "num_epochs = 10\n",
        "lr = 0.02\n",
        "batch_size = 256\n",
        "\n",
        "lambda_1 = 1.0\n",
        "lambda_2 = 5.0\n",
        "lambda_3 = 0.0\n",
        "sensitive_idx=2\n",
        "\n",
        "model = torch.nn.Sequential(\n",
        "  torch.nn.Linear(3, 8),\n",
        "  torch.nn.Sigmoid(),\n",
        "  torch.nn.Linear(8, 2),\n",
        ")\n",
        "\n",
        "opt  = torch.optim.Adam(model.parameters(), lr=lr)\n",
        "dl   = torch.utils.data.DataLoader(data_train, batch_size=batch_size, shuffle=True, drop_last=True)\n",
        "\n",
        "logs = []\n",
        "\n",
        "for i in range(num_epochs):\n",
        "  for x, y_t in dl:\n",
        "    opt.zero_grad()\n",
        "\n",
        "    y_hat  = model(x)\n",
        "    losses = loss(y_hat, y_t)\n",
        "\n",
        "    z      = losses.detach()\n",
        "    z_mean = z.mean()\n",
        "\n",
        "\n",
        "    # Calculate mean loss for each class\n",
        "    class_mean_loss = torch.zeros_like(z)\n",
        "    lambda_2v       = lambda_2 * torch.ones_like(z)\n",
        "    lambda_3v       = lambda_3 * torch.ones_like(z)\n",
        "\n",
        "\n",
        "    for c in x[:,sensitive_idx].unique():\n",
        "      mask = (x[:,sensitive_idx] == c)\n",
        "      class_mean_loss[mask] = z[mask].mean()\n",
        "\n",
        "    # Calculate weights\n",
        "    w = lambda_1 + lambda_2v * (class_mean_loss - z_mean) / z_mean + \\\n",
        "        lambda_3v * (z - class_mean_loss - z.var()/z_mean)\n",
        "    w = w.clip(0.0)\n",
        "\n",
        "    # Backpropagate and optimize    \n",
        "    (w * losses).mean().backward()\n",
        "    opt.step()"
      ],
      "execution_count": 281,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eAaA4nFChv7I",
        "outputId": "091c1d41-70d1-481d-dd0d-d2cee7bf75a7"
      },
      "source": [
        "# Evaluate model\n",
        "\n",
        "with torch.no_grad():\n",
        "  y_hat = model(X_test)\n",
        "\n",
        "y_class = y_hat.argmax(dim=1)\n",
        "\n",
        "\n",
        "print(\"Number of missclassifications\", (y_class != y_test).sum().item(), \"out of\", y_test.shape[0], \"or\", (y_class != y_test).sum().item() /  y_test.shape[0])"
      ],
      "execution_count": 277,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of missclassifications 316 out of 3200 or 0.09875\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "avmM4Ep0h10q",
        "outputId": "2e30bd4c-6089-434b-d57c-70d9962c9753"
      },
      "source": [
        "y_mask  = (y_class != y_test)\n",
        "x_group = X_test[:, 2][y_mask]\n",
        "\n",
        "print(\"Number of missclassifications for subgroup a\", (x_group < 0).sum().item(), \"out of\", y_test.shape[0], \"or\", (x_group < 0).sum().item() /  y_test.shape[0])\n",
        "print(\"Number of missclassifications for subgroup b\", (x_group > 0).sum().item(), \"out of\", y_test.shape[0], \"or\", (x_group > 0).sum().item() /  y_test.shape[0])"
      ],
      "execution_count": 278,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of missclassifications for subgroup a 168 out of 3200 or 0.0525\n",
            "Number of missclassifications for subgroup b 148 out of 3200 or 0.04625\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "mWan1eBIh6Ve",
        "outputId": "a5262f3f-598b-4879-b39f-15f014d84a62"
      },
      "source": [
        "# Select predicted prob by subgroup\n",
        "pred_prob   = torch.tensor([y[idx].item() for y, idx in zip(y_hat.softmax(dim=1), y_test)])\n",
        "pred_prob_a = pred_prob[y_mask][x_group < 0]\n",
        "pred_prob_b = pred_prob[y_mask][x_group > 0]\n",
        "\n",
        "plt.hist(pred_prob_a.numpy(), alpha=0.5, bins=30, density=True, label=\"subgroup a\")\n",
        "plt.hist(pred_prob_b.numpy(), alpha=0.5, bins=30, density=True, label=\"subgroup b\")\n",
        "plt.legend()\n",
        "sns.despine()"
      ],
      "execution_count": 279,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAARCklEQVR4nO3df3BVdXrH8c8DZCciaB2MIIICHX6W4EADDaOzklUcahB0RF0H2mZ1h6HWurUrZbEyWvuHtLW725nuaNG2trMOMoCI1Skt4xAVLWKIqLPgbgGRjWwxSSsUWoqQp38ksAET7kk4594n975fM8zk5p57zpMv4TNfvvd8n2vuLgBAXP0KXQAA4PwIagAIjqAGgOAIagAIjqAGgOAGZHHSOXPm+KZNm7I4NQAUK+vuiUxm1C0tLVmcFgBKEksfABAcQQ0AwRHUABBcJm8mduXLL79UU1OTjh8/nq9Llqzy8nKNGDFCZWVlhS4FQAryFtRNTU0aPHiwRo0aJbNu39zEBXJ3tba2qqmpSaNHjy50OQBSkLelj+PHj2vIkCGEdMbMTEOGDOF/LkARyesaNSGdH4wzUFx4MxEAgsvbGvW5frD5Z6me76HZ41I93+OPP65Bgwbp4YcfTvW8ANBTBQvqYnfq1Cn179+/0GUASNOWJ8//fM3yTC5bMksfx44dU21tra699lpNnjxZa9askSSNGjXqzJb3hoYGzZo168xrPvjgA82cOVNjx47Vs88+K0lqa2vT/fffrwkTJmj27Nm65ZZbtG7dujPnWrZsmaZNm6a1a9dq9erVqqys1OTJk7Vs2bIz5x00aNCZr9etW6e6ujpJUl1dnZYsWaKqqiqNGzdOr7766ld+jqNHj+rGG2/UtGnTVFlZqY0bN6Y6TgDiKZkZ9aZNmzR8+HC99tprkqTDhw/nfM2HH36obdu26dixY5o6dapqa2v1zjvvaP/+/dq1a5c+//xzTZw4Uffee++Z1wwZMkSNjY06ePCgqqurtWPHDl122WW6+eab9fLLL+u222477zX379+v7du3a+/evaqpqdGePXtUXl5+5vny8nJt2LBBl1xyiVpaWlRdXa158+bxBiJQxEpmRl1ZWanNmzdr2bJleuutt3TppZfmfM38+fN10UUX6fLLL1dNTY22b9+urVu36s4771S/fv00bNgw1dTUnPWau+++W5L03nvvadasWaqoqNCAAQO0cOFCvfnmmzmvedddd6lfv34aO3asxowZo48//vis591djzzyiKZMmaKbbrpJn332mQ4dOtSDkQDQ15RMUI8bN06NjY2qrKzUo48+qieeeEKSNGDAALW1tUnSV+49PneWmmTWevHFF+c8pvN5enrNF154Qc3NzdqxY4d27typoUOHcs80UORKJqgPHjyogQMHatGiRVq6dKkaGxslta8r79ixQ5K0fv36s16zceNGHT9+XK2traqvr9f06dN13XXXaf369Wpra9OhQ4dUX1/f5fVmzJihN954Qy0tLTp16pRWr16tG264QZI0dOhQ7d69W21tbdqwYcNZr1u7dq3a2tq0d+9e7du3T+PHjz/r+cOHD+uKK65QWVmZtmzZok8//TSN4QEQWMHWqNO+nS6Xjz76SEuXLlW/fv1UVlamp59+WpL02GOP6b777tOKFSvOeiNRkqZMmaKamhq1tLRoxYoVGj58uO644w69/vrrmjRpkkaOHKlp06Z1uYxy5ZVXauXKlaqpqZG7q7a2VvPnz5ckrVy5UnPnzlVFRYWqqqp09OjRM6+7+uqrNWPGDB05ckTPPPPMWevTkrRw4ULdeuutqqysVFVVlSZMmJDySAGIxtw99ZNWVVV5Q0PDWd/bvXu3Jk6cmPq1CuHo0aMaNGiQWltbNWPGDL399tsaNmzYBZ+3rq5Oc+fO1YIFCy74XMU03kAY2d6e1+3aasnc9ZGmuXPn6osvvtCJEye0YsWKVEIaALpDUPdCd+vSF+r555/P5LwA+raSeTMRAPoqghoAgiOoASA4ghoAgivcm4m5bnPpqZS7VkVtc1pfX6+nnnqqy4ZNAIoTM+qMnDp1qtAlACgSJRPUxdLmVJKOHDmi2tpajR8/XkuWLDnTqwRAcUq09GFmD0n6tiSX9JGkb7l7n+oEVCxtTiVp+/bt2rVrl6655hrNmTNHL730Uiq7GQHElHNGbWZXSXpQUpW7T5bUX9I3sy4sbcXS5lRqb/g0ZswY9e/fX/fcc4+2bt2acBQA9EVJlz4GSLrIzAZIGijpYHYlZaNY2pz2ti4AfVfOoHb3zyQ9JemApF9IOuzu/3rucWa22MwazKyhubk5/UovULG0OZXalz4++eQTtbW1ac2aNbr++usvdHgABJZzjdrMLpM0X9JoSV9IWmtmi9z9x52Pc/dVklZJ7d3zcl45ow+B7E6xtDmVpOnTp+uBBx7Qnj17VFNTo9tvvz3FkQIQTc42p2Z2p6Q57n5fx+PfllTt7vd39xranPYObU6B4AK3OT0gqdrMBkr6X0k3Smo4/0uKG21OAeRTzqB293fNbJ2kRkknJb2vjiWOUkWbUwD5lOg+and/TNJjF3oxd+cOhTzI4lN7ABRO3nYmlpeXq7W1lRDJmLurtbW1yzchAfRNeWvKNGLECDU1NSnirXvFpry8XCNGjCh0GQBSkregLisr0+jRo/N1OQAoGiXTlAkA+iqCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCSxTUZvYrZrbOzD42s91mNjPrwgAA7QYkPO6vJG1y9wVm9jVJAzOsCShdW57MfUzN8uzrQCg5g9rMLpX0dUl1kuTuJySdyLYsAMBpSZY+RktqlvT3Zva+mT1nZhdnXBcAoEOSoB4gaZqkp919qqRjkr537kFmttjMGsysobm5OeUyAaB0JQnqJklN7v5ux+N1ag/us7j7KnevcveqioqKNGsEgJKWM6jd/T8k/dzMxnd860ZJuzKtCgBwRtK7Pn5f0gsdd3zsk/St7EoCAHSWKKjdfaekqoxrAQB0gZ2JABAcQQ0AwRHUABAcQQ0AwRHUABBc0tvzAKC4JWmIVSDMqAEgOIIaAIIjqAEgOIIaAIIjqAEgOIIaAIIjqAEgOIIaAIIjqAEgOIIaAIIjqAEgOIIaAIKjKROAwknSCKlmefZ1BMeMGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiCGgCCI6gBIDiaMgHFptgaHSX5eYocM2oACI6gBoDgCGoACI6gBoDgCGoACI67PgAk8oPNP0t03EOzx2VcSelJPKM2s/5m9r6ZvZplQQCAs/Vk6eM7knZnVQgAoGuJgtrMRkiqlfRctuUAAM6VdEb9Q0l/JKmtuwPMbLGZNZhZQ3NzcyrFAQASBLWZzZX0ubvvON9x7r7K3avcvaqioiK1AgGg1CWZUV8naZ6Z7Zf0oqRvmNmPM60KAHBGztvz3H25pOWSZGazJD3s7osyrgsoTlEaDPWijuoDrWc93nb14rSqQQ5seAGA4Hq04cXd6yXVZ1IJAKBL7EwsMtF3j0Wvr5j8277W3AdJmjlmSKrXTfp3LH11OaUrSesr1M+bDyx9AEBwBDUABEdQA0BwBDUABEdQA0BwBDUABEdQA0BwBDUABEdQA0Bw7EzMSE92ZyWR5k696gOrpC3n2Z1Vszy1a6F4VR9YlfOYvty4KdJOR2bUABAcQQ0AwRHUABAcQQ0AwRHUABAcQQ0AwRHUABAcQQ0AwbHhBShxSTd2FEr0+vKBGTUABEdQA0BwBDUABMcaNZCWLU8W13UQBjNqAAiOoAaA4AhqAAiOoAaA4AhqAAiOuz6QirQ/egzALzGjBoDgCGoACI6gBoDgCGoACI6gBoDgCGoACI7b89A75zQGqj7w1ebu265enK9qEFT1gVWFLqEoMKMGgOByBrWZjTSzLWa2y8x+YmbfyUdhAIB2SZY+Tkr6rrs3mtlgSTvMbLO778q4NnRSajv/kv68D80el+r5eiLptZNK+tmAM8cMSfW6iC/njNrdf+HujR1f/7ek3ZKuyrowAEC7Hq1Rm9koSVMlvdvFc4vNrMHMGpqbm9OpDgCQPKjNbJCk9ZL+wN2PnPu8u69y9yp3r6qoqEizRgAoaYmC2szK1B7SL7j7S9mWBADoLMldHybpbyXtdvfvZ18SAKCzJDPq6yT9lqRvmNnOjj+3ZFwXAKBDztvz3H2rJMtDLQCALrAzEQCCI6gBIDiaMnVIeydcdOfbBbftZO6x6KoJU08ladiTs7HTOc2hulSzPGFFKAVJd4BGwowaAIIjqAEgOIIaAIIjqAEgOIIaAIIjqAEgOIIaAIIjqAEgODa89FCpfSRWdEk2LyTZwNMb5/4upLEJCOgKM2oACI6gBoDgCGoACI416j4klSZGKV0nX6LUEqUOlCZm1AAQHEENAMER1AAQHEENAMER1AAQXLi7PtLe+VcsH50FnNYXP0oKF4YZNQAER1ADQHAENQAER1ADQHAENQAER1ADQHDhbs/rq3I17UnSLCmNxj+RmgdFqSVfzayArDCjBoDgCGoACK7olz74jEMAfR0zagAIjqAGgOAIagAIjqAGgOAIagAIjqAGgOASBbWZzTGzn5rZHjP7XtZFAQB+KWdQm1l/ST+S9JuSJkm6x8wmZV0YAKBdkhn1DEl73H2fu5+Q9KKk+dmWBQA4LcnOxKsk/bzT4yZJv3HuQWa2WNLpzjZHzeynF15e6i6X1FKYS/9lYS6bXAHHJoJu/35KfFzOi7H5ikdOf9Gbsdnk7nO6eiK1LeTuvkpSjHZp3TCzBnevKnQdETE2XWNcusfYdC/tsUmy9PGZpJGdHo/o+B4AIA+SBPV7ksaa2Wgz+5qkb0p6JduyAACn5Vz6cPeTZvaApH+R1F/S37n7TzKvLBuhl2YKjLHpGuPSPcame6mOjbl7mucDAKSMnYkAEBxBDQDBFWVQ59rybmZfN7NGMztpZgsKUWMhJBiXPzSzXWb2oZm9bmbXFKLOQkgwNkvM7CMz22lmW0tpd27SFhJmdoeZuZmVxC17CX5n6sysueN3ZqeZfbvXF3P3ovqj9jc890oaI+lrkj6QNOmcY0ZJmiLpHyUtKHTNgcalRtLAjq9/V9KaQtcdaGwu6fT1PLVvTih47RHGpuO4wZLelLRNUlWh644wLpLqJP11Gtcrxhl1zi3v7r7f3T+U1FaIAgskybhscff/6Xi4Te33zJeCJGNzpNPDiyWVyrvwSVtI/KmkP5N0PJ/FFVBeW2sUY1B3teX9qgLVEklPx+U+Sf+caUVxJBobM/s9M9sr6c8lPZin2got59iY2TRJI939tXwWVmBJ/z3d0bGUuM7MRnbxfCLFGNS4QGa2SFKVpL8odC2RuPuP3P1XJS2T9Gih64nAzPpJ+r6k7xa6loD+SdIod58iabOkf+jtiYoxqNny3rVE42JmN0n6Y0nz3P3/8lRbofX0d+ZFSbdlWlEcucZmsKTJkurNbL+kakmvlMAbijl/Z9y9tdO/oeck/XpvL1aMQc2W967lHBczmyrpb9Qe0p8XoMZCSTI2Yzs9rJX073msr5DOOzbuftjdL3f3Ue4+Su3vbcxz94bClJs3SX5nruz0cJ6k3b29WGrd86Lwbra8m9kTkhrc/RUzmy5pg6TLJN1qZn/i7r9WwLIzl2Rc1L7UMUjSWjOTpAPuPq9gRedJwrF5oON/G19K+i9Jv1O4ivMn4diUnITj8qCZzZN0UtJ/qv0ukF5hCzkABFeMSx8AUFQIagAIjqAGgOAIagAIjqAGgOAIagAIjqAGgOD+H2T3chUk4cmVAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "7n1ysnDZjiQb",
        "outputId": "9950eed0-0272-4c31-cb3c-ab7901178140"
      },
      "source": [
        "# Select predicted prob by subgroup\n",
        "y_mask  = (y_class == y_test)\n",
        "x_group = X_test[:, 2][y_mask]\n",
        "\n",
        "\n",
        "pred_prob   = torch.tensor([y[idx].item() for y, idx in zip(y_hat.softmax(dim=1), y_test)])\n",
        "pred_prob_a = pred_prob[y_mask][x_group < 0]\n",
        "pred_prob_b = pred_prob[y_mask][x_group > 0]\n",
        "\n",
        "plt.hist(pred_prob_a.numpy(), alpha=0.5, bins=30, density=True, label=\"subgroup a\")\n",
        "plt.hist(pred_prob_b.numpy(), alpha=0.5, bins=30, density=True, label=\"subgroup b\")\n",
        "plt.legend()\n",
        "sns.despine()"
      ],
      "execution_count": 280,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAASq0lEQVR4nO3df2xVZZ7H8c8XqKmKv1IrqLUCCT+XouCFLdGsVMQwFkXHn0Q329WdhjXOD3dkGV2JrslE/zDLbrITTR1dJ7MOKiA6q7vMEoeqKIoFQUeKBBCdwqy0zSoDG0Ztv/vHvbQFL72ncM+5z719v5KGe7nnnvv1of349DnP8xxzdwEAwjWk0AUAAPpHUANA4AhqAAgcQQ0AgSOoASBww+I46dy5c3316tVxnBoASpUd64VYetQdHR1xnBYABiWGPgAgcAQ1AASOoAaAwMVyMTGbr7/+Wm1tbTp06FBSHzlolZeXq6qqSmVlZYUuBUAeJBbUbW1tOu200zRq1CiZHfPiJk6Qu6uzs1NtbW0aPXp0ocsBkAeJDX0cOnRIFRUVhHTMzEwVFRX85gKUkETHqAnpZNDOQGnhYiIABC6xMeqjLV2zPa/nu2fOuLye76GHHtLw4cN177335vW8ADBQBQvqUtfV1aWhQ4cWugwA/Vn7SLTj6u6Lt44cBs3Qx8GDB1VfX6+LLrpIkydP1vPPPy9JGjVqVM+S95aWFs2aNavnPVu2bNHMmTM1duxYPfnkk5Kk7u5u3XXXXZowYYLmzJmjq6++WitWrOg51+LFizVt2jQtX75cy5YtU01NjSZPnqzFixf3nHf48OE9j1esWKGGhgZJUkNDgxYuXKhUKqVx48bplVde+dZ/x4EDBzR79mxNmzZNNTU1evnll/PaTgDCM2h61KtXr9Z5552nV199VZL05Zdf5nzPBx98oHfeeUcHDx7U1KlTVV9fr7ffflu7d+/W1q1btW/fPk2cOFF33HFHz3sqKiq0adMm7d27V7W1tdq4caPOOussXXXVVXrppZd03XXX9fuZu3fv1oYNG7Rz507V1dVpx44dKi8v73m9vLxcq1at0umnn66Ojg7V1tbq2muv5QIiUMJy9qjNbLyZbe7ztd/MfpREcflUU1OjNWvWaPHixXrzzTd1xhln5HzP/PnzdfLJJ+vss89WXV2dNmzYoHXr1ummm27SkCFDNHLkSNXV1R3xnltuuUWS9N5772nWrFmqrKzUsGHDdNttt+mNN97I+Zk333yzhgwZorFjx2rMmDHatm3bEa+7u+6//35NmTJFV155pfbs2aPPP/98AC0BoNjkDGp3/9jdL3b3iyVdIun/JK2KvbI8GzdunDZt2qSamho98MADevjhhyVJw4YNU3d3tyR9a+7x0b3UKL3WU089Necxfc8z0M989tln1d7ero0bN2rz5s0aMWIEc6aBEjfQMerZkna6+6dxFBOnvXv36pRTTtHtt9+uRYsWadOmTZLS48obN26UJK1cufKI97z88ss6dOiQOjs71dzcrOnTp+vSSy/VypUr1d3drc8//1zNzc1ZP2/GjBl6/fXX1dHRoa6uLi1btkyXX365JGnEiBFqbW1Vd3e3Vq068v95y5cvV3d3t3bu3Kldu3Zp/PjxR7z+5Zdf6pxzzlFZWZnWrl2rTz8tun8KAAM00DHqWyUty8cH53s6XS4ffvihFi1apCFDhqisrEyPP/64JOnBBx/UnXfeqSVLlhxxIVGSpkyZorq6OnV0dGjJkiU677zzdMMNN+i1117TpEmTdMEFF2jatGlZh1HOPfdcPfroo6qrq5O7q76+XvPnz5ckPfroo5o3b54qKyuVSqV04MCBnvdVV1drxowZ2r9/v5544okjxqcl6bbbbtM111yjmpoapVIpTZgwIc8tBSA05u7RDjQ7SdJeSX/m7t8aFDWzRkmNklRdXX3J0T291tZWTZw48YQLDsGBAwc0fPhwdXZ2asaMGXrrrbc0cuTIEz5vQ0OD5s2bpxtvvPGEz1VK7Q3EJqzpecccWx1Ij/o7kjZlC2lJcvcmSU2SlEqloqV/kZo3b56++OILffXVV1qyZEleQhoAjmUgQb1AeRr2KHbHGpc+Uc8880ws5wVQ3CJdTDSzUyXNkfRivOUAAI4WqUft7gclVcRcCwAgi0GzhBwAihVBDQCBK9xeH1GnxUSV5+kzoW5z2tzcrMceeyzrhk0AShM96ph0dXUVugQAJWLQBHWpbHMqSfv371d9fb3Gjx+vhQsX9uxVAqA0sc1pP0Lc5lSSNmzYoK1bt+rCCy/U3Llz9eKLL+ZlNSOAMA2aHnWpbHMqpTd8GjNmjIYOHaoFCxZo3bp1EVsBQDEaNEFdKtucHm9dAIrXoAnqUtnmVEoPfXzyySfq7u7W888/r8suu+xEmwdAwAo3Rp3wzSJLZZtTSZo+fbruvvtu7dixQ3V1dbr++uvz2FIAQhN5m9OBSKVS3tLScsTfldK2m2xzCpSIEtzmFBlscwogSQT1cWCbUwBJSvRiYhzDLPg22hkoLYkFdXl5uTo7OwmRmLm7Ojs7s16EBFCcEhv6qKqqUltbm9rb25P6yEGrvLxcVVVVhS4DKJx8b/pWYIkFdVlZmUaPHp3UxwFAyRg0C14AoFgx6wNA8SixIY2o6FEDQOCi3oX8TDNbYWbbzKzVzGbGXRgAIC3q0Me/SFrt7jea2UmSTomxJgBAHzmD2szOkPQXkhokyd2/kvRVvGUBAA6LMvQxWlK7pH8zs/fN7Odm9q1Nl82s0cxazKyFudIAkD9RgnqYpGmSHnf3qZIOSvrJ0Qe5e5O7p9w9VVlZmecyAWDwihLUbZLa3P3dzPMVSgc3ACABOYPa3f9H0u/N7PCtRmZL2hprVQCAHlFnfXxf0rOZGR+7JP11fCUBAPqKFNTuvllSKuZaAABZsDIRAAJHUANA4AhqAAgcu+cBSNzSNdsjHXfPnHExV1Ic6FEDQOAIagAIHEENAIEjqAEgcAQ1AASOoAaAwBHUABA4ghoAAseCFwAFVftZ07FfXFuRXCEBo0cNAIEjqAEgcAQ1AASOMWoARW/9rs5Ix80cU5xj3vSoASBwBDUABI6gBoDARRqjNrPdkv4oqUvSN+7OjW4BICEDuZhY5+4dsVUCAMiKoQ8ACFzUoHZJ/21mG82sMdsBZtZoZi1m1tLe3p6/CgFgkIs69HGZu+8xs3MkrTGzbe7+Rt8D3L1JUpMkpVIpz3OdAIpA1JvWYmAi9ajdfU/mz32SVkmaEWdRAIBeOYPazE41s9MOP5Z0laTfxV0YACAtytDHCEmrzOzw8b9y99WxVgUA6JEzqN19l6SLEqgFAJAF0/MAIHAENQAEjm1OAQQr6valpY4eNQAEjqAGgMAR1AAQOIIaAALHxUQAsaj9rKnQJZQMetQAEDiCGgACR1ADQOAYowaQE/tMFxY9agAIHEENAIEjqAEgcAQ1AASOoAaAwBHUABA4ghoAAhd5HrWZDZXUImmPu8+LryQAiEfUGxHMHFMRcyUDM5Ae9Q8ltcZVCAAgu0g9ajOrklQv6aeS/i7WigAgNGsfiXZc3X2xfHzUHvU/S/p7Sd3HOsDMGs2sxcxa2tvb81IcACBCUJvZPEn73H1jf8e5e5O7p9w9VVlZmbcCAWCwi9KjvlTStWa2W9Jzkq4ws3+PtSoAQI+cQe3u97l7lbuPknSrpN+6++2xVwYAkMQ8agAI3oD2o3b3ZknNsVQCoChwL8Tk0aMGgMAR1AAQOIIaAAJHUANA4Li5LQAcJbTNm+hRA0DgCGoACBxBDQCBI6gBIHAENQAEjqAGgMAR1AAQOIIaAAJHUANA4AhqAAgcS8iBQWzpmu2FLgER0KMGgMAR1AAQOIY+AEjiFlsho0cNAIHLGdRmVm5mG8xsi5l9ZGb/mERhAIC0KEMff5J0hbsfMLMySevM7L/c/Z2YawMAKEJQu7tLOpB5Wpb58jiLAgD0ijRGbWZDzWyzpH2S1rj7u1mOaTSzFjNraW9vz3edADBoRQpqd+9y94slVUmaYWaTsxzT5O4pd09VVlbmu04AGLQGNOvD3b+QtFbS3HjKAQAcLcqsj0ozOzPz+GRJcyRti7swAEBalFkf50r6hZkNVTrYX3D3V+ItCwBwWJRZHx9ImppALQCALFhCDhSR49ntrr+l4bUnUgwSwxJyAAgcPWogAOwLjf7QowaAwBHUABA4ghoAAkdQA0DgCGoACBxBDQCBI6gBIHAENQAEjqAGgMAR1AAQOIIaAAJHUANA4AhqAAgcQQ0AgSOoASBw7EcNFKn+7tyC0hLlLuQXmNlaM9tqZh+Z2Q+TKAwAkBalR/2NpB+7+yYzO03SRjNb4+5bY64NAKBodyH/g6Q/ZB7/0cxaJZ0viaAGcjh8i61cwxR9bzL7TnVjjBWhGA3oYqKZjZI0VdK7cRQDAPi2yBcTzWy4pJWSfuTu+7O83iipUZKqq6vzViCQpKg3mb1nzriYKwF6RQpqMytTOqSfdfcXsx3j7k2SmiQplUp53ioEQrb2kX5frv2sM6FCUMqizPowSU9JanX3f4q/JABAX1HGqC+V9JeSrjCzzZmvq2OuCwCQEWXWxzpJlkAtAIAsWEIOAIEjqAEgcOz1ARyH3oUszOpA/AhqIDBstoSjEdQIUr4XnkQ9HxAixqgBIHAENQAEjqAGgMAR1AAQOIIaAAJHUANA4Jieh0QxTQ4YOHrUABA4etQIWq5VeuufSv/JfQZRyuhRA0DgCGoACBxDH0AWbIyEkBDUyIuBzubIdxASrChlDH0AQODoUWfke1tNAMiXnD1qM3vazPaZ2e+SKAgAcKQoQx/PSJobcx0AgGPIOfTh7m+Y2aj4SxmcCjXkwlAPUDzyNkZtZo2SGiWpuro6X6fF8Vj7SM5Daj/rZDUfUCTyFtTu3iSpSZJSqZTn67wYmKVrtnNnbKDEMOsDudFDBwqKoC4SbA8KDF5Rpuctk7Re0ngzazOzO+MvCwBwWJRZHwuSKOSw9U/dG+m4mXc+FnMlGCiWcQPxYOijiBCEwOBEUKNfzCIBCo9NmQAgcAQ1AASueIc+IsztlaSl39wQcyHH+Nw12yONKdeK+/0B6F/xBjVOGBcngeJQtEG9flfEC1xsOwKgyDFGDQCBK9oedVRRf72POk4cdUFObaSjACA3etQAELiS71EXAy7qAegPPWoACBxBDQCBI6gBIHCMUWcwTgwgVPSoASBwBDUABI6gBoDAEdQAEDiCGgACFymozWyumX1sZjvM7CdxFwUA6JUzqM1sqKSfSfqOpEmSFpjZpLgLAwCkRelRz5C0w913uftXkp6TND/esgAAh0VZ8HK+pN/3ed4m6c+PPsjMGiUd3iv0gJl9fJw1nS2p4zjfW0poh160RS/aIi3Qdrj/RN682t3nZnshbysT3b1J0gkv7zOzFndP5aGkokY79KItetEWaYOtHaIMfeyRdEGf51WZvwMAJCBKUL8naayZjTazkyTdKunX8ZYFADgs59CHu39jZndL+o2koZKedvePYqyJ3ZHSaIdetEUv2iJtULWDuXuhawAA9IOViQAQOIIaAAJXkKDOtSTdzBrMrN3MNme+/qYQdSYhyvJ8M7vZzLaa2Udm9quka0xKhO+LpX2+J7ab2ReFqDMJEdqi2szWmtn7ZvaBmV1diDrjFqEdLjSz1zJt0GxmVYWoM3bunuiX0hckd0oaI+kkSVskTTrqmAZJ/5p0bYG2xVhJ70s6K/P8nELXXai2OOr47yt9YbvgtRfo+6JJ0t9mHk+StLvQdReoHZZL+qvM4ysk/bLQdcfxVYgeNUvSe0Vpi+9J+pm7/68kufu+hGtMykC/LxZIWpZIZcmL0hYu6fTM4zMk7U2wvqREaYdJkn6bebw2y+sloRBBnW1J+vlZjrsh8+vMCjO7IMvrpSBKW4yTNM7M3jKzd8ws6xLTEhD1+0JmdqGk0er9AS01UdriIUm3m1mbpP9U+jeMUhOlHbZI+m7m8fWSTjOzigRqS1SoFxP/Q9Iod58iaY2kXxS4nkIapvTwxyyle5FPmtmZBa2o8G6VtMLduwpdSAEtkPSMu1dJulrSL80s1J/nON0r6XIze1/S5Uqvmi6574tC/MPmXJLu7p3u/qfM059LuiSh2pIWZXl+m6Rfu/vX7v6JpO1KB3epGchWBbeqdIc9pGhtcaekFyTJ3ddLKld6o6JSEiUr9rr7d919qqR/yPxdyV1kLkRQ51ySbmbn9nl6raTWBOtLUpTl+S8p3ZuWmZ2t9FDIriSLTEikrQrMbIKksyStT7i+JEVpi88kzZYkM5uodFC3J1pl/KJkxdl9fpO4T9LTCdeYiMSD2t2/kXR4SXqrpBfc/SMze9jMrs0c9oPMVLQtkn6g9CyQkhOxLX4jqdPMtip9sWSRu3cWpuL4RGwLKf3D+pxnLvOXooht8WNJ38v8jCyT1FBqbRKxHWZJ+tjMtksaIemnBSk2ZiwhB4DADcaLDwBQVAhqAAgcQQ0AgSOoASBwBDUABI6gBoDAEdQAELj/B2jd6PyC0ylnAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lV7Po8lg8VIM"
      },
      "source": [
        "**Questions:**\n",
        "- Why do we use `z` variable instead of `losses` variable?\n",
        "- How can we enforce subgroup fairness and individual fairness at the same time?\n",
        "- Does it scale if we have more than 2 subgroups?\n",
        "\n",
        "\n",
        "**What did we learn (\"the take home\" messages)?**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DMhsOnT0BYM2"
      },
      "source": [
        "## Bonus Questions\n",
        "\n",
        "- Why the last method is called moments penalization?\n",
        "- What about the third moment?\n",
        "- In settings with multiple subgroups, how can we even further enforce fairness? (hint: loss function)\n",
        "- Why does the method also works for regression?\n",
        "- Explain the original CVaR method ([equation 27](http://proceedings.mlr.press/v97/williamson19a/williamson19a.pdf))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GCAQso3jAfS0"
      },
      "source": [
        "## References\n",
        "\n",
        "1. Michele Donini, et al. \"Empirical Risk Minimization under Fairness Constraints.\" – the dataset\n",
        "1. Robert Williamson, Aditya Menon \"Fairness risk measures\" – Conditional Value at Risk\n",
        "1. Anonymous Authors – \"The weighted mean trick: Optimization strategies for fairness and robustness\" – Moments penalization method\n",
        "\n",
        "This code is freely available on github.\n",
        "\n",
        "https://github.com/valeriu-balaban/fair-ml-presentation/blob/main/Fair_ML_Presentation.ipynb "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZX5AAE6rCIwk"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}