9-Non-Compliance-and-LATE.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dipping our Toes into a Heterogeneous World\n",
    "\n",
    "Previously, we've seen Instrumental Variables through a more traditional lens. IV was seen as some sort of natural experiment we can leverage. In contrast, modern IV practice draws a lot of insight from medical sciences. It partitions the world into 3 kids of subjects, depending on how they respond to the instrument variable.\n",
    "\n",
    "1. Compliers\n",
    "2. Never Takers\n",
    "3. Always Takers\n",
    "4. Defiers\n",
    "\n",
    "This naming comes from pharmaceutical science. Imagine you are conducting an experiment to test the effect of a new drug on some illness. Each subject gets assigned to a treatment: a drug or a placebo. The compliers are the subjects that stick to what was assigned to them. If they get the placebo, they take it; if they get the drug, they also take it. The never takers are those subjects that refuse to take their medicine. Even if they are assigned the new drug, they won't take it. On the other hand, the always takers are those that can somehow get the new drug even if they were assigned to the placebo. Finally, Defiers are the ones that take the treatment if assigned to the control and take the control if assigned the treatment. You can think of them as that annoying child that does the opposite of what it is told. In practice, they are not so common (the defiers, not the kids), so we will often ignore them.\n",
    "\n",
    "![img](./data/img/late/defiers.png)\n",
    "\n",
    "Modern IV treats the instrument as a quasi-experimental design where compliance is not perfect. By doing so, it distinguishes internally from externally valid causal effect. Just as a reminder, an internally valid effect is one we are able to identify. It is valid in that particular setting, with that particular data. In IV, it would be the treatment effect on those where the instrument changes the treatment. External validity, on the other hand, would be concerned with the predictive power of that causal effect. It would ask the question of whether we could generalize the effect we found in this sample for other populations. For example, let's say you conducted an RCT in your university to figure out if people are generous or not, when given an incentive to donate. The experiment is well designed, but you only invite economics students to participate. You then find that all of them are selfish jerks. This is an internally valid conclusion. It's valid for those data points. But can you, from that experiment, infer that humanity is selfish? That's hardly the case. So we would question whether your experiment has external validity to generalize its results. Anyway, back to IV.\n",
    "\n",
    "To make it more concrete, let's consider a case where you want to boost user engagement measured by in app purchase. One way to do that, is by asking your marketing department to come up with a push you can use to engage your users. They come up with a brilliant design and very fancy user interactions. With that push, you move on to design a randomised trial. You select 10000 random customers and, for each of them, you assign the push with 50% probability. However, when you execute the test, you notice that some customers who were assigned to receive the push are not receiving it. When you talk to the engineers, they say that it is because they probably have an older phone that doesn't support the kind of push the marketing team designed. \n",
    "\n",
    "At first, you might think that this is no big deal. Instead of using treatment assigned as your treatment variable, you can just use the treatment received, right? It turns out it is not that simple. If you plot the causal graph of this whole situation, it would look something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy import stats\n",
    "from matplotlib import style\n",
    "import seaborn as sns\n",
    "from matplotlib import pyplot as plt\n",
    "import statsmodels.formula.api as smf\n",
    "from linearmodels.iv import IV2SLS\n",
    "import graphviz as gr\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "style.use(\"fivethirtyeight\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"218pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 217.62 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-184 213.6163,-184 213.6163,4 -4,4\"/>\n",
       "<!-- push assigned -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>push assigned</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"59.48\" cy=\"-162\" rx=\"59.4599\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"59.48\" y=\"-157.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">push assigned</text>\n",
       "</g>\n",
       "<!-- push delivered -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>push delivered</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"83.48\" cy=\"-90\" rx=\"61.8481\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"83.48\" y=\"-85.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">push delivered</text>\n",
       "</g>\n",
       "<!-- push assigned&#45;&gt;push delivered -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>push assigned&#45;&gt;push delivered</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M65.5362,-143.8314C68.1312,-136.0463 71.2223,-126.7729 74.1017,-118.1347\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"77.5003,-119.0069 77.3422,-108.4133 70.8595,-116.7933 77.5003,-119.0069\"/>\n",
       "</g>\n",
       "<!-- in app purchase -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>in app purchase</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"128.48\" cy=\"-18\" rx=\"65.7062\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"128.48\" y=\"-13.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">in app purchase</text>\n",
       "</g>\n",
       "<!-- push delivered&#45;&gt;in app purchase -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>push delivered&#45;&gt;in app purchase</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M94.6036,-72.2022C99.85,-63.8079 106.2164,-53.6218 112.0076,-44.3558\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"114.9963,-46.1776 117.3284,-35.8425 109.0603,-42.4675 114.9963,-46.1776\"/>\n",
       "</g>\n",
       "<!-- income -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>income</title>\n",
       "<ellipse fill=\"none\" stroke=\"#0000ff\" cx=\"173.48\" cy=\"-162\" rx=\"36.2732\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"173.48\" y=\"-157.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">income</text>\n",
       "</g>\n",
       "<!-- income&#45;&gt;push delivered -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>income&#45;&gt;push delivered</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M153.9513,-146.3771C141.9637,-136.787 126.3802,-124.3202 112.9166,-113.5493\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"114.8538,-110.6169 104.8586,-107.1029 110.4809,-116.083 114.8538,-110.6169\"/>\n",
       "</g>\n",
       "<!-- income&#45;&gt;in app purchase -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>income&#45;&gt;in app purchase</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M170.9448,-143.7268C168.0824,-125.2955 162.7207,-96.1857 154.48,-72 151.4078,-62.9835 147.208,-53.5101 143.0798,-45.0596\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"146.1505,-43.3758 138.5135,-36.0323 139.9041,-46.5354 146.1505,-43.3758\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x12a416358>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "g = gr.Digraph()\n",
    "\n",
    "g.edge(\"push assigned\", \"push delivered\")\n",
    "g.edge(\"push delivered\", \"in app purchase\")\n",
    "g.edge(\"income\", \"in app purchase\")\n",
    "g.edge(\"income\", \"push delivered\")\n",
    "g.node(\"income\", color=\"blue\")\n",
    "g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Up on the causal graph, you have the push assignment. This is random by design, so nothing is causing it. Then, you have a node for if the push was delivered. Not everyone that was assigned to receive the push did it, so you have non compliance here. More specifically, you have some never takers: those that don't receive the treatment even if assigned to it. You also have reasons to suspect that this non compliance is not simply by chance. Since people with older phones are the one that don't get the push, you can argue that income is also causing push delivery. The richer the person, the more likely it is that he or she has a nicer phone, which in turn makes it more likely that he or she will receive the push. Finally, you have the outcome variable, in app purchase. Keep in mind that we don't know income, so we can't control it. With this in mind, let's examine what would happen if we simply used the push assigned as the treatment variable and if instead we used push delivered as the treatment. \n",
    "\n",
    "In the first case, we would estimate the causal effect by the following difference in means:\n",
    "\n",
    "$\n",
    "ATE = E[Y | pushAssigned=1] - E[Y | pushAssigned=0]\n",
    "$\n",
    "\n",
    "As we are very aware by now, this is only unbiased estimate for \\\\(E[Y_1] - E[Y_0]\\\\) if the bias \\\\(E[Y_0|pushAssigned=0] - E[Y_0|pushAssigned=1]\\\\) is zero. Since `pushAssigned` is random, we know that the bias is zero. Does this settle it then? Not quite. You see, if we do this, we are actually answering a different question than the one we intended to. We would be finding the **causal effect of treatment assignment**, not of the treatment itself. But are they different or can we extrapolate the causal effect of treatment assignment to the ATE? In other words, is the causal effect of treatment assignment an unbiased estimate of the ATE? \n",
    "\n",
    "As it turns out, it is not. Because of non compliance, the outcome of those assigned to the treatment will be pushed in the direction of the outcome of those assigned the control. Non compliance flips the treatment unintendedly, making the treatment and control more similar in the outcome. Don't confuse this with similarity on the variables. We want treated and control to be similar on the variables. That will make them comparable. What we don't want is for them to be similar in the outcome if there is indeed a treatment effect.\n",
    "\n",
    "To see this, suppose first we have always takers. Some of them will be assigned to the control by chance. But those that are, will take the treatment anyway. This makes them essentially a treated group that is mixed up with the control. As a result of this mix, the causal effect will be harder to find when we have non compliance.\n",
    "\n",
    "![img](./data/img/late/always_takers.png)\n",
    "\n",
    "By the same reasoning, never takers will make those assigned to the treatment look a little bit like the untreated, because they don't take the treatment even if assigned to it. In this sense, **the causal effect of treatment assignment is biased towards zero** because non compliance shrinks the detectable impact. Another way to see this is to imagine an extreme case. Let's say that non compliance is really high. Treatment assignment says nothing about treatment received. Treatment received is purely random in this case. In IV language, that would be saying we have a very weak 1st stage. Using `Z` to denote the treatment assignment, we would have that\n",
    "\n",
    "$\n",
    "E[Y|Z=1] - E[Y|Z=0] = 0\n",
    "$\n",
    "\n",
    "There would be no more causal link of treatment assignment to the outcome. Z would be just a meaningless random variable floating there. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"247pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 247.08 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-184 243.083,-184 243.083,4 -4,4\"/>\n",
       "<!-- push assigned -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>push assigned</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"59.48\" cy=\"-162\" rx=\"59.4599\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"59.48\" y=\"-157.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">push assigned</text>\n",
       "</g>\n",
       "<!-- push delivered -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>push delivered</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"128.48\" cy=\"-90\" rx=\"61.8481\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"128.48\" y=\"-85.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">push delivered</text>\n",
       "</g>\n",
       "<!-- in app purchase -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>in app purchase</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"173.48\" cy=\"-18\" rx=\"65.7062\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"173.48\" y=\"-13.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">in app purchase</text>\n",
       "</g>\n",
       "<!-- push delivered&#45;&gt;in app purchase -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>push delivered&#45;&gt;in app purchase</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M139.6036,-72.2022C144.85,-63.8079 151.2164,-53.6218 157.0076,-44.3558\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"159.9963,-46.1776 162.3284,-35.8425 154.0603,-42.4675 159.9963,-46.1776\"/>\n",
       "</g>\n",
       "<!-- income -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>income</title>\n",
       "<ellipse fill=\"none\" stroke=\"#0000ff\" cx=\"173.48\" cy=\"-162\" rx=\"36.2732\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"173.48\" y=\"-157.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">income</text>\n",
       "</g>\n",
       "<!-- income&#45;&gt;push delivered -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>income&#45;&gt;push delivered</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M162.5867,-144.5708C157.3039,-136.1184 150.8457,-125.7851 144.9756,-116.3931\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"147.8535,-114.3939 139.5855,-107.7689 141.9175,-118.1039 147.8535,-114.3939\"/>\n",
       "</g>\n",
       "<!-- income&#45;&gt;in app purchase -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>income&#45;&gt;in app purchase</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M184.2576,-144.7385C190.0168,-134.4847 196.5168,-120.9847 199.48,-108 203.0397,-92.401 203.0397,-87.599 199.48,-72 197.3607,-62.7132 193.4322,-53.1628 189.2595,-44.725\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"192.2859,-42.9603 184.5281,-35.7445 186.0928,-46.2231 192.2859,-42.9603\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x12a416550>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "g = gr.Digraph()\n",
    "\n",
    "g.node(\"push assigned\")\n",
    "g.edge(\"push delivered\", \"in app purchase\")\n",
    "g.edge(\"income\", \"in app purchase\")\n",
    "g.edge(\"income\", \"push delivered\")\n",
    "g.node(\"income\", color=\"blue\")\n",
    "g"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, so we've ruled out using the causal effect of the assignment as a way to estimate the causal effect of the treatment. So, how about just using the treatment received? \n",
    "\n",
    "$\n",
    "ATE = E[Y | push=1] - E[Y | push=0]\n",
    "$\n",
    "\n",
    "Once again, we need to think if this is biased or not, or if \\\\(E[Y_0|push=0] = E[Y_0|push=1]\\\\). Just by looking at the causal graph above, we know it is not the case. We have that unmeasured confounder, income, lurking around, and it is sure to mess things up. As we've said earlier, we know that failure in push delivery, in our case, is caused by customers having older phones. Which means that we probably have  \\\\(E[Y_0|push=0] < E[Y_0|push=1]\\\\). We think this is the case because customers with less money both have older phones which will lead to \\\\(push=0\\\\) and also a lower potential in app purchase \\\\(Y_0\\\\). \n",
    "\n",
    "Bummer! We can't use treatment assigned nor treatment received to estimate our ATE. But fortunately, we know what we can use: Instrumental Variables. Here, treatment assigned is the perfect instrument for the treatment. It is as good as random and it only causes in-app purchase by the virtue of the treatment.\n",
    "\n",
    "## Local Average Treatment Effect: LATE\n",
    "\n",
    "Local average treatment effect makes it explicit the population for which we can estimate the causal effect. It is also another way to view IV, which gives other cool intuitions we can use. In modern IV, we see the instrument as starting a causal chain: Z causes T which causes Y. In this context, the exclusion restriction means that Z does not cause Y, unless by its effect on T. The 1st stage is now viewed as the causal effect of Z on T. We also rewrite potential outcomes with a double indexing notation, where the first index is denote de counterfactual for the instrument and the second index, for the treatment\n",
    "\n",
    "$\n",
    "\\text{Potential Outcome}=\\begin{cases}\n",
    "Y_i(1, 1) \\ \\text{if } T_i=1, \\ Z_i=1\\\\\n",
    "Y_i(1, 0) \\ \\text{if } T_i=1, \\ Z_i=0\\\\\n",
    "Y_i(0, 1) \\ \\text{if } T_i=0, \\ Z_i=1\\\\\n",
    "Y_i(0, 0) \\ \\text{if } T_i=0, \\ Z_i=0\\\\\n",
    "\\end{cases}\n",
    "$\n",
    "\n",
    "In a sense, the treatment becomes the outcome, at least in the first stage. This means we can also write it with potential outcome notation:\n",
    "\n",
    "$\n",
    "\\text{Potential Treatment}=\\begin{cases}\n",
    "T_0 \\ \\text{if } Z_i=0 \\\\\n",
    "T_1 \\ \\text{if } Z_i=1\n",
    "\\end{cases}\n",
    "$\n",
    "\n",
    "![img](./data/img/late/double_index.png)\n",
    "\n",
    "Instrumental Variable assumptions can now be rewritten as follows\n",
    "\n",
    "1. \\\\(T_{0i}, T_{1i} \\perp Z_i \\\\) and \\\\(Y_i(T_{1i},1), Y_i(T_{0i},0) \\perp Z_i \\\\). This is the independence Assumption. This says that the instrument is as good as randomly assigned. In other words, Z, the instrument, is not correlated with the potential treatments, which is the same as saying that people in different instrument groups are comparable. \n",
    "\n",
    "2. \\\\(Y_i(1, 0)=Y_i(1, 1)=Y_{i1}\\\\) and \\\\(Y_i(1, 0)=Y_i(1, 1)=Y_{i0}\\\\). This is the exclusion restriction. It says that if I'm looking at the potential outcome for the treated, it is the same for both instrument groups. In other words, the instrument does not affect the potential outcome, which is the same as saying that the instrument only affects the outcome through the treatment.\n",
    "\n",
    "3. \\\\(E[T_{1i}-T_{0i}] \\neq 0\\\\). This is the existence of a 1st stage. It is saying that the potential outcome of the 1st stage, that is, the potential treatment, are NOT the same. Another way of saying this is that the instrument does affect the treatment.\n",
    "\n",
    "4. \\\\(T_{i1} > T_{i0}\\\\). This is the monotonicity assumption. It is saying that if everyone had the instrument turned on, the treatment level would be higher than if everyone had the treatment turned off. \n",
    "\n",
    "Now, let's review the Wald estimator to gain some further intuition on IV:\n",
    "\n",
    "$\n",
    "ATE = \\dfrac{E[Y|Z=1]-E[Y|Z=0]}{E[T|Z=1]-E[T|Z=0]}\n",
    "$\n",
    "\n",
    "Let's take the first bit of it, \\\\(E[Y|Z=1]\\\\). Using the exclusion restriction, we can rewrite Y in terms of potential outcome like this.\n",
    "\n",
    "$\n",
    "E[Y_i|Z_i=1]=E[Y_{i0} + T_{i1}(Y_{i1} - Y_{i0})|Z=1]\n",
    "$\n",
    "\n",
    "Using independence, we can take out the conditioning on Z.\n",
    "\n",
    "$\n",
    "E[Y_i|Z_i=1]=E[Y_{i0} + T_{i1}(Y_{i1} - Y_{i0})]\n",
    "$\n",
    "\n",
    "With a similar argument, we get that\n",
    "\n",
    "$\n",
    "E[Y_i|Z_i=0]=E[Y_{i0} + T_{i0}(Y_{i1} - Y_{i0})]\n",
    "$\n",
    "\n",
    "We can now rewrite the numerator of the Wald estimator like this\n",
    "\n",
    "$\n",
    "E[Y|Z=1]-E[Y|Z=0] = E[(Y_{i1}-Y_{i0})(T_{i1}-T_{i0})]\n",
    "$\n",
    "\n",
    "Using monotonicity, we know that \\\\(T_{i1}-T_{i0}\\\\) is 0 or 1, so\n",
    "\n",
    "$\n",
    "E[(Y_{i1}-Y_{i0})(T_{i1}-T_{i0})] = E[(Y_{i1}-Y_{i0})|T_{i1}>T_{i0}]P(T_{i1}>T_{i0})\n",
    "$\n",
    "\n",
    "Using a similar argument to tackle the denominator, we get that\n",
    "\n",
    "$\n",
    "E[T|Z=1]-E[T|Z=0]=E[T_{i1}-T_{i0}]=P(T_{i1}>T_{i0})\n",
    "$\n",
    "\n",
    "All of this allows us to look at the Wald estimator like this:\n",
    "\n",
    "$\n",
    "ATE = \\dfrac{E[(Y_{i1}-Y_{i0})|T_{i1}>T_{i0}]P(T_{i1}>T_{i0})}{P(T_{i1}>T_{i0})}=E[(Y_{i1}-Y_{i0})|T_{i1}>T_{i0}]\n",
    "$\n",
    "\n",
    "That is, the ATE as estimated by IV is the ATE on the subpopulation where \\\\(T_{i1}>T_{i0}\\\\). And if you think about compliance, which population is this? This is the population where those that get the instrument turned on have the treatment level higher than if they had the instrument turned off. In other words, this is the compiler population. Just so we can remember,\n",
    "\n",
    "1. Compilers means that \\\\(T_{i1}>T_{i0}\\\\)\n",
    "2. Never Takers \\\\(T_{i1}=T_{i0}=0\\\\)\n",
    "3. Always Takers \\\\(T_{i1}=T_{i0}=1\\\\)\n",
    "\n",
    "The conclusion of this is that IV says nothing about the effect on never takers, always takers or defiers, because the treatment is unchanged for them! **IV only finds the treatment effect for the compliers**.\n",
    "\n",
    "## Effect on Engagement\n",
    "\n",
    "Let's see how all of this plays out in the case study where we try to estimate the effect of a push on in-app purchase. The causal graph is the one we depicted above, so I won't repeat it here. The data we have is on both push assigned, the randomised instrument, and push delivered, the treatment variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>in_app_purchase</th>\n",
       "      <th>push_assigned</th>\n",
       "      <th>push_delivered</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>47</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>43</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>51</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>79</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   in_app_purchase  push_assigned  push_delivered\n",
       "0               47              1               1\n",
       "1               43              1               0\n",
       "2               51              1               1\n",
       "3               49              0               0\n",
       "4               79              0               0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv(\"./data/app_engagement_push.csv\")\n",
    "    \n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's run OLS to see what it would give us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Parameter Estimates</caption>\n",
       "<tr>\n",
       "         <td></td>        <th>Parameter</th> <th>Std. Err.</th> <th>T-stat</th>  <th>P-value</th> <th>Lower CI</th> <th>Upper CI</th>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>       <td>69.292</td>    <td>0.3624</td>   <td>191.22</td>  <td>0.0000</td>   <td>68.581</td>   <td>70.002</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>push_assigned</th>   <td>-17.441</td>   <td>0.5702</td>   <td>-30.590</td> <td>0.0000</td>   <td>-18.559</td>  <td>-16.324</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>push_delivered</th>  <td>27.600</td>    <td>0.6124</td>   <td>45.069</td>  <td>0.0000</td>   <td>26.399</td>   <td>28.800</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.table.SimpleTable'>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ols = IV2SLS.from_formula(\"in_app_purchase ~ 1 + push_assigned + push_delivered\", data).fit()\n",
    "ols.summary.tables[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OLS is saying that the treatment effect is R$ 27.60, that is, the push increases in-app purchase by 27.6 reais. However, we have reasons to believe this is a biased estimate. We know that older phones are having trouble in receiving the push, so, probably, richer customers, with newer phones, are the compilers. Since the ones that get the treatment also have more money, we believe this bias is positive and the true impact of the push is lower. In other words, we probably have \\\\(E[Y_0|T=0] < E[Y_0|T=1]\\\\).\n",
    "\n",
    "Now, let's try to estimate this effect with Instrumental Variables. First, let's run the first stage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Parameter Estimates</caption>\n",
       "<tr>\n",
       "        <td></td>        <th>Parameter</th> <th>Std. Err.</th> <th>T-stat</th> <th>P-value</th>  <th>Lower CI</th>  <th>Upper CI</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>      <td>0.0000</td>   <td>7.408e-11</td> <td>0.0000</td> <td>1.0000</td>  <td>-1.452e-10</td> <td>1.452e-10</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>push_assigned</th>  <td>0.7176</td>    <td>0.0064</td>   <td>112.07</td> <td>0.0000</td>    <td>0.7050</td>    <td>0.7301</td>  \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.table.SimpleTable'>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_stage = IV2SLS.from_formula(\"push_delivered ~ 1 + push_assigned\", data).fit()\n",
    "first_stage.summary.tables[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like we have a strong first stage. Those that get assigned to get the push get it 71.8% of the time. This means that we have something like 28% of never takers. We also have strong reasons to believe there are no never always takers, since the intercept parameter is estimated to be zero. This means that no one get's the push if it is not assigned to it. Given the design of our experiment, this is expected. \n",
    "\n",
    "Let's now run the reduced form:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Parameter Estimates</caption>\n",
       "<tr>\n",
       "        <td></td>        <th>Parameter</th> <th>Std. Err.</th> <th>T-stat</th> <th>P-value</th> <th>Lower CI</th> <th>Upper CI</th>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>      <td>69.292</td>    <td>0.3624</td>   <td>191.22</td> <td>0.0000</td>   <td>68.581</td>   <td>70.002</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>push_assigned</th>  <td>2.3636</td>    <td>0.5209</td>   <td>4.5376</td> <td>0.0000</td>   <td>1.3427</td>   <td>3.3845</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.table.SimpleTable'>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reduced_form = IV2SLS.from_formula(\"in_app_purchase ~ 1 + push_assigned\", data).fit()\n",
    "reduced_form.summary.tables[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reduced form shows that the causal effect of treatment assignment is 2.36. This means that assigning someone to receive the push increases the in-app purchase by 2.36 reais. \n",
    "\n",
    "If we divide the reduced form by the first stage, we scale the effect of the instrument by the units of the treatment, we get \\\\(2.3636/0.7176=3.29\\\\). Running the 2SLS, we get these same estimates, with the bonus of correct standard errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Parameter Estimates</caption>\n",
       "<tr>\n",
       "         <td></td>        <th>Parameter</th> <th>Std. Err.</th> <th>T-stat</th> <th>P-value</th> <th>Lower CI</th> <th>Upper CI</th>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>       <td>69.292</td>    <td>0.3624</td>   <td>191.22</td> <td>0.0000</td>   <td>68.581</td>   <td>70.002</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>push_delivered</th>  <td>3.2938</td>    <td>0.7165</td>   <td>4.5974</td> <td>0.0000</td>   <td>1.8896</td>   <td>4.6981</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.table.SimpleTable'>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iv = IV2SLS.from_formula(\"in_app_purchase ~ 1 + [push_delivered ~ push_assigned]\", data).fit()\n",
    "iv.summary.tables[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows that the result with 2SLS is much lower than the one we got with OLS: 3.29 against 27.60. This makes sense, since the causal effect estimated with OLS is positively biased. We also need to remember about LATE. 3.29 is the average causal effect on compilers. Unfortunately, we can't say anything about those never takers. This means that we are estimating the effect on the richer segment of the population that have newer phones.\n",
    "\n",
    "## Key Ideas\n",
    "\n",
    "Here we've looked at a more modern view of Instrumental Variables. We saw how IV can be seen as a causal chain where the instrument causes the treatment which causes the outcome. In this view, we took a look at compliance to understand ATE in IV estimation and we've figured out that it was the LATE on compilers.\n",
    "\n",
    "## References\n",
    "\n",
    "I like to think of this entire series as a tribute to Joshua Angrist, Alberto Abadie and Christopher Walters for their amazing Econometrics class. Most of the ideas here are taken from their classes at the American Economic Association. Watching them is what is keeping me sane during this tough year of 2020.\n",
    "* [Cross-Section Econometrics](https://www.aeaweb.org/conference/cont-ed/2017-webcasts)\n",
    "* [Mastering Mostly Harmless Econometrics](https://www.aeaweb.org/conference/cont-ed/2020-webcasts)\n",
    "\n",
    "I'll also like to reference the amazing books from Angrist. They have shown me that Econometrics, or 'Metrics as they call it, is not only extremely useful but also profoundly fun.\n",
    "\n",
    "* [Mostly Harmless Econometrics](https://www.mostlyharmlesseconometrics.com/)\n",
    "* [Mastering 'Metrics](https://www.masteringmetrics.com/)\n",
    "\n",
    "My final reference is Miguel Hernan and Jamie Robins' book. It has been my trustworthy companion in the most thorny causal questions I had to answer.\n",
    "\n",
    "* [Causal Inference Book](https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/)\n",
    "\n",
    "\n",
    "![img](./data/img/poetry.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}