W1D5 Post-Course Update

tutorials/W1D5_Microcircuits/W1D5_Tutorial1.ipynb

@@ -1166,6 +1166,18 @@
     "interact(plot_kurtosis, theta_value = slider)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "execution": {}
+   },
+   "source": [
+    "<details>\n",
+    "    <summary>Kurtosis value behaviour</summary>\n",
+    "    You might notice that, at first, the kurtosis value decreases (around till $\\theta = 140$), and then it drastically increases (reflecting the desired sparsity property). If we take a closer look at the kurtosis formula, it measures the expected value (average) of standardized data values raised to the 4th power. That being said, if the data point lies in the range of standard deviation, it doesn’t contribute to the kurtosis value almost at all (something less than 1 to the fourth degree is small), and most of the contribution is produced by extreme outliers (lying far away from the range of standard deviation). So, the main characteristic it measures is the tailedness of the data - it will be high when the power of criticality of outliers will overweight the “simple” points (as kurtosis is an average metric for all points). What happens is that with $\\theta \\le 120$, outliers don't perform that much to the kurtosis.\n",
+    "</details>\n"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

tutorials/W1D5_Microcircuits/W1D5_Tutorial2.ipynb

@@ -638,6 +638,8 @@
     "\n",
     "$$\\hat{x} = \\frac{x}{f(||x||)}$$\n",
     "\n",
+    "There are indeed many options for the specific form of the denominator here; still, what we want to highlight is the essential divisive nature of the normalization.\n",
+    "\n",
     "Evidence suggests that normalization provides a useful inductive bias in artificial and natural systems. However, do we need a dedicated computation that implements normalization?\n",
     "\n",
     "Let's explore if ReLUs can estimate a normalization-like function. Specifically, we will see if a fully-connected one-layer network can estimate $y=\\frac{1}{x+\\epsilon}$ function.\n",

tutorials/W1D5_Microcircuits/further_reading.md

@@ -14,6 +14,8 @@
 - [Flexible gating of contextual influences in natural vision](https://pubmed.ncbi.nlm.nih.gov/26436902/)
 - [Normalization as a canonical neural computation](https://www.nature.com/articles/nrn3136)
 - [Attention-related changes in correlated neuronal activity arise from normalization mechanisms](https://www.nature.com/articles/nn.4572)
+- [Spatially tuned normalization explains attention modulation variance within neurons](https://journals.physiology.org/doi/full/10.1152/jn.00218.2017)
+- [Attention-related changes in correlated neuronal activity arise from normalization mechanisms](https://www.nature.com/articles/nn.4572)
 
 ## Tutorial 3: Attention
 

tutorials/W1D5_Microcircuits/instructor/W1D5_Tutorial1.ipynb

@@ -1168,6 +1168,18 @@
     "interact(plot_kurtosis, theta_value = slider)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "execution": {}
+   },
+   "source": [
+    "<details>\n",
+    "    <summary>Kurtosis value behaviour</summary>\n",
+    "    You might notice that, at first, the kurtosis value decreases (around till $\\theta = 140$), and then it drastically increases (reflecting the desired sparsity property). If we take a closer look at the kurtosis formula, it measures the expected value (average) of standardized data values raised to the 4th power. That being said, if the data point lies in the range of standard deviation, it doesn’t contribute to the kurtosis value almost at all (something less than 1 to the fourth degree is small), and most of the contribution is produced by extreme outliers (lying far away from the range of standard deviation). So, the main characteristic it measures is the tailedness of the data - it will be high when the power of criticality of outliers will overweight the “simple” points (as kurtosis is an average metric for all points). What happens is that with $\\theta \\le 120$, outliers don't perform that much to the kurtosis.\n",
+    "</details>\n"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

tutorials/W1D5_Microcircuits/instructor/W1D5_Tutorial2.ipynb

@@ -638,6 +638,8 @@
     "\n",
     "$$\\hat{x} = \\frac{x}{f(||x||)}$$\n",
     "\n",
+    "There are indeed many options for the specific form of the denominator here; still, what we want to highlight is the essential divisive nature of the normalization.\n",
+    "\n",
     "Evidence suggests that normalization provides a useful inductive bias in artificial and natural systems. However, do we need a dedicated computation that implements normalization?\n",
     "\n",
     "Let's explore if ReLUs can estimate a normalization-like function. Specifically, we will see if a fully-connected one-layer network can estimate $y=\\frac{1}{x+\\epsilon}$ function.\n",

tutorials/W1D5_Microcircuits/student/W1D5_Tutorial1.ipynb

@@ -1151,6 +1151,18 @@
     "interact(plot_kurtosis, theta_value = slider)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "execution": {}
+   },
+   "source": [
+    "<details>\n",
+    "    <summary>Kurtosis value behaviour</summary>\n",
+    "    You might notice that, at first, the kurtosis value decreases (around till $\\theta = 140$), and then it drastically increases (reflecting the desired sparsity property). If we take a closer look at the kurtosis formula, it measures the expected value (average) of standardized data values raised to the 4th power. That being said, if the data point lies in the range of standard deviation, it doesn’t contribute to the kurtosis value almost at all (something less than 1 to the fourth degree is small), and most of the contribution is produced by extreme outliers (lying far away from the range of standard deviation). So, the main characteristic it measures is the tailedness of the data - it will be high when the power of criticality of outliers will overweight the “simple” points (as kurtosis is an average metric for all points). What happens is that with $\\theta \\le 120$, outliers don't perform that much to the kurtosis.\n",
+    "</details>\n"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

tutorials/W1D5_Microcircuits/student/W1D5_Tutorial2.ipynb

@@ -638,6 +638,8 @@
     "\n",
     "$$\\hat{x} = \\frac{x}{f(||x||)}$$\n",
     "\n",
+    "There are indeed many options for the specific form of the denominator here; still, what we want to highlight is the essential divisive nature of the normalization.\n",
+    "\n",
     "Evidence suggests that normalization provides a useful inductive bias in artificial and natural systems. However, do we need a dedicated computation that implements normalization?\n",
     "\n",
     "Let's explore if ReLUs can estimate a normalization-like function. Specifically, we will see if a fully-connected one-layer network can estimate $y=\\frac{1}{x+\\epsilon}$ function.\n",