docs/banking.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Open Diffix demo - pg_diffix\n",
        "\n",
        "This notebook demonstrates how to use the `pg_diffix`, a PostgreSQL extention that implements Diffix Elm anonymization.\n",
        "\n",
        "> Diffix Elm used here is missing several features of the next version Fir, which is still a work in progress.\n",
        "\n",
        "The notebook demonstrates\n",
        "1. The SQL limitations imposed by Diffix Elm\n",
        "2. How Diffix Elm perturbs data to achieve anonymization\n",
        "\n",
        "Feel free to use this notebook as a starting point for playing with Diffix Elm.\n",
        "\n",
        "## For more information\n",
        "\n",
        "To read more about Diffix Elm, visit [open-diffix.org](https://open-diffix.org).\n",
        "A good overview of Diffix Elm can be found [here](https://www.open-diffix.org/blog/diffix-elm-automates-what-statistics-offices-have-been-doing-for-decades/).\n",
        "A detailed description is [available on ArXiv](https://arxiv.org/abs/2201.04351).\n",
        "Besides including a **full specification**, the latter includes a complete privacy analysis and guidance for writing a risk assessment.\n",
        "\n",
        "## How to run\n",
        "\n",
        "To run this notebook locally you need `psycopg2` and `ipython-sql` installed. For the meta-commands you need `pgspecial`.\n",
        "\n",
        "Below you can find connection details for 3 users in this database instance:\n",
        "\n",
        "- `direct_user` with direct (non-anonymized) access to `banking`\n",
        "- `trusted_user` with anonymized access to `banking` in trusted mode\n",
        "- `untrusted_user` with anonymized access to `banking` in untrusted mode"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Connected to database.\n"
          ]
        }
      ],
      "source": [
        "%load_ext sql\n",
        "%sql postgresql://direct_user:demo@demo-pg.open-diffix.org/banking\n",
        "%sql postgresql://trusted_user:demo@demo-pg.open-diffix.org/banking\n",
        "%sql postgresql://untrusted_user:demo@demo-pg.open-diffix.org/banking\n",
        "%config SqlMagic.displaycon = False # No connection string in output.\n",
        "%config SqlMagic.feedback = False   # No Done, rows affected.\n",
        "print('Connected to database.')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The dataset has the following tables:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>Schema</th>\n",
              "        <th>Name</th>\n",
              "        <th>Type</th>\n",
              "        <th>Owner</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>public</td>\n",
              "        <td>accounts</td>\n",
              "        <td>table</td>\n",
              "        <td>postgres</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>public</td>\n",
              "        <td>accounts_receivables</td>\n",
              "        <td>table</td>\n",
              "        <td>postgres</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>public</td>\n",
              "        <td>clients</td>\n",
              "        <td>table</td>\n",
              "        <td>postgres</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>public</td>\n",
              "        <td>credit_cards</td>\n",
              "        <td>table</td>\n",
              "        <td>postgres</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>public</td>\n",
              "        <td>dispositions</td>\n",
              "        <td>table</td>\n",
              "        <td>postgres</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>public</td>\n",
              "        <td>loans</td>\n",
              "        <td>table</td>\n",
              "        <td>postgres</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>public</td>\n",
              "        <td>loss_events</td>\n",
              "        <td>table</td>\n",
              "        <td>postgres</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>public</td>\n",
              "        <td>orders</td>\n",
              "        <td>table</td>\n",
              "        <td>postgres</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>public</td>\n",
              "        <td>transactions</td>\n",
              "        <td>table</td>\n",
              "        <td>postgres</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('public', 'accounts', 'table', 'postgres'),\n",
              " ('public', 'accounts_receivables', 'table', 'postgres'),\n",
              " ('public', 'clients', 'table', 'postgres'),\n",
              " ('public', 'credit_cards', 'table', 'postgres'),\n",
              " ('public', 'dispositions', 'table', 'postgres'),\n",
              " ('public', 'loans', 'table', 'postgres'),\n",
              " ('public', 'loss_events', 'table', 'postgres'),\n",
              " ('public', 'orders', 'table', 'postgres'),\n",
              " ('public', 'transactions', 'table', 'postgres')]"
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%sql \\dt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can explore columns of tables with `\\d <table>`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>Column</th>\n",
              "        <th>Type</th>\n",
              "        <th>Modifiers</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>loan_id</td>\n",
              "        <td>integer</td>\n",
              "        <td> not null</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>account_id</td>\n",
              "        <td>integer</td>\n",
              "        <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>date</td>\n",
              "        <td>text</td>\n",
              "        <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>amount</td>\n",
              "        <td>integer</td>\n",
              "        <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>duration</td>\n",
              "        <td>integer</td>\n",
              "        <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>payments</td>\n",
              "        <td>double precision</td>\n",
              "        <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>status</td>\n",
              "        <td>text</td>\n",
              "        <td></td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[['loan_id', 'integer', ' not null'],\n",
              " ['account_id', 'integer', ''],\n",
              " ['date', 'text', ''],\n",
              " ['amount', 'integer', ''],\n",
              " ['duration', 'integer', ''],\n",
              " ['payments', 'double precision', ''],\n",
              " ['status', 'text', '']]"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%sql \\d loans"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Examples\n",
        "\n",
        "Diffix Elm supports a very limited subset of SQL.\n",
        "\n",
        "It supports only one aggregate, `count()`. Specifically, it supports `count(*)`, `count(column)`, and `count(distinct column)`.\n",
        "\n",
        "It supports the following generalization functions for numeric and text columns:\n",
        "\n",
        "- `diffix.floor_by(numeric_column, K)`\n",
        "- `diffix.round_by(numeric_column, K)`\n",
        "- `diffix.ceil_by(numeric_column, K)`\n",
        "- `substring(text_column, index, length)`\n",
        "\n",
        "Let's see what it looks like when data is queried by the trusted user (receives anonymized output) and the direct user (receives raw data)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>status</th>\n",
              "        <th>real_count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>A</td>\n",
              "        <td>203</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>B</td>\n",
              "        <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>C</td>\n",
              "        <td>403</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>D</td>\n",
              "        <td>45</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('A', 203), ('B', 31), ('C', 403), ('D', 45)]"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql direct_user@banking\n",
        "SELECT status, count(*) AS real_count\n",
        "FROM loans\n",
        "GROUP BY status\n",
        "ORDER BY status ASC"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>status</th>\n",
              "        <th>anon_count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>A</td>\n",
              "        <td>202</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>B</td>\n",
              "        <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>C</td>\n",
              "        <td>402</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>D</td>\n",
              "        <td>44</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('A', 202), ('B', 30), ('C', 402), ('D', 44)]"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT status, count(*) AS anon_count\n",
        "FROM loans\n",
        "GROUP BY status\n",
        "ORDER BY status ASC"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The two queries above are identical, but the anonymized results are slightly perturbed.\n",
        "\n",
        "Diffix Elm perturbs data in three ways:\n",
        "\n",
        "- Adding noise to counts\n",
        "- Suppressing column values that pertain to too few individuals\n",
        "- Hiding the impact of individuals that contribute disproportionately to counts\n",
        "\n",
        "The following gives examples of these.\n",
        "\n",
        "## Adding noise\n",
        "\n",
        "Diffix Elm adds noise to the results of aggregates.\n",
        "This is why the counts in the above two queries differ.\n",
        "\n",
        "## Suppressing values with few individuals\n",
        "\n",
        "Let us take another look at the `loans` table we queried above.\n",
        "The following query shows that many of the loan dates are unique:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>date</th>\n",
              "        <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>971019</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>951002</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>980522</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>970519</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>980129</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>981201</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>950922</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>940825</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>981029</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>961024</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('971019', 1),\n",
              " ('951002', 1),\n",
              " ('980522', 1),\n",
              " ('970519', 1),\n",
              " ('980129', 1),\n",
              " ('981201', 1),\n",
              " ('950922', 1),\n",
              " ('940825', 1),\n",
              " ('981029', 1),\n",
              " ('961024', 1)]"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql direct_user@banking\n",
        "SELECT date, count(*)\n",
        "FROM loans\n",
        "GROUP BY date\n",
        "HAVING count(*) = 1\n",
        "LIMIT 10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In fact 457 out of the 682 loans in the table (or approximately 67%) had a date that only appeared once:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>num_occurrences</th>\n",
              "        <th>count_loans_sharing_date</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>457</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>84</td>\n",
              "        <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>15</td>\n",
              "        <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>3</td>\n",
              "        <td>4</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[(457, 1), (84, 2), (15, 3), (3, 4)]"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql direct_user@banking\n",
        "SELECT count(*) num_occurrences, count_loans_sharing_date\n",
        "FROM (\n",
        "  SELECT date, count(*) AS count_loans_sharing_date\n",
        "  FROM loans\n",
        "  GROUP BY date\n",
        ") t\n",
        "GROUP BY count_loans_sharing_date\n",
        "ORDER BY count(*) DESC"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This means that, without anonymization, if we know the date on which somebody went to the bank to take out a loan,\n",
        "we could most likely learn the amount of the loan and other information:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>date</th>\n",
              "        <th>amount</th>\n",
              "        <th>duration</th>\n",
              "        <th>status</th>\n",
              "        <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>930705</td>\n",
              "        <td>96396</td>\n",
              "        <td>12</td>\n",
              "        <td>B</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>930711</td>\n",
              "        <td>165960</td>\n",
              "        <td>36</td>\n",
              "        <td>A</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>930728</td>\n",
              "        <td>127080</td>\n",
              "        <td>60</td>\n",
              "        <td>A</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>930803</td>\n",
              "        <td>105804</td>\n",
              "        <td>36</td>\n",
              "        <td>A</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>930906</td>\n",
              "        <td>274740</td>\n",
              "        <td>60</td>\n",
              "        <td>A</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>930913</td>\n",
              "        <td>87840</td>\n",
              "        <td>24</td>\n",
              "        <td>A</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>930915</td>\n",
              "        <td>52788</td>\n",
              "        <td>12</td>\n",
              "        <td>A</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>930924</td>\n",
              "        <td>174744</td>\n",
              "        <td>24</td>\n",
              "        <td>B</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>931013</td>\n",
              "        <td>154416</td>\n",
              "        <td>48</td>\n",
              "        <td>A</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>931104</td>\n",
              "        <td>117024</td>\n",
              "        <td>24</td>\n",
              "        <td>A</td>\n",
              "        <td>1</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('930705', 96396, 12, 'B', 1),\n",
              " ('930711', 165960, 36, 'A', 1),\n",
              " ('930728', 127080, 60, 'A', 1),\n",
              " ('930803', 105804, 36, 'A', 1),\n",
              " ('930906', 274740, 60, 'A', 1),\n",
              " ('930913', 87840, 24, 'A', 1),\n",
              " ('930915', 52788, 12, 'A', 1),\n",
              " ('930924', 174744, 24, 'B', 1),\n",
              " ('931013', 154416, 48, 'A', 1),\n",
              " ('931104', 117024, 24, 'A', 1)]"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql direct_user@banking\n",
        "SELECT date, amount, duration, status, count(*)\n",
        "FROM loans\n",
        "GROUP BY date, amount, duration, status\n",
        "ORDER BY date\n",
        "LIMIT 10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If we know that an individual has taken a loan on Sept. 6, 1993, then we learn that it is a 60 month loan for 274740 dollars.\n",
        "\n",
        "When Diffix is used to anonymize the data, the system detects output rows that can be linked to a single or a small number of individuals,\n",
        "and suppresses that information.\n",
        "\n",
        "We can see this in practice by repeating the two queries we just ran on the raw dataset, but protected by Diffix Elm."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>date</th>\n",
              "        <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>*</td>\n",
              "        <td>667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>971208</td>\n",
              "        <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>960429</td>\n",
              "        <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>970307</td>\n",
              "        <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>970317</td>\n",
              "        <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>970503</td>\n",
              "        <td>3</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('*', 667),\n",
              " ('971208', 3),\n",
              " ('960429', 3),\n",
              " ('970307', 3),\n",
              " ('970317', 3),\n",
              " ('970503', 3)]"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT date, count(*)\n",
        "FROM loans\n",
        "GROUP BY date"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here we see that almost all of the data has been suppressed. \n",
        "Counts for only five dates are displayed.\n",
        "These are dates where there are enough distinct individuals with loans to show the data.\n",
        "All other dates have one or very loans. Diffix Elm automatically recognizes this and suppresses the dates,\n",
        "merging them into a single bin with the value '`*`' which we call the _suppression bin_."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>date</th>\n",
              "        <th>amount</th>\n",
              "        <th>duration</th>\n",
              "        <th>status</th>\n",
              "        <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>*</td>\n",
              "        <td>None</td>\n",
              "        <td>None</td>\n",
              "        <td>*</td>\n",
              "        <td>683</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('*', None, None, '*', 683)]"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT date, amount, duration, status, count(*)\n",
        "FROM loans\n",
        "GROUP BY date, amount, duration, status"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The query above, which attempts to display all columns, suppresses even more aggressively:\n",
        "all of the data is suppressed and merged into a single suppression bin.\n",
        "This is because the four column values taken together are distinct for every individual. \n",
        "\n",
        "The text columns of suppressed bins are denoted with the value '`*`' (this is configurable), while numeric columns are denoted with `NULL`.\n",
        "Diffix Elm provides a custom function, `diffix.is_suppress_bin(*)`, which returns `True` for the suppression bin, and `False` otherwise:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>date</th>\n",
              "        <th>count</th>\n",
              "        <th>is_suppress_bin</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>*</td>\n",
              "        <td>667</td>\n",
              "        <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>971208</td>\n",
              "        <td>3</td>\n",
              "        <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>960429</td>\n",
              "        <td>3</td>\n",
              "        <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>970307</td>\n",
              "        <td>3</td>\n",
              "        <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>970317</td>\n",
              "        <td>3</td>\n",
              "        <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>970503</td>\n",
              "        <td>3</td>\n",
              "        <td>False</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('*', 667, True),\n",
              " ('971208', 3, False),\n",
              " ('960429', 3, False),\n",
              " ('970307', 3, False),\n",
              " ('970317', 3, False),\n",
              " ('970503', 3, False)]"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT date, count(*), diffix.is_suppress_bin(*)\n",
        "FROM loans\n",
        "GROUP BY date"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Generalization\n",
        "\n",
        "The queries above are anonymous, but they are also useless. All of the data has been suppressed.\n",
        "**Generalization** is used to avoid suppression.\n",
        "\n",
        "For example, rather than attempt to count the number of loans for individual days,\n",
        "we can generalize these to years using `substring()`:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>year</th>\n",
              "        <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>93</td>\n",
              "        <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>94</td>\n",
              "        <td>102</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>98</td>\n",
              "        <td>159</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>96</td>\n",
              "        <td>113</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>95</td>\n",
              "        <td>91</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>97</td>\n",
              "        <td>198</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('93', 21), ('94', 102), ('98', 159), ('96', 113), ('95', 91), ('97', 198)]"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT substring(date, 1, 2) AS year, count(*)\n",
        "FROM loans\n",
        "GROUP BY substring(date, 1, 2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here we see that while no data has been suppressed, it comes at the cost of a loss of precision.\n",
        "We can improve this by generalizing to month (here showing only the first 10 rows):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>year_month</th>\n",
              "        <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>*</td>\n",
              "        <td>24</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>9312</td>\n",
              "        <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>9401</td>\n",
              "        <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>9403</td>\n",
              "        <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>9405</td>\n",
              "        <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>9406</td>\n",
              "        <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>9407</td>\n",
              "        <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>9408</td>\n",
              "        <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>9409</td>\n",
              "        <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>9410</td>\n",
              "        <td>6</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('*', 24),\n",
              " ('9312', 7),\n",
              " ('9401', 5),\n",
              " ('9403', 7),\n",
              " ('9405', 10),\n",
              " ('9406', 13),\n",
              " ('9407', 13),\n",
              " ('9408', 7),\n",
              " ('9409', 11),\n",
              " ('9410', 6)]"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT substring(date, 1, 4) AS year_month, count(*)\n",
        "FROM loans\n",
        "GROUP BY substring(date, 1, 4)\n",
        "ORDER BY substring(date, 1, 4)\n",
        "LIMIT 10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here only 3.5% of the data is suppressed (roughly 24 of 682 loans). \n",
        "\n",
        "Diffix Elm allows generalization of both text and numeric columns.\n",
        "In the following, we use the `difix.floor_by(col, amount)` function to generate loan amount bins of 50000:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>amount</th>\n",
              "        <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>None</td>\n",
              "        <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>0.0</td>\n",
              "        <td>125</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>50000.0</td>\n",
              "        <td>178</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>100000.0</td>\n",
              "        <td>89</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>150000.0</td>\n",
              "        <td>103</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>200000.0</td>\n",
              "        <td>57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>250000.0</td>\n",
              "        <td>41</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>300000.0</td>\n",
              "        <td>35</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>350000.0</td>\n",
              "        <td>25</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>400000.0</td>\n",
              "        <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>450000.0</td>\n",
              "        <td>11</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[(None, 5),\n",
              " (0.0, 125),\n",
              " (50000.0, 178),\n",
              " (100000.0, 89),\n",
              " (150000.0, 103),\n",
              " (200000.0, 57),\n",
              " (250000.0, 41),\n",
              " (300000.0, 35),\n",
              " (350000.0, 25),\n",
              " (400000.0, 10),\n",
              " (450000.0, 11)]"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT diffix.floor_by(amount, 50000) AS amount, count(*)\n",
        "FROM loans\n",
        "GROUP BY diffix.floor_by(amount, 50000)\n",
        "ORDER BY diffix.floor_by(amount, 50000)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Post Processing\n",
        "\n",
        "Although Diffix Elm places strong SQL restrictions on anonymizing queries,\n",
        "it allows arbitrary SQL when post-processing the result of an anonymized query.\n",
        "\n",
        "The following post processes the anonymizing sub-query to make the output more readable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>year</th>\n",
              "        <th>amount</th>\n",
              "        <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>Other</td>\n",
              "        <td>Other</td>\n",
              "        <td>37</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>1993</td>\n",
              "        <td>50k - 100k</td>\n",
              "        <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>1994</td>\n",
              "        <td>0k - 50k</td>\n",
              "        <td>16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>1994</td>\n",
              "        <td>50k - 100k</td>\n",
              "        <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>1994</td>\n",
              "        <td>100k - 150k</td>\n",
              "        <td>14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>1994</td>\n",
              "        <td>150k - 200k</td>\n",
              "        <td>16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>1994</td>\n",
              "        <td>200k - 250k</td>\n",
              "        <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>1994</td>\n",
              "        <td>250k - 300k</td>\n",
              "        <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>1994</td>\n",
              "        <td>300k - 350k</td>\n",
              "        <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>1995</td>\n",
              "        <td>0k - 50k</td>\n",
              "        <td>21</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('Other', 'Other', 37),\n",
              " ('1993', '50k - 100k', 8),\n",
              " ('1994', '0k - 50k', 16),\n",
              " ('1994', '50k - 100k', 30),\n",
              " ('1994', '100k - 150k', 14),\n",
              " ('1994', '150k - 200k', 16),\n",
              " ('1994', '200k - 250k', 9),\n",
              " ('1994', '250k - 300k', 8),\n",
              " ('1994', '300k - 350k', 3),\n",
              " ('1995', '0k - 50k', 21)]"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT\n",
        "  CASE WHEN anonymized.year = '*' THEN 'Other'\n",
        "  ELSE '19' || anonymized.year\n",
        "  END AS year,\n",
        "  CASE WHEN anonymized.amount IS NULL THEN 'Other'\n",
        "  ELSE anonymized.amount/1000 || 'k - ' || (anonymized.amount/1000 + 50) || 'k'\n",
        "  END AS amount,\n",
        "  count\n",
        "FROM (\n",
        "  SELECT substring(date, 1, 2) as year, diffix.floor_by(amount, 50000) AS amount, count(*) AS count\n",
        "  FROM loans\n",
        "  GROUP BY 1, 2\n",
        ") AS anonymized\n",
        "ORDER BY anonymized.year ASC, anonymized.amount ASC\n",
        "LIMIT 10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The following counts the number of unsuppressed bins in an output (including the suppression bin itself):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>number_of_unsuppressed_bins_including_suppress_bin</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>59</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[(59,)]"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT count(*) AS number_of_unsuppressed_bins_including_suppress_bin\n",
        "FROM (\n",
        "  SELECT substring(date, 1, 4), count(*)\n",
        "  FROM loans\n",
        "  GROUP BY substring(date, 1, 4)\n",
        ") t"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The following computes the min and max loan amount bins. Note that this may not the true min and max,\n",
        "but simply the min and max of the bin value for unsuppressed bins. The true values may be more or less."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>min_loan_amount</th>\n",
              "        <th>max_loan_amount</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>0.0</td>\n",
              "        <td>550000.0</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[(0.0, 550000.0)]"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT min(amount_floor) AS min_loan_amount,\n",
        "       max(amount_ceiling) AS max_loan_amount\n",
        "FROM (\n",
        "  SELECT diffix.floor_by(amount, 50000) AS amount_floor,\n",
        "\t     diffix.ceil_by(amount, 50000) AS amount_ceiling,\n",
        "\t     count(*)\n",
        "  FROM loans\n",
        "  GROUP BY 1, 2\n",
        ") t"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The following computes the average number of transactions per user."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>average_trans_per_user</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>234</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[(234,)]"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT total_transactions/total_users AS average_trans_per_user\n",
        "FROM (\n",
        "  SELECT count(DISTINCT account_id) AS total_users,\n",
        "         count(*) AS total_transactions\n",
        "  FROM transactions\n",
        ") t"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Proportional Noise\n",
        "\n",
        "Diffix Elm uses noise to hide the presence of absence of individuals.\n",
        "If individuals contribute more to a count, then Diffix Elm recognizes this and increases the noise proprotionally.\n",
        "\n",
        "As an example, consider the following query made on the `transactions` table for both the raw and anonymized answers.\n",
        "This query counts the number of distinct individuals (as identified by their account_id) for each transaction type."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>type</th>\n",
              "        <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>PRIJEM</td>\n",
              "        <td>4500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>VYBER</td>\n",
              "        <td>1144</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>VYDAJ</td>\n",
              "        <td>4500</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('PRIJEM', 4500), ('VYBER', 1144), ('VYDAJ', 4500)]"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql direct_user@banking\n",
        "SELECT type, count(DISTINCT account_id)\n",
        "FROM transactions\n",
        "GROUP BY type\n",
        "ORDER BY type"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>type</th>\n",
              "        <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>PRIJEM</td>\n",
              "        <td>4501</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>VYBER</td>\n",
              "        <td>1143</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>VYDAJ</td>\n",
              "        <td>4500</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('PRIJEM', 4501), ('VYBER', 1143), ('VYDAJ', 4500)]"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT type, count(DISTINCT account_id)\n",
        "FROM transactions\n",
        "GROUP BY type\n",
        "ORDER BY type"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The amount of noise is very small. This is because each individual contributes only one to the count.\n",
        "The amount of noise needed to hide one user is quite small (roughly plus or minus five).\n",
        "\n",
        "Now consider counting the number of transactions rather than the number of distince individuals:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>type</th>\n",
              "        <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>PRIJEM</td>\n",
              "        <td>405083</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>VYBER</td>\n",
              "        <td>16666</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>VYDAJ</td>\n",
              "        <td>634571</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('PRIJEM', 405083), ('VYBER', 16666), ('VYDAJ', 634571)]"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql direct_user@banking\n",
        "SELECT type, count(*)\n",
        "FROM transactions\n",
        "GROUP BY type\n",
        "ORDER BY type"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>type</th>\n",
              "        <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>PRIJEM</td>\n",
              "        <td>405145</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>VYBER</td>\n",
              "        <td>16647</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>VYDAJ</td>\n",
              "        <td>634406</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('PRIJEM', 405145), ('VYBER', 16647), ('VYDAJ', 634406)]"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql trusted_user@banking\n",
        "SELECT type, count(*)\n",
        "FROM transactions\n",
        "GROUP BY type\n",
        "ORDER BY type"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here the counts differ by 62, 19, and 165 respectively.\n",
        "To see why, let's look at the average number of transactions per individual per type:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "vscode": {
          "languageId": "sql"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table>\n",
              "    <tr>\n",
              "        <th>type</th>\n",
              "        <th>average_transactions</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>PRIJEM</td>\n",
              "        <td>90</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>VYBER</td>\n",
              "        <td>14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "        <td>VYDAJ</td>\n",
              "        <td>141</td>\n",
              "    </tr>\n",
              "</table>"
            ],
            "text/plain": [
              "[('PRIJEM', 90), ('VYBER', 14), ('VYDAJ', 141)]"
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%sql direct_user@banking\n",
        "SELECT type, count_transactions/count_users AS average_transactions\n",
        "FROM (\n",
        "  SELECT type,\n",
        "         count(DISTINCT account_id) AS count_users,\n",
        "         count(*) AS count_transactions,\n",
        "         count(*)/count(distinct account_id) AS avg\n",
        "  FROM transactions\n",
        "  GROUP BY 1\n",
        ") t"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We see that the amount of noise applied by Diffix Elm is proportional to the amount contributed by individuals."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "orig_nbformat": 3
  },
  "nbformat": 4,
  "nbformat_minor": 2
}