Skip to content

Commit

Permalink
Merge pull request #5 from o19s/actually_show_overall_jaccard_and_rbo
Browse files Browse the repository at this point in the history 
Tweaks from David and Scott
  • Loading branch information
@epugh
epugh authored Feb 19, 2024
2 parents c19bede + 98a0ccc commit 5c6665a
Show file tree
Hide file tree
Showing 3 changed files with 154 additions and 115 deletions.
29 changes: 10 additions & 19 deletions jupyterlite/files/examples/Fleiss Kappa.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
"cells": [
{
"cell_type": "markdown",
"source": "# Fleiss' Kappa \nTo understand how much your raters what? Scott, need some text!\n\nPlease copy this example and customize it for your own purposes!",
"source": "# Fleiss' Kappa \nTo understand how much your judges agree with each other. It is meant to be used with more than two judges.\n\nRead https://www.datanovia.com/en/blog/kappa-coefficient-interpretation/ to learn more.\n\nPlease copy this example and customize it for your own purposes!",
"metadata": {},
"id": "bd7e4efa-eb00-451e-984d-ed6646d8e25f"
},
Expand Down Expand Up @@ -51,11 +51,11 @@
},
{
"cell_type": "code",
"source": "QUEPID_BOOK_NUM = 25\n\n# Not needed if running within Quepid JupyterLite\n# QUEPID_API_TOKEN = \"\"",
"source": "QUEPID_BOOK_NUM = 25",
"metadata": {
"trusted": true
},
"execution_count": 3,
"execution_count": 2,
"outputs": [],
"id": "71803a49-4065-4adf-a69e-cb0fe2d00f22"
},
Expand All @@ -71,7 +71,7 @@
"metadata": {
"trusted": true
},
"execution_count": 4,
"execution_count": 3,
"outputs": [],
"id": "31193536-98eb-4b46-ab98-af04ee07c6d3"
},
Expand All @@ -81,7 +81,7 @@
"metadata": {
"trusted": true
},
"execution_count": 5,
"execution_count": null,
"outputs": [],
"id": "8fef6231-daa8-467f-ac57-13a144e8a356"
},
Expand All @@ -97,7 +97,7 @@
"metadata": {
"trusted": true
},
"execution_count": 6,
"execution_count": null,
"outputs": [],
"id": "9a8561fd-2dbf-477e-9ac1-4df6d5ebdc91"
},
Expand All @@ -113,7 +113,7 @@
"metadata": {
"trusted": true
},
"execution_count": 7,
"execution_count": null,
"outputs": [],
"id": "a7598308-129b-4628-ad3a-fc3d703f8205"
},
Expand All @@ -129,22 +129,13 @@
"metadata": {
"trusted": true
},
"execution_count": 8,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<IPython.core.display.Markdown object>",
"text/markdown": "## Fleiss' Kappa: -0.3333"
},
"metadata": {}
}
],
"execution_count": null,
"outputs": [],
"id": "25a613f9"
},
{
"cell_type": "markdown",
"source": "_This notebook was last updated 17-FEB-2024_",
"source": "_This notebook was last updated 19-FEB-2024_",
"metadata": {},
"id": "5704579e-2321-4629-8de0-6608b428e2b6"
},
Expand Down
69 changes: 62 additions & 7 deletions jupyterlite/files/examples/Jaccard and RBO Comparison.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,12 @@
"cells": [
{
"cell_type": "markdown",
"source": "# Jaccard and RBO Comparison \nTo understand the magnatude of changes to your query result sets, you can compare multiple snapshots together, either from the same case or different cases. \n\nThis notebook provides both Jaccard and Rank Biased Overlap (RBO) metrics.\n\nPlease copy this example and customize it for your own purposes!",
"source": "# Jaccard and RBO Comparison \nTo understand the magnatude of changes to your query result sets, you can compare multiple snapshots to each other.\n\nThis notebook provides both Jaccard and Rank Biased Overlap (RBO) metrics.\n\nPlease copy this example and customize it for your own purposes!",
"metadata": {}
},
{
"cell_type": "code",
"source": "from js import fetch\nfrom typing import List, Optional, Union\n\nimport json\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\n\nimport piplite\nawait piplite.install('seaborn')\nawait piplite.install('rbo')\n\nimport rbo\nimport seaborn as sns\n\nimport os\n\nos.environ[\"TQDM_DISABLE\"] = \"1\"",
"source": "from js import fetch\nfrom typing import List, Optional, Union\n\nimport json\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\n\nimport piplite\nawait piplite.install('seaborn')\nawait piplite.install('rbo')\n\nimport rbo\nimport seaborn as sns\n\nimport os",
"metadata": {
"trusted": true
},
Expand Down Expand Up @@ -72,16 +72,30 @@
}
]
},
{
"cell_type": "code",
"source": "os.environ[\"TQDM_DISABLE\"] = \"1\"",
"metadata": {
"trusted": true
},
"execution_count": 5,
"outputs": []
},
{
"cell_type": "code",
"source": "def jaccard(l1, l2, max_n):\n if len(l1) == 0 and len(l2) == 0:\n return 1\n max_len = min(len(l1), len(l2), max_n)\n set1 = set(l1[:max_len])\n set2 = set(l2[:max_len])\n intersection = len(set1.intersection(set2))\n union = len(set1) + len(set2) - intersection\n return float(intersection) / union\n\nasync def load_snapshots(case_id1, snapshot_id1, case_id2, snapshot_id2):\n df_a = await load_snapshot(case_id1, snapshot_id1)\n df_b = await load_snapshot(case_id2, snapshot_id2)\n return df_a.merge(df_b, on='query')\n\nasync def compare(case_id1, snapshot_id1, case_id2, snapshot_id2):\n df = await load_snapshots(case_id1, snapshot_id1, case_id2, snapshot_id2)\n \n df['jaccard'] = df.apply(lambda row: jaccard(row['docs_x'], row['docs_y'], 10), axis=1)\n df['rbo'] = df.apply(lambda row: rbo.RankingSimilarity(row['docs_x'], row['docs_y']).rbo(), axis=1)\n df['score_delta'] = df['score_y'] - df['score_x']\n df.name = f\"Case {case_id1} snapshot {snapshot_id1} vs. case {case_id1} snapshot {snapshot_id2}\"\n return df\n\n\n\nawait compare(case_id1=6789, snapshot_id1=2471, case_id2=6789, snapshot_id2=2472)",
"metadata": {
"trusted": true
},
"execution_count": 7,
"execution_count": 6,
"outputs": [
{
"execution_count": 7,
"name": "stderr",
"text": "/lib/python3.11/site-packages/rbo/rbo.py:129: TqdmMonitorWarning: tqdm:disabling monitor support (monitor_interval = 0) due to:\ncan't start new thread\n for d in tqdm(range(1, k), disable=~self.verbose):\n",
"output_type": "stream"
},
{
"execution_count": 6,
"output_type": "execute_result",
"data": {
"text/plain": " num_results_x score_x \\\nquery \nprojector screen 1 1.0 \nnotebook 1 1.0 \niphone 8 1 1.0 \nprinter 1 1.0 \ncomputer 1 1.0 \n... ... ... \nwindows 10 1 1.0 \nmicrowave 1 1.0 \nbluetooth speakers 1 1.0 \ncoffee 1 1.0 \nvans 1 1.0 \n\n docs_x \\\nquery \nprojector screen [1069226, 47471, 490523, 1229109, 1229118, 325... \nnotebook [3851056, 3959000, 1550833, 1684763, 1675257, ... \niphone 8 [2048598, 1648546, 79524888, 1857711, 3613408,... \nprinter [3849563, 2225354, 1569761, 798960, 377837, 13... \ncomputer [560468, 532095, 560475, 523407, 693956, 56047... \n... ... \nwindows 10 [4481689, 3902727, 1560529, 1797902, 3155116, ... \nmicrowave [79513345, 4020048, 1768856, 2936032] \nbluetooth speakers [1993197, 3537784, 279672, 2663204, 558184, 33... \ncoffee [1996660, 2102472, 79583150, 1357989, 656359, ... \nvans [78503576, 79118095, 77388459, 78322005, 79013... \n\n num_results_y score_y \\\nquery \nprojector screen 1 1.0 \nnotebook 1 1.0 \niphone 8 1 1.0 \nprinter 1 1.0 \ncomputer 1 1.0 \n... ... ... \nwindows 10 1 1.0 \nmicrowave 1 1.0 \nbluetooth speakers 1 1.0 \ncoffee 1 1.0 \nvans 1 1.0 \n\n docs_y \\\nquery \nprojector screen [1069226, 47471, 490523, 1229109, 1229118, 325... \nnotebook [3851056, 3959000, 1550833, 1684763, 1675257, ... \niphone 8 [2048598, 1648546, 79524888, 1857711, 3613408,... \nprinter [3849563, 2225354, 1569761, 798960, 377837, 13... \ncomputer [560468, 532095, 560475, 523407, 693956, 56047... \n... ... \nwindows 10 [4481689, 3902727, 1560529, 1797902, 3155116, ... \nmicrowave [79513345, 4020048, 1768856, 2936032] \nbluetooth speakers [1993197, 3537784, 279672, 2663204, 558184, 33... \ncoffee [1996660, 2102472, 79583150, 1357989, 656359, ... \nvans [78503576, 79118095, 77388459, 78322005, 79013... \n\n jaccard rbo score_delta \nquery \nprojector screen 1.0 1.0 0.0 \nnotebook 1.0 1.0 0.0 \niphone 8 1.0 1.0 0.0 \nprinter 1.0 1.0 0.0 \ncomputer 1.0 1.0 0.0 \n... ... ... ... \nwindows 10 1.0 1.0 0.0 \nmicrowave 1.0 1.0 0.0 \nbluetooth speakers 1.0 1.0 0.0 \ncoffee 1.0 1.0 0.0 \nvans 1.0 1.0 0.0 \n\n[135 rows x 9 columns]",
Expand All @@ -93,11 +107,45 @@
},
{
"cell_type": "code",
"source": "import matplotlib\nmatplotlib.rc_file_defaults()\n\ndef plot_compare(df):\n figure, axes = plt.subplots(1, 3, figsize=(10, 4))\n figure.suptitle(df.name)\n\n sns.barplot(ax=axes[0], x=df['score_delta'], y=df.index, width=0.3, color='darkgrey')\n axes[0].set(xlim=(-1, 1))\n axes[0].set_xlabel('Change in Score')\n axes[0].set_ylabel('')\n axes[0].set_facecolor((0.90, 0.90, 0.90))\n axes[0].grid(True)\n axes[0].spines['top'].set_visible(False)\n axes[0].spines['right'].set_visible(False)\n axes[0].spines['bottom'].set_visible(False)\n axes[0].spines['left'].set_visible(False)\n axes[0].set_axisbelow(True)\n axes[0].xaxis.grid(color='w', linestyle='solid')\n axes[0].yaxis.grid(color='w', linestyle='solid')\n \n sns.heatmap(df[['jaccard']], ax=axes[1], cmap='crest', annot=True, xticklabels=False, yticklabels=False)\n axes[1].set_xlabel('Jaccard Similiarity')\n axes[1].set_ylabel('')\n \n sns.heatmap(df[['rbo']], ax=axes[2], cmap='crest', annot=True, xticklabels=False, yticklabels=False)\n axes[2].set_xlabel('Rank Biased Overlap')\n axes[2].set_ylabel('')\n \n plt.show()\n \ndf = await compare(case_id1=6789, snapshot_id1=2471, case_id2=6789, snapshot_id2=2473)\nplot_compare(df)",
"source": "import matplotlib\nmatplotlib.rc_file_defaults()\n\ndef plot_compare(df):\n figure, axes = plt.subplots(1, 3, figsize=(10, 4))\n figure.suptitle(df.name)\n\n sns.barplot(ax=axes[0], x=df['score_delta'], y=df.index, width=0.3, color='darkgrey')\n axes[0].set(xlim=(-1, 1))\n axes[0].set_xlabel('Change in Score')\n axes[0].set_ylabel('')\n axes[0].set_facecolor((0.90, 0.90, 0.90))\n axes[0].grid(True)\n axes[0].spines['top'].set_visible(False)\n axes[0].spines['right'].set_visible(False)\n axes[0].spines['bottom'].set_visible(False)\n axes[0].spines['left'].set_visible(False)\n axes[0].set_axisbelow(True)\n axes[0].xaxis.grid(color='w', linestyle='solid')\n axes[0].yaxis.grid(color='w', linestyle='solid')\n \n sns.heatmap(df[['jaccard']], ax=axes[1], cmap='crest', annot=True, xticklabels=False, yticklabels=False)\n axes[1].set_xlabel('Jaccard Similiarity')\n axes[1].set_ylabel('')\n \n sns.heatmap(df[['rbo']], ax=axes[2], cmap='crest', annot=True, xticklabels=False, yticklabels=False)\n axes[2].set_xlabel('Rank Biased Overlap')\n axes[2].set_ylabel('')\n \n plt.show()\n \ndf = await compare(case_id1=6789, snapshot_id1=2471, case_id2=6789, snapshot_id2=2473)\n",
"metadata": {
"trusted": true
},
"execution_count": 6,
"execution_count": 7,
"outputs": []
},
{
"cell_type": "markdown",
"source": "## Overall Jaccard and RBO Scores",
"metadata": {}
},
{
"cell_type": "code",
"source": "print(f\"Overall Jaccard Score: {df['jaccard'].mean()}\\nOverall RBO Score: {df['rbo'].mean()}\")",
"metadata": {
"trusted": true
},
"execution_count": 8,
"outputs": [
{
"name": "stdout",
"text": "Overall Jaccard Score: 1.0\nOverall RBO Score: 1.0\n",
"output_type": "stream"
}
]
},
{
"cell_type": "markdown",
"source": "## Query Level Jaccard and RBO Scores",
"metadata": {}
},
{
"cell_type": "code",
"source": "plot_compare(df)",
"metadata": {
"trusted": true
},
"execution_count": 9,
"outputs": [
{
"output_type": "display_data",
Expand All @@ -111,8 +159,15 @@
},
{
"cell_type": "markdown",
"source": "_This notebook was last updated 16-FEB-2024_",
"source": "_This notebook was last updated 19-FEB-2024_",
"metadata": {}
},
{
"cell_type": "code",
"source": "",
"metadata": {},
"execution_count": null,
"outputs": []
}
]
}
Loading

0 comments on commit 5c6665a

Please sign in to comment.