assignment-1/3_Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 1 - Part 2\n",
    "\n",
    "Evaluation of retrieval results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "RANKING_FILE = \"data/baseline.txt\"  # file with the document rankings\n",
    "QRELS_FILE = \"data/qrels2.csv\"  # file with the relevance judgments (ground truth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO** Complete the function that calculates evaluation metrics for a given a ranking (`ranking`) against the ground truth (`gt`). It should return the results as a dictionary, where the key is the retrieval metric.\n",
    "\n",
    "(Hint: see [Task 1 of Practicum 2](https://github.com/kbalog/uis-dat630-fall2017/blob/master/practicum/practicum-2/solutions/1_Eval_binary.ipynb).)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def eval_query(ranking, gt):\n",
    "    p10, ap, rr = 0, 0, 0\n",
    "\n",
    "    # TODO\n",
    "    \n",
    "    return {\"P10\": p10, \"AP\": ap, \"RR\": rr}\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function that evaluates an output file, which contains rankings for a set of queries.\n",
    "\n",
    "**TODO** It is almost complete, you just need to add the computation of mean scores (over the entire query set)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def eval(gt_file, output_file):\n",
    "    # load data from ground truth file\n",
    "    gt = {}  # holds a list of relevant documents for each queryID\n",
    "    with open(gt_file, \"r\") as fin:\n",
    "        header = fin.readline().strip()\n",
    "        if header != \"queryID,docIDs\":\n",
    "            raise Exception(\"Incorrect file format!\")\n",
    "        for line in fin.readlines():\n",
    "            qid, docids = line.strip().split(\",\")\n",
    "            gt[qid] = docids.split()\n",
    "            \n",
    "    # load data from output file\n",
    "    output = {}\n",
    "    with open(output_file, \"r\") as fin:\n",
    "        header = fin.readline().strip()\n",
    "        if header != \"QueryId,DocumentId\":\n",
    "            raise Exception(\"Incorrect file format!\")\n",
    "        for line in fin.readlines():\n",
    "            qid, docid = line.strip().split(\",\")\n",
    "            if qid not in output:\n",
    "                output[qid] = []\n",
    "            output[qid].append(docid)\n",
    "    \n",
    "    # evaluate each query that is in the ground truth\n",
    "    print(\"  QID  P@10   (M)AP  (M)RR\")\n",
    "    sum_p10, sum_ap, sum_rr = 0, 0, 0\n",
    "    for qid in sorted(gt.keys()):\n",
    "        res = eval_query(output.get(qid, []), gt.get(qid, []))\n",
    "        print(\"%5s %6.3f %6.3f %6.3f\" % (qid, res[\"P10\"], res[\"AP\"], res[\"RR\"]))\n",
    "    \n",
    "    # TODO compute averages over the entire query set\n",
    "    \n",
    "    # print averages\n",
    "    print(\"%5s %6.3f %6.3f %6.3f\" % (\"ALL\", sum_p10, sum_ap, sum_rr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Main"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  QID  P@10   (M)AP  (M)RR\n",
      "  303  0.000  0.000  0.000\n",
      "  307  0.000  0.000  0.000\n",
      "  310  0.000  0.000  0.000\n",
      "  314  0.000  0.000  0.000\n",
      "  322  0.000  0.000  0.000\n",
      "  325  0.000  0.000  0.000\n",
      "  330  0.000  0.000  0.000\n",
      "  336  0.000  0.000  0.000\n",
      "  341  0.000  0.000  0.000\n",
      "  344  0.000  0.000  0.000\n",
      "  347  0.000  0.000  0.000\n",
      "  353  0.000  0.000  0.000\n",
      "  354  0.000  0.000  0.000\n",
      "  362  0.000  0.000  0.000\n",
      "  363  0.000  0.000  0.000\n",
      "  367  0.000  0.000  0.000\n",
      "  372  0.000  0.000  0.000\n",
      "  374  0.000  0.000  0.000\n",
      "  383  0.000  0.000  0.000\n",
      "  389  0.000  0.000  0.000\n",
      "  393  0.000  0.000  0.000\n",
      "  399  0.000  0.000  0.000\n",
      "  401  0.000  0.000  0.000\n",
      "  404  0.000  0.000  0.000\n",
      "  408  0.000  0.000  0.000\n",
      "  409  0.000  0.000  0.000\n",
      "  416  0.000  0.000  0.000\n",
      "  419  0.000  0.000  0.000\n",
      "  426  0.000  0.000  0.000\n",
      "  427  0.000  0.000  0.000\n",
      "  433  0.000  0.000  0.000\n",
      "  435  0.000  0.000  0.000\n",
      "  436  0.000  0.000  0.000\n",
      "  439  0.000  0.000  0.000\n",
      "  443  0.000  0.000  0.000\n",
      "  448  0.000  0.000  0.000\n",
      "  622  0.000  0.000  0.000\n",
      "  625  0.000  0.000  0.000\n",
      "  638  0.000  0.000  0.000\n",
      "  639  0.000  0.000  0.000\n",
      "  648  0.000  0.000  0.000\n",
      "  650  0.000  0.000  0.000\n",
      "  651  0.000  0.000  0.000\n",
      "  658  0.000  0.000  0.000\n",
      "  689  0.000  0.000  0.000\n",
      "  ALL  0.000  0.000  0.000\n"
     ]
    }
   ],
   "source": [
    "eval(QRELS_FILE, RANKING_FILE)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}