assignment-1/2_Baseline_retrieval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 1 - Part 1\n",
    "\n",
    "Baseline retrieval using Elasticsearch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from elasticsearch import Elasticsearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "INDEX_NAME = \"aquaint\"\n",
    "DOC_TYPE = \"doc\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "QUERY_FILE = \"data/queries.txt\"  # make sure the query file exists on this location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "OUTPUT_FILE = \"data/baseline.txt\"  # output the ranking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the queries from the file\n",
    "\n",
    "See the assignment description for the format of the query file [here](https://github.com/kbalog/uis-dat630-fall2017/tree/master/assignment-1#queries)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def load_queries(query_file):\n",
    "    queries = {}\n",
    "    with open(query_file, \"r\") as fin:\n",
    "        for line in fin.readlines():\n",
    "            qid, query = line.strip().split(\" \", 1)\n",
    "            queries[qid] = query\n",
    "    return queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "queries = load_queries(QUERY_FILE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perform retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO** Generate a ranking for each query and output the results to `OUTPUT_FILE`\n",
    "\n",
    "See the assignment description for the format of the output file [here](https://github.com/kbalog/uis-dat630-fall2017/tree/master/assignment-1#output-file-format)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ranking documents for [303] 'Hubble Telescope Achievements'\n",
      "Ranking documents for [307] 'New Hydroelectric Projects'\n",
      "Ranking documents for [310] 'Radio Waves and Brain Cancer'\n",
      "Ranking documents for [314] 'Marine Vegetation'\n",
      "Ranking documents for [322] 'International Art Crime'\n",
      "Ranking documents for [325] 'Cult Lifestyles'\n",
      "Ranking documents for [330] 'Iran-Iraq Cooperation'\n",
      "Ranking documents for [336] 'Black Bear Attacks'\n",
      "Ranking documents for [341] 'Airport Security'\n",
      "Ranking documents for [344] 'Abuses of E-Mail'\n",
      "Ranking documents for [345] 'Overseas Tobacco Sales'\n",
      "Ranking documents for [347] 'Wildlife Extinction'\n",
      "Ranking documents for [353] 'Antarctica exploration'\n",
      "Ranking documents for [354] 'journalist risks'\n",
      "Ranking documents for [362] 'human smuggling'\n",
      "Ranking documents for [363] 'transportation tunnel disasters'\n",
      "Ranking documents for [367] 'piracy'\n",
      "Ranking documents for [372] 'Native American casino'\n",
      "Ranking documents for [374] 'Nobel prize winners'\n",
      "Ranking documents for [375] 'hydrogen energy'\n",
      "Ranking documents for [378] 'euro opposition'\n",
      "Ranking documents for [383] 'mental illness drugs'\n",
      "Ranking documents for [389] 'illegal technology transfer'\n",
      "Ranking documents for [393] 'mercy killing'\n",
      "Ranking documents for [394] 'home schooling'\n",
      "Ranking documents for [397] 'automobile recalls'\n",
      "Ranking documents for [399] 'oceanographic vessels'\n",
      "Ranking documents for [401] 'foreign minorities, Germany'\n",
      "Ranking documents for [404] 'Ireland, peace talks'\n",
      "Ranking documents for [408] 'tropical storms'\n",
      "Ranking documents for [409] 'legal, Pan Am, 103'\n",
      "Ranking documents for [416] 'Three Gorges Project'\n",
      "Ranking documents for [419] 'recycle, automobile tires'\n",
      "Ranking documents for [426] 'law enforcement, dogs'\n",
      "Ranking documents for [427] 'UV damage, eyes'\n",
      "Ranking documents for [433] 'Greek, philosophy, stoicism'\n",
      "Ranking documents for [435] 'curbing population growth'\n",
      "Ranking documents for [436] 'railway accidents'\n",
      "Ranking documents for [439] 'inventions, scientific discoveries'\n",
      "Ranking documents for [443] 'U.S., investment, Africa'\n",
      "Ranking documents for [448] 'ship losses'\n",
      "Ranking documents for [622] 'price fixing'\n",
      "Ranking documents for [625] 'arrests bombing WTC'\n",
      "Ranking documents for [638] 'wrongful convictions'\n",
      "Ranking documents for [639] 'consumer on-line shopping'\n",
      "Ranking documents for [648] 'family leave law'\n",
      "Ranking documents for [650] 'tax evasion indicted'\n",
      "Ranking documents for [651] 'U.S. ethnic population'\n",
      "Ranking documents for [658] 'teenage pregnancy'\n",
      "Ranking documents for [689] 'family-planning aid'\n"
     ]
    }
   ],
   "source": [
    "for q_id, query in queries.items():\n",
    "    print(\"Ranking documents for [%s] '%s'\" % (q_id, query))\n",
    "    # TODO generate ranking\n",
    "    # TODO write results to file"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}