notebooks/ibmseti_get_data_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# IBM SETI Tutorial  \n",
    "### Transfer Data  to Local Object Store\n",
    "\n",
    "This tutorial builds on the information presented in the [introduction to the HTTP API notebook](https://github.com/ibm-cds-labs/seti_at_ibm/blob/master/notebooks/ibmseti_intro_to_http_api.ipynb).\n",
    "\n",
    "### Goal\n",
    "\n",
    "The goal is to use the REST API to extract a set of data from the REST API and store it into our Object Store. We want to retain the raw data and SignalDB rows in order to later perform data analysis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Interesting Target\n",
    "\n",
    "Start with the coordinates for our interesting target, Kepler 1229b. We ensured that data existed for this celestial location (RA, DEC)  in the SETI Public data set in the [HTTP API introduction notebook](https://github.com/ibm-cds-labs/seti_at_ibm/blob/master/notebooks/ibmseti_intro_to_http_api.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "ra=19.832\n",
    "dec=46.997"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "### Strategy:\n",
    "    \n",
    "    * Build RDD with meta-data container/objectname\n",
    "    * map function to get temporary URL and data\n",
    "    * save data to Object Store in various ways\n",
    "        * RDD as pickled Hadoop file on Object Store\n",
    "        * Individual files on Object Store\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Build RDD with meta-data\n",
    "\n",
    "Retrieve the meta data for each raw data object. \n",
    "\n",
    "Use the skip parameter to paginate through the results and extract all of the SignalDB rows for our particular RA/DEC coordinate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "skip = 0\n",
    "skip_delta = 2000\n",
    "all_rows = []\n",
    "\n",
    "while True:\n",
    "    params = {'skip':skip}\n",
    "    r = requests.get('https://setigopublic.mybluemix.net/v1/aca/meta/{}/{}?limit=2000'.format(ra,dec), \n",
    "                     params=params)\n",
    "    r.raise_for_status()\n",
    "    \n",
    "    if r.json()['returned_num_rows'] == 0:\n",
    "        break\n",
    "        \n",
    "    all_rows += r.json()['rows']\n",
    "    skip += skip_delta\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "392"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(all_rows)  #We have 392 rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rdd = sc.parallelize( all_rows ) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Pull data into RDD\n",
    "\n",
    "We now want to get the raw data and combine it with the SignalDB data. The SignalDB data should be retained since it could contain useful information to be used as features in a machine-learning analysis. You should at least retain the celestial coordinates of the raw data since one characteristic of an expected SETI signal is multiple observations of a signal from the same location in the sky. \n",
    "\n",
    "Recall that each raw data file can show up multiple times in SignalDB, for various resasons. We want to package all of these rows together with a single raw data file.\n",
    "\n",
    "##### GroupBy\n",
    "\n",
    "To do this, we use `groupBy` to re-organize the data returned from the setigopublic server into `(K,  <iterator V>)` pairs, where `K` is the concatenation of the `<containter>-<objectname>`, which should be completely unique, and `<iterator V>` is an iterator over the SignalDB rows. \n",
    "\n",
    "Recall that each `row` returned by the API server is dictionary where each key is the name of a column in the SignalDB. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Group By Raw Data File Name\n",
    "\n",
    "This creates an RDD of `(K, <iterator V>)` rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rdd_klv = rdd.groupBy(lambda row: row['container'] + '-' + row['objectname'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Add Temporary URLs to RDD\n",
    "From this `(K, <iterator V>)` pair, we then grab the data with the temporary URLs. \n",
    "\n",
    "We define a function to request the temporary URLs. \n",
    "\n",
    "Also, we modify each row into a `(K, V)` pair such that `V` is now a list containing the HTTP status code, the temprorary URL and the iterable list of the SignalDB rows. The key, `K`, becomes just the name of the raw data file. \n",
    "\n",
    "##### Access Token\n",
    "\n",
    "Before requesting the temporary URLs, you must first attain an access token. This will require an IBM DSX account. \n",
    "\n",
    "Click here: https://setigopublic.mybluemix.net/token\n",
    "\n",
    "Copy the token returned below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "access_token='1234567890abcdefg'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def add_temp_urls(row):\n",
    "    '''\n",
    "    This function will get a temporary URL to the data. \n",
    "    \n",
    "    Note the new format of each Row.\n",
    "    '''\n",
    "    \n",
    "    #each row in RDD is a tuple2, The first element is the container-objectname\n",
    "    container, objectname = row[0].split('-',1) \n",
    "    \n",
    "    temp_url_api = 'https://setigopublic.mybluemix.net/v1/data/url/{}/{}'.format(container, objectname)\n",
    "    r = requests.get(temp_url_api, params={'access_token':access_token})\n",
    "    \n",
    "    temp_url = ''\n",
    "    if r.status_code == 200:\n",
    "        temp_url = r.json()['temp_url']\n",
    "        \n",
    "    #Object names look like: 2013-01-05/act1779/2013-01-05_07-21-33_UTC.act1779.dx3016.id-15.R.archive-compamp\n",
    "    #We just want `2013-01-05_07-12-33_UTC.act1779.dx3016.id-15.R.archive-compamp`\n",
    "    newkey = objectname.split('/')[-1]  \n",
    "    \n",
    "    return (newkey, [r.status_code, temp_url, row[1]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rdd_with_url = rdd_klv.map(add_temp_urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "##### Cache and Count the RDD. \n",
    "\n",
    "The RDD is cached here so that Spark doesn't obtain the temporary data URLs for a second time when the data is saved to Object Storage later in this notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PythonRDD[5] at RDD at PythonRDD.scala:43"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd_with_url.cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6.07 ms, sys: 7 ms, total: 13.1 ms\n",
      "Wall time: 1min 20s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "206"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time rdd_with_url.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Note that 206 rows were returned here whereas we initiall started with 392 rows. This is because out of the 392 rows returned above, 186 of them referred to the same data file as another.  Grouping the `rdd` by `container-objectname`, removed the duplicate raw data files, but still let us retain each of the SignalDB rows for that file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Get Data\n",
    "\n",
    "Note that we again transform our RDD Row. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def get_data(row):\n",
    "    '''\n",
    "    We use the temporary URL to pull the data into each Row.\n",
    "    '''\n",
    "    r = requests.get(row[1][1])\n",
    "\n",
    "    # here we transform the data to something a little easier to use\n",
    "    retDataVal = {\"aca_file_name\":row[0],\n",
    "                  \"target_name\":\"kepler1229b\", #could be helpful later to attach name\n",
    "                  \"raw_data\":r.content, \n",
    "                  \"http_status\":r.status_code, \n",
    "                  \"signaldb_rows\":row[1][2]\n",
    "                 }\n",
    "\n",
    "    return retDataVal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#filter out rows where HTTP requests did not return 200\n",
    "rdd_with_data = rdd_with_url.filter(lambda x: x[1][0] == 200)\\\n",
    "                .map(get_data)\\\n",
    "                .filter(lambda x: x['http_status'] == 200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Various Ways To Save"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Using Object Storage\n",
    "\n",
    "Moving the data form the SETI Object Storage to your personal Object Storage will provide the best performance for your analysis. This tutorial assumes that you've provisioned your services via IBM Data Science Experience, which automatically generates a Spark and Object Storage service for you. \n",
    "\n",
    " \n",
    "#### Insert Credentials from this notebook\n",
    "\n",
    "These instructions assume you're running this notebook from within DSX.\n",
    "\n",
    "1. Click the ![10/01](../img/1001.png \"10/01\") button in the set of icons on the top right to reveal the files in your Object Storage container associated with this notebook.\n",
    "> If there appears to be no Object Storage linked with your account, save and leave this notebook, go to your DSX project settings and add a new Object Storage instance.  \n",
    "3. From any file listed in the right-side panel, select `Insert to Code` and then choose `Insert Credentials`.\n",
    "> If there are no data shown, the easiest way to obtain your credentials is by adding a small file to your object storage. From your desktop, drag in a small text file. Then you may obtain credentials via the `Insert to Code` pull-down. \n",
    "\n",
    "##### Alternatively, from Bluemix\n",
    "\n",
    "Alternatively, you can obtain the credentials from Bluemix. \n",
    "1. Log into your IBM Bluemix account: https://bluemix.net. (Use your DSX credentials. Your Bluemix account was automatically created for you.) \n",
    "2. Scroll down and select your Object Storage service.\n",
    "3. Select `Service Credentials` and then `View Credentials`. Any set of credentials should work. \n",
    "\n",
    "**Note in the commented code below:** set the `credentials_1['container']` key to the name of an existing container in your Object Storage if it is not already set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "credentials_1 = {\n",
    "  'auth_uri':'',\n",
    "  'global_account_auth_uri':'',\n",
    "  'username':'xx',\n",
    "  'password':\"xx\",\n",
    "  'auth_url':'https://identity.open.softlayer.com',\n",
    "  'project':'xx',\n",
    "  'project_id':'xx',\n",
    "  'region':'dallas',\n",
    "  'user_id':'xx',\n",
    "  'domain_id':'xx',\n",
    "  'domain_name':'xx',\n",
    "  'container':'seti_raw_data',  #Need to make sure this container exists in Object Store! \n",
    "  'tenantId':'xx'\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Pickle RDD to Object Store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#!pip install --user --upgrade ibmos2spark\n",
    "#This is a tool that configures the Spark Hadoop configuration, needed for connecting to Object Store via the swift driver.\n",
    "#https://github.com/ibm-cds-labs/ibmos2spark\n",
    "\n",
    "import ibmos2spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "configuration_name = 'my_dsx_object_storage' \n",
    "bmos = ibmos2spark.bluemix(sc, credentials_1, configuration_name) #note, I use version \"2d\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 9.33 ms, sys: 4.14 ms, total: 13.5 ms\n",
      "Wall time: 1min 35s\n"
     ]
    }
   ],
   "source": [
    "%time rdd_with_data.saveAsPickleFile(bmos.url(credentials_1['container'], 'kepler1229b.sigdb.archive-compamps.rdd.dict.pickle'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6.31 ms, sys: 2.43 ms, total: 8.74 ms\n",
      "Wall time: 9.99 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "206"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time rdd_with_data.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "###### Quick Verification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.29 ms, sys: 2 ms, total: 5.28 ms\n",
      "Wall time: 5.18 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "206"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rdd_read_data = sc.pickleFile(bmos.url(credentials_1['container'], 'kepler1229b.sigdb.archive-compamps.rdd.dict.pickle'))\n",
    "%time rdd_read_data.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rdd_read_data_firsttwo = rdd_read_data.take(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pyspark.resultiterable.ResultIterable'>\n",
      "\n",
      "{u'inttimes': 94, u'pperiods': None, u'drifthzs': 0.041, u'tgtid': 150096, u'sigreason': u'Confrm', u'freqmhz': 9567.273377852, u'dec2000deg': 46.997, u'container': u'setiCompAmp', u'objectname': u'2014-10-07/act37464/2014-10-07_04-20-00_UTC.act37464.dx1014.id-1.R.archive-compamp', u'ra2000hr': 19.832, u'npul': None, u'acttype': u'target', u'power': 49.339, u'widhz': 0.087, u'catalog': u'keplerHZ', u'snr': 0.181, u'uniqueid': u'kepler8ghz_37464_1014_1_2281530', u'beamno': 1, u'sigclass': u'Cand', u'sigtyp': u'CwC', u'tscpeldeg': 0, u'pol': u'both', u'candreason': u'Confrm', u'time': u'2014-10-07T04:20:00Z', u'tscpazdeg': 0}\n"
     ]
    }
   ],
   "source": [
    "print type(rdd_read_data_firsttwo[1]['signaldb_rows'])\n",
    "print ''\n",
    "print list(rdd_read_data_firsttwo[1]['signaldb_rows'])[0]    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2014-10-07_04-20-00_UTC.act37464.dx1014.id-1.R.archive-compamp 1061928\n"
     ]
    }
   ],
   "source": [
    "print rdd_read_data_firsttwo[1]['aca_file'], len(rdd_read_data_firsttwo[1]['raw_data'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Save individual files\n",
    "\n",
    "Saving files individually will not be the most performant. **Do not do this.** Optimal object sizes to load from Object Store to Spark are in the 64 to 128 MB range. You are encouraged to save the entire RDD as a single pickled object. \n",
    "\n",
    "We leave this method here for instructional purposes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#!pip install --user --upgrade python-swiftclient\n",
    "\n",
    "import swiftclient.client as swiftclient\n",
    "\n",
    "conn = swiftclient.Connection(\n",
    "    key=credentials_1['password'],\n",
    "    authurl=credentials_1['auth_url']+\"/v3\",\n",
    "    auth_version='3',\n",
    "    os_options={\n",
    "        \"project_id\": credentials_1['project_id'],\n",
    "        \"user_id\": credentials_1['user_id'],\n",
    "        \"region_name\": credentials_1['region']})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cPickle \n",
    "\n",
    "def pickle_rows_to_objects(row):\n",
    "    pickled_row = cPickle.dumps(row, protocol=cPickle.HIGHEST_PROTOCOL)\n",
    "    #just to be clear\n",
    "    new_file_name = 'single_files/' + row['aca_file'] + '.pickle_with_sigdb'\n",
    "    resp = {}\n",
    "    etag = conn.put_object(credentials_1['container'], new_file_name , pickled_row, response_dict=resp)\n",
    "    return (new_file_name, etag, resp, len(pickled_row))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rdd_pickle_rows_to_objects = rdd_with_data.map(pickle_rows_to_objects) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 8.43 ms, sys: 4.96 ms, total: 13.4 ms\n",
      "Wall time: 46.6 s\n"
     ]
    }
   ],
   "source": [
    "%time results = rdd_pickle_rows_to_objects.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "###### Quick Verification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(u'single_files/2014-06-13_11-26-48_UTC.act18681.dx2018.id-2.L.archive-compamp.pickle_with_sigdb',\n",
       " '216b6bd2fb5b95ea106c3197da1d7633',\n",
       " {'headers': {u'content-length': u'0',\n",
       "   u'content-type': u'text/html; charset=UTF-8',\n",
       "   u'date': u'Tue, 16 Aug 2016 19:06:20 GMT',\n",
       "   u'etag': u'216b6bd2fb5b95ea106c3197da1d7633',\n",
       "   u'last-modified': u'Tue, 16 Aug 2016 19:06:21 GMT',\n",
       "   u'x-trans-id': u'tx4dbfb9263c094ba8be4ea-0057b3642b'},\n",
       "  'reason': 'Created',\n",
       "  'response_dicts': [{'headers': {u'content-length': u'0',\n",
       "     u'content-type': u'text/html; charset=UTF-8',\n",
       "     u'date': u'Tue, 16 Aug 2016 19:06:20 GMT',\n",
       "     u'etag': u'216b6bd2fb5b95ea106c3197da1d7633',\n",
       "     u'last-modified': u'Tue, 16 Aug 2016 19:06:21 GMT',\n",
       "     u'x-trans-id': u'tx4dbfb9263c094ba8be4ea-0057b3642b'},\n",
       "    'reason': 'Created',\n",
       "    'status': 201}],\n",
       "  'status': 201},\n",
       " 1063278)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rdd_read_data_pickle_with_sigdb = sc.binaryFiles(bmos.url(credentials_1['container'], 'single_files/*'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 15.6 ms, sys: 8.1 ms, total: 23.7 ms\n",
      "Wall time: 16.4 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "206"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time rdd_read_data_pickle_with_sigdb.count()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}