examples/data-enrichment/data_enrichment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Enrichment - Geo Location Data\n",
    "\n",
    "This example demonstrates how to use GlassFlow to enrich events data by calling a Geo Location API during the transformation stage. \n",
    "\n",
    "## Pre-requisites\n",
    "\n",
    "- Create your free GlassFlow account via the [GlassFlow WebApp](https://app.glassflow.dev).\n",
    "- Get your [Personal Access Token](https://app.glassflow.dev/profile) to authorize the Python SDK to interact with GlassFlow Cloud.\n",
    "- Get your Geoapify API key from https://myprojects.geoapify.com/ . This key is used in the transformation stage to get geo location data\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "%pip install \"glassflow>=2.0.8\" pandas Faker"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glassflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fill credentials\n",
    "# Please edit this variable with your own personal access token from https://app.glassflow.dev/profile\n",
    "personal_access_token = \"\"\n",
    "GEOAPI_KEY = \"\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = glassflow.GlassFlowClient(\n",
    "    personal_access_token=personal_access_token\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the space named \"examples\" (or create one if no space is found)\n",
    "list_spaces = client.list_spaces()\n",
    "\n",
    "space_name = \"examples\"\n",
    "for s in list_spaces.spaces:\n",
    "    if s[\"name\"] == space_name:\n",
    "        space = glassflow.Space(\n",
    "            personal_access_token=client.personal_access_token,\n",
    "            id=s[\"id\"], \n",
    "            name=s[\"name\"]\n",
    "        )\n",
    "        break\n",
    "else:\n",
    "    space = client.create_space(name=space_name)\n",
    "\n",
    "print(f\"Space \\\"{space.name}\\\" with ID: {space.id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformation Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pycat transform.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Env Variables needed for transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env_vars = [{\n",
    "  \"name\": \"GEOAPI_KEY\",\n",
    "  \"value\": GEOAPI_KEY\n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requirements txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"requirements.txt\") as f:\n",
    "    requirements_txt = f.read()\n",
    "display(requirements_txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_name = \"data_enrichment-example\"\n",
    "\n",
    "pipeline = client.create_pipeline(\n",
    "    name=pipeline_name, \n",
    "    transformation_file='transform.py',\n",
    "    space_id=space.id, \n",
    "    env_vars=env_vars, \n",
    "    requirements=requirements_txt\n",
    ")\n",
    "print(\"Pipeline ID:\", pipeline.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Produce data and send it to your pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a dummy data generator using python faker library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from faker import Faker\n",
    "\n",
    "def geo_data_generator():\n",
    "    fake = Faker()\n",
    "    return {\n",
    "        'address': fake.address(),\n",
    "        'source': 'example-pipeline'\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Get pipeline data source object to publish events to the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_source = pipeline.get_source()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate some data and send it to the pipeline. Store it locally to compare\n",
    "n_events = 10\n",
    "input_events = []\n",
    "for i in range(n_events):\n",
    "    event = geo_data_generator()\n",
    "    input_events.append(event)\n",
    "    data_source.publish(event)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Display data sent to the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "display(pd.DataFrame(input_events))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Consume events from the pipeline \n",
    "\n",
    "Get pipeline data sink to consume the transformed events from the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_sink = pipeline.get_sink()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_events = []\n",
    "while True:\n",
    "    resp = data_sink.consume()\n",
    "    if resp.status_code == 200:\n",
    "        output_events.append(resp.json())\n",
    "    if len(output_events) == n_events:\n",
    "        # all events have been consumed\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "display(pd.DataFrame(output_events))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the pipeline on the web-UI\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_url = f\"https://app.glassflow.dev/pipelines/{pipeline.id}\"\n",
    "print(pipeline_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}