Exploratory_Data_Analysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/EAName/yelp-reviews/blob/main/Exploratory_Data_Analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# EDA notebook"
      ],
      "metadata": {
        "id": "Uex-J9oEMyKY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!git status"
      ],
      "metadata": {
        "id": "7Cyu8DgeZJrm",
        "outputId": "b59f6736-6754-44b3-8ffe-fb6863ce7fb4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "fatal: not a git repository (or any of the parent directories): .git\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%cd {repository}"
      ],
      "metadata": {
        "id": "jyK13TTUZuoE",
        "outputId": "9a1c8444-3179-47c5-eebd-5d1bd885fc01",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[Errno 2] No such file or directory: '{repository}'\n",
            "/content\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Ingest Data"
      ],
      "metadata": {
        "id": "TRpI1R2fQAQt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive, files\n",
        "drive.mount('/content/drive', force_remount = True)"
      ],
      "metadata": {
        "id": "Ule19piAP9jl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import matplotlib\n",
        "from matplotlib import pyplot as plt\n",
        "import seaborn as sns\n",
        "import statistics\n",
        "import numpy as np"
      ],
      "metadata": {
        "id": "mh_tkmBoQG0-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# First drop yelp_dataset into your drive (Add shortcut to Drive)\n",
        "business_json_path = '/content/gdrive/MyDrive/yelp_dataset/yelp_academic_dataset_business.json'\n",
        "df_b = pd.read_json(business_json_path, lines=True)"
      ],
      "metadata": {
        "id": "qN1asA02QKGx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#keep only restaurant category\n",
        "df_b = df_b[df_b['categories'].str.contains(\n",
        "              'restaurant|Restaurant|RESTAURANT',\n",
        "              case=False, na=False)]"
      ],
      "metadata": {
        "id": "tB3qP_u1QZUN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_b.head()"
      ],
      "metadata": {
        "id": "7HqMKHkbQcAN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Elaine's DF Clean of Business\n"
      ],
      "metadata": {
        "id": "Tz1bqjBYMpuV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Drop Canada and other states"
      ],
      "metadata": {
        "id": "plm0ynljNGFP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# look at rows with NC, CO, HI, MT, XMS to see if they are mis-labeled or if they should be dropped\n",
        "\n",
        "df_b.loc[df_b['state'].isin([\"NC\", \"CO\", \"HI\", \"MT\", \"XMS\"])]"
      ],
      "metadata": {
        "id": "T3D64c9pMzS-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Remove these 5 states since they don't belong to a market "
      ],
      "metadata": {
        "id": "0a_IGbqhM89u"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Drop rows for restaurants in outlier states (4 rows) & Canada\n",
        "\n",
        "#dfb_restaurants1.loc[dfb_restaurants1['state'].isin([\"NC\", \"CO\", \"HI\", \"MT\", \"XMS\"])]\n",
        "\n",
        "removestates = [\"NC\", \"CO\", \"HI\", \"MT\", \"XMS\", \"AB\"]\n",
        "\n",
        "df_b = df_b[~ df_b_['state'].isin(removestates)]"
      ],
      "metadata": {
        "id": "sTtIVbX-NNl-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Total restaurants remaining should be 49,871"
      ],
      "metadata": {
        "id": "95abN3cURqSV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Normalize cities and states into DMA markets with market column\n",
        "For CA cities, I checked that Reno, NV should only contain NV rows and all CA rows belong to Santa Barbara"
      ],
      "metadata": {
        "id": "lXvCnLN_Nknk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Test that CA zipcodes belonging to Reno DMA are not in the df\n",
        "\n",
        "renoziptest = [\"93512\", \"93517\", \"93529\", \"93541\", \"93546\", \"95646\", \"96009\", \"96056\", \"96068\", \"96107\", \"96109\", \"96113\", \"96114\", \"96117\", \"96119\", \"96120\", \"96121\", \"96123\", \"96127\", \n",
        "               \"96128\", \"96130\", \"96132\", \"96133\", \"96136\", \"96150\", \"96151\", \"96152\", \"96154\", \"96155\", \"96156\", \"96157\", \"96158\"]"
      ],
      "metadata": {
        "id": "e9SnLF7xTvml"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "renotestdf = df_b[df_b['postal_code'].isin(renoziptest)]\n",
        "renotestdf.shape"
      ],
      "metadata": {
        "id": "w5oCMq7yT2NF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Get list of all states\n",
        "\n",
        "df_b.state.value_counts()"
      ],
      "metadata": {
        "id": "TleRCiFAN4x9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#create market dictionary\n",
        "\n",
        "market = {'PA': 'Philadelphia', 'DE': 'Philadelphia', 'NJ': 'Philadelphia', 'FL': 'Tampa', 'IN': 'Indianapolis', 'TN': 'Nashville', 'AZ': 'Tucson', 'LA': 'New Orleans', 'MO': 'St. Louis', 'IL': 'St. Louis',\n",
        "          'NV': 'Reno', 'CA': 'Santa Barbara', 'ID': 'Boise'}"
      ],
      "metadata": {
        "id": "lF2a7BOgNf6-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_b['market']= df_b['state'].map(market)"
      ],
      "metadata": {
        "id": "X7Z0IEFCNsHu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Remove rows to narrow \"Restaurant\" Definition to Sit-down restaurants"
      ],
      "metadata": {
        "id": "QHHjuqU4OjHt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Create DF to isolate the rows we want to remove\n",
        "#Please add more as needed\n",
        "\n",
        "df_bfast = df_b[df_b['categories'].str.contains('Fast Food|Bakery|Bakeries|Food Trucks|Street Vendors|Coffee| Coffee & Tea|Caterers',\n",
        "                                                                                 case=False, na=False)]\n",
        "df_bfast.shape"
      ],
      "metadata": {
        "id": "YJ32tUweOhdX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_b = df_b.drop(df_bfast.index)\n",
        "df_b.shape"
      ],
      "metadata": {
        "id": "h4qdjcNEPpNu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Total rows remaining should be 36,923"
      ],
      "metadata": {
        "id": "xboE5G5zRfGv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Check whether top fast food names are still in the df\n",
        "\n",
        "fast_list = [\"McDonald's\", \"Subway\", \"Taco Bell\", \"Burger King\", \"Wendy's\", \"Wawa\",\"Domino's Pizza\", \"Pizza Hut\", \"Papa John's Pizza\", \"Jimmy John's\",\n",
        "             \"KFC\", \"Chick-fil-A\", \"Popeyes Louisiana Kitchen\", \"Chipotle Mexican Grill\", \"Starbucks\", \"Panera Bread\", \"Sonic Drive-In\", \"Arby's\",\n",
        "             \"Panda Express\", \"Waffle House\", \"IHOP\", \"Jack in the Box\", \"Jersey Mike's Subs\" , \"Steak ’n Shake\", \"Five Guys\", \"Dairy Queen\" , \"QDOBA Mexican Eats\", \n",
        "             \"Little Caesars Pizza\", \"Hardee's\", \"Wingstop\", \"Firehouse Subs\", \"Dairy Queen Grill & Chill\", \"Buffalo Wild Wings\", \"Checkers\", \"China Wok\", \n",
        "             \"White Castle\", \"Imo's Pizza\", \"Papa Murphy's\", \"PrimoHoagies\", \"Saladworks\", \"Marco's Pizza\", \"QuikTrip\", \"Philly Pretzel Factory\", \"Boston Market\"\n",
        "              \"Moe's Southwest Grill\", \"Church's Chicken\", \"Culver's\", \"Rally's\", \"Cracker Barrel Old Country Store\", \"Tropical Smoothie Café\", \n",
        "             \"Hungry Howie's Pizza & Subs\", \"McAlister's Deli\", \"Jet's Pizza\", \"Hooters\", \"Little Caesars\", \"St. Louis Bread\", \"Friendly's\", \n",
        "             \"Long John Silver's\", \"Dickey's Barbecue Pit\", \"Zaxby's Chicken Fingers & Buffalo Wings\", \"Captain D's\", \"Chuck E. Cheese\", \"Carl's Jr\",\n",
        "             \"MOD Pizza\", \"Einstein Bros. Bagels\", \"Honey Baked Ham Company\", \"eegee's\", \"Manhattan Bagel\", \"Raising Cane's Chicken Fingers\", \"Noodles & Company\", \n",
        "             \"Smashburger\", \"Penn Station East Coast Subs\", \"Tijuana Flats\",  \"Westshore Pizza\", \"Zoes Kitchen\", \"Port of Subs\", \"Quiznos\", \n",
        "             \"Shake Shack\", \"Fazoli's\"]"
      ],
      "metadata": {
        "id": "aq7ueZhrR7L1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(df_b[df_b['name'].isin(fast_list)]))"
      ],
      "metadata": {
        "id": "F_3h6AXmSCQt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "There are still 1846 rows with the chains in fast_list. We may need to check if we can delete from Pizza and Sandwich category. The only issue with deleting Pizza is that it may delete \"Italian\" also."
      ],
      "metadata": {
        "id": "YgXgUTkxSIpl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Larger Chains that are still kept within the Total restaurants list to provide comparison for sit-down restaurants:\n",
        "\n",
        "\n",
        "\n",
        "*   Applebee's Grill + Bar\n",
        "*   Chili's\n",
        "* Denny's\n",
        "* First Watch\n",
        "* Outback Steakhouse\n",
        "* China King - there is a chain but there are also independents\n",
        "* Olive Garden Italian Restaurant\n",
        "* LongHorn Steakhouse\n",
        "* Red Lobster\n",
        "* Ruby Tuesday\n",
        "* TGI Fridays\n",
        "* Bob Evans\n",
        "* Red Robin Gourmet Burgers and Brews\n",
        "* Beef 'O'Brady's\n",
        "* Texas Roadhouse\n",
        "* Carrabba's Italian Grill\n",
        "* China House\n",
        "* O'Charley's Restaurant & Bar\n",
        "* Bonefish Grill\n",
        "* New China\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "Wjq_Y8r7SWHF"
      }
    }
  ]
}