Copy_of_fcc_book_recommendation_knn_4.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uGd4NYQX1Rf_"
      },
      "source": [
        "*Note: You are currently reading this using Google Colaboratory which is a cloud-hosted version of Jupyter Notebook. This is a document containing both text cells for documentation and runnable code cells. If you are unfamiliar with Jupyter Notebook, watch this 3-minute introduction before starting this challenge: https://www.youtube.com/watch?v=inN8seMm7UI*\n",
        "\n",
        "---\n",
        "\n",
        "In this challenge, you will create a book recommendation algorithm using **K-Nearest Neighbors**.\n",
        "\n",
        "You will use the [Book-Crossings dataset](http://www2.informatik.uni-freiburg.de/~cziegler/BX/). This dataset contains 1.1 million ratings (scale of 1-10) of 270,000 books by 90,000 users.\n",
        "\n",
        "After importing and cleaning the data, use `NearestNeighbors` from `sklearn.neighbors` to develop a model that shows books that are similar to a given book. The Nearest Neighbors algorithm measures distance to determine the “closeness” of instances.\n",
        "\n",
        "Create a function named `get_recommends` that takes a book title (from the dataset) as an argument and returns a list of 5 similar books with their distances from the book argument.\n",
        "\n",
        "This code:\n",
        "\n",
        "`get_recommends(\"The Queen of the Damned (Vampire Chronicles (Paperback))\")`\n",
        "\n",
        "should return:\n",
        "\n",
        "```\n",
        "[\n",
        "  'The Queen of the Damned (Vampire Chronicles (Paperback))',\n",
        "  [\n",
        "    ['Catch 22', 0.793983519077301],\n",
        "    ['The Witching Hour (Lives of the Mayfair Witches)', 0.7448656558990479],\n",
        "    ['Interview with the Vampire', 0.7345068454742432],\n",
        "    ['The Tale of the Body Thief (Vampire Chronicles (Paperback))', 0.5376338362693787],\n",
        "    ['The Vampire Lestat (Vampire Chronicles, Book II)', 0.5178412199020386]\n",
        "  ]\n",
        "]\n",
        "```\n",
        "\n",
        "Notice that the data returned from `get_recommends()` is a list. The first element in the list is the book title passed in to the function. The second element in the list is a list of five more lists. Each of the five lists contains a recommended book and the distance from the recommended book to the book passed in to the function.\n",
        "\n",
        "If you graph the dataset (optional), you will notice that most books are not rated frequently. To ensure statistical significance, remove from the dataset users with less than 200 ratings and books with less than 100 ratings.\n",
        "\n",
        "The first three cells import libraries you may need and the data to use. The final cell is for testing. Write all your code in between those cells."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y1onB6kUvo4Z"
      },
      "outputs": [],
      "source": [
        "# import libraries (you may add additional imports but you may not have to)\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from scipy.sparse import csr_matrix\n",
        "from sklearn.neighbors import NearestNeighbors\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iAQGqqO_vo4d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e74b4ad6-9e9c-4bf3-af04-d0c89e4e47b7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-01-19 22:29:53--  https://cdn.freecodecamp.org/project-data/books/book-crossings.zip\n",
            "Resolving cdn.freecodecamp.org (cdn.freecodecamp.org)... 104.26.2.33, 172.67.70.149, 104.26.3.33, ...\n",
            "Connecting to cdn.freecodecamp.org (cdn.freecodecamp.org)|104.26.2.33|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 26085508 (25M) [application/zip]\n",
            "Saving to: ‘book-crossings.zip’\n",
            "\n",
            "book-crossings.zip  100%[===================>]  24.88M  29.1MB/s    in 0.9s    \n",
            "\n",
            "2022-01-19 22:29:55 (29.1 MB/s) - ‘book-crossings.zip’ saved [26085508/26085508]\n",
            "\n",
            "Archive:  book-crossings.zip\n",
            "  inflating: BX-Book-Ratings.csv     \n",
            "  inflating: BX-Books.csv            \n",
            "  inflating: BX-Users.csv            \n"
          ]
        }
      ],
      "source": [
        "# get data files\n",
        "!wget https://cdn.freecodecamp.org/project-data/books/book-crossings.zip\n",
        "\n",
        "!unzip book-crossings.zip\n",
        "\n",
        "books_filename = 'BX-Books.csv'\n",
        "ratings_filename = 'BX-Book-Ratings.csv'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NClILWOiEd6Q"
      },
      "outputs": [],
      "source": [
        "# import csv data into dataframes\n",
        "df_books = pd.read_csv(\n",
        "    books_filename,\n",
        "    encoding = \"ISO-8859-1\",\n",
        "    sep=\";\",\n",
        "    header=0,\n",
        "    names=['isbn', 'title', 'author'],\n",
        "    usecols=['isbn', 'title', 'author'],\n",
        "    dtype={'isbn': 'str', 'title': 'str', 'author': 'str'})\n",
        "\n",
        "df_ratings = pd.read_csv(\n",
        "    ratings_filename,\n",
        "    encoding = \"ISO-8859-1\",\n",
        "    sep=\";\",\n",
        "    header=0,\n",
        "    names=['user', 'isbn', 'rating'],\n",
        "    usecols=['user', 'isbn', 'rating'],\n",
        "    dtype={'user': 'int32', 'isbn': 'str', 'rating': 'float32'})"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df = df_ratings\n",
        "user_counts = df['user'].value_counts()\n",
        "user_keep = user_counts[user_counts >= 200].index\n",
        "isbn_counts = df['isbn'].value_counts()\n",
        "isbn_keep = isbn_counts[isbn_counts >= 100].index\n",
        "df = df[(df.user.isin(user_keep)) & (df.isbn.isin(isbn_keep))]\n",
        "\n",
        "# merge the two dataframes\n",
        "df_merge = pd.merge(df, df_books, on='isbn')\n",
        "\n",
        "# clean data and remove duplicate\n",
        "df_merge = df_merge.replace({' :': \": \", ' \\s+': ' '}, regex=True).drop('author', axis=1).drop_duplicates(['title','user'])\n",
        "print(df_merge.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lLCIPIOFhVsJ",
        "outputId": "773b06eb-a152-40a6-b962-0318006a49c3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(49136, 4)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xAcXjkCFCh0A",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cdf17a3e-59f2-45eb-dc0a-15705115422c"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NearestNeighbors(metric='cosine', n_neighbors=6)"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "# add your code here - consider creating a new cell for each section of code\n",
        "# filter and keep users with >=200 rating and books with >= 100 rating\n",
        "df = df_ratings\n",
        "user_counts = df['user'].value_counts()\n",
        "user_keep = user_counts[user_counts >= 200].index\n",
        "isbn_counts = df['isbn'].value_counts()\n",
        "isbn_keep = isbn_counts[isbn_counts >= 100].index\n",
        "df = df[(df.user.isin(user_keep)) & (df.isbn.isin(isbn_keep))]\n",
        "\n",
        "# merge the two dataframes\n",
        "df_merge = pd.merge(df, df_books, on='isbn')\n",
        "\n",
        "# clean data and remove duplicate\n",
        "df_merge = df_merge.replace({' :': \": \", ' \\s+': ' '}, regex=True).drop('author', axis=1).drop_duplicates(['title','user'])\n",
        "\n",
        "# generate matrix for knn analysis\n",
        "df_merge_pivot = df_merge.pivot(index = 'title', columns = 'user', values = 'rating').fillna(0)\n",
        "df_merge_matrix = csr_matrix(df_merge_pivot.values)\n",
        "\n",
        "# knn model generation\n",
        "#model_knn = NearestNeighbors(metric = 'cosine', algorithm = 'brute')\n",
        "model_knn = NearestNeighbors(metric = 'cosine',n_neighbors=6, algorithm='auto')\n",
        "model_knn.fit(df_merge_matrix)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f5ZUd-L1SQz7"
      },
      "outputs": [],
      "source": [
        "# function to return recommended books - this will be tested\n",
        "def get_recommends(book = \"\"):\n",
        "  distances, indices = model_knn.kneighbors(df_merge_pivot.loc[book].values.reshape(1, -1))\n",
        "#  recommended_books = [book, sum([[[df_merge_pivot.index[indices.flatten()[i]], distances.flatten()[i]]] for i in range(5, 0, -1)], [])]\n",
        "  recommended_books = [book, []]\n",
        "  for i in range(5, 0 , -1):\n",
        "    books = df_merge_pivot.index[indices.flatten()[i]]\n",
        "    dist = distances.flatten()[i]\n",
        "    recommended_books[1].append([books, dist])\n",
        "  return recommended_books"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eat9A2TKawHU"
      },
      "source": [
        "Use the cell below to test your function. The `test_book_recommendation()` function will inform you if you passed the challenge or need to keep trying."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jd2SLCh8oxMh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6f6ca098-b941-458d-991a-4419e4eec0a5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[\"Where the Heart Is (Oprah's Book Club (Paperback))\", [[\"I'll Be Seeing You\", 0.8016211], ['The Weight of Water', 0.77085835], ['The Surgeon', 0.7699411], ['I Know This Much Is True', 0.7677075], ['The Lovely Bones: A Novel', 0.7234864]]]\n",
            "You passed the challenge! 🎉🎉🎉🎉🎉\n"
          ]
        }
      ],
      "source": [
        "books = get_recommends(\"Where the Heart Is (Oprah's Book Club (Paperback))\")\n",
        "print(books)\n",
        "\n",
        "def test_book_recommendation():\n",
        "  test_pass = True\n",
        "  recommends = get_recommends(\"Where the Heart Is (Oprah's Book Club (Paperback))\")\n",
        "  if recommends[0] != \"Where the Heart Is (Oprah's Book Club (Paperback))\":\n",
        "    test_pass = False\n",
        "  recommended_books = [\"I'll Be Seeing You\", 'The Weight of Water', 'The Surgeon', 'I Know This Much Is True']\n",
        "  recommended_books_dist = [0.8, 0.77, 0.77, 0.77]\n",
        "  for i in range(2):\n",
        "    if recommends[1][i][0] not in recommended_books:\n",
        "      test_pass = False\n",
        "    if abs(recommends[1][i][1] - recommended_books_dist[i]) >= 0.05:\n",
        "      test_pass = False\n",
        "  if test_pass:\n",
        "    print(\"You passed the challenge! 🎉🎉🎉🎉🎉\")\n",
        "  else:\n",
        "    print(\"You haven't passed yet. Keep trying!\")\n",
        "\n",
        "test_book_recommendation()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}