10_4_2_How_to_Transcribe_Text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix 10.4.2: How to transcribe documents with Claude\n",
    "\n",
    "Claude 3 is great at reading unstructured text and information within images and PDFs and turning it into structured text. We'll take a look at a few examples but first let's setup the code we need to run the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pip install -qUr requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import json\n",
    "from datetime import datetime\n",
    "from IPython.display import Image\n",
    "from botocore.exceptions import ClientError\n",
    "\n",
    "session = boto3.Session()\n",
    "region = session.region_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "modelId = 'anthropic.claude-3-sonnet-20240229-v1:0'\n",
    "#modelId = 'anthropic.claude-3-haiku-20240307-v1:0'\n",
    "\n",
    "print(f'Using modelId: {modelId}')\n",
    "print('Using region: ', region)\n",
    "\n",
    "bedrock_client = boto3.client(service_name = 'bedrock-runtime', region_name = region,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transcribing typed text\n",
    "\n",
    "The advantage of using Claude 3 over traditional OCR systems is that you can specify exactly what you want to transcribe due to Claude 3's advanced reasoning capabilities. For this image, let’s transcribe just the code in the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "Image(filename='./images/transcribe/stack_overflow.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"./images/transcribe/stack_overflow.png\", \"rb\") as f:\n",
    "    image_file = f.read()\n",
    "\n",
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"text\": \"Transcribe the code in the answer. Only output the code.\"},\n",
    "            {\"image\": {\n",
    "                \"format\": 'png',\n",
    "                \"source\": {\"bytes\": image_file }\n",
    "                },\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "}\n",
    "\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "# Extract the generated text content from the response\n",
    "output_message = response['output']['message']['content'][0]['text']\n",
    "\n",
    "# Return the generated text content\n",
    "print(output_message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transcribing handwritten text\n",
    "\n",
    "That's good but let's try something a little harder. Claude 3 excels at transcribing handwritten text as well. Let's ask Claude 3 to transcribe this handwritten prescription note."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "Image(filename='./images/transcribe/school_notes.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"./images/transcribe/school_notes.png\", \"rb\") as f:\n",
    "    image_file = f.read()\n",
    "\n",
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"text\": \"Transcribe this text. Only output the text and nothing else.\"},\n",
    "            {\"image\": {\n",
    "                \"format\": 'png',\n",
    "                \"source\": {\"bytes\": image_file }\n",
    "                },\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "}\n",
    "\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "# Extract the generated text content from the response\n",
    "output_message = response['output']['message']['content'][0]['text']\n",
    "\n",
    "# Return the generated text content\n",
    "print(output_message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transcribing forms\n",
    "How about we try a combination of typed and handwritten text? This is common across a variety of documents like insurance and report forms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='./images/transcribe/vehicle_form.jpg') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"./images/transcribe/vehicle_form.jpg\", \"rb\") as f:\n",
    "    image_file = f.read()\n",
    "\n",
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"text\": \"Transcribe this form exactly.\"},\n",
    "            {\"image\": {\n",
    "                \"format\": 'jpeg',\n",
    "                \"source\": {\"bytes\": image_file }\n",
    "                },\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "}\n",
    "\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "# Extract the generated text content from the response\n",
    "output_message = response['output']['message']['content'][0]['text']\n",
    "\n",
    "# Return the generated text content\n",
    "print(output_message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complicated document QA\n",
    "With Claude 3 we can go beyond just transcription and ask specific questions about our information in our unstructured documents. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='./images/transcribe/page.jpeg') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"./images/transcribe/page.jpeg\", \"rb\") as f:\n",
    "    image_file = f.read()\n",
    "\n",
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"text\": \"Which is the most critical issue for live rep support?\"},\n",
    "            {\"image\": {\n",
    "                \"format\": 'jpeg',\n",
    "                \"source\": {\"bytes\": image_file }\n",
    "                },\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "}\n",
    "\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "# Extract the generated text content from the response\n",
    "output_message = response['output']['message']['content'][0]['text']\n",
    "\n",
    "# Return the generated text content\n",
    "print(output_message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unstructured information -> JSON\n",
    "\n",
    "Let's take a look at how you can use Claude to turn unstructured information in an image into a structured JSON output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='./images/transcribe/org_chart.jpeg') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"./images/transcribe/org_chart.jpeg\", \"rb\") as f:\n",
    "    image_file = f.read()\n",
    "\n",
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"text\": \"Turn this org chart into JSON indicating who reports to who. Only output the JSON and nothing else.\"},\n",
    "            {\"image\": {\n",
    "                \"format\": 'jpeg',\n",
    "                \"source\": {\"bytes\": image_file }\n",
    "                },\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "}\n",
    "\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "# Extract the generated text content from the response\n",
    "output_message = response['output']['message']['content'][0]['text']\n",
    "\n",
    "# Return the generated text content\n",
    "print(output_message)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_tensorflow2_p310",
   "language": "python",
   "name": "conda_tensorflow2_p310"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}