Clinical Trials Gen AI workshop Part 2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clinical Trials Gen AI Workshop Part 2\n",
    "\n",
    "**Note:** This notebook was designed to be run in a SageMaker Studio running *Data Science 3.0* on an *ml.t3.medium* instance with 5GB of storage although other configurations may be supported.\n",
    "\n",
    "## Introduction\n",
    "\n",
    "This is continuation of Part 1, we will move the use-case closer to production by focusing on evaluation logic, chaining LLMs and cost analysis. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the real world, patient records typically live in datastore outside of the LLM. In this notebook we will explore querying the data directly from Amazon Healthlake (FHIR Datastore) or from JSON files in FHIR standard. \n",
    "\n",
    "**If you prefer to query Amazon Healthlake resource to get FHIR records follow the procedure below. If not jump down to FHIR Record [Section](#read_FHIR_records)**\n",
    "\n",
    "Let continue to get our patient records! \n",
    " \n",
    " _Note: Patient records are synthetic and are not real patient data and are generated by [Synthea](https://github.com/synthetichealth/synthea). All pricing information is provided as an example, refer to https://aws.amazon.com/bedrock/pricing/ for most up to date pricing information._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ensure we have neccessary packages, you use the requirements.txt for virtual environments\n",
    "%pip install boto3 awscrt ipykernel ipython ipywidgets requests\n",
    "# you may need to restart kernal for the environment to recognize installed packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Amazon Healthlake - FHIR Datastore\n",
    "\n",
    "Lets run the cell below to construct a FHIR query to get records related to 3 patient ids and store the results in `patients_records` directory. We will use `helper_packages` to sign the FHIR API request to Amazon Healthlake.\n",
    "\n",
    "_Note: You will need to uncomment the code to run the code._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Query FHIR Records for Amazon HealthLake\n",
    "# # uncomment to run\n",
    "# import boto3\n",
    "# client =  boto3.client('healthlake')\n",
    "# datastore_resp = client.list_fhir_datastores(\n",
    "#     Filter={\n",
    "#         'DatastoreName': 'demo-store'\n",
    "#     }\n",
    "# )\n",
    "# # Selection of pre-selected FHIR Records in Amazon Healthlake\n",
    "# patient_id = [\"20a70ecf-c423-4318-82c3-40542074d6a8\",\"04c704c4-5d2d-4308-9c33-1690a6e47a6b\",\"fddf3bac-f14d-45a9-a0b0-690435ea799b\" , \"9ae87a5e-0cd2-4573-b0b6-c37ae5e5e894\"]\n",
    "# print(\"FHIR Database URLs:\")\n",
    "# for id in patient_id:\n",
    "#     datastore_url = datastore_resp['DatastorePropertiesList'][0]['DatastoreEndpoint']\n",
    "#     full = f\"{datastore_url}Patient\"\n",
    "#     payload = [\n",
    "#         (\"identifier\",f\"{id}\"),\n",
    "#         (\"_revinclude\", 'Encounter:patient'),\n",
    "#         (\"_revinclude\", \"AllergyIntolerance:patient\"),\n",
    "#         (\"_revinclude\", \"CarePlan:patient\"),\n",
    "#         (\"_revinclude\", \"Condition:patient\"),\n",
    "#         (\"_revinclude\", \"Encounter:patient\"),\n",
    "#         (\"_revinclude\", \"DiagnosticReport:patient\"),\n",
    "#         (\"_revinclude\", \"Observation:patient\"),\n",
    "#         (\"_revinclude\", \"Medication:patient\"),\n",
    "#         (\"_revinclude\", \"MedicationDispense:patient\"),\n",
    "#         (\"_revinclude\", \"Procedure:patient\")\n",
    "#     ]\n",
    "#     payload_str =\"\"\n",
    "#     for i in payload:\n",
    "#         key=(i[0])\n",
    "#         val=(i[1])\n",
    "#         payload_str = payload_str + f\"&{key}={val}\"\n",
    "#     # The first element does not require '&' sign\n",
    "#     payload_str = (payload_str.replace('&','',1)) \n",
    "#     # Construct the full url with query '?'\n",
    "#     url = full+'?'+payload_str\n",
    "#     print(url)\n",
    "#     from helper_packages.sigv4a_sign import SigV4ASign\n",
    "#     import requests\n",
    "\n",
    "\n",
    "#     headers = SigV4ASign().get_headers_basic('healthlake', 'us-east-1', 'GET', url)\n",
    "#     r = requests.get(url, headers=headers)\n",
    "#     # For each patient record write to file in patient_records\n",
    "    # with open(f'patients_records/{id}.json','w') as f:\n",
    "    #     f.write(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='read_fhir_records'></a>\n",
    "### Read FHIR records from JSON files\n",
    "\n",
    "Lets run the cells below to read the patient record and store in memory for quick access. You may open the files directly to familiarize yourself with the patient's history. \n",
    "\n",
    "Can you identify the main differences between the patients? \n",
    "\n",
    "Patient Details:\n",
    "| Patient ID | Name | Sex | Date of Birth | Short History |\n",
    "| -- | -- | -- | -- | -- |\n",
    "| **04c704c4-5d2d-4308-9c33-1690a6e47a6b** |  Mr. Dexter530 Little434 |  Male | Date of Birth: 1997-11-22 |  history of common flu symptoms in good general health |\n",
    "| **20a70ecf-c423-4318-82c3-40542074d6a8** | Dorene845 Fadel536 |  Female | Date of Birth: 2015-05-01 | common flu symptoms in good general health |\n",
    "| **fddf3bac-f14d-45a9-a0b0-690435ea799b** | Ronnie7 Greenfelder433 |  Female | Date of Birth: 1984-08-31 | Several illness including breast cancer |\n",
    "| **9ae87a5e-0cd2-4573-b0b6-c37ae5e5e894** |  Mrs. Queenie922 Bechtelar572 |  Female | Date of Birth: 1983-09-17 | obesity and related conditions |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# File FHIR Resources (Option 2)\n",
    "import os\n",
    "import json\n",
    "# Get records from folder patient_records into memory\n",
    "list_files = os.listdir('patients_records')\n",
    "patient_data = {}\n",
    "for i in list_files:\n",
    "    if '.json' in i:\n",
    "        with open(f'patients_records/{i}') as f:\n",
    "            result = json.load(f)\n",
    "            # Condense JSON Object \n",
    "            result = json.dumps(result)\n",
    "            patient_data[i] = result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h1>Choose Patient Record</h1>\n",
       "<h3>Select a patient record to continue, you may revisit this cell to choose another patient record.</h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0710889e99f64c88acb250c89fc03148",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Button(description='04c704c4-5d2d-4308-9c33-1690a6e47a6b.json', style=ButtonStyle()), Button(de…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display UI to Switch Between Patient Records\n",
    "from helper_packages.choice import Prompt\n",
    "from IPython.display import HTML, display\n",
    "patient_choice = Prompt(patient_data)\n",
    "display(HTML(\"<h1>Choose Patient Record</h1>\\n<h3>Select a patient record to continue, you may revisit this cell to choose another patient record.</h3>\"))\n",
    "# If the button does not load up, you need to reload the window\n",
    "display(patient_choice.get_buttons())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Study Information from ClinicalTrials.gov\n",
    "\n",
    "In the next cell, we will download Study details directly from ClinicalTrials.gov. We have opted to download only `EligibilityCriteria`,`HealthyVolunteers`,`Sex`,`MinimumAge`,`MaximumAge`,`BriefTitle`, and `BriefSummary` sections.\n",
    "\n",
    "Study Details\n",
    "\n",
    "| Study ID | Name |\n",
    "| -- | --|\n",
    "| NCT04510376 | Allergy Potential of Omeza Collagen Matrix in Human Subjects Using the Skin Prick Method |\n",
    "| NCT02340468 | Breast Tumor Oxygenation During Exercise |\n",
    "| NCT05174689 | Epigenetic Regulation of Exercise Induced Asthma |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h1>Choose Clinical Study</h1>\n",
       "<h3>Select a Clinical Study to continue, you may revisit this cell to choose another Clinical Study</h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0a49039fc6524909b4b0f9faa91ff1d8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Button(description='NCT04510376', style=ButtonStyle()), Button(description='NCT02340468', style…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display UI to Switch Between Clinical Studies\n",
    "from helper_packages.choice import Prompt\n",
    "from IPython.display import HTML, display\n",
    "import requests\n",
    "\n",
    "list_studies = ['NCT04510376', 'NCT02340468','NCT05174689']\n",
    "study = {}\n",
    "for i in list_studies:\n",
    "    study[i] = requests.get(f\"https://clinicaltrials.gov/api/int/studies/download/{i}?format=json&fields=EligibilityCriteria%2CHealthyVolunteers%2CSex%2CMinimumAge%2CMaximumAge%2CBriefTitle%2CBriefSummary\").text\n",
    "study_choice = Prompt(study)\n",
    "display(HTML(\"<h1>Choose Clinical Study</h1>\\n<h3>Select a Clinical Study to continue, you may revisit this cell to choose another Clinical Study</h3>\"))\n",
    "display(study_choice.get_buttons())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If the clincial Trials urls have network issues uncomment and run this cell. If not skip this cell.\n",
    "\n",
    "# import os\n",
    "# list_files = os.listdir('study')\n",
    "# study_data = {}\n",
    "# for i in list_files:\n",
    "#     if '.json' in i:\n",
    "#         with open(f'study/{i}','r') as f:\n",
    "#             result = json.load(f)\n",
    "#             result = json.dumps(result)\n",
    "#             study_data[i] = result\n",
    "# study_choice_manual = Prompt(study_data)\n",
    "# display(HTML(\"<h1>Choose Clincial Study</h1>\\n<h3>Select a Clincial Study to continue, you may revisit this cell to choose another Clincial Study</h3>\"))\n",
    "# display(study_choice_manual.get_buttons())\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prompt Engineering \n",
    "\n",
    "In this section we will deconstruct the prompt to better understand why we did it this way:\n",
    "\n",
    "- Provide dynamic information or important information in xml tags - It is important to provide prompts in ways the model is most [familiar with](https://docs.anthropic.com/claude/docs/use-xml-tags). Claude was trained on prompts with XML tags. \n",
    "\n",
    "- _\"You are medical researcher checking if a patient is eligible for a clinical study.\"_ - Give LLM a [role](https://docs.anthropic.com/claude/docs/give-claude-a-role#when-to-use-role-prompting) in this highly technical task.  \n",
    "\n",
    "- _FHIR healthcare record with <patient> tags_ - Help the LLM identity the type of information they are given. Validate the model's knowledge in one-off prompts.\n",
    "- _\"Please follow these steps to evaluate if patient is suitable for study in <study> tags:\"_ - In highly technical workflow, we can teach the model on how to [think](https://docs.anthropic.com/claude/docs/let-claude-think#how-to-prompt-for-thinking-step-by-step) \n",
    "    - It is important to guide the LLM to make decisions based on its understanding and if information is missing educate the model on what path it should take. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "import json\n",
    "from botocore.config import Config\n",
    "from helper_packages.Tokencounter import PrettyPrintModel\n",
    "# To ensure boto3 has time to get a response \n",
    "config = Config(read_timeout=1000)\n",
    "\n",
    "# Get choice from buttons above\n",
    "result = patient_choice.get_choice()\n",
    "trial = study_choice.get_choice()\n",
    "\n",
    "\n",
    "today = datetime.today().strftime('%Y-%m-%d')\n",
    "input_prompt = f\"\"\"\n",
    "You are medical researcher checking if a patient is eligible for a clinical study. You are given FHIR healthcare record within <patient> tags.\n",
    "\n",
    "<patient>{result}</patient>\n",
    "\n",
    "Please follow these steps to evaluate if patient is suitable for study in <study> tags:\n",
    "\n",
    "Step 1: Calculate the age of the patient based on today's date of {today}.\n",
    "Step 2: Validate if patient age is eligible for study. If patient does not met study requirements then patient is not eligible then skip other steps.\n",
    "Step 3: Validate if patient mets study gender requirements if patient does not met gender requirements then patient is not eligible then skip other steps.\n",
    "Step 4: Validate if patient mets all inclusion requirements.\n",
    "Step 5: If there is no information about whether the patient mets the criteria then patient is not eligible for study then say patient not eligible and skip other steps.\n",
    "Step 6: If there is no indication patient mets inclusion requirements then say patient is not eligible for study then skips other steps.\n",
    "Step 7: If patient mets any exclusion requirements then patient is not eligible then skip other steps.\n",
    "Step 8: If study accepts health patients and patient is healthy then patient is eligible for study. If patient health status is unclear, then patient is not eligible.\n",
    "\n",
    "\n",
    "<study>{trial}</study>\n",
    "\n",
    "\n",
    "Please provide your reasoning in <reason> tag and if the patient is eligible as \"true\" and If candidate is not eligible as \"false\" in <result> tag and if the patient is maybe eligible as as \"possible\" in <result> tag.\n",
    "\"\"\"\n",
    "import boto3\n",
    "import json\n",
    "brt = boto3.client(service_name='bedrock-runtime',config=config)\n",
    "\n",
    "\n",
    "body = json.dumps({\n",
    "    \"anthropic_version\": \"bedrock-2023-05-31\",\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\":[\n",
    "                {\"type\": \"text\",\"text\": input_prompt}\n",
    "            ]\n",
    "        }\n",
    "    ],\n",
    "    \"max_tokens\": 10000,\n",
    "    \"temperature\": 0.0,\n",
    "    \"top_p\": 0.9,\n",
    "    \"top_k\": 50\n",
    "})\n",
    "\n",
    "model_id = 'anthropic.claude-3-sonnet-20240229-v1:0'\n",
    "accept = 'application/json'\n",
    "content_type = 'application/json'\n",
    "print(f\"Invoking {model_id}.....\")\n",
    "print(f'Study: {json.loads(trial)[\"protocolSection\"][\"identificationModule\"][\"briefTitle\"]}')\n",
    "print(f'Patient Name: {json.loads(result)[\"entry\"][0][\"resource\"][\"name\"]}')\n",
    "response = brt.invoke_model(body=body, modelId=model_id, accept=accept, contentType=content_type)\n",
    "# Print input prompt\n",
    "# print(input_prompt)\n",
    "# print(\"=\" * 70)\n",
    "single_token = PrettyPrintModel(response,model_id)\n",
    "print(single_token)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review Claude Sonnet Response\n",
    "\n",
    "In the response we can view the model's reasoning and final output. **Do you agree with the thought process it took to arrive at the conclusion in the `<result>` tag?**\n",
    "\n",
    "We also see the cost of running the prompt 1,000 times or trying to match 1,000 similar sized FHIR records with this study. In the real world we may get a list of 100 patients and check if they are matching against 1 study and vice versa. \n",
    "\n",
    "In all of these cases it is important to track the number of token and its effect on total cost."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chain Together Models\n",
    "\n",
    "Similar to writing prompts to LLM chat bot and guiding it along a workflow, we can use output of one model and feed it to another model. In the below cell, we will take an approach of summarizing the patients FHIR record using [Claude Haku](https://www.anthropic.com/news/claude-3-family). This will help with lowering the overall number of token being sent to next model (Sonnet).\n",
    "\n",
    "In the next cells, notice the **time to execute the prompt** and **costs**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Chain together requests\n",
    "\n",
    "\n",
    "modelId_chain_1='anthropic.claude-3-haiku-20240307-v1:0'\n",
    "# Get choice from buttons above\n",
    "result = patient_choice.get_choice()\n",
    "\n",
    "input_prompt_chain_1 = f\"\"\"\n",
    "Summarize the following FHIR record in tags <patient> and focus on extracting information related to patient, medication list and past illness with resolutions.\n",
    "\n",
    "<patient>{result}</patient>\n",
    "\n",
    "here is output example\n",
    "\n",
    "<information>contain patient information</information> \n",
    "<medication>contain medication list<medication> \n",
    "<history>contain history with time it took to resolve<history> \n",
    "\n",
    "\"\"\"\n",
    "accept = 'application/json'\n",
    "contentType = 'application/json'\n",
    "print(f'Invoke model: {modelId_chain_1}')\n",
    "body_chain_1 = json.dumps({\n",
    "    \"anthropic_version\": \"bedrock-2023-05-31\",\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\":[\n",
    "                {\"type\": \"text\",\"text\": input_prompt_chain_1}\n",
    "            ]\n",
    "        }\n",
    "    ],\n",
    "    \"max_tokens\": 10000,\n",
    "    \"temperature\": 0.0,\n",
    "    \"top_p\": 0.9,\n",
    "    \"top_k\": 50\n",
    "})\n",
    "print(f\"Invoking {model_id}.....\")\n",
    "response_chain_1 = brt.invoke_model(body=body_chain_1, modelId=modelId_chain_1, accept=accept, contentType=contentType)\n",
    "# Print prompt input \n",
    "# print(input_prompt_chain_1)\n",
    "# print(\"=\" * 70)\n",
    "token_chain_1 =PrettyPrintModel(response_chain_1,modelId_chain_1)\n",
    "print(token_chain_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chaining Models Together - Review Haiku\n",
    "\n",
    "We are first invoking Haiku model to summarize a patient's FHIR record before passing it along to next model. [Haiku](https://www.anthropic.com/news/claude-3-haiku) is great at quickly analyzing large datasets. In leveraging lower priced model to summarize the output, we can lower the total cost overall by sending the summary to more intelligent model for reasoning.  \n",
    "\n",
    "A patient medical history can be quite large, especially patients with illness so we need to keep in mind the context window for each model. Both Haiku and Sonnet has large context window of 200K tokens. In the process of using two models, we can optimize the large input (FHIR Record) to Haiku (lower price) and smaller input (Summary + Study) to Sonnet (complex reasoning)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Chain together requests\n",
    "\n",
    "model_id_chain_2 = 'anthropic.claude-3-sonnet-20240229-v1:0'\n",
    "# Get choice from buttons above\n",
    "trial_chain_2 = study_choice.get_choice()\n",
    "today = datetime.today().strftime('%Y-%m-%d')\n",
    "input_prompt_chain_2 = f\"\"\"\n",
    "You are medical researcher checking if a patient is eligible  for a clincial study. You are given summarized healthcare information in <patient> tags.\n",
    "\n",
    "{token_chain_1.content}\n",
    "\n",
    "\n",
    "Please follow these steps to evaluate if patient is suitable for study in <study> tags:\n",
    "\n",
    "Step 1: Calculate the age of the patient based on today's date of {today}.\n",
    "Step 2: Validate if patient age is eligible for study. If patient does not met study requirements then patient is not eligible then skip other steps.\n",
    "Step 3: Validate if patient mets study gender requirements if patient does not met gender requirements then patient is not eligible then skip other steps.\n",
    "Step 4: Validate if patient mets all inclusion requirements.\n",
    "Step 5: If there is no information about whether the patient mets the criteria then patient is not eligible for study then say patient not eligible and skip other steps.\n",
    "Step 6: If there is no indication patient mets inclusion requirements then say patient is not eligible for study then skips other steps.\n",
    "Step 7: If patient mets any exclusion requirements then patient is not eligible then skip other steps.\n",
    "Step 8: If study accepts health patients and patient is healthy then patient is eligible for study. If patient health status is unclear, then patient is not eligible.\n",
    "\n",
    "\n",
    "<study>{trial_chain_2}</study>\n",
    "\n",
    "\n",
    "Please provide your reasoning in <reason> tag and if the patient is eligible as \"true\" and If candidate is not eligible as \"false\" in <result> tag and if the patient is maybe eligible as as \"possible\" in <result> tag.\n",
    "\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "accept = 'application/json'\n",
    "contentType = 'application/json'\n",
    "print(f'Invoke model: {model_id_chain_2}')\n",
    "body_chain_2 = json.dumps({\n",
    "    \"anthropic_version\": \"bedrock-2023-05-31\",\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\":[\n",
    "                {\"type\": \"text\",\"text\": input_prompt_chain_2}\n",
    "            ]\n",
    "        }\n",
    "    ],\n",
    "    \"max_tokens\": 10000,\n",
    "    \"temperature\": 0.0,\n",
    "    \"top_p\": 0.9,\n",
    "    \"top_k\": 50\n",
    "})\n",
    "print(f\"Invoking {model_id}.....\")\n",
    "response_chain_2 = brt.invoke_model(body=body_chain_2, modelId=model_id_chain_2, accept=accept, contentType=contentType)\n",
    "\n",
    "# Optional\n",
    "# print(input_prompt_chain_2)\n",
    "print(\"=\" * 70)\n",
    "token_chain_2 = PrettyPrintModel(response_chain_2,model_id_chain_2)\n",
    "print(token_chain_2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chaining Models Together - Review Sonnet\n",
    "\n",
    "We have completed the chain by taking the summarization of FHIR record focusing on specific conditions and gave it as input to Sonnet model.\n",
    "\n",
    "Sonnet is more intelligent in Haiku and is better [choice](https://aws.amazon.com/about-aws/whats-new/2024/03/anthropics-claude-3-sonnet-model-amazon-bedrock/) for complex reasoning and analysis. \n",
    "\n",
    "See the results for performance and cost in following cell!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import locale\n",
    "locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')\n",
    "price_input = token_chain_1.raw_input_cost + token_chain_2.raw_input_cost\n",
    "price_output = token_chain_1.raw_output_cost + token_chain_2.raw_output_cost\n",
    "print(\"The estimated price for 1000 similar patients evaluated for the study. All prices are provided as examples, refer to https://aws.amazon.com/bedrock/pricing/ for most up to pricing\\n\")\n",
    "print(\"Single Model\")\n",
    "print(f\"Total Estimated Price\\nInput: {single_token.input_cost}\\nOutput: {single_token.output_cost}\\nTotal {locale.currency(single_token.raw_input_cost+single_token.raw_output_cost)}\\nLatency {single_token.latency}ms\\n\\n\")\n",
    "print(\"Chaining Model\")\n",
    "print(f\"Total Estimated Price\\nInput: {locale.currency(price_input)}\\nOutput: {locale.currency(price_output)}\\nTotal {locale.currency(price_input+price_output)}\\nLatency {token_chain_1.latency + token_chain_2.latency}ms\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}