-
Notifications
You must be signed in to change notification settings - Fork 661
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Last site change was 591064944.
- Loading branch information
Showing
3 changed files
with
95 additions
and
78 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -48,12 +48,12 @@ | |
| "source": [ | ||
| "<table class=\"tfo-notebook-buttons\" align=\"left\">\n", | ||
| " <td>\n", | ||
| " <a target=\"_blank\" href=https://ai.google.dev/docs/semantic_retriever\"><img src=\"https://ai.google.dev/static/site-assets/images/docs/notebook-site-button.png\" height=\"32\" width=\"32\" />View on Generative AI</a>\n", | ||
| " <a target=\"_blank\" href=\"https://ai.google.dev/docs/semantic_retriever\"><img src=\"https://ai.google.dev/static/site-assets/images/docs/notebook-site-button.png\" height=\"32\" width=\"32\" />View on Generative AI</a>\n", | ||
| " <td>\n", | ||
| " <a target=\"_blank\" href=\"https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/docs/semantic_retriever.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n", | ||
| " </td>\n", | ||
| " <td>\n", | ||
| " <a target=\"_blank\" href=\" https://github.com/google/generative-ai-docs/blob/main/site/en/docs/semantic_retriever.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n", | ||
| " <a target=\"_blank\" href=\"https://github.com/google/generative-ai-docs/blob/main/site/en/docs/semantic_retriever.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n", | ||
| " </td>\n", | ||
| "</table>" | ||
| ] | ||
|
|
@@ -113,22 +113,24 @@ | |
| "source": [ | ||
| "## Authenticate\n", | ||
| "\n", | ||
| "The Google Semantic Retriever API lets you perform semantic search on your own data. Since it's **your data**, this needs stricter access controls than API keys. Authenticate with OAuth with your [user credentials](#setup_oauth_with_user_credentials) or through [service accounts](#setup_oauth_using_service_accounts). As a first step, [enable the Generative Language API](../tutorials/oauth_quickstart#1_enable_the_api)." | ||
| "The Google Semantic Retriever API lets you perform semantic search on your own data. Since it's **your data**, this needs stricter access controls than API keys. Authenticate with OAuth with [service accounts](#scrollTo=eLjhFIOQ7_Dk) or through your [user credentials](#scrollTo=9YGv4x9ehLba). As a first step, [enable the Generative Language API](https://ai.google.dev/docs/oauth_quickstart#enable-api)." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": { | ||
| "id": "9YGv4x9ehLba" | ||
| "id": "eLjhFIOQ7_Dk" | ||
| }, | ||
| "source": [ | ||
| "### Setup OAuth with user credentials\n", | ||
| "### Setup OAuth using service accounts (Preferred)\n", | ||
| "\n", | ||
| "Follow the steps below from the [OAuth Quickstart](/docs/oauth_quickstart) to setup OAuth authentication.\n", | ||
| "Follow the steps below to setup OAuth using service accounts:\n", | ||
| "\n", | ||
| "1. [Configure the OAuth consent screen](/docs/oauth_quickstart#2_configure_the_oauth_consent_screen).\n", | ||
| "1. Create the Service Account by following the [documentation](https://developers.google.com/identity/protocols/oauth2/service-account#creatinganaccount){:.external}.\n", | ||
| "\n", | ||
| "1. [Authorize credentials for a desktop application](/docs/oauth_quickstart#3_authorize_credentials_for_a_desktop_application). To run this notebook in Colab, first rename your credential file (usually `client_secret_*.json`) to just `client_secret.json`. Then upload the file by using the file icon on the left sidebar, then the upload icon, as shown in the screenshot below.\n", | ||
| " * After creating the service account, generate a service account key.\n", | ||
| "\n", | ||
| "1. Upload your service account file by using the file icon on the left sidebar, then the upload icon, as shown in the screenshot below.\n", | ||
| "\n", | ||
| "<img width=400 src=\"https://developers.generativeai.google/tutorials/images/colab_upload.png\">" | ||
| ] | ||
|
|
@@ -137,67 +139,69 @@ | |
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "id": "9C3X6r1dueO4" | ||
| "id": "akwWUSrp8Bx2" | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Replace TODO-your-project-name with the project used in the OAuth Quickstart\n", | ||
| "project_name = \"TODO-your-project-name\" # @param {type:\"string\"}\n", | ||
| "# Replace [email protected] with the email added as a test user in the OAuth Quickstart\n", | ||
| "email = \"[email protected]\" # @param {type:\"string\"}\n", | ||
| "# Rename the uploaded file to `client_secret.json` OR\n", | ||
| "# Change the variable `client_file_name` in the code below.\n", | ||
| "client_file_name = \"client_secret.json\"\n", | ||
| "!pip install -U google-auth-oauthlib" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "id": "2jZmqVCj8FKa" | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Rename the uploaded file to `service_account_key.json` OR\n", | ||
| "# Change the variable `service_account_file_name` in the code below.\n", | ||
| "service_account_file_name = 'service_account_key.json'\n", | ||
| "\n", | ||
| "# IMPORTANT: Follow the instructions from the output - you must copy the command\n", | ||
| "# to your terminal and copy the output after authentication back here.\n", | ||
| "!gcloud config set project $project_name\n", | ||
| "!gcloud config set account $email\n", | ||
| "from google.oauth2 import service_account\n", | ||
| "\n", | ||
| "# NOTE: The simplified project setup in this tutorial triggers a \"Google hasn't verified this app.\" dialog.\n", | ||
| "# This is normal, click \"Advanced\" -> \"Go to [app name] (unsafe)\"\n", | ||
| "!gcloud auth application-default login --no-browser --client-id-file=$client_file_name --scopes=\"https://www.googleapis.com/auth/generative-language.retriever,https://www.googleapis.com/auth/cloud-platform\"" | ||
| "credentials = service_account.Credentials.from_service_account_file(service_account_file_name)\n", | ||
| "\n", | ||
| "scoped_credentials = credentials.with_scopes(\n", | ||
| " ['https://www.googleapis.com/auth/cloud-platform', 'https://www.googleapis.com/auth/generative-language.retriever'])" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": { | ||
| "id": "lybTOlN9gz0I" | ||
| "id": "Qqjs2tvvicqq" | ||
| }, | ||
| "source": [ | ||
| "Initialize the client library." | ||
| "Initialize the client library using the service account credentials." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "id": "9MJbDTpMmlKJ" | ||
| "id": "P719DMtK8t-p" | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import google.ai.generativelanguage as glm\n", | ||
| "\n", | ||
| "generative_service_client = glm.GenerativeServiceClient()\n", | ||
| "retriever_service_client = glm.RetrieverServiceClient()\n", | ||
| "permission_service_client = glm.PermissionServiceClient()" | ||
| "generative_service_client = glm.GenerativeServiceClient(credentials=scoped_credentials)\n", | ||
| "retriever_service_client = glm.RetrieverServiceClient(credentials=scoped_credentials)\n", | ||
| "permission_service_client = glm.PermissionServiceClient(credentials=scoped_credentials)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": { | ||
| "id": "eLjhFIOQ7_Dk" | ||
| "id": "9YGv4x9ehLba" | ||
| }, | ||
| "source": [ | ||
| "### Setup OAuth using service accounts\n", | ||
| "### Setup OAuth with user credentials\n", | ||
| "\n", | ||
| "Optionally, you can use OAuth using service accounts:\n", | ||
| "Follow the steps below from the [OAuth Quickstart](https://ai.google.dev/docs/oauth_quickstart) to setup OAuth authentication.\n", | ||
| "\n", | ||
| "1. Create the Service Account by following the [documentation](https://developers.google.com/identity/protocols/oauth2/service-account#creatinganaccount){:.external}.\n", | ||
| "1. [Configure the OAuth consent screen](https://ai.google.dev/docs/oauth_quickstart#configure-oauth).\n", | ||
| "\n", | ||
| " * After creating the service account, generate a service account key.\n", | ||
| "\n", | ||
| "1. Upload your service account file by using the file icon on the left sidebar, then the upload icon, as shown in the screenshot below.\n", | ||
| "1. [Authorize credentials for a desktop application](https://ai.google.dev/docs/oauth_quickstart#authorize-credentials). To run this notebook in Colab, first rename your credential file (usually `client_secret_*.json`) to just `client_secret.json`. Then upload the file by using the file icon on the left sidebar, then the upload icon, as shown in the screenshot below.\n", | ||
| "\n", | ||
| "<img width=400 src=\"https://developers.generativeai.google/tutorials/images/colab_upload.png\">" | ||
| ] | ||
|
|
@@ -206,54 +210,50 @@ | |
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "id": "akwWUSrp8Bx2" | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!pip install -U google-auth-oauthlib" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "id": "2jZmqVCj8FKa" | ||
| "id": "9C3X6r1dueO4" | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Rename the uploaded file to `service_account_secret.json` OR\n", | ||
| "# Change the variable `service_account_file_name` in the code below.\n", | ||
| "service_account_file_name = 'service_account_secret.json'\n", | ||
| "\n", | ||
| "from google.oauth2 import service_account\n", | ||
| "# Replace TODO-your-project-name with the project used in the OAuth Quickstart\n", | ||
| "project_name = \"TODO-your-project-name\" # @param {type:\"string\"}\n", | ||
| "# Replace [email protected] with the email added as a test user in the OAuth Quickstart\n", | ||
| "email = \"[email protected]\" # @param {type:\"string\"}\n", | ||
| "# Rename the uploaded file to `client_secret.json` OR\n", | ||
| "# Change the variable `client_file_name` in the code below.\n", | ||
| "client_file_name = \"client_secret.json\"\n", | ||
| "\n", | ||
| "credentials = service_account.Credentials.from_service_account_file(service_account_file_name)\n", | ||
| "# IMPORTANT: Follow the instructions from the output - you must copy the command\n", | ||
| "# to your terminal and copy the output after authentication back here.\n", | ||
| "!gcloud config set project $project_name\n", | ||
| "!gcloud config set account $email\n", | ||
| "\n", | ||
| "scoped_credentials = credentials.with_scopes(\n", | ||
| " ['https://www.googleapis.com/auth/cloud-platform', 'https://www.googleapis.com/auth/generative-language.retriever'])" | ||
| "# NOTE: The simplified project setup in this tutorial triggers a \"Google hasn't verified this app.\" dialog.\n", | ||
| "# This is normal, click \"Advanced\" -> \"Go to [app name] (unsafe)\"\n", | ||
| "!gcloud auth application-default login --no-browser --client-id-file=$client_file_name --scopes=\"https://www.googleapis.com/auth/generative-language.retriever,https://www.googleapis.com/auth/cloud-platform\"" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": { | ||
| "id": "Qqjs2tvvicqq" | ||
| "id": "lybTOlN9gz0I" | ||
| }, | ||
| "source": [ | ||
| "Initialize the client library using the service account credentials." | ||
| "Initialize the client library." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "id": "P719DMtK8t-p" | ||
| "id": "9MJbDTpMmlKJ" | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import google.ai.generativelanguage as glm\n", | ||
| "generative_service_client = glm.GenerativeServiceClient(credentials=scoped_credentials)\n", | ||
| "retriever_service_client = glm.RetrieverServiceClient(credentials=scoped_credentials)\n", | ||
| "permission_service_client = glm.PermissionServiceClient(credentials=scoped_credentials)" | ||
| "\n", | ||
| "generative_service_client = glm.GenerativeServiceClient()\n", | ||
| "retriever_service_client = glm.RetrieverServiceClient()\n", | ||
| "permission_service_client = glm.PermissionServiceClient()" | ||
| ] | ||
| }, | ||
| { | ||
|
|
@@ -796,7 +796,7 @@ | |
| "\n", | ||
| "When `answerable_probability` is low, some clients may wish to:\n", | ||
| "* Display a message to the effect of \"We couldn't answer that question\" to the end user.\n", | ||
| "* Fall back to a general-purpose LLM that answers the question from world knowledge. The threshold and nature of such fallbacks will depend on individual use cases. A value of `answerable_probablitity` <= 0.5 is a good starting threshold.\n" | ||
| "* Fall back to a general-purpose LLM that answers the question from world knowledge. The threshold and nature of such fallbacks will depend on individual use cases. A value of `answerable_probability` <= 0.5 is a good starting threshold.\n" | ||
| ] | ||
| }, | ||
| { | ||
|
|
@@ -829,7 +829,7 @@ | |
| "outputs": [], | ||
| "source": [ | ||
| "user_query = \"What is the purpose of Project IDX?\"\n", | ||
| "answer_style = \"ABSTRACTIVE\"\n", | ||
| "answer_style = \"ABSTRACTIVE\" # Or VERBOSE, EXTRACTIVE\n", | ||
| "MODEL_NAME = \"models/aqa\"\n", | ||
| "\n", | ||
| "# Make the request\n", | ||
|
|
@@ -879,7 +879,7 @@ | |
| "source": [ | ||
| "user_query = \"What is AQA from Google?\"\n", | ||
| "user_query_content = glm.Content(parts=[glm.Part(text=user_query)])\n", | ||
| "answer_style = \"ABSTRACTIVE\"\n", | ||
| "answer_style = \"VERBOSE\" # or ABSTRACTIVE, EXTRACTIVE\n", | ||
| "MODEL_NAME = \"models/aqa\"\n", | ||
| "\n", | ||
| "# Create the grounding inline passages\n", | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.