From e06296dcf50d783ee858abd4c32744c6413d036e Mon Sep 17 00:00:00 2001 From: Mark McDonald Date: Tue, 9 Apr 2024 16:49:16 +0800 Subject: [PATCH 1/3] Syncing notebooks From cl/622940573 --- site/en/tutorials/prompting_with_media.ipynb | 406 +++++++++++++++++++ 1 file changed, 406 insertions(+) create mode 100644 site/en/tutorials/prompting_with_media.ipynb diff --git a/site/en/tutorials/prompting_with_media.ipynb b/site/en/tutorials/prompting_with_media.ipynb new file mode 100644 index 000000000..95c2dc36f --- /dev/null +++ b/site/en/tutorials/prompting_with_media.ipynb @@ -0,0 +1,406 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "Tce3stUlHN0L" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "tuOe1ymfHZPu" + }, + "outputs": [], + "source": [ + "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "084u8u0DpBlo" + }, + "source": [ + "# Prompting with media files\n", + "\n", + "The Gemini API supports prompting with text, image, and audio data, also known as *multimodal* prompting. You can include text, image,\n", + "and audio in your prompts. For small images, you can point the Gemini model\n", + "directly to a local file when providing a prompt. For larger images, videos\n", + "(sequences of image frames), and audio, upload the files with the [File\n", + "API](https://ai.google.dev/api/rest/v1beta/files) before including them in\n", + "prompts.\n", + "\n", + "The File API lets you store up to 20GB of files per project, with each file not\n", + "exceeding 2GB in size. Files are stored for 48 hours and can be accessed with\n", + "your API key for generation within that time period. It is available at no cost in all regions where the [Gemini API is\n", + "available](https://ai.google.dev/available_regions).\n", + "\n", + "For information on valid file formats (MIME types) and supported models, see [Supported file formats](#supported_file_formats).\n", + "\n", + "Note: Videos must be converted into image frames before uploading to the File\n", + "API.\n", + "\n", + "This guide shows how to use the File API to upload a media file and include it in a `GenerateContent` call to the Gemini API. For more information, see the [code\n", + "samples](https://github.com/google-gemini/gemini-api-cookbook/tree/main/quickstarts/file-api).\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VxCstRHvpX0r" + }, + "source": [ + "## Setup\n", + "\n", + "Before you use the File API, you need to install the Gemini API SDK package and configure an API key. This section describes how to complete these setup steps." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "G6J_rV2ipmj_" + }, + "source": [ + "### Install the Python SDK and import packages\n", + "\n", + "The Python SDK for the Gemini API is contained in the [google-generativeai](https://pypi.org/project/google-generativeai/) package. Install the dependency using pip." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "mN8x8DPgu9Kv" + }, + "outputs": [], + "source": [ + "!pip install -q -U google-generativeai" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NInUZ4hwDq6d" + }, + "source": [ + "Import the necessary packages." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0x3xmmWrDtEH" + }, + "outputs": [], + "source": [ + "import google.generativeai as genai\n", + "from IPython.display import Markdown" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "l8g4hTRotheH" + }, + "source": [ + "### Setup your API key\n", + "\n", + "The File API uses API keys for authentication and access. Uploaded files are associated with the project linked to the API key. Unlike other Gemini APIs that use API keys, your API key also grants access to data you've uploaded to the File API, so take extra care in keeping your API key secure. For more on keeping your keys\n", + "secure, see [Best practices for using API\n", + "keys](https://support.google.com/googleapi/answer/6310037).\n", + "\n", + "Store your API key in a Colab Secret named `GOOGLE_API_KEY`. If you don't already have an API key, or are unfamiliar with Colab Secrets, refer to the [Authentication quickstart](https://github.com/google-gemini/gemini-api-cookbook/blob/main/quickstarts/Authentication.ipynb)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "d6lYXRcjthKV" + }, + "outputs": [], + "source": [ + "from google.colab import userdata\n", + "GOOGLE_API_KEY=userdata.get('GOOGLE_API_KEY')\n", + "\n", + "genai.configure(api_key=GOOGLE_API_KEY)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c-z4zsCUlaru" + }, + "source": [ + "## Upload a file to the File API\n", + "\n", + "The File API lets you upload a variety of multimodal MIME types, including images and audio formats. The File API handles inputs that can be used to generate content with [`model.generateContent`](https://ai.google.dev/api/rest/v1/models/generateContent) or [`model.streamGenerateContent`](https://ai.google.dev/api/rest/v1/models/streamGenerateContent).\n", + "\n", + "The File API accepts files under 2GB in size and can store up to 20GB of files per project. Files last for 2 days and cannot be downloaded from the API." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "o1K81yn9mFBo" + }, + "source": [ + "First, we will prepare a sample image to upload to the API.\n", + "\n", + "To upload your own file, see the [Appendix section](#uploading_files_to_colab)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "lC6sS6DnmGmi" + }, + "outputs": [], + "source": [ + "!curl -o image.jpg https://storage.googleapis.com/generativeai-downloads/images/jetpack.jpg" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rI84z01ZmSyF" + }, + "source": [ + "Next, we'll upload that file to the File API." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "N9NxXGZKKusG" + }, + "outputs": [], + "source": [ + "sample_file = genai.upload_file(path=\"image.jpg\",\n", + " display_name=\"Sample drawing\")\n", + "\n", + "print(f\"Uploaded file '{sample_file.display_name}' as: {sample_file.uri}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cto22vhKOvGQ" + }, + "source": [ + "The `response` shows that the File API stored the specified `display_name` for the uploaded file and a `uri` to reference the file in Gemini API calls. Use `response` to track how uploaded files are mapped to URIs.\n", + "\n", + "Depending on your use cases, you could store the URIs in structures such as a `dict` or a database." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ds5iJlaembWe" + }, + "source": [ + "## Get file\n", + "\n", + "After uploading the file, you can verify the API has successfully received the files by calling `files.get`.\n", + "\n", + "It lets you get the file metadata that have been uploaded to the File API that are associated with the Cloud project your API key belongs to. Only the `name` (and by extension, the `uri`) are unique. Only use the `displayName` to identify files if you manage uniqueness yourself." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "kLFsVLFHOWSV" + }, + "outputs": [], + "source": [ + "file = genai.get_file(name=sample_file.name)\n", + "print(f\"Retrieved file '{file.display_name}' as: {sample_file.uri}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EPPOECHzsIGJ" + }, + "source": [ + "## Generate content\n", + "\n", + "After uploading the file, you can make `GenerateContent` requests that reference the File API URI. In this example, we create prompt that starts with a text followed by the uploaded image." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ZYVFqmLkl5nE" + }, + "outputs": [], + "source": [ + "# Set the model to Gemini 1.5 Pro.\n", + "model = genai.GenerativeModel(model_name=\"models/gemini-1.5-pro-latest\")\n", + "\n", + "response = model.generate_content([\"Describe the image with a creative description.\", sample_file])\n", + "\n", + "Markdown(\">\" + response.text)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IrPDYdQSKTg4" + }, + "source": [ + "## Delete files\n", + "\n", + "Files are automatically deleted after 2 days. You can also manually delete them using `files.delete()`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "d4eO8ZXoKdZf" + }, + "outputs": [], + "source": [ + "genai.delete_file(sample_file.name)\n", + "print(f'Deleted {sample_file.display_name}.')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OxgouC6i7exO" + }, + "source": [ + "## Supported file formats\n", + "\n", + "Gemini models support prompting with multiple file formats. This section explains considerations in using general media formats for prompting, specifically image, audio, and video files. You can use media files for prompting only with specific model versions, as shown in the following table.\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
ModelImagesAudioVideo
Gemini 1.5 Pro (release 008 and later)✔ (3600 max image files)✔✔
Gemini Pro Vision✔ (16 max image files)
\n", + "\n", + "### Image formats\n", + "\n", + "You can use image data for prompting with the `gemini-pro-vision` and `gemini-1.5-pro ` models. When you use images for prompting, they are subject to the following limitations and requirements:\n", + "\n", + "- Images must be in one of the following image data [MIME types](https://developers.google.com/drive/api/guides/ref-export-formats):\n", + " - PNG - image/png\n", + " - JPEG - image/jpeg\n", + " - WEBP - image/webp\n", + " - HEIC - image/heic\n", + " - HEIF - image/heif\n", + "- Maximum of 16 individual images for the `gemini-pro-vision` and 3600 images for `gemini-1.5-pro`\n", + "- No specific limits to the number of pixels in an image; however, larger images are scaled down to fit a maximum resolution of 3072 x 3072 while preserving their original aspect ratio.\n", + "\n", + "### Audio formats\n", + "\n", + "You can use audio data for prompting with the `gemini-1.5-pro` model. When you use audio for prompting, they are subject to the following limitations and requirements:\n", + "\n", + "- Audio data is supported in the following common audio format [MIME types](https://developers.google.com/drive/api/guides/ref-export-formats):\n", + " - WAV - audio/wav\n", + " - MP3 - audio/mp3\n", + " - AIFF - audio/aiff\n", + " - AAC - audio/aac\n", + " - OGG Vorbis - audio/ogg\n", + " - FLAC - audio/flac\n", + "- The maximum supported length of audio data in a single prompt is 9.5 hours.\n", + "- Audio files are resampled down to a 16 Kbps data resolution, and multiple channels of audio are combined into a single channel.\n", + "- There is no specific limit to the number of audio files in a single prompt, however the total combined length of all audio files in a single prompt cannot exceed 9.5 hours.\n", + "\n", + "### Video formats\n", + "\n", + "You can use video data for prompting with the `gemini-1.5-pro` model. However, video file formats are not supported as direct inputs by the Gemini API. You can use video data as prompt input by breaking down the video into a series of still frame images and a separate audio file. This approach lets you manage the amount of data, and the level of detail provided by the video, by choosing how many frames per second are included in your prompt from the video file.\n", + "\n", + "Note: Video files added to a prompt as constituent parts, audio file and image frames, are considered as separate prompt data inputs by the model. For this reason, requests or questions that specify the time when *both* an audio snippet and video frames appear in the source video may not produce useful results." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "uploading_files_to_colab" + }, + "source": [ + "## Appendix: Uploading files to Colab\n", + "\n", + "This notebook uses the File API with files that were downloaded from the internet. If you're running this in Colab and want to use your own files, you first need to upload them to the colab instance.\n", + "\n", + "First, click **Files** on the left sidebar, then click the **Upload** button:\n", + "\n", + "\n", + "\n", + "Next, we'll upload that file to the File API. In the form for the code cell below, enter the filename for the file you uploaded and provide an appropriate display name for the file, then run the cell.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "2QGBpboMmxHG" + }, + "outputs": [], + "source": [ + "my_filename = \"gemini_logo.png\" # @param {type:\"string\"}\n", + "my_file_display_name = \"Gemini Logo\" # @param {type:\"string\"}\n", + "\n", + "my_file = genai.upload_file(path=my_filename,\n", + " display_name=my_file_display_name)\n", + "print(f\"Uploaded file '{my_file.display_name}' as: {my_file.uri}\")" + ] + } + ], + "metadata": { + "colab": { + "name": "prompting_with_media.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} From 977ac086f18e9f503122fd03e627ee208e86fa15 Mon Sep 17 00:00:00 2001 From: Mark McDonald Date: Tue, 9 Apr 2024 17:14:18 +0800 Subject: [PATCH 2/3] Add buttons --- site/en/tutorials/prompting_with_media.ipynb | 29 ++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/site/en/tutorials/prompting_with_media.ipynb b/site/en/tutorials/prompting_with_media.ipynb index 95c2dc36f..3d40e42c3 100644 --- a/site/en/tutorials/prompting_with_media.ipynb +++ b/site/en/tutorials/prompting_with_media.ipynb @@ -37,8 +37,33 @@ "id": "084u8u0DpBlo" }, "source": [ - "# Prompting with media files\n", - "\n", + "# Prompting with media files" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZFWzQEqNosrS" + }, + "source": [ + "\n", + " \n", + " \n", + "
\n", + " View on ai.google.dev\n", + " \n", + " Run in Google Colab\n", + " \n", + " View source on GitHub\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3c5e92a74e64" + }, + "source": [ "The Gemini API supports prompting with text, image, and audio data, also known as *multimodal* prompting. You can include text, image,\n", "and audio in your prompts. For small images, you can point the Gemini model\n", "directly to a local file when providing a prompt. For larger images, videos\n", From 9065f6cb370887f871b7da70c9df7ae0f0a6a31f Mon Sep 17 00:00:00 2001 From: Mark McDonald Date: Tue, 9 Apr 2024 17:16:29 +0800 Subject: [PATCH 3/3] nblint fixes --- site/en/tutorials/prompting_with_media.ipynb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/site/en/tutorials/prompting_with_media.ipynb b/site/en/tutorials/prompting_with_media.ipynb index 3d40e42c3..e677706b4 100644 --- a/site/en/tutorials/prompting_with_media.ipynb +++ b/site/en/tutorials/prompting_with_media.ipynb @@ -187,7 +187,7 @@ "id": "o1K81yn9mFBo" }, "source": [ - "First, we will prepare a sample image to upload to the API.\n", + "First, you will prepare a sample image to upload to the API.\n", "\n", "To upload your own file, see the [Appendix section](#uploading_files_to_colab)." ] @@ -209,7 +209,7 @@ "id": "rI84z01ZmSyF" }, "source": [ - "Next, we'll upload that file to the File API." + "Next, you'll upload that file to the File API." ] }, { @@ -270,7 +270,7 @@ "source": [ "## Generate content\n", "\n", - "After uploading the file, you can make `GenerateContent` requests that reference the File API URI. In this example, we create prompt that starts with a text followed by the uploaded image." + "After uploading the file, you can make `GenerateContent` requests that reference the File API URI. In this example, you create prompt that starts with a text followed by the uploaded image." ] }, { @@ -396,7 +396,7 @@ "\n", "\n", "\n", - "Next, we'll upload that file to the File API. In the form for the code cell below, enter the filename for the file you uploaded and provide an appropriate display name for the file, then run the cell.\n" + "Next, you'll upload that file to the File API. In the form for the code cell below, enter the filename for the file you uploaded and provide an appropriate display name for the file, then run the cell.\n" ] }, {