[draft] add flow sample with new llm

docs/reference/index.md

@@ -46,6 +46,7 @@ python-library-reference/promptflow-azure/promptflow
 :maxdepth: 1
 
 tools-reference/llm-tool
+tools-reference/llm-vision-tool
 tools-reference/prompt-tool
 tools-reference/python-tool
 tools-reference/serp-api-tool

docs/reference/tools-reference/llm-tool.md

@@ -2,20 +2,21 @@
 
 ## Introduction
 Prompt flow LLM tool enables you to leverage widely used large language models like [OpenAI](https://platform.openai.com/) or [Azure OpenAI (AOAI)](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/overview) for natural language processing. 
+> [!NOTE]
+> The previous version of the LLM tool is now being deprecated. Please upgrade to latest [promptflow-tools](https://pypi.org/project/promptflow-tools/) package to consume new llm tools.
 
 Prompt flow provides a few different LLM APIs:
 - **[Completion](https://platform.openai.com/docs/api-reference/completions)**: OpenAI's completion models generate text based on provided prompts.
 - **[Chat](https://platform.openai.com/docs/api-reference/chat)**: OpenAI's chat models facilitate interactive conversations with text-based inputs and responses.
-> [!NOTE]
-> We now remove the `embedding` option from LLM tool api setting. You can use embedding api with [Embedding tool](https://github.com/microsoft/promptflow/blob/main/docs/reference/tools-reference/embedding_tool.md).
 
 
 ## Prerequisite
-Create OpenAI resources:
+Create OpenAI or Azure OpenAI resources:
 
 - **OpenAI**
 
     Sign up account [OpenAI website](https://openai.com/)
+
     Login and [Find personal API key](https://platform.openai.com/account/api-keys)
 
 - **Azure OpenAI (AOAI)**
@@ -66,16 +67,15 @@ Setup connections to provisioned resources in prompt flow.
 | presence\_penalty      | float       | value that controls the model's behavior with regards to repeating phrases. Default is 0.      | No       |
 | frequency\_penalty     | float       | value that controls the model's behavior with regards to generating rare phrases. Default is 0.| No       |
 | logit\_bias            | dictionary  | the logit bias for the language model. Default is empty dictionary.                            | No       |
-| function\_call         | object      | value that controls which function is called by the model. Default is null.                    | No       |
-| functions              | list        | a list of functions the model may generate JSON inputs for. Default is null.                   | No       |
+| tool\_choice           | object      | value that controls which tool is called by the model. Default is null.                        | No       |
+| tools                  | list        | a list of tools the model may generate JSON inputs for. Default is null.                       | No       |
 | response_format        | object      | an object specifying the format that the model must output. Default is null.                   | No       |
 
 ## Outputs
 
-| API        | Return Type | Description                              |
-|------------|-------------|------------------------------------------|
-| Completion | string      | The text of one predicted completion     |
-| Chat       | string      | The text of one response of conversation |
+| Return Type | Description                                                          |
+|-------------|----------------------------------------------------------------------|
+| string      | The text of one predicted completion or response of conversation     |
 
 
 ## How to use LLM Tool?

docs/reference/tools-reference/llm-vision-tool.md

@@ -0,0 +1,51 @@
+# LLM Vision
+
+## Introduction
+Prompt flow LLM vision tool enables you to leverage your AzureOpenAI GPT-4 Turbo or OpenAI's GPT-4 with vision to analyze images and provide textual responses to questions about them.
+
+## Prerequisites
+Create OpenAI or Azure OpenAI resources:
+
+- **OpenAI**
+
+    Sign up account [OpenAI website](https://openai.com/)
+
+    Login and [Find personal API key](https://platform.openai.com/account/api-keys)
+
+- **Azure OpenAI (AOAI)**
+
+    Create Azure OpenAI resources with [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal)
+
+    Browse to [Azure OpenAI Studio](https://oai.azure.com/) and sign in with the credentials associated with your Azure OpenAI resource. During or after the sign-in workflow, select the appropriate directory, Azure subscription, and Azure OpenAI resource.
+
+    Under Management select Deployments and Create a GPT-4 Turbo with Vision deployment by selecting model name: `gpt-4` and model version `vision-preview`.
+
+
+
+## **Connections**
+
+Setup connections to provisioned resources in prompt flow.
+
+| Type        | Name     | API KEY  | API Type | API Version |
+|-------------|----------|----------|----------|-------------|
+| OpenAI      | Required | Required | -        | -           |
+| AzureOpenAI | Required | Required | Required | Required    |
+
+## Inputs
+
+| Name                    | Type        | Description                                                                                    | Required |
+|-------------------------|-------------|------------------------------------------------------------------------------------------------|----------|
+| model, deployment\_name | string      | the language model to use                                                                      | Yes      |
+| prompt                  | string      | The text prompt that the language model will use to generate it's response.                    | Yes      |
+| max\_tokens             | integer     | the maximum number of tokens to generate in the response. Default is 512.                      | No       |
+| temperature             | float       | the randomness of the generated text. Default is 1.                                            | No       |
+| stop                    | list        | the stopping sequence for the generated text. Default is null.                                 | No       |
+| top_p                   | float       | the probability of using the top choice from the generated tokens. Default is 1.               | No       |
+| presence\_penalty       | float       | value that controls the model's behavior with regards to repeating phrases. Default is 0.      | No       |
+| frequency\_penalty      | float       | value that controls the model's behavior with regards to generating rare phrases. Default is 0.| No       |
+
+## Outputs
+
+| Return Type | Description                              |
+|-------------|------------------------------------------|
+| string      | The text of one response of conversation |

examples/flows/standard/web-classification-with-new-llm/README.md

@@ -0,0 +1,129 @@
+# Web Classification
+
+This is a flow demonstrating multi-class classification with LLM. Given an url, it will classify the url into one web category with just a few shots, simple summarization and classification prompts.
+
+## Tools used in this flow
+- LLM Tool
+- Python Tool
+
+## What you will learn
+
+In this flow, you will learn
+- how to compose a classification flow with LLM.
+- how to feed few shots to LLM classifier.
+
+## Prerequisites
+
+Install promptflow sdk and other dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+## Getting Started
+
+### 1. Setup connection
+
+If you are using Azure Open AI, prepare your resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
+
+```bash
+# Override keys with --set to avoid yaml file changes
+pf connection create --file ../../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
+```
+
+If you using OpenAI, sign up account [OpenAI website](https://openai.com/), login and [find personal API key](https://platform.openai.com/account/api-keys).
+
+```shell
+pf connection create --file ../../../connections/openai.yml --set api_key=<your_api_key>
+```
+
+### 2. Configure the flow with your connection
+`flow.dag.yaml` is already configured with connection named `open_ai_connection`.
+
+### 3. Test flow with single line data
+
+```bash
+# test with default input value in flow.dag.yaml
+pf flow test --flow .
+# test with user specified inputs
+pf flow test --flow . --inputs url='https://www.youtube.com/watch?v=kYqRtjDBci8'
+```
+
+### 4. Run with multi-line data
+
+```bash
+# create run using command line args
+pf run create --flow . --data ./data.jsonl --column-mapping url='${data.url}' --stream
+
+# (Optional) create a random run name
+run_name="web_classification_"$(openssl rand -hex 12)
+# create run using yaml file, run_name will be used in following contents, --name is optional
+pf run create --file run.yml --stream --name $run_name
+```
+
+You can also skip providing `column-mapping` if provided data has same column name as the flow.
+Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
+
+```bash
+# list run
+pf run list
+# show run
+pf run show --name $run_name
+# show run outputs
+pf run show-details --name $run_name
+```
+
+### 5. Run with classification evaluation flow
+
+create `evaluation` run:
+```bash
+# (Optional) save previous run name into variable, and create a new random run name for further use
+prev_run_name=$run_name
+run_name="classification_accuracy_"$(openssl rand -hex 12)
+# create run using command line args
+pf run create --flow ../../evaluation/eval-classification-accuracy --data ./data.jsonl --column-mapping groundtruth='${data.answer}' prediction='${run.outputs.category}' --run $prev_run_name --stream
+# create run using yaml file, --name is optional
+pf run create --file run_evaluation.yml --run $prev_run_name --stream --name $run_name
+```
+
+```bash
+pf run show-details --name $run_name
+pf run show-metrics --name $run_name
+pf run visualize --name $run_name
+```
+
+
+### 6. Submit run to cloud
+```bash
+# set default workspace
+az account set -s <your_subscription_id>
+az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
+
+# create run
+pfazure run create --flow . --data ./data.jsonl --column-mapping url='${data.url}' --stream
+
+# (Optional) create a new random run name for further use
+run_name="web_classification_"$(openssl rand -hex 12)
+
+# create run using yaml file, --name is optional
+pfazure run create --file run.yml --name $run_name
+
+
+pfazure run stream --name $run_name
+pfazure run show-details --name $run_name
+pfazure run show-metrics --name $run_name
+
+
+# (Optional) save previous run name into variable, and create a new random run name for further use
+prev_run_name=$run_name
+run_name="classification_accuracy_"$(openssl rand -hex 12)
+
+# create evaluation run, --name is optional
+pfazure run create --flow ../../evaluation/eval-classification-accuracy --data ./data.jsonl --column-mapping groundtruth='${data.answer}' prediction='${run.outputs.category}' --run $prev_run_name
+pfazure run create --file run_evaluation.yml --run $prev_run_name --stream --name $run_name
+
+pfazure run stream --name $run_name
+pfazure run show --name $run_name
+pfazure run show-details --name $run_name
+pfazure run show-metrics --name $run_name
+pfazure run visualize --name $run_name
+```

examples/flows/standard/web-classification-with-new-llm/classify_with_llm.jinja2

@@ -0,0 +1,21 @@
+# system:
+Your task is to classify a given url into one of the following categories:
+Movie, App, Academic, Channel, Profile, PDF or None based on the text content information.
+The classification will be based on the url, the webpage text content summary, or both.
+
+# user:
+The selection range of the value of "category" must be within "Movie", "App", "Academic", "Channel", "Profile", "PDF" and "None".
+The selection range of the value of "evidence" must be within "Url", "Text content", and "Both".
+Here are a few examples:
+{% for ex in examples %}
+URL: {{ex.url}}
+Text content: {{ex.text_content}}
+OUTPUT:
+{"category": "{{ex.category}}", "evidence": "{{ex.evidence}}"}
+
+{% endfor %}
+
+For a given URL and text content, classify the url to complete the category and indicate evidence:
+URL: {{url}}
+Text content: {{text_content}}.
+OUTPUT:

examples/flows/standard/web-classification-with-new-llm/convert_to_dict.py

@@ -0,0 +1,12 @@
+import json
+
+from promptflow.core import tool
+
+
+@tool
+def convert_to_dict(input_str: str):
+    try:
+        return json.loads(input_str)
+    except Exception as e:
+        print("The input is not valid, error: {}".format(e))
+        return {"category": "None", "evidence": "None"}

examples/flows/standard/web-classification-with-new-llm/data.jsonl

@@ -0,0 +1,3 @@
+{"url": "https://www.youtube.com/watch?v=kYqRtjDBci8", "answer": "Channel", "evidence": "Both"}
+{"url": "https://arxiv.org/abs/2307.04767", "answer": "Academic", "evidence": "Both"}
+{"url": "https://play.google.com/store/apps/details?id=com.twitter.android", "answer": "App", "evidence": "Both"}

examples/flows/standard/web-classification-with-new-llm/fetch_text_content_from_url.py

@@ -0,0 +1,30 @@
+import bs4
+import requests
+
+from promptflow.core import tool
+
+
+@tool
+def fetch_text_content_from_url(url: str):
+    # Send a request to the URL
+    try:
+        headers = {
+            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
+            "Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.35"
+        }
+        response = requests.get(url, headers=headers)
+        if response.status_code == 200:
+            # Parse the HTML content using BeautifulSoup
+            soup = bs4.BeautifulSoup(response.text, "html.parser")
+            soup.prettify()
+            return soup.get_text()[:2000]
+        else:
+            msg = (
+                f"Get url failed with status code {response.status_code}.\nURL: {url}\nResponse: "
+                f"{response.text[:100]}"
+            )
+            print(msg)
+            return "No available content"
+    except Exception as e:
+        print("Get url failed with error: {}".format(e))
+        return "No available content"

examples/flows/standard/web-classification-with-new-llm/flow.dag.yaml

@@ -0,0 +1,88 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
+environment:
+  python_requirements_txt: requirements.txt
+inputs:
+  url:
+    type: string
+    default: https://play.google.com/store/apps/details?id=com.twitter.android
+outputs:
+  category:
+    type: string
+    reference: ${convert_to_dict.output.category}
+  evidence:
+    type: string
+    reference: ${convert_to_dict.output.evidence}
+nodes:
+- name: fetch_text_content_from_url
+  type: python
+  source:
+    type: code
+    path: fetch_text_content_from_url.py
+  inputs:
+    url: ${inputs.url}
+- name: summarize_text_content
+  use_variants: true
+- name: prepare_examples
+  type: python
+  source:
+    type: code
+    path: prepare_examples.py
+  inputs: {}
+- name: classify_with_llm
+  type: llm
+  source:
+    type: code
+    path: classify_with_llm.jinja2
+  inputs:
+    deployment_name: gpt-35-turbo
+    model: gpt-3.5-turbo
+    max_tokens: 128
+    temperature: 0.2
+    url: ${inputs.url}
+    text_content: ${summarize_text_content.output}
+    examples: ${prepare_examples.output}
+  connection: open_ai_connection
+  api: chat
+- name: convert_to_dict
+  type: python
+  source:
+    type: code
+    path: convert_to_dict.py
+  inputs:
+    input_str: ${classify_with_llm.output}
+node_variants:
+  summarize_text_content:
+    default_variant_id: variant_0
+    variants:
+      variant_0:
+        node:
+          name: summarize_text_content
+          type: custom_llm
+          source:
+            type: package_with_prompt
+            path: summarize_text_content.jinja2
+            tool: promptflow.tools.llm.llm
+          inputs:
+            connection: open_ai_connection
+            api: chat
+            deployment_name: gpt-35-turbo
+            model: gpt-3.5-turbo
+            max_tokens: 128
+            temperature: 0.2
+            text: ${fetch_text_content_from_url.output}
+      variant_1:
+        node:
+          name: summarize_text_content
+          type: custom_llm
+          source:
+            type: package_with_prompt
+            path: summarize_text_content__variant_1.jinja2
+            tool: promptflow.tools.llm.llm
+          inputs:
+            connection: open_ai_connection
+            api: chat
+            deployment_name: gpt-35-turbo
+            model: gpt-3.5-turbo
+            max_tokens: 256
+            temperature: 0.3
+            text: ${fetch_text_content_from_url.output}