From d9f0bb6edf9dda5683efe8094616a9659fa4a09a Mon Sep 17 00:00:00 2001 From: Philip Gao Date: Thu, 16 Nov 2023 19:27:07 +0800 Subject: [PATCH] Fix replaying test documentation (#1153) # Description Fix replaying test documentation # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --- docs/dev/dev_setup.md | 4 ++ docs/dev/replay-e2e-test-for-pfazure.md | 25 --------- docs/dev/replay-e2e-test.md | 51 +++++++++++++++++++ src/promptflow/.env.example | 5 ++ src/promptflow/tests/sdk_cli_test/conftest.py | 2 + 5 files changed, 62 insertions(+), 25 deletions(-) delete mode 100644 docs/dev/replay-e2e-test-for-pfazure.md create mode 100644 docs/dev/replay-e2e-test.md create mode 100644 src/promptflow/.env.example diff --git a/docs/dev/dev_setup.md b/docs/dev/dev_setup.md index b92cc0a077c..183fee3a8b0 100644 --- a/docs/dev/dev_setup.md +++ b/docs/dev/dev_setup.md @@ -72,6 +72,10 @@ After above setup process is finished. You can use `pytest` command to run test, ![img2](../media/dev_setup/set_up_pycharm_2.png) +### Record and replay tests + +Please refer to [Replay End-to-End Tests](./replay-e2e-test.md) to learn how to record and replay tests. + ## How to write docstring. A clear and consistent API documentation is crucial for the usability and maintainability of our codebase. Please refer to [API Documentation Guidelines](./documentation_guidelines.md) to learn how to write docstring when developing the project. diff --git a/docs/dev/replay-e2e-test-for-pfazure.md b/docs/dev/replay-e2e-test-for-pfazure.md deleted file mode 100644 index c2c63c74b2f..00000000000 --- a/docs/dev/replay-e2e-test-for-pfazure.md +++ /dev/null @@ -1,25 +0,0 @@ -# Replay end-to-end test for pfazure - -This document introduces replay test for those tests located in [end-to-end tests for pfazure](../../src/promptflow/tests/sdk_cli_azure_test/e2etests/). - -## Why we need replay test - -End-to-end tests for pfazure targets to test the behavior that prompt flow SDK/CLI interacts with service, this process can be time consuming, error prone, and require credentials (which is unavailable to pull request from forked repository); all of these go against our intention for a smooth develop experience. - -Therefore, we introduce replay test, which leverage [VCR.py](https://pypi.org/project/vcrpy/) to record all required network traffic to local files and replay during tests. In this way, we avoid the need of credentials, speed up and stabilize the test process. - -## 3 test modes - -- `live`: tests run against real backend, which is the way traditional end-to-end test do. -- `record`: tests run against real backend, and network traffic will be sanitized (filter sensitive and unnecessary requests/responses) and recorded to local files (recordings). -- `replay`: there is no real network traffic between SDK/CLI and backend, tests run against local recordings. - -## How to record a test - -We are planning to build a workflow to automatically update recordings on behalf of you, but it's still work in progress. Before that, you need to manually record it(them) locally. - -To record a test, all you need is specifying environment variable `PROMPT_FLOW_TEST_MODE` to `'record'`; if you have a `.env` file, we recommend to specify it there. Then, just run the test that you want to record. Once the test completed, there should be one new YAML file located in `src/promptflow/tests/test_configs/recordings/`, containing the network traffic of the test - Congratulations! You have just recorded a test! - -## How to run test(s) in replay mode - -Specify environment variable `PROMPT_FLOW_TEST_MODE` to `'replay'` and run the test(s). diff --git a/docs/dev/replay-e2e-test.md b/docs/dev/replay-e2e-test.md new file mode 100644 index 00000000000..9f719f2ad2f --- /dev/null +++ b/docs/dev/replay-e2e-test.md @@ -0,0 +1,51 @@ +# Replay end-to-end tests + +* This document introduces replay tests for those located in [sdk_cli_azure_test](../../src/promptflow/tests/sdk_cli_azure_test/e2etests/) and [sdk_cli_test](../../src/promptflow/tests/sdk_cli_test/e2etests/). +* The primary purpose of replay tests is to avoid the need for credentials, Azure workspaces, OpenAI tokens, and to directly test prompt flow behavior. +* Although there are different techniques behind recording/replaying, there are some common steps to run the tests in replay mode. +* The key handle of replay tests is the environment variable `PROMPT_FLOW_TEST_MODE`. + +## How to run tests in replay mode + +After cloning the full repo and setting up the proper test environment following [dev_setup.md](./dev_setup.md), run the following command in the root directory of the repo: + +1. If you have changed/affected tests in __sdk_cli_test__ : Copy or rename the file [dev-connections.json.example](../../src/promptflow/dev-connections.json.example) to `connections.json` in the same folder. +2. In your Python environment, set the environment variable `PROMPT_FLOW_TEST_MODE` to `'replay'` and run the test(s). + +These tests should work properly without any real connection settings. + +## Test modes + +There are 3 representative values of the environment variable `PROMPT_FLOW_TEST_MODE` +- `live`: Tests run against the real backend, which is the way traditional end-to-end tests do. +- `record`: Tests run against the real backend, and network traffic will be sanitized (filter sensitive and unnecessary requests/responses) and recorded to local files (recordings). +- `replay`: There is no real network traffic between SDK/CLI and the backend, tests run against local recordings. + +## Update test recordings + +To record a test, don’t forget to clone the full repo and set up the proper test environment following [dev_setup.md](./dev_setup.md): +1. Prepare some data. + * If you have changed/affected tests in __sdk_cli_test__: Copy or rename the file [dev-connections.json.example](../../src/promptflow/dev-connections.json.example) to `connections.json` in the same folder. + * If you have changed/affected tests in __sdk_cli_azure_test__: prepare your Azure ML workspace, make sure your Azure CLI logged in, and set the environment variable `PROMPT_FLOW_SUBSCRIPTION_ID`, `PROMPT_FLOW_RESOURCE_GROUP_NAME`, `PROMPT_FLOW_WORKSPACE_NAME` and `PROMPT_FLOW_RUNTIME_NAME` (if needed) pointing to your workspace. +2. Record the test. + * Specify the environment variable `PROMPT_FLOW_TEST_MODE` to `'record'`. If you have a `.env` file, we recommend specifying it there. Here is an example [.env file](../../src/promptflow/.env.example). Then, just run the test that you want to record. +3. Once the test completed. + * If you have changed/affected tests in __sdk_cli_azure_test__: There should be one new YAML file located in `src/promptflow/tests/test_configs/recordings/`, containing the network traffic of the test. + * If you have changed/affected tests in __sdk_cli_test__: There may be changes in the folder `src/promptflow/tests/test_configs/node_recordings/`. Don’t worry if there are no changes, because similar LLM calls may have been recorded before. + +## Techniques behind replay test + +### Sdk_cli_azure_test + +End-to-end tests for pfazure aim to test the behavior of the PromptFlow SDK/CLI as it interacts with the service. This process can be time-consuming, error-prone, and require credentials (which are unavailable to pull requests from forked repositories); all of these go against our intention for a smooth development experience. + +Therefore, we introduce replay tests, which leverage [VCR.py](https://pypi.org/project/vcrpy/) to record all required network traffic to local files and replay during tests. In this way, we avoid the need for credentials, speed up, and stabilize the test process. + +### Sdk_cli_test + +sdk_cli_test often doesn’t use a real backend. It will directly invokes LLM calls from localhost. Thus the key target of replay tests is to avoid the need for OpenAI tokens. If you have OpenAI / Azure OpenAI tokens yourself, you can try recording the tests. Record Storage will not record your own LLM connection, but only the inputs and outputs of the LLM calls. + +There are also limitations. Currently, recorded calls are: +* AzureOpenAI calls +* OpenAI calls +* tool name "fetch_text_content_from_url" and tool name "my_python_tool" \ No newline at end of file diff --git a/src/promptflow/.env.example b/src/promptflow/.env.example new file mode 100644 index 00000000000..d729df5974e --- /dev/null +++ b/src/promptflow/.env.example @@ -0,0 +1,5 @@ +DEFAULT_SUBSCRIPTION_ID="your-subscription-id" +DEFAULT_RESOURCE_GROUP_NAME="your-resource-group-name" +DEFAULT_WORKSPACE_NAME="your-workspace-name" +DEFAULT_RUNTIME_NAME="test-runtime-ci" +PROMPT_FLOW_TEST_MODE="replay" diff --git a/src/promptflow/tests/sdk_cli_test/conftest.py b/src/promptflow/tests/sdk_cli_test/conftest.py index 198a0736f2b..c183a57e5c8 100644 --- a/src/promptflow/tests/sdk_cli_test/conftest.py +++ b/src/promptflow/tests/sdk_cli_test/conftest.py @@ -159,7 +159,9 @@ def mock_invoke_tool(self, node, func, kwargs): if ( node.provider == "AzureOpenAI" + or node.provider == "OpenAI" or func_wo_partial.__qualname__.startswith("AzureOpenAI") + or func_wo_partial.__qualname__.startswith("OpenAI") or func_wo_partial.__qualname__ == "fetch_text_content_from_url" or func_wo_partial.__qualname__ == "my_python_tool" ):