Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add client for Azure-hosted Llama models #871

Closed
drdavella opened this issue Oct 8, 2024 · 0 comments · Fixed by #872
Closed

Add client for Azure-hosted Llama models #871

drdavella opened this issue Oct 8, 2024 · 0 comments · Fixed by #872
Assignees

Comments

@drdavella
Copy link
Member

We would like to support codemods that use Llama models hosted in Azure.

We have proposed a spec update here that introduces two new environment variables: pixee/codemodder-specs#39

CODEMODDER_AZURE_LLAMA_API_KEY=<KEY>
CODEMODDER_AZURE_LLAMA_ENDPOINT=<ENDPOINT>

If only one or the other variable is present, we should raise an exception.

If both of these variables are present, we need to create a new llama client using the Azure AI Inference package.

Use of the client looks something like this:

# pip install azure-ai-inference
import os
from azure.ai.inference import ChatCompletionsClient
from azure.core.credentials import AzureKeyCredential

api_key = os.getenv("<API key>", '')
if not api_key:
  raise Exception("A key should be provided to invoke the endpoint")

client = ChatCompletionsClient(
    endpoint='<endpoint URL>',
    credential=AzureKeyCredential(api_key)
)

The behavior of this client is completely orthogonal to the OpenAI/Azure OpenAI client. One or the other or both can be configured at the same time. The client should live on the CodemodExecutionContext. Implementing this may lead to some breaking API changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants