Agent / Tool usage using Anthropic and Gemini models on custom endpoints

[jameslamine]

We use AWS Bedrock and Vertex via a custom endpoint through PortKey. We're not able to get tool usage / agent features like MCP working via these custom endpoints.

For example, our Claude models hit PortKey which then hits AWS Bedrock.

Any idea on what might be going wrong?

When trying to do a function call we get this error:

2025-02-26 18:06:25 error: [api/server/controllers/agents/client.js #sendCompletion] Unhandled error type 400 bedrock error: The content field in the Message object at messages.1 is empty. Add a ContentBlock object to the content field and try again.
2025-02-26 18:06:25 error: [handleAbortError] AI response error; aborting request: 400 bedrock error: The content field in the Message object at messages.1 is empty. Add a ContentBlock object to the content field and try again.

[jameslamine]

I think the issue might be that custom endpoints use the OpenAI agent and some models expect different format / parameters for tool calling.

When I manually craft requests with CURL I'm able to get tool calling to work.

[danny-avila]

Without seeing the actual usage, it's hard to tell. Bedrock/VertexAI work well without needing a proxy:

[danny-avila]

Using MCP, VertexAI:

Bedrock:

[jameslamine]

Thanks @danny-avila, let me explain our setup in more detail.

In our company's enterprise setup, we are not allowed to hit Bedrock, Vertex or OpenAI directly. We must go though our company's PortKey proxy - this adds features such as per-application API keys, cost tracking, rate-limiting and observability which are important in an enterprise setting. LibreChat is just one of many applications at our company which makes calls to PortKey.

In my LibreChat config, I have it set up as a custom endpoint like this, and then have a premade modelSpec for each model.

That's what our prod config looks like. Right now we have agents disabled, but locally I have enabled agents and am experimenting with them. But it looks like when using a custom endpoint, tool calling won't work for Anthropic models though bedrock through a custom endpoint.

Is there a way to set a custom endpoint but use the bedrock client?

version: 1.2.0

interface:
  endpointsMenu: false
  modelSelect: false
  parameters: true
  sidePanel: true
  presets: true
  prompts: true
  bookmarks: true
  multiConvo: true
  agents: false

registration:
  socialLogins: ["openid"]
  allowedDomains:
    - "doordash.com"

endpoints:
  agents:
    capabilities:
      - "actions"
      - "tools"
  custom:
    - name: "OpenAI"
      apiKey: "${PORTKEY_OPENAI_VIRTUAL_KEY}"
      baseURL: "${PORTKEY_URL}"
      models:
        default: ["gpt-4o", "gpt-4o-mini"]
        fetch: false
      headers:
        x-portkey-api-key: "${PORTKEY_API_KEY}"
        x-portkey-virtual-key: "${PORTKEY_OPENAI_VIRTUAL_KEY}"
        # Do not track setting which disables logging of user messages
        x-portkey-debug: "${PORTKEY_DEBUG}"
      titleConvo: true
      titleModel: "gpt-4o-mini"
      summarize: false
      modelDisplayLabel: "OpenAI"
      iconURL: "openAI"

    - name: "OpenAI-high"
      apiKey: "${PORTKEY_OPENAI_VIRTUAL_KEY}"
      baseURL: "${PORTKEY_URL}"
      models:
        default: ["o1", "o1-mini", "o3-mini"]
        fetch: false
      headers:
        x-portkey-api-key: "${PORTKEY_API_KEY}"
        x-portkey-virtual-key: "${PORTKEY_OPENAI_VIRTUAL_KEY}"
        # Do not track setting which disables logging of user messages
        x-portkey-debug: "${PORTKEY_DEBUG}"
      addParams:
        reasoning_effort: "high"
      titleConvo: true
      titleModel: "gpt-4o-mini"
      summarize: false
      modelDisplayLabel: "OpenAI"
      iconURL: "openAI"

    - name: "Anthropic"
      apiKey: "${PORTKEY_AWS_BEDROCK_VIRTUAL_KEY}"
      baseURL: "${PORTKEY_URL}"
      models:
        default: ["us.anthropic.claude-3-7-sonnet-20250219-v1:0", "anthropic.claude-3-5-sonnet-20241022-v2:0", "anthropic.claude-3-5-haiku-20241022-v1:0"]
        fetch: false
      headers:
        x-portkey-api-key: "${PORTKEY_API_KEY}"
        x-portkey-virtual-key: "${PORTKEY_AWS_BEDROCK_VIRTUAL_KEY}"
        # Do not track setting which disables logging of user messages
        x-portkey-debug: "${PORTKEY_DEBUG}"
      titleConvo: true
      titleModel: "anthropic.claude-3-5-haiku-20241022-v1:0"
      titleMessageRole: "user"
      summarize: false

    - name: "Google Gemini"
      apiKey: "${PORTKEY_VERTEX_AI_VIRTUAL_KEY}"
      baseURL: "${PORTKEY_URL}"
      models:
        default: ["gemini-1.5-pro", "gemini-2.0-flash-001", "gemini-1.5-flash"]
        fetch: false
      headers:
        "x-portkey-api-key": "${PORTKEY_API_KEY}"
        "x-portkey-virtual-key": "${PORTKEY_VERTEX_AI_VIRTUAL_KEY}"
        # Do not track setting which disables logging of user messages
        x-portkey-debug: "${PORTKEY_DEBUG}"
      titleConvo: true
      titleModel: "gemini-1.5-flash"
      titleMessageRole: "user"
      summarize: false
      modelDisplayLabel: "Gemini"

modelSpecs:
  enforce: false
  prioritize: true
  list:
    - name: "us.anthropic.claude-3-7-sonnet-20250219-v1:0"
      label: "Claude 3.7 Sonnet"
      description: "Best all-around model"
      iconURL: "anthropic"
      preset:
        append_current_datetime: true
        endpoint: "Anthropic"
        model: "us.anthropic.claude-3-7-sonnet-20250219-v1:0"
        modelLabel: "Claude"

    - name: "o3-mini-high"
      label: "o3-mini-high"
      iconURL: "openAI"
      preset:
        append_current_datetime: true
        addParams:
          reasoning_effort: "high"
        endpoint: "OpenAI-high"
        model: "o3-mini"
        modelLabel: "o3-mini-high"

    - name: "gemini-2.0-flash"
      label: "Gemini 2.0 Flash"
      preset:
        append_current_datetime: true
        endpoint: "Google Gemini"
        model: "gemini-2.0-flash-001"
        modelLabel: "Gemini 2.0 Flash"

    - name: "gpt-4o"
      label: "GPT-4o"
      iconURL: "openAI"
      preset:
        append_current_datetime: true
        endpoint: "OpenAI"
        model: "gpt-4o"

    - name: "gemini-1.5-pro"
      label: "Gemini 1.5 Pro"
      preset:
        append_current_datetime: true
        endpoint: "Google Gemini"
        model: "gemini-1.5-pro"
        modelLabel: "Gemini Pro"

    - name: "o1-high"
      label: "OpenAI o1"
      preset:
        endpoint: "OpenAI-high"
        model: "o1"
        modelLabel: "o1"

    - name: "anthropic.claude-3-5-haiku-20241022-v1:0"
      label: "Claude 3.5 Haiku"
      iconURL: "anthropic"
      preset:
        append_current_datetime: true
        endpoint: "Anthropic"
        model: "anthropic.claude-3-5-haiku-20241022-v1:0"
        modelLabel: "Claude Haiku"

    - name: "gpt-4o-mini"
      label: "GPT-4o mini"
      iconURL: "openAI"
      preset:
        append_current_datetime: true
        endpoint: "OpenAI"
        model: "gpt-4o-mini"
        modelLabel: "GPT-4o mini"