Inconsistent behavior when passing in Pydantic models for structured generation

[ericmjl]

UPDATE: I have a GitHub gist to illustrate the issue: https://gist.github.com/ericmjl/f50e1cc716e566770ead755c9c1e28a3

I recently tried the following code:

# Direct litellm completion approach
from pydantic import BaseModel
from litellm import completion


class Person(BaseModel):
    name: str
    age: int
    occupation: str
    hobbies: list[str]


response = completion(
    model="anthropic/claude-3-5-sonnet-20240620",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that generates information about a person.",
        },
        {"role": "user", "content": "Generate a person"},
    ],
    response_format=Person,  # Pass the Pydantic model directly
)

And in response I got the following error message:

{
	"name": "BadRequestError",
	"message": "litellm.BadRequestError: AnthropicException - {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"tools.0.input_schema: JSON schema is invalid - please consult https://json-schema.org or our documentation at https://docs.anthropic.com/en/docs/tool-use\"}}",
	"stack": "---------------------------------------------------------------------------
HTTPStatusError                           Traceback (most recent call last)
File ~/github/software/llamabot/.pixi/envs/notebooks/lib/python3.12/site-packages/litellm/llms/anthropic/chat/handler.py:566, in AnthropicChatCompletion.completion(self, model, messages, api_base, custom_prompt_dict, model_response, print_verbose, encoding, api_key, logging_obj, optional_params, timeout, acompletion, litellm_params, logger_fn, headers, client)
    565 try:
--> 566     response = client.post(
    567         api_base,
    568         headers=headers,
    569         data=json.dumps(data),
    570         timeout=timeout,
    571     )
    572 except Exception as e:
File ~/github/software/llamabot/.pixi/envs/notebooks/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py:389, in HTTPHandler.post(self, url, data, json, params, headers, stream, timeout)

388         setattr(e, "message", e.response.text)

--> 389     raise e

390 except Exception as e:
File ~/github/software/llamabot/.pixi/envs/notebooks/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py:375, in HTTPHandler.post(self, url, data, json, params, headers, stream, timeout)

374 response = self.client.send(req, stream=stream)

--> 375 response.raise_for_status()

376 return response
File ~/github/software/llamabot/.pixi/envs/notebooks/lib/python3.12/site-packages/httpx/_models.py:763, in Response.raise_for_status(self)

762 message = message.format(self, error_type=error_type)

--> 763 raise HTTPStatusError(message, request=request, response=self)
HTTPStatusError: Client error '400 Bad Request' for url 'https://api.anthropic.com/v1/messages'

For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
During handling of the above exception, another exception occurred:
AnthropicError                            Traceback (most recent call last)

File ~/github/software/llamabot/.pixi/envs/notebooks/lib/python3.12/site-packages/litellm/main.py:1769, in completion(model, messages, timeout, temperature, top_p, n, stream, stream_options, stop, max_completion_tokens, max_tokens, modalities, prediction, audio, presence_penalty, frequency_penalty, logit_bias, user, response_format, seed, tools, tool_choice, logprobs, top_logprobs, parallel_tool_calls, deployment_id, extra_headers, functions, function_call, base_url, api_version, api_key, model_list, **kwargs)

1767         api_base += "/v1/messages"

-> 1769     response = anthropic_chat_completions.completion(

1770         model=model,

1771         messages=messages,

1772         api_base=api_base,

1773         acompletion=acompletion,

1774         custom_prompt_dict=litellm.custom_prompt_dict,

1775         model_response=model_response,

1776         print_verbose=print_verbose,

1777         optional_params=optional_params,

1778         litellm_params=litellm_params,

1779         logger_fn=logger_fn,

1780         encoding=encoding,  # for calculating input/output tokens

1781         api_key=api_key,

1782         logging_obj=logging,

1783         headers=headers,

1784         timeout=timeout,

1785         client=client,

1786     )

1787 if optional_params.get("stream", False) or acompletion is True:

1788     ## LOGGING
File ~/github/software/llamabot/.pixi/envs/notebooks/lib/python3.12/site-packages/litellm/llms/anthropic/chat/handler.py:581, in AnthropicChatCompletion.completion(self, model, messages, api_base, custom_prompt_dict, model_response, print_verbose, encoding, api_key, logging_obj, optional_params, timeout, acompletion, litellm_params, logger_fn, headers, client)

580                 error_text = getattr(error_response, "text", error_text)

--> 581             raise AnthropicError(

582                 message=error_text,

583                 status_code=status_code,

584                 headers=error_headers,

585             )

587 return self._process_response(

588     model=model,

589     response=response,

(...)

599     json_mode=json_mode,

600 )
AnthropicError: {"type":"error","error":{"type":"invalid_request_error","message":"tools.0.input_schema: JSON schema is invalid - please consult https://json-schema.org or our documentation at https://docs.anthropic.com/en/docs/tool-use\"}}
During handling of the above exception, another exception occurred:
BadRequestError                           Traceback (most recent call last)

Cell In[5], line 13

9     occupation: str

10     hobbies: list[str]

---> 13 response = completion(

14     model="anthropic/claude-3-sonnet-20240320",

15     messages=[

16         {

17             "role": "system",

18             "content": "You are a helpful assistant that generates information about a person.",

19         },

20         {"role": "user", "content": "Generate a person"},

21     ],

22     response_format=Person,  # Pass the Pydantic model directly

23 )
File ~/github/software/llamabot/.pixi/envs/notebooks/lib/python3.12/site-packages/litellm/utils.py:959, in client..wrapper(*args, **kwargs)

955 if logging_obj:

956     logging_obj.failure_handler(

957         e, traceback_exception, start_time, end_time

958     )  # DO NOT MAKE THREADED - router retry fallback relies on this!

--> 959 raise e
File ~/github/software/llamabot/.pixi/envs/notebooks/lib/python3.12/site-packages/litellm/utils.py:848, in client..wrapper(*args, **kwargs)

846         print_verbose(f"Error while checking max token limit: {str(e)}")

847 # MODEL CALL

--> 848 result = original_function(*args, **kwargs)

849 end_time = datetime.datetime.now()

850 if "stream" in kwargs and kwargs["stream"] is True:
File ~/github/software/llamabot/.pixi/envs/notebooks/lib/python3.12/site-packages/litellm/main.py:3033, in completion(model, messages, timeout, temperature, top_p, n, stream, stream_options, stop, max_completion_tokens, max_tokens, modalities, prediction, audio, presence_penalty, frequency_penalty, logit_bias, user, response_format, seed, tools, tool_choice, logprobs, top_logprobs, parallel_tool_calls, deployment_id, extra_headers, functions, function_call, base_url, api_version, api_key, model_list, **kwargs)

3030     return response

3031 except Exception as e:

3032     ## Map to OpenAI Exception

-> 3033     raise exception_type(

3034         model=model,

3035         custom_llm_provider=custom_llm_provider,

3036         original_exception=e,

3037         completion_kwargs=args,

3038         extra_kwargs=kwargs,

3039     )
File ~/github/software/llamabot/.pixi/envs/notebooks/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py:2122, in exception_type(model, original_exception, custom_llm_provider, completion_kwargs, extra_kwargs)

2120 if exception_mapping_worked:

2121     setattr(e, "litellm_response_headers", litellm_response_headers)

-> 2122     raise e

2123 else:

2124     for error_type in litellm.LITELLM_EXCEPTION_TYPES:
File ~/github/software/llamabot/.pixi/envs/notebooks/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py:469, in exception_type(model, original_exception, custom_llm_provider, completion_kwargs, extra_kwargs)

464 elif (

465     original_exception.status_code == 400

466     or original_exception.status_code == 413

467 ):

468     exception_mapping_worked = True

--> 469     raise BadRequestError(

470         message=f"AnthropicException - {error_str}",

471         model=model,

472         llm_provider="anthropic",

473     )

474 elif original_exception.status_code == 404:

475     exception_mapping_worked = True
BadRequestError: litellm.BadRequestError: AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"tools.0.input_schema: JSON schema is invalid - please consult https://json-schema.org or our documentation at https://docs.anthropic.com/en/docs/tool-use\"}}"

}

On the other hand, when I do a drop-in replacement of the model name with got-4o, everything runs perfectly fine:

response = completion(
    model="gpt-4o",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that generates information about a person.",
        },
        {"role": "user", "content": "Generate a person"},
    ],
    response_format=Person,  # Pass the Pydantic model directly
)

It appears that to solve the problem with anthropic models, I actually need to do the model JSON schema dump and pass that into the response format keyword argument:

response = completion(
    model="anthropic/claude-3-5-sonnet-20240620",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that generates information about a person.",
        },
        {"role": "user", "content": "Generate a person"},
    ],
    response_format=Person.model_json_schema(),
)

I believe it should be better that the two behaviors are harmonized and actually made consistent. In case it is helpful, my opinion is that the first format is better. There's fewer things for us to worry about. We should just use be passing in the object rather than the model JSON schema.

[ericmjl]

Might be related to this issue, @ishaan-jaff @krrishdholakia?

[krrishdholakia]

can you use litellm.set_verbose = True and share the debug logs

[ericmjl]

Hmmm, I suspect something changed between version 1.3x.y and 1.5x.y I recently just upgraded to 1.56.10 and the error goes away. I apologize for not first checking package versions, should have done that first!