Using vLLM-hosted models

[ArturDev42]

Is your feature request related to a problem? Please describe.
I would like to create agent workflows with models hosted by vLLM. I also need to use function calling.

Describe the solution you'd like
On Swarm Models I read that OpenAI, HuggingFace and Anthropic are supported. I am not sure about vLLM?

Describe alternatives you've considered
Previously, I was using OpenAI Swarm, but it did not seem to support vLLM-hosted models.

Additional context
Basic usage of vLLM is as follows: Start a server with

vllm serve meta-llama/Llama-3.1-8B-Instruct \  
    --enable-auto-tool-choice \  
    --tool-call-parser llama3_json \  
    --chat-template /home/.../repos/vllm/examples/tool_chat_template_llama3.1_json.jinja \  
    --max_model_len 36192

And then use the model client with openai.OpenAI(base_url="http://localhost:8000/v1", api_key="dummy").

Thanks for your help!

[github-actions]

Hello there, thank you for opening an Issue ! 🙏🏻 The team was notified and they will get back to you asap.

[chethanuk]

vLLM does support function calling through its OpenAI-compatible server interface, specifically for models that can handle tool/function calling

Then you can use the OpenAI client exactly as you showed, or swarms https://docs.swarms.world/en/latest/swarms/models/openai_function_caller/#openaifunctioncaller

Do try and if you face any error please ping thanks :)