Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Add guided-* Parameters to Sampling Parameters #5433

Closed
zhanghx0905 opened this issue Jun 12, 2024 · 1 comment
Closed

[Feature]: Add guided-* Parameters to Sampling Parameters #5433

zhanghx0905 opened this issue Jun 12, 2024 · 1 comment

Comments

@zhanghx0905
Copy link

🚀 The feature, motivation and pitch

The openai endpoint of vllm now supports passing guided grammar in the request body. However, the current sampling parameters do not include corresponding parameters for guided grammar. This limitation prevents me from using guided grammar while utilizing the LLM Engine.

Please add support for guided-* parameters in the sampling parameters to enable the use of guided grammar with the LLM Engine.

With the addition of guided-* parameters in the sampling parameters, users will be able to leverage guided grammar in conjunction with the LLM Engine, enhancing the flexibility and functionality of the model.

Alternatives

No response

Additional context

No response

@zhanghx0905
Copy link
Author

duplicate #5423

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant