Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Klat does not handle vLLM deployment updates #163

Open
NeonDaniel opened this issue Feb 6, 2025 · 1 comment
Open

[BUG] Klat does not handle vLLM deployment updates #163

NeonDaniel opened this issue Feb 6, 2025 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@NeonDaniel
Copy link
Member

Description

After restarting vLLM, Klat continues to request the old models rather than the updated ones.

Steps to Reproduce

  • Update a vLLM deployment
  • Restart the neon-llm-vllm container
  • Request one of the vLLM chatbots in Klat
  • Observe errors logged in neon-llm-vllm container (also reported to Sentry)

Relevant Code

No response

Other Notes

Previously, these invalid requests would fail; they now log errors with NeonGeckoCom/neon-llm-vllm#9

@NeonDaniel NeonDaniel added bug Something isn't working triage New issue needs review and removed triage New issue needs review labels Feb 6, 2025
@NeonDaniel
Copy link
Member Author

@NeonKirill if you can point me at the place(s) where Klat receives available models from the vLLM container, I can probably troubleshoot and resolve this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants