Don't attempt batching with InstructLab's llama-cpp-python #358
Conversation
Try harder to check for InstructLab's default llama-cpp-python during our batching check so that we can avoid throwing assertion errors on the server when we probe for batching support. See instructlab/instructlab#1748 for user reports of this issue. Signed-off-by: Ben Browning <[email protected]>
|
There doesn't appear to be a robust way to identify llama-cpp-python as a HTTP client. However, InstructLab's most common happy path has users generating data against a llama-cpp-python managed by InstructLab and we CAN reliably identify that. We'll do that by looking for the API root welcome message and disable batching without actually attempting the batching request probe. That batching request attempt is what throws the assertion error in llama-cpp-python, and this change will avoid throwing those errors for anyone using |
|
We get lots of user reports via Slack and in this and other repos about the assertion error with llama-cpp-python. This gets rid of that, and will make for a much cleaner initial user experience. Otherwise, we often have users that think things are broken with data generation. |
Try harder to check for InstructLab's default llama-cpp-python during our batching check so that we can avoid throwing assertion errors on the server when we probe for batching support.
See instructlab/instructlab#1748 for user reports of this issue.