Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow chat streaming to use tools #1088

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

Jeadie
Copy link
Contributor

@Jeadie Jeadie commented Jan 23, 2025

Description

  • Add tool_calls to .chunk.delta within ChatCompletionChunkResponse (i.e. /v1/chat/completion streaming).
  • This is done by checking whether the stream could be parsed to the tool call structure (could be meaning that we'd get an EOF, not a format error, when parsing it to a tool structure).

Copy link

github-actions bot commented Jan 23, 2025

Code Metrics Report
  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                2           35           28            0            7
 Dockerfile              1           41           22           10            9
 JSON                   12          105          104            0            1
 Python                 69         2926         2534           77          315
 Shell                   1           58           22           18           18
 Plain Text              3         3723            0         2413         1310
 TOML                   18          645          574            2           69
 YAML                    2           21           19            2            0
-------------------------------------------------------------------------------
 Jupyter Notebooks       4            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          205          178            1           26
 (Total)                            282          210           32           40
-------------------------------------------------------------------------------
 Markdown               47         3826            0         2907          919
 |- BASH                 6          103          100            0            3
 |- JSON                 1           12           12            0            0
 |- Python               7          121          109            0           12
 |- Rust                15          512          433            0           79
 |- TOML                 2           75           63            0           12
 (Total)                           4649          717         2907         1025
-------------------------------------------------------------------------------
 Rust                  311       100379        89907         2007         8465
 |- Markdown           151         1702           25         1552          125
 (Total)                         102081        89932         3559         8590
===============================================================================
 Total                 470       111759        93210         7436        11113
===============================================================================
  

Copy link
Owner

@EricLBuehler EricLBuehler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Just a small comment.

mistralrs-core/src/pipeline/sampling.rs Outdated Show resolved Hide resolved
@Jeadie Jeadie requested a review from EricLBuehler January 31, 2025 03:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants