[MODULE] Add module on inference #150
Conversation
| response = generator( | ||
| "Write a short poem about coding:", | ||
| max_new_tokens=100, | ||
| do_sample=True, | ||
| temperature=0.7 | ||
| ) |
There was a problem hiding this comment.
isn't 100 tokens a bit small?
7_inference/inference_pipeline.md
Outdated
| Here's how to integrate a pipeline into a Flask application: | ||
|
|
||
| ```python | ||
| from flask import Flask, request, jsonify |
There was a problem hiding this comment.
why are you using Flask and not something more modern like FastAPI?
There was a problem hiding this comment.
It's just in the pipeline documentation
| @@ -0,0 +1,137 @@ | |||
| # Text Generation Inference (TGI) | |||
|
|
|||
| Text Generation Inference (TGI) is a toolkit developed by Hugging Face for deploying and serving Large Language Models (LLMs). It's designed to enable high-performance text generation for popular open-source LLMs. TGI is used in production by Hugging Chat - An open-source interface for open-access models. | |||
There was a problem hiding this comment.
I think it would be fair to mention other providers like vLLM and Ollama too, as being the same but different?
|
|
||
| LLM inference can be categorized into two main approaches: simple pipeline-based inference for development and testing, and optimized serving solutions for production deployments. We'll cover both approaches, starting with the simpler pipeline approach and moving to production-ready solutions. | ||
|
|
||
| ## Contents |
There was a problem hiding this comment.
should we consider showing and mentioning inference options again like adapters and structured generation?
There was a problem hiding this comment.
I agree. It would be very helpful to include controlled inference generation, which is particular crucial when integrating LLMs with downstream systems (e.g. grammars with llama.cpp, FSM with outlines or logit processing techniques in general).
Here's an example on how to control inference using Smol model + LogitsProcessor class from the Transformers library:
https://www.tamingllms.com/notebooks/structured_output.html#logit-post-processing
There was a problem hiding this comment.
This is nice suggestion, but I would add it in a subsequent PR and a separate page.
| from huggingface_hub import InferenceClient | ||
|
|
||
| client = InferenceClient( | ||
| base_url="http://localhost:8080/v1/", |
There was a problem hiding this comment.
Perhaps it would be helpful to use working API URLs such that sample code runs. Plus explain that in HuggingFace one can run inference on Serverless Inference API, dedicated APIs etc.
This is a module on inference techniques like pipeline and TGI.