Update pyproject.toml + tools Falcon 3 addition #402
Conversation
| checkpoint_repository: | ||
| repo: tiiuae/Falcon3-10B-Instruct | ||
| source: HF | ||
| max_seq_len: 8192 |
There was a problem hiding this comment.
Should this match the runtime tokens?
There was a problem hiding this comment.
| enable_chunked_context: true | ||
| kv_cache_free_gpu_mem_fraction: 0.62 | ||
| request_default_max_tokens: 1000 | ||
| total_token_limit: 500000 |
There was a problem hiding this comment.
Should this be a round number?
There was a problem hiding this comment.
total_token_limit defines a briton setting for how many tokens are queued inside C++. If queueing too many requests, we overload runtime. Not important, as long as it's in a decent range.
total_token_limit = 500000 is default.
kv_cache_free_gpu_mem_fraction: 0.62 is a hard setting.
Falcon-3-10B needs around 20GB Vram for weights. We have 20GB Vram left.
Of the rest of the 20GB we allocate 62% to Falcon-10B. The rest of 7.6GB go to the 1B model. 2 GB out of the 7.6Gb will go to the weights, while 5.6GB are for activations and KV cache.
truss==0.9.49via git! Git still pins revisions!Adds falcon: