A HuggingFace model handler would be elite

[pablogranolabar]

You know what would be amazing, would be Python bindings for this. To expose it as a standard HF model object that could be instantiated just like a CUDA placed model. That would open the door for infinite new applications, really.

[thomasantony]

I have a kind-of-experimental Python bindings here: https://github.com/thomasantony/llamacpp-python . Not sure what it would take to make it compatible with HF though.

[amartinr]

You know what would be amazing, would be Python bindings for this. To expose it as a standard HF model object that could be instantiated just like a CUDA placed model. That would open the door for infinite new applications, really.

It would be top notch, as quantized models are becoming increasingly popular on HF, in part thanks to the interest created by llama.cpp and similar projects. I started quantizing other people's models, but now I resorted to use other people's quantized models, as you only need to know which model they quantized to get the corresponding config.json and tokenizer.model files.