How to run Qwen using Executorch?

[Arya-Hari]

📚 The doc issue

Hi! I just wanted to how, how would I go about running Qwen using executorch? I was able to create the .pte file for Qwen. The example for Llama had a step 'Create a llama runner for android'. Do we have to do something similar for Qwen by creating a custom runner? Also the Qwen repository on Hugging Face Hub does not have a 'tokenizer.model' file, but the Llama example requires it for running inference using the adb shell. How to navigate around this?

Suggest a potential alternative/fix

No response

[kimishpatel]

i dont know the details how to run Qwen and whether there is any significant difference compared to llama as far as model's interface is concerned.

Also when you say you were abel to export the model, can you detail the steps you took. If you can run exported qwen model using https://github.com/pytorch/executorch/blob/main/examples/models/llama/runner/eager.py#L103 then highly likely that you can run via cpp runner. But you do need tokenizer, so not sure how hf runs this model

[SS-JIA]

@Arya-Hari for some more context, the llama_runner binary used in our examples is heavily tailored to the llama model architecture. So as Kimish mentioned, depending on the interface of Qwen compared to llama you may not be able to re-use the llama_runner binary. If you are familiar with the interface of the model, then the best way would be to fork or modify the llama_runner binary for the Qwen model; essentially creating a custom runner as you mentioned.

[mergennachin]

@guangy10, Is there guidelines on how to leverage from recent Hugging Face (huggingface/transformers#32253, huggingface/transformers#34102) and optimum integrations (https://huggingface.co/docs/optimum/main/en/exporters/executorch/usage_guides/export_a_model)