A Huggingface Space for test please

[Pevernow]

No description provided.

[lewtun]

Although we haven't made a Space to demo these methods, you can find a notebook to explore with here: huggingface/cookbook#262

Of course, if someone in the community wishes to implement a Space, we'd be happy to amplify it!

[sergiopaniego]

Hi @lewtun!
I was considering generating the HF Space, but it might be challenging with the current implementation.
This is the issue:
vllm-project/vllm#3510

I've directly tried to generate it and failed 😄

Traceback (most recent call last):
  File "/home/user/app/app.py", line 24, in <module>
    llm = LLM(
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 177, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 571, in from_engine_args
    engine_config = engine_args.create_engine_config()
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 899, in create_engine_config
    device_config = DeviceConfig(device=self.device)
  File "/usr/local/lib/python3.10/site-packages/vllm/config.py", line 1088, in __init__
    raise RuntimeError("Failed to infer device type")
RuntimeError: Failed to infer device type

What do you think? Is there a possible workaround?