How's the speed compare with vanilla candle

[MonolithFoundation]

Candle supports both CUDA and CPU. How does its speed compare to the original?

[guoqingbao]

Candle supports both CUDA and CPU. How does its speed compare to the original?

Candle-vLLm uses Paged Attention (which is suitable for long sentence generation) and the performance of Candle-vLLm is quite similar to vLLM, e.g., it can give you over 110 tokens/s for LLaMa3.1 8B model on A100.