Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to build CUDA backend without Flash attention #11946

Open
slaren opened this issue Feb 18, 2025 · 1 comment
Open

Add option to build CUDA backend without Flash attention #11946

slaren opened this issue Feb 18, 2025 · 1 comment
Labels
enhancement New feature or request

Comments

@slaren
Copy link
Member

slaren commented Feb 18, 2025

          @slaren Honestly, I think Flash Attention should be an optional feature in ggml since it doesn't introduce significant performance improvements, and the binary size has increased considerably—not to mention the compilation time, which, even though I only compile it for my GPU architecture, still takes 20 minutes on an i5-12400. It is not related to this PR, but it would be good to take it into account.

Originally posted by @FSSRepo in #11867 (comment)

@slaren slaren added the enhancement New feature or request label Feb 18, 2025
@bssrdf
Copy link
Contributor

bssrdf commented Feb 18, 2025

I can get that FA can be built optionally to reduce build time. But saying 'it doesn't introduce significant performance improvements' is a bit misleading. On my 4090, I got 47 T/S with FA on and 37 T/S off. SD generation also got a speed up with FA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants