Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Specscheduler new attention #1434

Merged
merged 38 commits into from
Aug 14, 2024
Merged

Specscheduler new attention #1434

merged 38 commits into from
Aug 14, 2024

Conversation

chenzhuofu
Copy link
Collaborator

@chenzhuofu chenzhuofu commented Jul 9, 2024 •
edited by wmdi
Loading

Description of changes:

Adopt flashinfer into self-attention.

Related Issues:

Linked Issues:

  • Issue #

Issues closed by this PR:

  • Closes #

This change is Reviewable

chenzhuofu and others added 25 commits June 29, 2024 02:06
@chenzhuofu
feat: update deps/flashinfer
7e02ef0
@chenzhuofu
feat: update flashinfer
be64b3b
@chenzhuofu
fix: now can get correct result, but has performance problem
7a4543b
@chenzhuofu
fix: update_custom_mask performance
e1c8145
@chenzhuofu
chore: minor
483b71b
@chenzhuofu
chore: add perf code
0a117fb
@zikun-li
Merge branch 'specscheduler' into specscheduler-new-attention
ac343ee
@chenzhuofu
feat: add attention metadata
3e9dc00
@chenzhuofu
feat: add AttentionMetaData
a498e6f
@chenzhuofu
feat: tree_verify_attn use global attentionmetadata
2903996
@chenzhuofu
feat: move attentionmetasize to global computing
fed9e5c
@chenzhuofu
chore: minor
7c64f36
@chenzhuofu
chore: remove unused
37d3e3d
@chenzhuofu
feat: add spec_inc_attn backup
9940ff5
@chenzhuofu
feat: SSM use flashinfer kernel
5262af7
@chenzhuofu
fix: SSM don't use cudaGraph
dfe4bec
@chenzhuofu
chore: remove redundant code
e08f06e
@chenzhuofu
chore: comment out minor
ed544e9
@chenzhuofu
feat: attention adapt to cudaGraph
214aed6
@chenzhuofu
fix: split handler_collections for prompt/decode phases
020129b
@chenzhuofu
chore: tree verify cannot use cudaGraph
0150501
@chenzhuofu
feat: move all flashinfer-related states to global (tree search atten…
d4d66af 
…tion)
@chenzhuofu
fix: use identical attention_meta instance across all FFHandlers
b4cc53b
@chenzhuofu
feat: enable cudaGraph in tree search mode
79ea1d9
@chenzhuofu
chore: minor
8d52e4f
@chenzhuofu chenzhuofu requested a review from zikun-li July 9, 2024 19:13
@chenzhuofu chenzhuofu self-assigned this Jul 9, 2024
@chenzhuofu chenzhuofu changed the title Specscheduler new attention [DO NOT MERGE] Specscheduler new attention Aug 2, 2024
@chenzhuofu chenzhuofu changed the title [DO NOT MERGE] Specscheduler new attention Specscheduler new attention Aug 13, 2024
@chenzhuofu
Copy link
Collaborator Author

I think this can be reviewed now. @jiazhihao
The LLM cudaGraph will not bring notable improvement, and it requires some changes in custom AllReduce I developed in the other branch. So we may leave it disabled for now.

@jiazhihao jiazhihao merged commit 8dbe98d into specscheduler Aug 14, 2024
25 of 32 checks passed
@chenzhuofu chenzhuofu deleted the new-attn branch August 14, 2024 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jiazhihao jiazhihao jiazhihao left review comments

@zikun-li zikun-li zikun-li approved these changes

Assignees

@chenzhuofu chenzhuofu

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@chenzhuofu @jiazhihao @zikun-li