Skip to content

Commit

Permalink
remove a residual line from when requests only stored the last token …
Browse files Browse the repository at this point in the history
…in order to fix a masking issue
  • Loading branch information
renxida committed Jan 28, 2025
1 parent c495afb commit 84563c9
Showing 1 changed file with 1 addition and 4 deletions.
5 changes: 1 addition & 4 deletions shortfin/python/shortfin_apps/llm/components/service.py
Original file line number Diff line number Diff line change
Expand Up @@ -401,10 +401,7 @@ async def run(self):
m.fill(
1 # Must pad with a nonzero value because a division by 0 during softmax floods clobber page (page 0) in cache with NaN values.
)
m.items = [
req.start_position + len(req.input_token_ids)
for req in self.exec_requests
]
m.items = [req.start_position + 1 for req in self.exec_requests]
seq_lens_host.copy_to(seq_lens)

# Populate cache pages.
Expand Down

0 comments on commit 84563c9

Please sign in to comment.