do_sample is True, nan appears in "eager" mode

[sunyanguomt]

HI
When i run demp.py, in attn_implementation="eager", the following problem occurs。

Traceback (most recent call last):
File "demo.py", line 44, in
output_ids = model.generate(input_ids, images=[video_tensor], modalities=["video"],beacon_skip_first=beacon_skip_first,beacon_skip_last=beacon_skip_last, **gen_kwargs)
File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/envs/py38/lib/python3.8/site-packages/videoxl/model/language_model/llava_qwen.py", line 1830, in generate
return super().generate(position_ids=position_ids, attention_mask=attention_mask,inputs=inputs,beacon_skip_first=beacon_skip_first, beacon_skip_last= beacon_skip_last, **kwargs)
File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/envs/py38/lib/python3.8/site-packages/transformers/generation/utils.py", line 2024, in generate
result = self._sample(
File "/opt/conda/envs/py38/lib/python3.8/site-packages/transformers/generation/utils.py", line 3020, in _sample
next_tokens = torch.multinomial(probs.to("cpu"), num_samples=1).squeeze(1).to("musa")
RuntimeError: probability tensor contains either inf, nan or element < 0

Then I found him in this position，this softmax apperas nan

This problem does not occur when using “sdpa” mode

[shuyansy]

In my implementation, the model is running with "flash-attention" mode. It may not work in other modes

[ApolloRay]

HI When i run demp.py, in attn_implementation="eager", the following problem occurs。

Traceback (most recent call last): File "demo.py", line 44, in output_ids = model.generate(input_ids, images=[video_tensor], modalities=["video"],beacon_skip_first=beacon_skip_first,beacon_skip_last=beacon_skip_last, **gen_kwargs) File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/opt/conda/envs/py38/lib/python3.8/site-packages/videoxl/model/language_model/llava_qwen.py", line 1830, in generate return super().generate(position_ids=position_ids, attention_mask=attention_mask,inputs=inputs,beacon_skip_first=beacon_skip_first, beacon_skip_last= beacon_skip_last, **kwargs) File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/opt/conda/envs/py38/lib/python3.8/site-packages/transformers/generation/utils.py", line 2024, in generate result = self._sample( File "/opt/conda/envs/py38/lib/python3.8/site-packages/transformers/generation/utils.py", line 3020, in _sample next_tokens = torch.multinomial(probs.to("cpu"), num_samples=1).squeeze(1).to("musa") RuntimeError: probability tensor contains either inf, nan or element < 0

Then I found him in this position，this softmax apperas nan

This problem does not occur when using “sdpa” mode

where can i change the flash-implementation？
same problem