Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

warpscan: try float accumulation for bf16 and float16 #4

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

warpscan: try float accumulation for bf16 and float16 #4

wants to merge 1 commit into from

Conversation

proger
Copy link
Owner

@proger proger commented Feb 4, 2024

@sustcsonglin has suggested that float accumulation might improve stability of the implementation. The current test I'm trying using to see this is:

python -m pytest tests -s -v  -k forward | grep 'max abs error'

The results are so far the same, perhaps I need to find better test conditions to see this:

tests/test_eq.py::test_eq_forward[dtype0-32-3407-scan] max abs error 7.152557373046875e-07 seqlen 32 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-32-4-scan] max abs error 4.76837158203125e-07 seqlen 32 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-32-42-scan] max abs error 7.152557373046875e-07 seqlen 32 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-32-57-scan] max abs error 7.152557373046875e-07 seqlen 32 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-64-3407-scan] max abs error 7.152557373046875e-07 seqlen 64 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-64-4-scan] max abs error 7.152557373046875e-07 seqlen 64 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-64-42-scan] max abs error 7.152557373046875e-07 seqlen 64 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-64-57-scan] max abs error 7.152557373046875e-07 seqlen 64 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-128-3407-scan] max abs error 9.5367431640625e-07 seqlen 128 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-128-4-scan] max abs error 7.152557373046875e-07 seqlen 128 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-128-42-scan] max abs error 7.152557373046875e-07 seqlen 128 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-128-57-scan] max abs error 9.5367431640625e-07 seqlen 128 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-256-3407-scan] max abs error 7.152557373046875e-07 seqlen 256 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-256-4-scan] max abs error 9.5367431640625e-07 seqlen 256 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-256-42-scan] max abs error 9.5367431640625e-07 seqlen 256 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-256-57-scan] max abs error 7.152557373046875e-07 seqlen 256 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-512-3407-scan] max abs error 7.152557373046875e-07 seqlen 512 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-512-4-scan] max abs error 9.5367431640625e-07 seqlen 512 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-512-42-scan] max abs error 9.5367431640625e-07 seqlen 512 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-512-57-scan] max abs error 7.152557373046875e-07 seqlen 512 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-1024-3407-scan] max abs error 9.5367431640625e-07 seqlen 1024 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-1024-4-scan] max abs error 9.5367431640625e-07 seqlen 1024 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-1024-42-scan] max abs error 9.5367431640625e-07 seqlen 1024 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-1024-57-scan] max abs error 9.5367431640625e-07 seqlen 1024 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-2048-3407-scan] max abs error 9.5367431640625e-07 seqlen 2048 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-2048-4-scan] max abs error 9.5367431640625e-07 seqlen 2048 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-2048-42-scan] max abs error 9.5367431640625e-07 seqlen 2048 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-2048-57-scan] max abs error 9.5367431640625e-07 seqlen 2048 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-4096-3407-scan] max abs error 9.5367431640625e-07 seqlen 4096 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-4096-4-scan] max abs error 9.5367431640625e-07 seqlen 4096 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-4096-42-scan] max abs error 9.5367431640625e-07 seqlen 4096 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-4096-57-scan] max abs error 9.5367431640625e-07 seqlen 4096 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-8192-3407-scan] max abs error 9.5367431640625e-07 seqlen 8192 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-8192-4-scan] max abs error 9.5367431640625e-07 seqlen 8192 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-8192-42-scan] max abs error 9.5367431640625e-07 seqlen 8192 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-8192-57-scan] max abs error 9.5367431640625e-07 seqlen 8192 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-16384-3407-scan] max abs error 9.5367431640625e-07 seqlen 16384 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-16384-4-scan] max abs error 9.5367431640625e-07 seqlen 16384 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-16384-42-scan] max abs error 9.5367431640625e-07 seqlen 16384 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-16384-57-scan] max abs error 9.5367431640625e-07 seqlen 16384 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-32768-3407-scan] max abs error 9.5367431640625e-07 seqlen 32768 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-32768-4-scan] max abs error 9.5367431640625e-07 seqlen 32768 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-32768-42-scan] max abs error 9.5367431640625e-07 seqlen 32768 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-32768-57-scan] max abs error 1.430511474609375e-06 seqlen 32768 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-65536-3407-scan] max abs error 9.5367431640625e-07 seqlen 65536 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-65536-4-scan] max abs error 1.430511474609375e-06 seqlen 65536 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-65536-42-scan] max abs error 9.5367431640625e-07 seqlen 65536 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype0-65536-57-scan] max abs error 1.430511474609375e-06 seqlen 65536 dtype torch.float32
tests/test_eq.py::test_eq_forward[dtype1-32-3407-scan] max abs error 0.03125 seqlen 32 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-32-4-scan] max abs error 0.03125 seqlen 32 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-32-42-scan] max abs error 0.03125 seqlen 32 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-32-57-scan] max abs error 0.03125 seqlen 32 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-64-3407-scan] max abs error 0.03125 seqlen 64 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-64-4-scan] max abs error 0.03125 seqlen 64 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-64-42-scan] max abs error 0.03125 seqlen 64 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-64-57-scan] max abs error 0.03125 seqlen 64 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-128-3407-scan] max abs error 0.03125 seqlen 128 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-128-4-scan] max abs error 0.03125 seqlen 128 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-128-42-scan] max abs error 0.03125 seqlen 128 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-128-57-scan] max abs error 0.03125 seqlen 128 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-256-3407-scan] max abs error 0.03125 seqlen 256 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-256-4-scan] max abs error 0.03125 seqlen 256 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-256-42-scan] max abs error 0.03125 seqlen 256 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-256-57-scan] max abs error 0.03125 seqlen 256 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-512-3407-scan] max abs error 0.046875 seqlen 512 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-512-4-scan] max abs error 0.03125 seqlen 512 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-512-42-scan] max abs error 0.046875 seqlen 512 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-512-57-scan] max abs error 0.03125 seqlen 512 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-1024-3407-scan] max abs error 0.03125 seqlen 1024 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-1024-4-scan] max abs error 0.03125 seqlen 1024 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-1024-42-scan] max abs error 0.03125 seqlen 1024 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-1024-57-scan] max abs error 0.046875 seqlen 1024 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-2048-3407-scan] max abs error 0.03125 seqlen 2048 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-2048-4-scan] max abs error 0.046875 seqlen 2048 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-2048-42-scan] max abs error 0.03125 seqlen 2048 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-2048-57-scan] max abs error 0.046875 seqlen 2048 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-4096-3407-scan] max abs error 0.046875 seqlen 4096 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-4096-4-scan] max abs error 0.046875 seqlen 4096 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-4096-42-scan] max abs error 0.046875 seqlen 4096 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-4096-57-scan] max abs error 0.046875 seqlen 4096 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-8192-3407-scan] max abs error 0.0625 seqlen 8192 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-8192-4-scan] max abs error 0.046875 seqlen 8192 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-8192-42-scan] max abs error 0.046875 seqlen 8192 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-8192-57-scan] max abs error 0.0625 seqlen 8192 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-16384-3407-scan] max abs error 0.0625 seqlen 16384 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-16384-4-scan] max abs error 0.0625 seqlen 16384 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-16384-42-scan] max abs error 0.046875 seqlen 16384 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-16384-57-scan] max abs error 0.0625 seqlen 16384 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-32768-3407-scan] max abs error 0.0625 seqlen 32768 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-32768-4-scan] max abs error 0.0625 seqlen 32768 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-32768-42-scan] max abs error 0.0625 seqlen 32768 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-32768-57-scan] max abs error 0.0625 seqlen 32768 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-65536-3407-scan] max abs error 0.0625 seqlen 65536 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-65536-4-scan] max abs error 0.0625 seqlen 65536 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-65536-42-scan] max abs error 0.0625 seqlen 65536 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype1-65536-57-scan] max abs error 0.0625 seqlen 65536 dtype torch.bfloat16
tests/test_eq.py::test_eq_forward[dtype2-32-3407-scan] max abs error 0.00390625 seqlen 32 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-32-4-scan] max abs error 0.00390625 seqlen 32 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-32-42-scan] max abs error 0.00390625 seqlen 32 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-32-57-scan] max abs error 0.00390625 seqlen 32 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-64-3407-scan] max abs error 0.00390625 seqlen 64 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-64-4-scan] max abs error 0.00390625 seqlen 64 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-64-42-scan] max abs error 0.00390625 seqlen 64 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-64-57-scan] max abs error 0.00390625 seqlen 64 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-128-3407-scan] max abs error 0.00390625 seqlen 128 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-128-4-scan] max abs error 0.00390625 seqlen 128 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-128-42-scan] max abs error 0.00390625 seqlen 128 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-128-57-scan] max abs error 0.005859375 seqlen 128 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-256-3407-scan] max abs error 0.00390625 seqlen 256 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-256-4-scan] max abs error 0.005859375 seqlen 256 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-256-42-scan] max abs error 0.00390625 seqlen 256 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-256-57-scan] max abs error 0.00390625 seqlen 256 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-512-3407-scan] max abs error 0.00390625 seqlen 512 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-512-4-scan] max abs error 0.0078125 seqlen 512 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-512-42-scan] max abs error 0.00390625 seqlen 512 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-512-57-scan] max abs error 0.00390625 seqlen 512 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-1024-3407-scan] max abs error 0.00390625 seqlen 1024 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-1024-4-scan] max abs error 0.005859375 seqlen 1024 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-1024-42-scan] max abs error 0.005859375 seqlen 1024 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-1024-57-scan] max abs error 0.00390625 seqlen 1024 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-2048-3407-scan] max abs error 0.005859375 seqlen 2048 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-2048-4-scan] max abs error 0.005859375 seqlen 2048 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-2048-42-scan] max abs error 0.0078125 seqlen 2048 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-2048-57-scan] max abs error 0.00390625 seqlen 2048 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-4096-3407-scan] max abs error 0.0078125 seqlen 4096 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-4096-4-scan] max abs error 0.005859375 seqlen 4096 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-4096-42-scan] max abs error 0.005859375 seqlen 4096 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-4096-57-scan] max abs error 0.0078125 seqlen 4096 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-8192-3407-scan] max abs error 0.005859375 seqlen 8192 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-8192-4-scan] max abs error 0.0078125 seqlen 8192 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-8192-42-scan] max abs error 0.005859375 seqlen 8192 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-8192-57-scan] max abs error 0.0078125 seqlen 8192 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-16384-3407-scan] max abs error 0.005859375 seqlen 16384 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-16384-4-scan] max abs error 0.0078125 seqlen 16384 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-16384-42-scan] max abs error 0.0078125 seqlen 16384 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-16384-57-scan] max abs error 0.0078125 seqlen 16384 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-32768-3407-scan] max abs error 0.0078125 seqlen 32768 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-32768-4-scan] max abs error 0.0078125 seqlen 32768 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-32768-42-scan] max abs error 0.0078125 seqlen 32768 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-32768-57-scan] max abs error 0.0078125 seqlen 32768 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-65536-3407-scan] max abs error 0.0078125 seqlen 65536 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-65536-4-scan] max abs error 0.0078125 seqlen 65536 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-65536-42-scan] max abs error 0.0078125 seqlen 65536 dtype torch.float16
tests/test_eq.py::test_eq_forward[dtype2-65536-57-scan] max abs error 0.0078125 seqlen 65536 dtype torch.float16

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@proger