You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In this profile, the call tree displays "Total Size (bytes): 485,408,992" for the root node.
But if you run filteredThread.samples.weight.reduce((a, b) => a + b, 0) on the console, you get a sum of 485,409,290.
Those two values should be the same but they aren't. One ends in 8,992 (incorrect) and the other ends in 9,290 (correct).
The text was updated successfully, but these errors were encountered:
mstange
changed the title
Float32Array loses precision in call tree timings - incorrect byte sums displayed in call tree
Float32Array has insufficient precision in call tree timings - incorrect byte sums displayed in call tree
Jan 13, 2025
Profile: https://share.firefox.dev/40hmh0v
In this profile, the call tree displays "Total Size (bytes): 485,408,992" for the root node.
But if you run
filteredThread.samples.weight.reduce((a, b) => a + b, 0)
on the console, you get a sum of 485,409,290.Those two values should be the same but they aren't. One ends in 8,992 (incorrect) and the other ends in 9,290 (correct).
This happens because the call tree timings store the total value per node in a Float32Array. But 32-bit floating point values can only exactly represent integers up to 16,777,216.
I must have chosen Float32Array when I wasn't thinking of byte values.
┆Issue is synchronized with this Jira Task
The text was updated successfully, but these errors were encountered: