GS: Clamp ST coordinates to valid range in GS vertex trace.

[TJnotJT]

Description of Changes

In the vertex trace, clamp floating point UV coordinates to valid UV ranges [-2047, 2047].

Rationale behind Changes

In the vertex trace, input float texture coordinates can sometimes be buggy (for reasons not totally clear, it might be emulation or game bugs), so they might be outside the valid ranges that the GS can process. Casting such values to integer can causes issues down stream with functions that use the vertex trace min/max texture coordinates, so this should fix some of those issues. In all the example I found, the issue was with SW renderer not loading a full texture (because it depends on vertex trace calulations) and thus sampling from garbage values.

Suggested Testing Steps

Testing on GS dumps/games that are known to have buggy/huge texture coordinates as described above. I didn't find anything that was obviously broken in testing, though any help testing on more games/dumps or feedback on the changes would be greatly appreciated.

Here are the most visible fixes that I could find in testing (before is left and after is right)

Should be a slight blurring effect (Counter_Terrorist_Special_Forces_-_Fire_for_Effect_SLES-53046_20230830195729.gs.xz).

Should not have double eyebrows (Bakugan_Battle_Brawlers_shadows.gs.xz).

Should have a faint star-like patterns (Psychonauts_SLUS-21120_20231013005852.gs.xz).

Should have character's shadow (Tomb_Raider_-_Legend_SLUS-21203_20230905130255.gs.xz).

Should not have dark road (testdriveswminmax.gs.xz).

[lightningterror]

TODO: On the dumps that I saw a difference should be checked on a ps2 to compare how they should look like.

[TJnotJT]

TODO: On the dumps that I saw a difference should be checked on a ps2 to compare how they should look like.

Lightning, would you be able to provide the dump name (if it is in gspack-10-july-2024)? I will try to work on this further given the new info from the hardware tests Ziemas ran.

[lightningterror]

007 Agent Under Fire - door blend
187 - Ride or Die_SLUS-21116_In_Game
2002 FIFA World Cup
50 Cent - Bulletproof_SLUS-21315_20230329231035
Ace Combat 04 - Shattered Skies_SLUS-20152_20230511225232
Ace_Combat_04_-_Shattered_Skies_SLUS-20152_20230502010138
AFL Premiership 2007_SCES-54639
Alfa Romeo
Ape Escape 2_SLUS-20685_20220914200448
Dave_Mirra_Freestyle_BMX_2_SLES-50217_20230213220044

There's more but the list is huge, ace combat 4, ape escape 2 dave mirra seem broken.

[TJnotJT]

Thanks much, this should keep me busy for some time

[TJnotJT]

I've fixed a few issues and the dump run seems to look good now on my end.