Fix unsigned int overflow in libjpeg memory mgr.

When attempting to decode a malformed JPEG image (refer to
https://bugzilla.mozilla.org/show_bug.cgi?id=1295044) with dimensions
61472 x 32800, the maximum_space variable within the
realize_virt_arrays() function will exceed the maximum value of a 32-bit
integer and will wrap around.  The memory manager subsequently fails
with an "Insufficient memory" error (case 4, in alloc_large()), so this
commit simply causes that error to be triggered earlier, before UBSan
has a chance to complain.

Note that this issue did not ever represent an exploitable security
threat, because the POSIX-based memory manager that we use doesn't ever
do anything meaningful with the value of maximum_space.
jpeg_mem_available() simply sets avail_mem = maximum_space, so the
subsequent behavior of the memory manager is the same regardless of
whether maximum_space is correct or not.  This commit simply removes a
UBSan warning in order to make it easier to detect actual security
issues.

ChangeLog.md

@@ -62,6 +62,13 @@ have 2x1 luminance and 1x1 chrominance sampling factors, and 4:4:0 JPEGs have
 1x2 luminance and 1x1 chrominance sampling factors), but the JPEG specification
 and the libjpeg API both allow it.
 
+7. Fixed an unsigned integer overflow in the libjpeg memory manager, detected
+by the Clang undefined behavior sanitizer, that could be triggered by
+attempting to decompress a specially-crafted malformed JPEG image.  This issue
+affected only 32-bit code and did not pose a security threat, but removing the
+warning makes it easier to detect actual security issues, should they arise in
+the future.
+
 
 1.5.0
 =====

jmemmgr.c

@@ -32,6 +32,7 @@
 #include "jinclude.h"
 #include "jpeglib.h"
 #include "jmemsys.h"            /* import the system-dependent declarations */
+#include <stdint.h>
 
 #ifndef NO_GETENV
 #ifndef HAVE_STDLIB_H           /* <stdlib.h> should declare getenv() */
@@ -650,18 +651,26 @@ realize_virt_arrays (j_common_ptr cinfo)
   maximum_space = 0;
   for (sptr = mem->virt_sarray_list; sptr != NULL; sptr = sptr->next) {
     if (sptr->mem_buffer == NULL) { /* if not realized yet */
+      size_t new_space = (long) sptr->rows_in_array *
+                         (long) sptr->samplesperrow * sizeof(JSAMPLE);
+
       space_per_minheight += (long) sptr->maxaccess *
                              (long) sptr->samplesperrow * sizeof(JSAMPLE);
-      maximum_space += (long) sptr->rows_in_array *
-                       (long) sptr->samplesperrow * sizeof(JSAMPLE);
+      if (SIZE_MAX - maximum_space < new_space)
+        out_of_memory(cinfo, 10);
+      maximum_space += new_space;
     }
   }
   for (bptr = mem->virt_barray_list; bptr != NULL; bptr = bptr->next) {
     if (bptr->mem_buffer == NULL) { /* if not realized yet */
+      size_t new_space = (long) bptr->rows_in_array *
+                         (long) bptr->blocksperrow * sizeof(JBLOCK);
+
       space_per_minheight += (long) bptr->maxaccess *
                              (long) bptr->blocksperrow * sizeof(JBLOCK);
-      maximum_space += (long) bptr->rows_in_array *
-                       (long) bptr->blocksperrow * sizeof(JBLOCK);
+      if (SIZE_MAX - maximum_space < new_space)
+        out_of_memory(cinfo, 11);
+      maximum_space += new_space;
     }
   }