Honor max_memory_to_use/JPEGMEM/-maxmemory

This re-introduces a feature of the obsolete system-specific libjpeg
memory managers-- namely the ability to limit the amount of main memory
used by the library during decompression or multi-pass compression.
This is mainly beneficial for two reasons:

- Works around a 2 GB limit in libFuzzer
- Allows security-sensitive applications to set a memory limit for the
  JPEG decoder so as to work around the progressive JPEG exploit
  (LJT-01-004) described here:
  http://www.libjpeg-turbo.org/pmwiki/uploads/About/TwoIssueswiththeJPEGStandard.pdf

This commit also removes obsolete documentation regarding the MS-DOS
memory manager (which itself was removed long ago) and changes the
documentation of the -maxmemory switch and JPEGMEM environment variable
to reflect the fact that backing stores are never used in libjpeg-turbo.

Inspired by:
https://github.com/caolanm/libjpeg-turbo/commit/066fee2e7d6834f24838bc1896aa38ca77209e3c

Closes libjpeg-turbo#143

ChangeLog.md

@@ -44,6 +44,21 @@ if the library was built with certain compilers and optimization levels
 GCC 5.x or 6.x) and one of the underlying libjpeg API functions threw an error
 after a TurboJPEG API function allocated a local buffer.
 
+9. The libjpeg-turbo memory manager will now honor the `max_memory_to_use`
+structure member in jpeg\_memory\_mgr, which can be set to the maximum amount
+of memory (in bytes) that libjpeg-turbo should use during decompression or
+multi-pass (including progressive) compression.  This limit can also be set
+using the `JPEGMEM` environment variable or using the `-maxmemory` switch in
+cjpeg/djpeg/jpegtran (refer to the respective man pages for more details.)
+This has been a documented feature of libjpeg since v5, but the
+`malloc()`/`free()` implementation of the memory manager (jmemnobs.c) never
+implemented the feature.  Restricting libjpeg-turbo's memory usage is useful
+for two reasons:  it allows testers to more easily work around the 2 GB limit
+in libFuzzer, and it allows developers of security-sensitive applications to
+more easily defend against one of the progressive JPEG exploits (LJT-01-004)
+identified in
+[this report](http://www.libjpeg-turbo.org/pmwiki/uploads/About/TwoIssueswiththeJPEGStandard.pdf).
+
 
 1.5.1
 =====

cjpeg.1

@@ -1,4 +1,4 @@
-.TH CJPEG 1 "17 February 2016"
+.TH CJPEG 1 "18 March 2017"
 .SH NAME
 cjpeg \- compress an image file to a JPEG file
 .SH SYNOPSIS
@@ -202,7 +202,7 @@ Set limit for amount of memory to use in processing large images.  Value is
 in thousands of bytes, or millions of bytes if "M" is attached to the
 number.  For example,
 .B \-max 4m
-selects 4000000 bytes.  If more space is needed, temporary files will be used.
+selects 4000000 bytes.  If more space is needed, an error will occur.
 .TP
 .BI \-outfile " name"
 Send output image to the named file, not to standard output.

djpeg.1

@@ -1,4 +1,4 @@
-.TH DJPEG 1 "18 February 2016"
+.TH DJPEG 1 "18 March 2017"
 .SH NAME
 djpeg \- decompress a JPEG file to an image file
 .SH SYNOPSIS
@@ -185,7 +185,7 @@ Set limit for amount of memory to use in processing large images.  Value is
 in thousands of bytes, or millions of bytes if "M" is attached to the
 number.  For example,
 .B \-max 4m
-selects 4000000 bytes.  If more space is needed, temporary files will be used.
+selects 4000000 bytes.  If more space is needed, an error will occur.
 .TP
 .BI \-outfile " name"
 Send output image to the named file, not to standard output.

jmemnobs.c

@@ -3,8 +3,8 @@
  *
  * This file was part of the Independent JPEG Group's software:
  * Copyright (C) 1992-1996, Thomas G. Lane.
- * It was modified by The libjpeg-turbo Project to include only code and
- * information relevant to libjpeg-turbo.
+ * libjpeg-turbo Modifications:
+ * Copyright (C) 2017, D. R. Commander.
  * For conditions of distribution and use, see the accompanying README.ijg
  * file.
  *
@@ -15,7 +15,6 @@
  * This is very portable in the sense that it'll compile on almost anything,
  * but you'd better have lots of main memory (or virtual memory) if you want
  * to process big images.
- * Note that the max_memory_to_use option is ignored by this implementation.
  */
 
 #define JPEG_INTERNALS
@@ -66,14 +65,21 @@ jpeg_free_large (j_common_ptr cinfo, void *object, size_t sizeofobject)
 
 /*
  * This routine computes the total memory space available for allocation.
- * Here we always say, "we got all you want bud!"
  */
 
 GLOBAL(size_t)
 jpeg_mem_available (j_common_ptr cinfo, size_t min_bytes_needed,
                     size_t max_bytes_needed, size_t already_allocated)
 {
-  return max_bytes_needed;
+  if (cinfo->mem->max_memory_to_use) {
+    if (cinfo->mem->max_memory_to_use > already_allocated)
+      return cinfo->mem->max_memory_to_use - already_allocated;
+    else
+      return 0;
+  } else {
+    /* Here we always say, "we got all you want bud!" */
+    return max_bytes_needed;
+  }
 }
 
 

jpegtran.1

@@ -1,4 +1,4 @@
-.TH JPEGTRAN 1 "18 February 2016"
+.TH JPEGTRAN 1 "18 March 2017"
 .SH NAME
 jpegtran \- lossless transformation of JPEG files
 .SH SYNOPSIS
@@ -222,7 +222,7 @@ Set limit for amount of memory to use in processing large images.  Value is
 in thousands of bytes, or millions of bytes if "M" is attached to the
 number.  For example,
 .B \-max 4m
-selects 4000000 bytes.  If more space is needed, temporary files will be used.
+selects 4000000 bytes.  If more space is needed, an error will occur.
 .TP
 .BI \-outfile " name"
 Send output image to the named file, not to standard output.

libjpeg.txt

@@ -3,7 +3,7 @@ USING THE IJG JPEG LIBRARY
 This file was part of the Independent JPEG Group's software:
 Copyright (C) 1994-2013, Thomas G. Lane, Guido Vollbeding.
 libjpeg-turbo Modifications:
-Copyright (C) 2010, 2014-2016, D. R. Commander.
+Copyright (C) 2010, 2014-2017, D. R. Commander.
 Copyright (C) 2015, Google, Inc.
 For conditions of distribution and use, see the accompanying README.ijg file.
 
@@ -2942,13 +2942,6 @@ Some operating modes (eg, two-pass color quantization) require full-image
 buffers.  Such buffers are treated as "virtual arrays": only the current strip
 need be in memory, and the rest can be swapped out to a temporary file.
 
-If you use the simplest memory manager back end (jmemnobs.c), then no
-temporary files are used; virtual arrays are simply malloc()'d.  Images bigger
-than memory can be processed only if your system supports virtual memory.
-The other memory manager back ends support temporary files of various flavors
-and thus work in machines without virtual memory.  They may also be useful on
-Unix machines if you need to process images that exceed available swap space.
-
 When using temporary files, the library will make the in-memory buffers for
 its virtual arrays just big enough to stay within a "maximum memory" setting.
 Your application can set this limit by setting cinfo->mem->max_memory_to_use
@@ -2961,6 +2954,11 @@ that space allocated with alloc_small() is ignored, on the assumption that
 it's too small to be worth worrying about; so a reasonable safety margin
 should be left when setting max_memory_to_use.
 
+NOTE: Unless you develop your own memory manager back end, then temporary files
+will never be used.  The back end provided in libjpeg-turbo (jmemnobs.c) simply
+malloc()s and free()s virtual arrays, and an error occurs if the required
+memory exceeds the limit specified in cinfo->mem->max_memory_to_use.
+
 
 Memory usage
 ------------

structure.txt

@@ -832,21 +832,19 @@ read_backing_store,             manipulate a backing-store object
 write_backing_store,
 close_backing_store
 
-On some systems there will be more than one type of backing-store object
-(specifically, in MS-DOS a backing store file might be an area of extended
-memory as well as a disk file).  jpeg_open_backing_store is responsible for
-choosing how to implement a given object.  The read/write/close routines
-are method pointers in the structure that describes a given object; this
-lets them be different for different object types.
+On some systems there will be more than one type of backing-store object.
+jpeg_open_backing_store is responsible for choosing how to implement a given
+object.  The read/write/close routines are method pointers in the structure
+that describes a given object; this lets them be different for different object
+types.
 
 It may be necessary to ensure that backing store objects are explicitly
-released upon abnormal program termination.  For example, MS-DOS won't free
-extended memory by itself.  To support this, we will expect the main program
-or surrounding application to arrange to call self_destruct (typically via
-jpeg_destroy) upon abnormal termination.  This may require a SIGINT signal
-handler or equivalent.  We don't want to have the back end module install its
-own signal handler, because that would pre-empt the surrounding application's
-ability to control signal handling.
+released upon abnormal program termination.  To support this, we will expect
+the main program or surrounding application to arrange to call self_destruct
+(typically via jpeg_destroy) upon abnormal termination.  This may require a
+SIGINT signal handler or equivalent.  We don't want to have the back end module
+install its own signal handler, because that would pre-empt the surrounding
+application's ability to control signal handling.
 
 The IJG distribution includes several memory manager back end implementations.
 Usually the same back end should be suitable for all applications on a given

usage.txt

@@ -212,7 +212,7 @@ Switches for advanced users:
                         large images.  Value is in thousands of bytes, or
                         millions of bytes if "M" is attached to the number.
                         For example, -max 4m selects 4000000 bytes.  If more
-                        space is needed, temporary files will be used.
+                        space is needed, an error will occur.
 
         -verbose        Enable debug printout.  More -v's give more printout.
         or  -debug      Also, version information is printed at startup.
@@ -377,7 +377,7 @@ Switches for advanced users:
                         large images.  Value is in thousands of bytes, or
                         millions of bytes if "M" is attached to the number.
                         For example, -max 4m selects 4000000 bytes.  If more
-                        space is needed, temporary files will be used.
+                        space is needed, an error will occur.
 
         -verbose        Enable debug printout.  More -v's give more printout.
         or  -debug      Also, version information is printed at startup.
@@ -423,48 +423,23 @@ When producing a color-quantized image, "-onepass -dither ordered" is fast but
 much lower quality than the default behavior.  "-dither none" may give
 acceptable results in two-pass mode, but is seldom tolerable in one-pass mode.
 
-Two-pass color quantization requires a good deal of memory; on MS-DOS machines
-it may run out of memory even with -maxmemory 0.  In that case you can still
-decompress, with some loss of image quality, by specifying -onepass for
-one-pass quantization.
-
 To avoid the Unisys LZW patent (now expired), djpeg produces uncompressed GIF
 files.  These are larger than they should be, but are readable by standard GIF
 decoders.
 
 
 HINTS FOR BOTH PROGRAMS
 
-If more space is needed than will fit in the available main memory (as
-determined by -maxmemory), temporary files will be used.  (MS-DOS versions
-will try to get extended or expanded memory first.)  The temporary files are
-often rather large: in typical cases they occupy three bytes per pixel, for
-example 3*800*600 = 1.44Mb for an 800x600 image.  If you don't have enough
-free disk space, leave out -progressive and -optimize (for cjpeg) or specify
--onepass (for djpeg).
-
-On MS-DOS, the temporary files are created in the directory named by the TMP
-or TEMP environment variable, or in the current directory if neither of those
-exist.  Amiga implementations put the temp files in the directory named by
-JPEGTMP:, so be sure to assign JPEGTMP: to a disk partition with adequate free
-space.
-
-The default memory usage limit (-maxmemory) is set when the software is
-compiled.  If you get an "insufficient memory" error, try specifying a smaller
--maxmemory value, even -maxmemory 0 to use the absolute minimum space.  You
-may want to recompile with a smaller default value if this happens often.
+If the memory needed by cjpeg or djpeg exceeds the limit specified by
+-maxmemory, an error will occur.  You can leave out -progressive and -optimize
+(for cjpeg) or specify -onepass (for djpeg) to reduce memory usage.
 
 On machines that have "environment" variables, you can define the environment
 variable JPEGMEM to set the default memory limit.  The value is specified as
 described for the -maxmemory switch.  JPEGMEM overrides the default value
 specified when the program was compiled, and itself is overridden by an
 explicit -maxmemory switch.
 
-On MS-DOS machines, -maxmemory is the amount of main (conventional) memory to
-use.  (Extended or expanded memory is also used if available.)  Most
-DOS-specific versions of this software do their own memory space estimation
-and do not need you to specify -maxmemory.
-
 
 JPEGTRAN