diff --git a/ChangeLog.md b/ChangeLog.md index 620c2f8e4..07a6cf526 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -44,6 +44,21 @@ if the library was built with certain compilers and optimization levels GCC 5.x or 6.x) and one of the underlying libjpeg API functions threw an error after a TurboJPEG API function allocated a local buffer. +9. The libjpeg-turbo memory manager will now honor the `max_memory_to_use` +structure member in jpeg\_memory\_mgr, which can be set to the maximum amount +of memory (in bytes) that libjpeg-turbo should use during decompression or +multi-pass (including progressive) compression. This limit can also be set +using the `JPEGMEM` environment variable or using the `-maxmemory` switch in +cjpeg/djpeg/jpegtran (refer to the respective man pages for more details.) +This has been a documented feature of libjpeg since v5, but the +`malloc()`/`free()` implementation of the memory manager (jmemnobs.c) never +implemented the feature. Restricting libjpeg-turbo's memory usage is useful +for two reasons: it allows testers to more easily work around the 2 GB limit +in libFuzzer, and it allows developers of security-sensitive applications to +more easily defend against one of the progressive JPEG exploits (LJT-01-004) +identified in +[this report](http://www.libjpeg-turbo.org/pmwiki/uploads/About/TwoIssueswiththeJPEGStandard.pdf). + 1.5.1 ===== diff --git a/cjpeg.1 b/cjpeg.1 index d1dc30412..283fc81f3 100644 --- a/cjpeg.1 +++ b/cjpeg.1 @@ -1,4 +1,4 @@ -.TH CJPEG 1 "17 February 2016" +.TH CJPEG 1 "18 March 2017" .SH NAME cjpeg \- compress an image file to a JPEG file .SH SYNOPSIS @@ -202,7 +202,7 @@ Set limit for amount of memory to use in processing large images. Value is in thousands of bytes, or millions of bytes if "M" is attached to the number. For example, .B \-max 4m -selects 4000000 bytes. If more space is needed, temporary files will be used. +selects 4000000 bytes. If more space is needed, an error will occur. .TP .BI \-outfile " name" Send output image to the named file, not to standard output. diff --git a/djpeg.1 b/djpeg.1 index 7efde43b4..b1e7f76e9 100644 --- a/djpeg.1 +++ b/djpeg.1 @@ -1,4 +1,4 @@ -.TH DJPEG 1 "18 February 2016" +.TH DJPEG 1 "18 March 2017" .SH NAME djpeg \- decompress a JPEG file to an image file .SH SYNOPSIS @@ -185,7 +185,7 @@ Set limit for amount of memory to use in processing large images. Value is in thousands of bytes, or millions of bytes if "M" is attached to the number. For example, .B \-max 4m -selects 4000000 bytes. If more space is needed, temporary files will be used. +selects 4000000 bytes. If more space is needed, an error will occur. .TP .BI \-outfile " name" Send output image to the named file, not to standard output. diff --git a/jmemnobs.c b/jmemnobs.c index 5797198de..ac12afa51 100644 --- a/jmemnobs.c +++ b/jmemnobs.c @@ -3,8 +3,8 @@ * * This file was part of the Independent JPEG Group's software: * Copyright (C) 1992-1996, Thomas G. Lane. - * It was modified by The libjpeg-turbo Project to include only code and - * information relevant to libjpeg-turbo. + * libjpeg-turbo Modifications: + * Copyright (C) 2017, D. R. Commander. * For conditions of distribution and use, see the accompanying README.ijg * file. * @@ -15,7 +15,6 @@ * This is very portable in the sense that it'll compile on almost anything, * but you'd better have lots of main memory (or virtual memory) if you want * to process big images. - * Note that the max_memory_to_use option is ignored by this implementation. */ #define JPEG_INTERNALS @@ -66,14 +65,21 @@ jpeg_free_large (j_common_ptr cinfo, void *object, size_t sizeofobject) /* * This routine computes the total memory space available for allocation. - * Here we always say, "we got all you want bud!" */ GLOBAL(size_t) jpeg_mem_available (j_common_ptr cinfo, size_t min_bytes_needed, size_t max_bytes_needed, size_t already_allocated) { - return max_bytes_needed; + if (cinfo->mem->max_memory_to_use) { + if (cinfo->mem->max_memory_to_use > already_allocated) + return cinfo->mem->max_memory_to_use - already_allocated; + else + return 0; + } else { + /* Here we always say, "we got all you want bud!" */ + return max_bytes_needed; + } } diff --git a/jpegtran.1 b/jpegtran.1 index 7f3c8531e..631455b64 100644 --- a/jpegtran.1 +++ b/jpegtran.1 @@ -1,4 +1,4 @@ -.TH JPEGTRAN 1 "18 February 2016" +.TH JPEGTRAN 1 "18 March 2017" .SH NAME jpegtran \- lossless transformation of JPEG files .SH SYNOPSIS @@ -222,7 +222,7 @@ Set limit for amount of memory to use in processing large images. Value is in thousands of bytes, or millions of bytes if "M" is attached to the number. For example, .B \-max 4m -selects 4000000 bytes. If more space is needed, temporary files will be used. +selects 4000000 bytes. If more space is needed, an error will occur. .TP .BI \-outfile " name" Send output image to the named file, not to standard output. diff --git a/libjpeg.txt b/libjpeg.txt index 2aa102789..5181afc03 100644 --- a/libjpeg.txt +++ b/libjpeg.txt @@ -3,7 +3,7 @@ USING THE IJG JPEG LIBRARY This file was part of the Independent JPEG Group's software: Copyright (C) 1994-2013, Thomas G. Lane, Guido Vollbeding. libjpeg-turbo Modifications: -Copyright (C) 2010, 2014-2016, D. R. Commander. +Copyright (C) 2010, 2014-2017, D. R. Commander. Copyright (C) 2015, Google, Inc. For conditions of distribution and use, see the accompanying README.ijg file. @@ -2942,13 +2942,6 @@ Some operating modes (eg, two-pass color quantization) require full-image buffers. Such buffers are treated as "virtual arrays": only the current strip need be in memory, and the rest can be swapped out to a temporary file. -If you use the simplest memory manager back end (jmemnobs.c), then no -temporary files are used; virtual arrays are simply malloc()'d. Images bigger -than memory can be processed only if your system supports virtual memory. -The other memory manager back ends support temporary files of various flavors -and thus work in machines without virtual memory. They may also be useful on -Unix machines if you need to process images that exceed available swap space. - When using temporary files, the library will make the in-memory buffers for its virtual arrays just big enough to stay within a "maximum memory" setting. Your application can set this limit by setting cinfo->mem->max_memory_to_use @@ -2961,6 +2954,11 @@ that space allocated with alloc_small() is ignored, on the assumption that it's too small to be worth worrying about; so a reasonable safety margin should be left when setting max_memory_to_use. +NOTE: Unless you develop your own memory manager back end, then temporary files +will never be used. The back end provided in libjpeg-turbo (jmemnobs.c) simply +malloc()s and free()s virtual arrays, and an error occurs if the required +memory exceeds the limit specified in cinfo->mem->max_memory_to_use. + Memory usage ------------ diff --git a/structure.txt b/structure.txt index 296d1250f..f69c9d87e 100644 --- a/structure.txt +++ b/structure.txt @@ -832,21 +832,19 @@ read_backing_store, manipulate a backing-store object write_backing_store, close_backing_store -On some systems there will be more than one type of backing-store object -(specifically, in MS-DOS a backing store file might be an area of extended -memory as well as a disk file). jpeg_open_backing_store is responsible for -choosing how to implement a given object. The read/write/close routines -are method pointers in the structure that describes a given object; this -lets them be different for different object types. +On some systems there will be more than one type of backing-store object. +jpeg_open_backing_store is responsible for choosing how to implement a given +object. The read/write/close routines are method pointers in the structure +that describes a given object; this lets them be different for different object +types. It may be necessary to ensure that backing store objects are explicitly -released upon abnormal program termination. For example, MS-DOS won't free -extended memory by itself. To support this, we will expect the main program -or surrounding application to arrange to call self_destruct (typically via -jpeg_destroy) upon abnormal termination. This may require a SIGINT signal -handler or equivalent. We don't want to have the back end module install its -own signal handler, because that would pre-empt the surrounding application's -ability to control signal handling. +released upon abnormal program termination. To support this, we will expect +the main program or surrounding application to arrange to call self_destruct +(typically via jpeg_destroy) upon abnormal termination. This may require a +SIGINT signal handler or equivalent. We don't want to have the back end module +install its own signal handler, because that would pre-empt the surrounding +application's ability to control signal handling. The IJG distribution includes several memory manager back end implementations. Usually the same back end should be suitable for all applications on a given diff --git a/usage.txt b/usage.txt index 5abda4ede..ed97aa987 100644 --- a/usage.txt +++ b/usage.txt @@ -212,7 +212,7 @@ Switches for advanced users: large images. Value is in thousands of bytes, or millions of bytes if "M" is attached to the number. For example, -max 4m selects 4000000 bytes. If more - space is needed, temporary files will be used. + space is needed, an error will occur. -verbose Enable debug printout. More -v's give more printout. or -debug Also, version information is printed at startup. @@ -377,7 +377,7 @@ Switches for advanced users: large images. Value is in thousands of bytes, or millions of bytes if "M" is attached to the number. For example, -max 4m selects 4000000 bytes. If more - space is needed, temporary files will be used. + space is needed, an error will occur. -verbose Enable debug printout. More -v's give more printout. or -debug Also, version information is printed at startup. @@ -423,11 +423,6 @@ When producing a color-quantized image, "-onepass -dither ordered" is fast but much lower quality than the default behavior. "-dither none" may give acceptable results in two-pass mode, but is seldom tolerable in one-pass mode. -Two-pass color quantization requires a good deal of memory; on MS-DOS machines -it may run out of memory even with -maxmemory 0. In that case you can still -decompress, with some loss of image quality, by specifying -onepass for -one-pass quantization. - To avoid the Unisys LZW patent (now expired), djpeg produces uncompressed GIF files. These are larger than they should be, but are readable by standard GIF decoders. @@ -435,24 +430,9 @@ decoders. HINTS FOR BOTH PROGRAMS -If more space is needed than will fit in the available main memory (as -determined by -maxmemory), temporary files will be used. (MS-DOS versions -will try to get extended or expanded memory first.) The temporary files are -often rather large: in typical cases they occupy three bytes per pixel, for -example 3*800*600 = 1.44Mb for an 800x600 image. If you don't have enough -free disk space, leave out -progressive and -optimize (for cjpeg) or specify --onepass (for djpeg). - -On MS-DOS, the temporary files are created in the directory named by the TMP -or TEMP environment variable, or in the current directory if neither of those -exist. Amiga implementations put the temp files in the directory named by -JPEGTMP:, so be sure to assign JPEGTMP: to a disk partition with adequate free -space. - -The default memory usage limit (-maxmemory) is set when the software is -compiled. If you get an "insufficient memory" error, try specifying a smaller --maxmemory value, even -maxmemory 0 to use the absolute minimum space. You -may want to recompile with a smaller default value if this happens often. +If the memory needed by cjpeg or djpeg exceeds the limit specified by +-maxmemory, an error will occur. You can leave out -progressive and -optimize +(for cjpeg) or specify -onepass (for djpeg) to reduce memory usage. On machines that have "environment" variables, you can define the environment variable JPEGMEM to set the default memory limit. The value is specified as @@ -460,11 +440,6 @@ described for the -maxmemory switch. JPEGMEM overrides the default value specified when the program was compiled, and itself is overridden by an explicit -maxmemory switch. -On MS-DOS machines, -maxmemory is the amount of main (conventional) memory to -use. (Extended or expanded memory is also used if available.) Most -DOS-specific versions of this software do their own memory space estimation -and do not need you to specify -maxmemory. - JPEGTRAN