Merge 3.3

.gitignore

@@ -15,11 +15,13 @@ Doc/tools/jinja2/
 Doc/tools/pygments/
 Doc/tools/sphinx/
 Lib/lib2to3/*.pickle
+Lib/test/data/*
 Lib/_sysconfigdata.py
 Lib/plat-mac/errors.rsrc.df.rsrc
 Makefile
 Makefile.pre
 Misc/python.pc
+Misc/python-config.sh
 Modules/Setup
 Modules/Setup.config
 Modules/Setup.local
@@ -57,6 +59,8 @@ platform
 pybuilddir.txt
 pyconfig.h
 python
+python-config
+python-config.py
 python.exe
 python-gdb.py
 python.exe-gdb.py

.hgignore

@@ -18,6 +18,8 @@ platform$
 pyconfig.h$
 python$
 python.exe$
+python-config$
+python-config.py$
 reflog.txt$
 tags$
 Lib/plat-mac/errors.rsrc.df.rsrc
@@ -27,12 +29,14 @@ Doc/tools/jinja/
 Doc/tools/jinja2/
 Doc/tools/pygments/
 Misc/python.pc
+Misc/python-config.sh$
 Modules/Setup$
 Modules/Setup.config
 Modules/Setup.local
 Modules/config.c
 Modules/ld_so_aix$
 Parser/pgen$
+^lcov-report/
 ^core
 ^python-gdb.py
 ^python.exe-gdb.py
@@ -88,3 +92,7 @@ Modules/_testembed
 .coverage
 coverage/
 htmlcov/
+*.gcda
+*.gcno
+*.gcov
+coverage.info

.hgtouch

@@ -4,8 +4,8 @@
 
 Python/importlib.h: Lib/importlib/_bootstrap.py Modules/_freeze_importlib.c
 
-Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py
-Python/Python-ast.c: Include/ast.h
+Include/Python-ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py
+Python/Python-ast.c: Include/Python-ast.h
 
 Python/opcode_targets.h: Python/makeopcodetargets.py Lib/opcode.py
 

Doc/c-api/dict.rst

@@ -110,6 +110,15 @@ Dictionary Objects
    :c:type:`char\*`, rather than a :c:type:`PyObject\*`.
 
 
+.. c:function:: PyObject* PyDict_SetDefault(PyObject *p, PyObject *key, PyObject *default)
+
+   This is the same as the Python-level :meth:`dict.setdefault`.  If present, it
+   returns the value corresponding to *key* from the dictionary *p*.  If the key
+   is not in the dict, it is inserted with value *defaultobj* and *defaultobj*
+   is returned.  This function evaluates the hash function of *key* only once,
+   instead of evaluating it independently for the lookup and the insertion.
+
+
 .. c:function:: PyObject* PyDict_Items(PyObject *p)
 
    Return a :c:type:`PyListObject` containing all the items from the dictionary.

Doc/c-api/exceptions.rst

@@ -235,7 +235,7 @@ in various ways.  There is a separate error indicator for each thread.
 
    Similar to :c:func:`PyErr_SetFromErrnoWithFilenameObject`, but the filename
    is given as a C string.  *filename* is decoded from the filesystem encoding
-   (:func:`sys.getfilesystemencoding`).
+   (:func:`os.fsdecode`).
 
 
 .. c:function:: PyObject* PyErr_SetFromWindowsErr(int ierr)
@@ -267,7 +267,7 @@ in various ways.  There is a separate error indicator for each thread.
 
    Similar to :c:func:`PyErr_SetFromWindowsErrWithFilenameObject`, but the
    filename is given as a C string.  *filename* is decoded from the filesystem
-   encoding (:func:`sys.getfilesystemencoding`).  Availability: Windows.
+   encoding (:func:`os.fsdecode`).  Availability: Windows.
 
 
 .. c:function:: PyObject* PyErr_SetExcFromWindowsErrWithFilenameObject(PyObject *type, int ierr, PyObject *filename)
@@ -293,20 +293,27 @@ in various ways.  There is a separate error indicator for each thread.
    .. versionadded:: 3.3
 
 
-.. c:function:: void PyErr_SyntaxLocationEx(char *filename, int lineno, int col_offset)
+.. c:function:: void PyErr_SyntaxLocationObject(PyObject *filename, int lineno, int col_offset)
 
    Set file, line, and offset information for the current exception.  If the
    current exception is not a :exc:`SyntaxError`, then it sets additional
    attributes, which make the exception printing subsystem think the exception
-   is a :exc:`SyntaxError`. *filename* is decoded from the filesystem encoding
-   (:func:`sys.getfilesystemencoding`).
+   is a :exc:`SyntaxError`.
 
-   .. versionadded:: 3.2
+.. versionadded:: 3.4
+
+
+.. c:function:: void PyErr_SyntaxLocationEx(char *filename, int lineno, int col_offset)
+
+   Like :c:func:`PyErr_SyntaxLocationObject`, but *filename* is a byte string
+   decoded from the filesystem encoding (:func:`os.fsdecode`).
+
+.. versionadded:: 3.2
 
 
 .. c:function:: void PyErr_SyntaxLocation(char *filename, int lineno)
 
-   Like :c:func:`PyErr_SyntaxLocationExc`, but the col_offset parameter is
+   Like :c:func:`PyErr_SyntaxLocationEx`, but the col_offset parameter is
    omitted.
 
 
@@ -355,15 +362,22 @@ in various ways.  There is a separate error indicator for each thread.
    documentation.  There is no C API for warning control.
 
 
-.. c:function:: int PyErr_WarnExplicit(PyObject *category, const char *message, const char *filename, int lineno, const char *module, PyObject *registry)
+.. c:function:: int PyErr_WarnExplicitObject(PyObject *category, PyObject *message, PyObject *filename, int lineno, PyObject *module, PyObject *registry)
 
    Issue a warning message with explicit control over all warning attributes.  This
    is a straightforward wrapper around the Python function
    :func:`warnings.warn_explicit`, see there for more information.  The *module*
    and *registry* arguments may be set to *NULL* to get the default effect
-   described there. *message* and *module* are UTF-8 encoded strings,
-   *filename* is decoded from the filesystem encoding
-   (:func:`sys.getfilesystemencoding`).
+   described there.
+
+   .. versionadded:: 3.4
+
+
+.. c:function:: int PyErr_WarnExplicit(PyObject *category, const char *message, const char *filename, int lineno, const char *module, PyObject *registry)
+
+   Similar to :c:func:`PyErr_WarnExplicitObject` except that *message* and
+   *module* are UTF-8 encoded strings, and *filename* is decoded from the
+   filesystem encoding (:func:`os.fsdecode`).
 
 
 .. c:function:: int PyErr_WarnFormat(PyObject *category, Py_ssize_t stack_level, const char *format, ...)

Doc/c-api/init.rst

@@ -329,7 +329,11 @@ Process-wide parameters
 
 .. c:function:: void PySys_SetArgv(int argc, wchar_t **argv)
 
-   This function works like :c:func:`PySys_SetArgvEx` with *updatepath* set to 1.
+   This function works like :c:func:`PySys_SetArgvEx` with *updatepath* set
+   to 1 unless the :program:`python` interpreter was started with the
+   :option:`-I`.
+
+   .. versionchanged:: 3.4 The *updatepath* value depends on :option:`-I`.
 
 
 .. c:function:: void Py_SetPythonHome(wchar_t *home)
@@ -657,6 +661,20 @@ with sub-interpreters:
    made on the main thread.  This is mainly a helper/diagnostic function.
 
 
+.. c:function:: int PyGILState_Check()
+
+   Return 1 if the current thread is holding the GIL and 0 otherwise.
+   This function can be called from any thread at any time.
+   Only if it has had its Python thread state initialized and currently is
+   holding the GIL will it return 1.
+   This is mainly a helper/diagnostic function.  It can be useful
+   for example in callback contexts or memory allocation functions when
+   knowing that the GIL is locked can allow the caller to perform sensitive
+   actions or otherwise behave differently.
+
+   .. versionadded:: 3.4
+
+
 The following macros are normally used without a trailing semicolon; look for
 example usage in the Python source distribution.
 

Doc/c-api/memory.rst

@@ -84,15 +84,65 @@ the C library allocator as shown in the previous example, the allocated memory
 for the I/O buffer escapes completely the Python memory manager.
 
 
+Raw Memory Interface
+====================
+
+The following function sets are wrappers to the system allocator. These
+functions are thread-safe, the :term:`GIL <global interpreter lock>` does not
+need to be held.
+
+The default raw memory block allocator uses the following functions:
+:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when
+requesting zero bytes.
+
+.. versionadded:: 3.4
+
+.. c:function:: void* PyMem_RawMalloc(size_t n)
+
+   Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
+   allocated memory, or *NULL* if the request fails. Requesting zero bytes
+   returns a distinct non-*NULL* pointer if possible, as if
+   ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have
+   been initialized in any way.
+
+
+.. c:function:: void* PyMem_RawRealloc(void *p, size_t n)
+
+   Resizes the memory block pointed to by *p* to *n* bytes. The contents will
+   be unchanged to the minimum of the old and the new sizes. If *p* is *NULL*,
+   the call is equivalent to ``PyMem_RawMalloc(n)``; else if *n* is equal to
+   zero, the memory block is resized but is not freed, and the returned pointer
+   is non-*NULL*. Unless *p* is *NULL*, it must have been returned by a
+   previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`. If
+   the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p* remains
+   a valid pointer to the previous memory area.
+
+
+.. c:function:: void PyMem_RawFree(void *p)
+
+   Frees the memory block pointed to by *p*, which must have been returned by a
+   previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`.
+   Otherwise, or if ``PyMem_Free(p)`` has been called before, undefined
+   behavior occurs. If *p* is *NULL*, no operation is performed.
+
+
 .. _memoryinterface:
 
 Memory Interface
 ================
 
 The following function sets, modeled after the ANSI C standard, but specifying
 behavior when requesting zero bytes, are available for allocating and releasing
-memory from the Python heap:
+memory from the Python heap.
+
+The default memory block allocator uses the following functions:
+:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when
+requesting zero bytes.
+
+.. warning::
 
+   The :term:`GIL <global interpreter lock>` must be held when using these
+   functions.
 
 .. c:function:: void* PyMem_Malloc(size_t n)
 
@@ -155,6 +205,125 @@ versions and is therefore deprecated in extension modules.
 :c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`.
 
 
+Customize Memory Allocators
+===========================
+
+.. versionadded:: 3.4
+
+.. c:type:: PyMemAllocator
+
+   Structure used to describe a memory block allocator. The structure has
+   four fields:
+
+   +----------------------------------------------------------+---------------------------------------+
+   | Field                                                    | Meaning                               |
+   +==========================================================+=======================================+
+   | ``void *ctx``                                            | user context passed as first argument |
+   +----------------------------------------------------------+---------------------------------------+
+   | ``void* malloc(void *ctx, size_t size)``                 | allocate a memory block               |
+   +----------------------------------------------------------+---------------------------------------+
+   | ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block     |
+   +----------------------------------------------------------+---------------------------------------+
+   | ``void free(void *ctx, void *ptr)``                      | free a memory block                   |
+   +----------------------------------------------------------+---------------------------------------+
+
+.. c:type:: PyMemAllocatorDomain
+
+   Enum used to identify an allocator domain. Domains:
+
+   * :c:data:`PYMEM_DOMAIN_RAW`: functions :c:func:`PyMem_RawMalloc`,
+     :c:func:`PyMem_RawRealloc` and :c:func:`PyMem_RawFree`
+   * :c:data:`PYMEM_DOMAIN_MEM`: functions :c:func:`PyMem_Malloc`,
+     :c:func:`PyMem_Realloc` and :c:func:`PyMem_Free`
+   * :c:data:`PYMEM_DOMAIN_OBJ`: functions :c:func:`PyObject_Malloc`,
+     :c:func:`PyObject_Realloc` and :c:func:`PyObject_Free`
+
+
+.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
+
+   Get the memory block allocator of the specified domain.
+
+
+.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
+
+   Set the memory block allocator of the specified domain.
+
+   The new allocator must return a distinct non-NULL pointer when requesting
+   zero bytes.
+
+   For the :c:data:`PYMEM_DOMAIN_RAW` domain, the allocator must be
+   thread-safe: the :term:`GIL <global interpreter lock>` is not held when the
+   allocator is called.
+
+   If the new allocator is not a hook (does not call the previous allocator),
+   the :c:func:`PyMem_SetupDebugHooks` function must be called to reinstall the
+   debug hooks on top on the new allocator.
+
+
+.. c:function:: void PyMem_SetupDebugHooks(void)
+
+   Setup hooks to detect bugs in the following Python memory allocator
+   functions:
+
+   - :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`,
+     :c:func:`PyMem_RawFree`
+   - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Free`
+   - :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`,
+     :c:func:`PyObject_Free`
+
+   Newly allocated memory is filled with the byte ``0xCB``, freed memory is
+   filled with the byte ``0xDB``. Additionnal checks:
+
+   - detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
+     allocated by :c:func:`PyMem_Malloc`
+   - detect write before the start of the buffer (buffer underflow)
+   - detect write after the end of the buffer (buffer overflow)
+
+   The function does nothing if Python is not compiled is debug mode.
+
+
+Customize PyObject Arena Allocator
+==================================
+
+Python has a *pymalloc* allocator for allocations smaller than 512 bytes. This
+allocator is optimized for small objects with a short lifetime. It uses memory
+mappings called "arenas" with a fixed size of 256 KB. It falls back to
+:c:func:`PyMem_Malloc` and :c:func:`PyMem_Realloc` for allocations larger than
+512 bytes.  *pymalloc* is the default allocator used by
+:c:func:`PyObject_Malloc`.
+
+The default arena allocator uses the following functions:
+
+* :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows,
+* :c:func:`mmap` and :c:func:`munmap` if available,
+* :c:func:`malloc` and :c:func:`free` otherwise.
+
+.. versionadded:: 3.4
+
+.. c:type:: PyObjectArenaAllocator
+
+   Structure used to describe an arena allocator. The structure has
+   three fields:
+
+   +--------------------------------------------------+---------------------------------------+
+   | Field                                            | Meaning                               |
+   +==================================================+=======================================+
+   | ``void *ctx``                                    | user context passed as first argument |
+   +--------------------------------------------------+---------------------------------------+
+   | ``void* alloc(void *ctx, size_t size)``          | allocate an arena of size bytes       |
+   +--------------------------------------------------+---------------------------------------+
+   | ``void free(void *ctx, size_t size, void *ptr)`` | free an arena                         |
+   +--------------------------------------------------+---------------------------------------+
+
+.. c:function:: PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)
+
+   Get the arena allocator.
+
+.. c:function:: PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)
+
+   Set the arena allocator.
+
+
 .. _memoryexamples:
 
 Examples