Issue #15494: merge from 3.3

.gitignore

@@ -15,11 +15,13 @@ Doc/tools/jinja2/
 Doc/tools/pygments/
 Doc/tools/sphinx/
 Lib/lib2to3/*.pickle
+Lib/test/data/*
 Lib/_sysconfigdata.py
 Lib/plat-mac/errors.rsrc.df.rsrc
 Makefile
 Makefile.pre
 Misc/python.pc
+Misc/python-config.sh
 Modules/Setup
 Modules/Setup.config
 Modules/Setup.local
@@ -57,6 +59,8 @@ platform
 pybuilddir.txt
 pyconfig.h
 python
+python-config
+python-config.py
 python.exe
 python-gdb.py
 python.exe-gdb.py

.hgignore

@@ -18,6 +18,8 @@ platform$
 pyconfig.h$
 python$
 python.exe$
+python-config$
+python-config.py$
 reflog.txt$
 tags$
 Lib/plat-mac/errors.rsrc.df.rsrc
@@ -27,6 +29,7 @@ Doc/tools/jinja/
 Doc/tools/jinja2/
 Doc/tools/pygments/
 Misc/python.pc
+Misc/python-config.sh$
 Modules/Setup$
 Modules/Setup.config
 Modules/Setup.local

Doc/c-api/dict.rst

@@ -110,6 +110,15 @@ Dictionary Objects
    :c:type:`char\*`, rather than a :c:type:`PyObject\*`.
 
 
+.. c:function:: PyObject* PyDict_SetDefault(PyObject *p, PyObject *key, PyObject *default)
+
+   This is the same as the Python-level :meth:`dict.setdefault`.  If present, it
+   returns the value corresponding to *key* from the dictionary *p*.  If the key
+   is not in the dict, it is inserted with value *defaultobj* and *defaultobj*
+   is returned.  This function evaluates the hash function of *key* only once,
+   instead of evaluating it independently for the lookup and the insertion.
+
+
 .. c:function:: PyObject* PyDict_Items(PyObject *p)
 
    Return a :c:type:`PyListObject` containing all the items from the dictionary.

Doc/c-api/init.rst

@@ -654,6 +654,20 @@ with sub-interpreters:
    made on the main thread.  This is mainly a helper/diagnostic function.
 
 
+.. c:function:: int PyGILState_Check()
+
+   Return 1 if the current thread is holding the GIL and 0 otherwise.
+   This function can be called from any thread at any time.
+   Only if it has had its Python thread state initialized and currently is
+   holding the GIL will it return 1.
+   This is mainly a helper/diagnostic function.  It can be useful
+   for example in callback contexts or memory allocation functions when
+   knowing that the GIL is locked can allow the caller to perform sensitive
+   actions or otherwise behave differently.
+
+   .. versionadded:: 3.4
+
+
 The following macros are normally used without a trailing semicolon; look for
 example usage in the Python source distribution.
 

Doc/c-api/memory.rst

@@ -84,15 +84,65 @@ the C library allocator as shown in the previous example, the allocated memory
 for the I/O buffer escapes completely the Python memory manager.
 
 
+Raw Memory Interface
+====================
+
+The following function sets are wrappers to the system allocator. These
+functions are thread-safe, the :term:`GIL <global interpreter lock>` does not
+need to be held.
+
+The default raw memory block allocator uses the following functions:
+:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when
+requesting zero bytes.
+
+.. versionadded:: 3.4
+
+.. c:function:: void* PyMem_RawMalloc(size_t n)
+
+   Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
+   allocated memory, or *NULL* if the request fails. Requesting zero bytes
+   returns a distinct non-*NULL* pointer if possible, as if
+   ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have
+   been initialized in any way.
+
+
+.. c:function:: void* PyMem_RawRealloc(void *p, size_t n)
+
+   Resizes the memory block pointed to by *p* to *n* bytes. The contents will
+   be unchanged to the minimum of the old and the new sizes. If *p* is *NULL*,
+   the call is equivalent to ``PyMem_RawMalloc(n)``; else if *n* is equal to
+   zero, the memory block is resized but is not freed, and the returned pointer
+   is non-*NULL*. Unless *p* is *NULL*, it must have been returned by a
+   previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`. If
+   the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p* remains
+   a valid pointer to the previous memory area.
+
+
+.. c:function:: void PyMem_RawFree(void *p)
+
+   Frees the memory block pointed to by *p*, which must have been returned by a
+   previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`.
+   Otherwise, or if ``PyMem_Free(p)`` has been called before, undefined
+   behavior occurs. If *p* is *NULL*, no operation is performed.
+
+
 .. _memoryinterface:
 
 Memory Interface
 ================
 
 The following function sets, modeled after the ANSI C standard, but specifying
 behavior when requesting zero bytes, are available for allocating and releasing
-memory from the Python heap:
+memory from the Python heap.
+
+The default memory block allocator uses the following functions:
+:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when
+requesting zero bytes.
+
+.. warning::
 
+   The :term:`GIL <global interpreter lock>` must be held when using these
+   functions.
 
 .. c:function:: void* PyMem_Malloc(size_t n)
 
@@ -155,6 +205,125 @@ versions and is therefore deprecated in extension modules.
 :c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`.
 
 
+Customize Memory Allocators
+===========================
+
+.. versionadded:: 3.4
+
+.. c:type:: PyMemAllocator
+
+   Structure used to describe a memory block allocator. The structure has
+   four fields:
+
+   +----------------------------------------------------------+---------------------------------------+
+   | Field                                                    | Meaning                               |
+   +==========================================================+=======================================+
+   | ``void *ctx``                                            | user context passed as first argument |
+   +----------------------------------------------------------+---------------------------------------+
+   | ``void* malloc(void *ctx, size_t size)``                 | allocate a memory block               |
+   +----------------------------------------------------------+---------------------------------------+
+   | ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block     |
+   +----------------------------------------------------------+---------------------------------------+
+   | ``void free(void *ctx, void *ptr)``                      | free a memory block                   |
+   +----------------------------------------------------------+---------------------------------------+
+
+.. c:type:: PyMemAllocatorDomain
+
+   Enum used to identify an allocator domain. Domains:
+
+   * :c:data:`PYMEM_DOMAIN_RAW`: functions :c:func:`PyMem_RawMalloc`,
+     :c:func:`PyMem_RawRealloc` and :c:func:`PyMem_RawFree`
+   * :c:data:`PYMEM_DOMAIN_MEM`: functions :c:func:`PyMem_Malloc`,
+     :c:func:`PyMem_Realloc` and :c:func:`PyMem_Free`
+   * :c:data:`PYMEM_DOMAIN_OBJ`: functions :c:func:`PyObject_Malloc`,
+     :c:func:`PyObject_Realloc` and :c:func:`PyObject_Free`
+
+
+.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
+
+   Get the memory block allocator of the specified domain.
+
+
+.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
+
+   Set the memory block allocator of the specified domain.
+
+   The new allocator must return a distinct non-NULL pointer when requesting
+   zero bytes.
+
+   For the :c:data:`PYMEM_DOMAIN_RAW` domain, the allocator must be
+   thread-safe: the :term:`GIL <global interpreter lock>` is not held when the
+   allocator is called.
+
+   If the new allocator is not a hook (does not call the previous allocator),
+   the :c:func:`PyMem_SetupDebugHooks` function must be called to reinstall the
+   debug hooks on top on the new allocator.
+
+
+.. c:function:: void PyMem_SetupDebugHooks(void)
+
+   Setup hooks to detect bugs in the following Python memory allocator
+   functions:
+
+   - :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`,
+     :c:func:`PyMem_RawFree`
+   - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Free`
+   - :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`,
+     :c:func:`PyObject_Free`
+
+   Newly allocated memory is filled with the byte ``0xCB``, freed memory is
+   filled with the byte ``0xDB``. Additionnal checks:
+
+   - detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
+     allocated by :c:func:`PyMem_Malloc`
+   - detect write before the start of the buffer (buffer underflow)
+   - detect write after the end of the buffer (buffer overflow)
+
+   The function does nothing if Python is not compiled is debug mode.
+
+
+Customize PyObject Arena Allocator
+==================================
+
+Python has a *pymalloc* allocator for allocations smaller than 512 bytes. This
+allocator is optimized for small objects with a short lifetime. It uses memory
+mappings called "arenas" with a fixed size of 256 KB. It falls back to
+:c:func:`PyMem_Malloc` and :c:func:`PyMem_Realloc` for allocations larger than
+512 bytes.  *pymalloc* is the default allocator used by
+:c:func:`PyObject_Malloc`.
+
+The default arena allocator uses the following functions:
+
+* :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows,
+* :c:func:`mmap` and :c:func:`munmap` if available,
+* :c:func:`malloc` and :c:func:`free` otherwise.
+
+.. versionadded:: 3.4
+
+.. c:type:: PyObjectArenaAllocator
+
+   Structure used to describe an arena allocator. The structure has
+   three fields:
+
+   +--------------------------------------------------+---------------------------------------+
+   | Field                                            | Meaning                               |
+   +==================================================+=======================================+
+   | ``void *ctx``                                    | user context passed as first argument |
+   +--------------------------------------------------+---------------------------------------+
+   | ``void* alloc(void *ctx, size_t size)``          | allocate an arena of size bytes       |
+   +--------------------------------------------------+---------------------------------------+
+   | ``void free(void *ctx, size_t size, void *ptr)`` | free an arena                         |
+   +--------------------------------------------------+---------------------------------------+
+
+.. c:function:: PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)
+
+   Get the arena allocator.
+
+.. c:function:: PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)
+
+   Set the arena allocator.
+
+
 .. _memoryexamples:
 
 Examples

Doc/c-api/module.rst

@@ -35,13 +35,20 @@ There are only a few functions special to module objects.
       single: __name__ (module attribute)
       single: __doc__ (module attribute)
       single: __file__ (module attribute)
+      single: __package__ (module attribute)
+      single: __loader__ (module attribute)
 
    Return a new module object with the :attr:`__name__` attribute set to *name*.
-   Only the module's :attr:`__doc__` and :attr:`__name__` attributes are filled in;
-   the caller is responsible for providing a :attr:`__file__` attribute.
+   The module's :attr:`__name__`, :attr:`__doc__`, :attr:`__package__`, and
+   :attr:`__loader__` attributes are filled in (all but :attr:`__name__` are set
+   to ``None``); the caller is responsible for providing a :attr:`__file__`
+   attribute.
 
    .. versionadded:: 3.3
 
+   .. versionchanged:: 3.4
+      :attr:`__package__` and :attr:`__loader__` are set to ``None``.
+
 
 .. c:function:: PyObject* PyModule_New(const char *name)
 

Doc/c-api/object.rst

@@ -240,7 +240,7 @@ is considered sufficient for this determination.
    of the Python expression ``callable_object(*args)``.
 
 
-.. c:function:: PyObject* PyObject_CallFunction(PyObject *callable, char *format, ...)
+.. c:function:: PyObject* PyObject_CallFunction(PyObject *callable, const char *format, ...)
 
    Call a callable Python object *callable*, with a variable number of C arguments.
    The C arguments are described using a :c:func:`Py_BuildValue` style format
@@ -250,8 +250,11 @@ is considered sufficient for this determination.
    pass :c:type:`PyObject \*` args, :c:func:`PyObject_CallFunctionObjArgs` is a
    faster alternative.
 
+   .. versionchanged:: 3.4
+      The type of *format* was changed from ``char *``.
 
-.. c:function:: PyObject* PyObject_CallMethod(PyObject *o, char *method, char *format, ...)
+
+.. c:function:: PyObject* PyObject_CallMethod(PyObject *o, const char *method, const char *format, ...)
 
    Call the method named *method* of object *o* with a variable number of C
    arguments.  The C arguments are described by a :c:func:`Py_BuildValue` format
@@ -261,6 +264,9 @@ is considered sufficient for this determination.
    Note that if you only pass :c:type:`PyObject \*` args,
    :c:func:`PyObject_CallMethodObjArgs` is a faster alternative.
 
+   .. versionchanged:: 3.4
+      The types of *method* and *format* were changed from ``char *``.
+
 
 .. c:function:: PyObject* PyObject_CallFunctionObjArgs(PyObject *callable, ..., NULL)
 
@@ -342,6 +348,15 @@ is considered sufficient for this determination.
    returned.  This is the equivalent to the Python expression ``len(o)``.
 
 
+.. c:function:: Py_ssize_t PyObject_LengthHint(PyObject *o, Py_ssize_t default)
+
+   Return an estimated length for the object *o*. First trying to return its
+   actual length, then an estimate using ``__length_hint__``, and finally
+   returning the default value. On error ``-1`` is returned. This is the
+   equivalent to the Python expression ``operator.length_hint(o, default)``.
+
+   .. versionadded:: 3.4
+
 .. c:function:: PyObject* PyObject_GetItem(PyObject *o, PyObject *key)
 
    Return element of *o* corresponding to the object *key* or *NULL* on failure.