☑ What’s New in Python 3.5 - Other Additions

Having taken an article each to look at the two biggest enhancements in Python 3.5, this article covers a collection of the smaller additions to the langauge.

Matrix Multiplication Operator¶

This release adds a new operator @ for matrix multiplication. None of the modules in the Python standard library actually implement this operator, at least in this release, so it’s primarily for third party modules to use. First and foremost among these is NumPy, a popular library for scientific and applied mathematics within Python. One of its primary purposes is to add support for multidimensional matrices, so this operator suits it well.

To see how this works I whipped up my own simple Matrix class. This isn’t production quality and has very little error checking, it’s just for illustrative purposes. The key point to notice here is that it implements the __matmul__() method to support the @ operator. Similar to other infix operators, there’s also an __rmatmul__() method for reversed operation in cases where only the right-hand operand supports the method, and __imatmul__() to override how x @= y is handled. I didn’t bother to define these since in this case the automatic fallback options are sufficient.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

import copy

class Matrix:

    def __init__(self, rows):
        """Construct a matrix from specified list of rows."""
        self._rows, self._cols = len(rows), len(rows[0])
        self._matrix = copy.deepcopy(rows)

    def __repr__(self):
        return "Matrix({0!r})".format(self._matrix)

    def __str__(self):
        return "[" + "\n ".join(repr(i) for i in self._matrix) + "]"

    @classmethod
    def create_empty(cls, rows, cols, initial=0):
        """Create a uniform matrix of specified size from one value."""
        return cls([[initial] * cols for i in range(rows)])

    def __getitem__(self, index):
        return self._matrix[index[0]][index[1]]

    def __setitem__(self, index, value):
        self._matrix[index[0]][index[1]] = value

    def __matmul__(self, other):
        """Implement the matrix multiplication (@) operator."""
        if self._cols != other._rows:
            raise IndexError("Row/col count mismatch")
        # The inner loop constructs a result row, and the outer
        # loop repeats this for each row in the result.
        return Matrix([
                [sum(i * j for i, j in zip(
                    self.row(r_idx), other.col(c_idx)))
                 for c_idx in range(other._cols)]
            for r_idx in range(self._rows)])

    def row(self, r_idx):
        """Return an iterator over the specified row."""
        return iter(self._matrix[r_idx])

    def col(self, c_idx):
        """Return an iterator over the specified column."""
        return (self._matrix[i][c_idx] for i in range(self._rows))

Below you can see an example of it being used:

>>> m1 = Matrix([[1], [2], [3]])
>>> m2 = Matrix([[4, 5, 6]])
>>> m1[2, 0]
3
>>> m2[0, 1]
5
>>> print(m1 @ m2)
[[4, 5, 6]
 [8, 10, 12]
 [12, 15, 18]]
>>>
>>> print(m1 @ Matrix.create_empty(1, 4, initial=2))
[[2, 2, 2, 2]
 [4, 4, 4, 4]
 [6, 6, 6, 6]]
>>>
>>> m1 @ Matrix.create_empty(2, 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/andy/misc-files/blog/py35/matrix.py", line 30, in __matmul__
    raise IndexError("Row/col count mismatch")
IndexError: Row/col count mismatch

Handy to be aware of this operator, but unless you’re using NumPy or you’re implementing your own matrix class, it’s probably something of an obscure curiosity.

Argument Unpacking¶

There have been some improvements to the way that iterables can be unpacked when calling functions and at other times. To recap, for a long time Python has had the ability to capture variable argument lists passed into functions, whether specified positionally or by keyword. Conversely, it’s also been able to unpack a sequence or mapping into position or keyword arguments when calling the function.

>>> def func(*args, **kwargs):
...     print("Args: " + repr(args))
...     print("Keyword args: " + repr(kwargs))
...
>>> func("one", 2, three="four", five=6)
Args: ('one', 2)
Keyword args: {'five': 6, 'three': 'four'}
>>>
>>> def func2(arg1, arg2, arg3, arg4, arg5):
...     print(arg1, arg2, arg3, arg4, arg5)
...
>>> func2(*("a", "b", "c"), **{"arg4": "D", "arg5": "E"})
a b c D E

As per PEP 448 this release expands these facilities to address some limitations of the current approach. One of the main issues is if you are collecting parameters from multiple places, you’re forced to collapse them all into a temporary structure (typically list or dict) and pass that into the function in question. This is not difficult, but it makes for somewhat cumbersome code.

As of Python 35, however, it’s possible to specify these operators multiple times each and the results are merged for the actual function call:

>>> func(1, 2, *[3, 4], *[5, 6, 7],
...      **{"eight": 9, "ten": 11}, **{"twelve": 12})
Args: (1, 2, 3, 4, 5, 6, 7)
Keyword args: {'eight': 9, 'ten': 11, 'twelve': 12}

It’s still the case that positional arguments must precede keyword ones, and *-arguments must precede **-arguments. Also, I note with interest that it’s actually an error to specify the same keyword parameter twice, raising a TypeError, as opposed to the second instance silently replacing the first:

>>> func(**{"arg": 1}, **{"arg": 2})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: func() got multiple values for keyword argument 'arg'

Perhaps more usefully, this syntax has also been extended to be usable within the construction of literals for tuple, list, set and dict:

>>> [*range(3), *range(10, 5, -2)]
[0, 1, 2, 10, 8, 6]
>>> config = {"rundir": "/tmp", "threads": 5, "log_level": "INFO"}
>>> {**config, "log_level": "DEBUG", "user": "andy"}
{'threads': 5, 'rundir': '/tmp', 'log_level': 'DEBUG', 'user': 'andy'}
>>> {*config.keys(), "user"}
{'threads', 'rundir', 'user', 'log_level'}

As a further aside, you’ll notice that unlike in the function call case, it’s fine to have repeated keys in the dict constructor (e.g. "log_level" occurs twice above). This is useful as it means you can perform an “upsert” where you either replace or add an entry in the new copy of the dict that’s created.

This may be a small feature, but I can think of a number of places in code where it will be quite convenient to have such a concise expression of intent.

Operating System Interface¶

There are a couple of improvements to interactions with the operating system, one related to a faster way to iterate over directory listings and the other more graceful handling of interrupted system calls.

os.scandir()¶

There’s a new function os.scandir() which is roughly an improved version of os.listdir() which includes some file attribute information. This can significantly improve performance over performing separate calls to things like os.is_dir() or os.state() for each entry. The os.walk() implementation has been updated to take advantage of scandir(), which makes it 3-5x faster on POSIX systems and 7-20x faster on Windows.

As an example of the improvements in time, see the following snippet which shows code to count the number of subdirectories whose name doesn’t begin with a dot.

>>> timeit.timeit('sum(1 for i in os.scandir("/Users/andy")'
                     ' if i.is_dir() and not i.name.startswith("."))',
                  setup='import os', number=100000)
9.061285664036404
>>> timeit.timeit('sum(1 for i in os.listdir("/Users/andy")'
                     ' if os.path.isdir("/Users/andy/" + i)'
                     ' and not i.startswith("."))',
                  setup='import os', number=100000)
31.92237657099031

Automatically Retry Interrupted System Calls¶

Another handy feature in this release is automatic retry of EINTR error results. This occurs when a system call is interrupted, typically by a signal arriving in the process — the system call returns an error with errno set to EINTR. It’s worth noting that this behaviour depends on the system call — for example, write() calls which have already written some partial data won’t return an error, but instead indicate success and return the number of bytes that were actually written¹.

In Python 3.4 and earlier, this error code triggered an InterruptedError exception which the application had to handle appropriately. This can be quite annoying, particularly if you’re making these calls in quite a few different places. But if you don’t do it them your application is at risk of randomly suffering all kinds of odd bugs. To make things more complicated, many libraries don’t handle this error for you, so it can propogate out of almost anywhere to bite you.

The good news for Python programmers is that as of Python 3.5 they won’t generally need to worry about this thanks to the improvements proposed in PEP 475. The wrappers around the standard library will now catch the EINTR case, check if any pending signal handlers need calling, and then retry the operation behind the scenes. This means that InterruptedError should not occur anywhere any more and any code to handle it may be removed.

This applies to almost all library calls, so I’m not going to enumerate them all here. Notable exceptions are os.close() and os.dup2() due to the indeterminate state of the file descriptor under Linux if these functions return EINTR — these calls will now simply ignore EINTR instead of retrying.

StopIterator Handling Within Generators¶

Many people reading this will probably be aware of the StopIteration exception. Whilst we rarely deal with it explicitly, it’s used implicitly any time we use a generator — when __next__() is called on an iterator it’s expected to either return the next item, or raise StopIteration to indicate the iterator is exhausted.

As with any other exception, however, it’s possible that it can be raised unexpectedly, perhaps due to a bug in your code, or perhaps even a bug in another library outside your control. If this happens within the implementation of a generator, if the StopIteration exception isn’t caught then it’ll propogate out to the looping code which will interpret it as an end of the iterator. This silent swallowing of erroneously raised exceptions can make all sorts of bugs particularly difficult to track down, as opposed to the normal behaviour of getting an immediate interruption with a helpful backtrace to help you figure out what the issue is.

As a result of these issues, a change has been introduced which prevents a StopIteration that’s raised within a generator from propogating outside it, instead turning it into a RuntimeError. The original StopIteration is still included as a chained exception for debugging.

This change is not backwards-compatible, because there are some potential valid uses of the behaviour around StopIteration. Take the code below, for example — whilst I wouldn’t suggest this is a great implementation, it’s certainly not safe to rule out that people have written this sort of thing.

>>> def spacer(it, value):
...     while True:
...         yield next(it)
...         yield value
...
>>> list(spacer(iter(range(5)), 999))
[0, 999, 1, 999, 2, 999, 3, 999, 4, 999]

As a result, to activate this new behaviour you need to from __future__ import generator_stop. You can see the difference in behaviour below:

>>> from __future__ import generator_stop
>>>
>>> def spacer(it, value):
...     while True:
...         yield next(it)
...         yield value
...
>>> list(spacer(iter(range(5)), 999))
Traceback (most recent call last):
  File "<stdin>", line 3, in spacer
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: generator raised StopIteration

This behaviour will likely become default in some future release, so it’s worth updating any code that breaks. I suspect there’s not a huge amount of code that will be impacted, but if you want to check your code then be aware that if you do raise a StopIteration in a generator without the from __future__ import then it’ll raise a PendingDeprecationWarning if you have that warning enabled.

If you want some more detailed discussion of the motivation for this change, PEP 479 has some great discussion.

Multi-phase Extension Module Initialisation¶

There are some changes in 3.5 to the way that C extension modules are initialised to bring them more closely in line with the way standard Python modules are loaded. I’ll need to go into a certain level of detail to explain what this change is doing, but I’ll do my best to keep it moving along as I know not everyone deals with extension modules. You can always check out PEP 489 for more details.

As a reminder, loading a standard Python module proceeds in five stages since Python 3.4²:

1. First look for an already-imported version of this module in sys.modules. If found, this module is used and the import process is complete.
2. The import protocol first calls a series of finders to locate the requested module. As soon as one of them finds the module, it returns a ModuleSpec which has some metainformation about the module and the loader to actually load it.
3. The import machinery then invokes create_module() on the loader to create a new module object³. You can think of this as conceptually equivalent to the __new__() method of a class.
4. The module object is added to sys.modules under the fully-qualified name. Doing this before the next step avoids an infinite loop if the module indirectly causes itself to be re-imported.
5. The module object is then passed to the exec_module() method of the loader to actually load the code itself. This is conceptually similar to the __init__() method of a class. If it raises an exception then the module is removed from sys.modules again.

By comparison, extension modues have had a monolithic loading process prior to Python 3.5. The module exports an initialisation function named PyInit_modulename(), where modulname must match the filename. This is executed by the import machinery and expected to provide a fully initialised module object, combining steps 2 and 4 above into one call. Also, extension modules are not added to sys.modules at present.

To put this into more concrete terms, extension modules typically provide a static definition of a PyModuleDef structure which defines the details for the module such as the methods it provides and the docstring. This is passed into PyModule_Create() at the start of the initialisation function to create the module object. This is followed by any module-specific initialisation required and finally the module object is returned.

This process is still supported in Python 3.5, to avoid breaking all the existing extension modules, but modules can also now request multi-phase initialisation by just returning the PyModuleDef objec itself without yet creating the module object. It must be passed through PyModuleDef_Init() to ensure it’s a properly initialised Python object, however.

The remaining stages of initialisation are then performed according to callbacks provided by the module. These are specified in the PyModuleDef structure using the m_slots pointer, which is a pointer to a NULL-terminated array of PyModuleDef_Slot structures⁴. Each PyModuleDef_Slot has an integer ID to specify the type of slot and a void* value which is currently always interpreter as a function pointer.

In this release there are two valid values for the slot ID:

Py_mod_create

This is optional and if provided is expected to be a pointer to a function which is invoked to create the module object — the equivalent of the create_module() object of the loader. It’s passed the ModuleSpec instance and the PyModuleDef as returned from the initialisation function. If you don’t need a custom module object then just omit this, and the import machinery calls PyModule_New() to construct a standard module object.

Py_mod_exec

This slot specifies the equivalent of the exec_module() method of the loader. This is invoked after the module was added to sys.modules and it’s passed the module object that was created, and typically adds classes and constants to the module. Multiple slots of this type can be specified, and they’ll be executed in sequence, in the order that they appear in the array.

So that’s about the shape of it. I have a feeling a lot of people will just stick to the single initialisation approach because a lot of C extension modules are essentially just Python bindings around libraries in other languages, and these probably have little use for custom module objects. But it’s good to know the flexibility is there if you need it.

Smaller Improvements¶

As usual there are some smaller language changes that I wanted to give a shout out to, but not discuss in detail.

% Formatting for bytes

The % operator can now also be used with bytes and compatible values to construct a bytes value directly. Typically arguments will be either numeric of bytes themselves — str arguments can be used with the %a format specifier, but this passes them through repr() so it’s probably not what you want.

Approximate equality testing

There are two new functions math.isclose() and cmath.is_close() for testing approximate equality between two numeric values. There are two tolerances that can be specified, a relative tolerance and an absolute one — if the gap betwen the values is smaller than either tolerance then the function returns True, otherwise False. The relative tolerance is expressed as a percentage which is applied to the larger of the two values to derive another absolute tolerance. See PEP 485 for more details.

Removal of .pyo files

Prior to this release, compiled bytecode was cached in .pyc files normally and .pyo files for optimized code (if -O or -OO were specified). However, since the same extension is used for both levels of optimisation it’s not easy to tell which was used, and code may end up using a mixture of optimisation levels. In this release the .pyo extension is removed entirely and .pyc is used in all cases. Instead of differing in the extension, a new tag is added indicating the optimisation level, which means the interpreter can select the correct optimisation level in all cases. For a source file lib.py, the unoptimised bytecote will be in lib.cpython-35.pyc, and the optimised versions will be in lib.cpython-35.opt-1.pyc and lib.cpython-35.opt-2.pyc. PEP 488 has more.

New zipapp module

A new module has been added which provides both an API and CLI for creating Python Zip Applications. These are consist of a zip archive containing the main script as well as supporting modules, which can be executed directly without the need for decompressing it. Support for these was introduced in Python 2.6, although a lot of people don’t seem to know about it. In general this works by creating your application as a package, complete with __main__.py file and then just passing this package directory as an argument to python -m zipapp. This gives you a .pyz file which can be executed directly by the interpreter as could a standard script.

Conclusions¶

Nothing too earth-shattering in this collection, I’d say, and quite a few of these items are a little niche. The argument unpacking improvements will certainly be useful for writing concise code in particular cases, although I do wonder how useful that is for people who’ll want to make full use of the type-hinting system — I can imagine all of those varargs-style constructions make accurate type-hinting a little tricky. The creation of os.scandir(), and consequent improvements to os.walk() performance, are certainly very welcome. It’s not uncommon to want to trawl a large directory structure for some reason or other, and these heavily IO-bound cases can be pretty slow, hence any improvement is welcome.

The next article in the series will likely be the last one looking at changes in 3.5, and it’ll be looking at all the remaining standard library improvements.