☑ What’s New in Python 3.10 - Other New Features

In the previous article we took a look at a major new feature in Python 3.10, structural pattern matching. In this one we’ll look at the other new langauge and builtin features that have been introduced.

Parentheised Context Managers¶

We’ll ease in with a simple one, which is that parentheses can be used with context managers to break their declaration across multiple lines. This is generally most useful when dealing with multiple context managers. Let’s say that you need to take a mutex and then copy data from one file to another, you end up nesting your context managers like this.

def copy_file(self):
    with self.lock:
        with open(self.input_file, "rb") as in_fd:
            with open(self.output_file, "wb") as out_fd:
                shutil.copyfileobj(in_fd, out_fd)

To stop this sort of nesting needlessly extending the indentation, it’s long been possible to merge the context managers on to a single line, so the following two constructions are equivalent.

with A() as a:
    with B() as b:
        with C() as c:
            ...

with A() as a, B() as b, C() as c:
    ...

If you look back at the first example, however, you’ll see this rapidly gets unwieldy when you cram all those expressions on to one line. You can use backslashes to split lines, but generally parentheses tend to be easier for programmers, editors and formatters to deal with. Unfortunately parentheses haven’t been valid with context managers — until now.

def copy_file(self):
    with (
        self.lock,
        open(self.input_file, "rb") as in_fd,
        open(self.output_file, "wb") as out_fd,
    ):
        shutil.copyfileobj(in_fd, out_fd)

Not the biggest change, but since context managers are fairly frequently used in idiomatic Python code, it’s certainly nice to see this consistency introduced. As a point of interest, this change leverages the change in 3.9 from an LL parser. which is a context-free grammar, to a PEG grammar and recursive descent parser. If you’re not interested in parsers and grammars, however, you can safely ignore that sentence, but if you’re thirsting for more you can read details in PEP 617.

Better Error Reporting¶

A change that I think more or less everyone will find useful is that Python has improved the detail it includes in its reporting of syntax and other errors. I’ll run through a summary of the changes to each error type in the sections below.

Syntax Errors¶

One of the big improvements here is that unterminated brackets and quoted strings now report the location of the opening bracket or quote, as opposed to the previous behaviour of some arbitrary later point at which the issue was identified.

1
2

my_string = "Hello, world
print(my_string)

1
2
3
4
5
6

my_dict = {
    1: "one",
    2: "two",
    3: "three"
for key, value in my_dict.items():
    print(key, value)

$ python3.9 syntax1.py
  File "/private/tmp/syntax1.py", line 1
    my_string = "Hello, world
                             ^
SyntaxError: EOL while scanning string literal
$ python3.9 syntax2.py
  File "/private/tmp/syntax2.py", line 5
    for key, value in my_dict.items():
    ^
SyntaxError: invalid syntax
$
$ python3.10 syntax1.py
  File "/private/tmp/syntax1.py", line 1
    my_string = "Hello, world
                ^
SyntaxError: unterminated string literal (detected at line 1)
$ python3.10 syntax2.py
  File "/private/tmp/syntax2.py", line 1
    my_dict = {
              ^
SyntaxError: '{' was never closed

Another improvement is that the full text of the erroneous section of code is now highlighted instead of just a pointer to the start of the issue. In some cases this is going to be really handy for avoiding misunderstandings of the scope of the error.

1
2

for i in range(10):
    print f"Repetition {i+1}: hello, world"

$ python3.9 syntax3.py
  File "/private/tmp/syntax3.py", line 2
    print f"Repetition {i+1}: hello, world"
          ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(f"Repetition {i+1}: hello, world")?
$
$ python3.10 syntax3.py
  File "/private/tmp/syntax3.py", line 2
    print f"Repetition {i+1}: hello, world"
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?

Finally, a number of specific error messages for certain cases have been added for better clarity of what is the actual issue. Here are some examples.

>>> if 10 > 20
  File "<stdin>", line 1
    if 10 > 20
              ^
SyntaxError: expected ':'
>>>
>>> items = {1: "one", 2: "two" 3: "three}
  File "<stdin>", line 1
    items = {1: "one", 2: "two" 3: "three}
                          ^^^^^^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?
>>>
>>> try:
...     pass
... except RuntimeError, AssertionError as exc:
  File "<stdin>", line 3
    except RuntimeError, AssertionError as exc:
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: multiple exception types must be parenthesized
>>>
>>> x = ["A", "B", "C"]
>>> (*x)
  File "<stdin>", line 1
    (*x)
     ^^
SyntaxError: cannot use starred expression here
>>> (*x,)
('A', 'B', 'C')
>>>
>>> {n, x for n, x in enumerate("ABCDEF")}
  File "<stdin>", line 1
    {n, x for n, x in enumerate("ABCDEF")}
     ^^^^
SyntaxError: did you forget parentheses around the comprehension target?
>>> {(n, x) for n, x in enumerate("ABCDEF")}
{(3, 'D'), (2, 'C'), (4, 'E'), (5, 'F'), (1, 'B'), (0, 'A')}
>>>
>>> try:
...     raise Exception("?")
...
  File "<stdin>", line 3

    ^
SyntaxError: expected 'except' or 'finally' block
>>>
>>> if 10 = 10.0:
  File "<stdin>", line 1
    if 10 = 10.0:
       ^^
SyntaxError: cannot assign to literal here. Maybe you meant '==' instead of '='?
>>>
>>> {1: "one", 2 "two", 3: "three"}
  File "<stdin>", line 1
    {1: "one", 2 "two", 3: "three"}
               ^
SyntaxError: ':' expected after dictionary key

These are all brilliant, and I’m grateful to everyone who’s put in the painstaking effort to implement all these. I suspect these sorts of changes often don’t get the credit they deserve for undoubtedly saving countless cumulative hours of developer frustration over the coming years.

Other Errors¶

As well as SyntaxError, some of the other error types have had similar treatment, though not quite to the same extent.

IndentationError

Many of these errors now include more context about why an indented block was expected (e.g. expected an indented block after 'with' statement in line 34) and they also include the location of the statement triggering the indentation as well as the code that was expected to be indented.

AttributeError and NameError

These errors now include suggestions about similar names which you may have meant, which is quite useful for quickly identifying typos during coding (e.g. module 'sys' has no attribute 'stder'. Did you mean: 'stderr'?).

Though smaller in scope than the changes to SyntaxError, these are also very useful changes and much appreciated.

Type Hints¶

There are a collection of useful enhancements to type hints in this release, the most significant of which are covered in the sections below.

Union of Types¶

Perhaps inspired by the pattern matching alternation feature covered in the previous article, PEP 604 allows a similarly concise syntax for allowing a union of types.

def find_item(item: str | int, items: list[ str | int]) -> int:
    return items.index(item)

This is equivalent to the more verbose version below:

from typing import Union
def find_item(item: Union[str, int], items: list[Union[str, int]]) -> int:
    return items.index(item)

Essentially this was enabled by implementing the type.__or__() method, and also adding a new types.UnionType builtin. This is a different type to typing.Union, but interoperates with it.

>>> Union[int, str] == (int | str)
True
>>> Union[int, str] == int | str | float
False
>>> type(Union[int, str])
<class 'typing._UnionGenericAlias'>
>>> type(int | str)
<class 'types.UnionType'>

A slightly more fundamental change introduced by the same PEP is that type unions can now also be used with isinstance() and issubclass() checks.

>>> isinstance("hello", Union[int, str])
True

Callables: Parameter Specification Variables¶

There are two improvements covered by PEP 612 which relate to cases of using type hinting where callables are passed into functions, such as with decorators.

By way of background, in Python versions up to 3.9 there are two ways to specify a type hint for a callable. The first is is the syntax typing.Callable[int, list[int]] introduced back in Python 3.5. The other way is to define a typing.Protocol class with a callback type specified with __call__(). Support for Protocol was added in Python 3.8, although I must admit I didn’t cover this aspect in any detail in my article at the time — a good job my blog’s slogan is “that’s good enough” instead of “no detail left uncovered”.

The problem with both of these approaches is that they’re incapable of expressing relationships between the parameters to the outer function and the callable. You can fix the signature of the callable, but if you have a function which takes, say, *args and **kwargs, and you want to proxy that out to your outer function, then the relationship you’d like to express is that whatever the inner function takes, the outer function should also take.

Both of the these two improvements in Python 3.10 add approaches to deal with this case. The first is parameter specification variables, which are a more flexible equivalent of the existing typing.TypeVar. The example below demonstrates its use with a decorator.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

import logging
import typing

T = typing.TypeVar("T")
P = typing.ParamSpec("P")

def log_entry_exit(func: typing.Callable[P, T]) -> typing.Callable[P, T]:
    def inner(*args: P.args, **kwargs: P.kwargs) -> T:
        logging.info(f"Entering {func.__name__}")
        try:
            return func(*args, **kwargs)
        finally:
            logging.info(f"Leaving {func.__name__}")
    return inner

@log_entry_exit
def pow(a: int, b: int) -> int:
    return a ** b

logging.basicConfig(level=logging.INFO)
print(pow(2, 3))
print(pow(3, 4))

You can see that typing.ParamSpec is used quite similarly to TypeVar except that it represents an entire set of parameters, not just a single value. It’s also significantly more constrained, and can only be used as the first parameter to a callable or a few other related places.

Callables: Concatenate Operator¶

The second improvement caters for the case where a callable should be invoked with a proxied set of arguments, as above, but also with one or more additional arguments added. This is the typing.Concatenate operator, and it’s used with the ParamSpec variable introduced above to represents the parameter list to which additional parameters are being added.

Here’s an example below of how it could be used to easily pass a global lock into methods.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

from threading import Lock
from typing import Callable, Concatenate, ParamSpec, TypeVar

P = ParamSpec("P")
R = TypeVar("R")

_my_lock = Lock()
_my_queue = []

def with_lock(func: Callable[Concatenate[Lock, P], R]) -> Callable[P, R]:
    def inner(*args: P.args, **kwargs: P.kwargs) -> R:
        return func(_my_lock, *args, **kwargs)
    return inner

@with_lock
def append_item(lock, item):
    with lock:
        _my_queue.append(item)

append_item(123)
append_item(456)

I’ll be honest it’s not an example I’m particularly happy with, as in this case I strongly suspect you’d just wrap the whole lot up in a class and make the Lock instance a member. But it’s the example from the documentation, and right now I struggle to think of other uses for this — I’m sure they exist, however, and if you want to type check accurately then these are both useful tools in your toolbox.

Type Aliases¶

In my previous article on type hinting way back in Python 3.5, I briefly talked about creating type aliases by just assigning a type to a normal variable. This syntax was pretty flexible, but makes life hard for static type checkers — any assignment could be a real assignment, or just a type alias, and it can be quite hard to tell in some cases.

As a result, Python 3.10 introduces typing.TypeAlias to allow these to be explicitly annotated. I’ll demonstrate using an updated version of the example from the original article.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

from collections.abc import Iterable, Sequence
import math
from typing import TypeAlias

Point: TypeAlias = tuple[float, float, float]
Path: TypeAlias = Sequence[Point]

def line_length(x: Point, y: Point) -> float:
    sum_squares = sum((x[i] - y[i]) ** 2 for i in range(3))
    return math.sqrt(sum_squares)

def path_segments(path: Path) -> Iterable[tuple[Point, Point]]:
    x, y = iter(path), iter(path)
    next(y, None)
    return zip(x, y)

def path_length(path: Path) -> float:
    return sum(line_length(x, y) for x, y in path_segments(path))

line: Path = [(0, 0, 0), (3, 2, 0), (6, 9, 2), (2, 18, 4)]
print(line_length(line[0], line[1]))
print(list(path_segments(line)))
print(path_length(line))

User-Defined Type Guards¶

Static type checkers typically work by taking the types of variables provided by annotations as a starting point, and then narrowing them as a result of branching statements.

1
2
3
4
5
6
7
8

def my_function(argument: Optional[List[str]]) -> str:
    # Static checker knows 'argument' is either None or List[str] here
    if argument is None:
        # Static checker knows type of 'argument' is None here
        return "::"
    else:
        # Static checker knows type of 'argument' is List[str] here
        return ":" + ":".join(argument) + ":"

This is quite powerful in many cases, but it doesn’t reflect the role that user-defined type checking can perform. For example, see the code below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

def check_int_list(items: list[object]) -> bool:
    return all(isinstance(i, int) for i in items)

def my_function(argument: list[object]) -> object:
    if check_int_list(argument):
        return sum(argument)
    elif argument:
        return argument[0]
    else:
        return None

print(my_function([1, 2, 3]))

At runtime it works fine, but if you run it through mypy you get the following error.

$ mypy /tmp/userguard.py
/tmp/userguard.py:6: error: Argument 1 to "sum" has incompatible type "List[object]"; expected "Iterable[bool]"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

This is because mypy isn’t aware of the type validation that our code in check_int_list() is performing. However, as of Python3.10 we can use a new TypeGuard annotation to express this. For any function which returns bool, you can annotate its return type as TypeGuard[T] and in the branch case where the return is True, static checkers will behave as if the checked variable was of the asserted type T. Here’s the example above modified to be acceptable to mypy.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

from typing import TypeGuard

def check_int_list(items: list[object]) -> TypeGuard[list[int]]:
    return all(isinstance(i, int) for i in items)

def my_function(argument: list[object]) -> object:
    if check_int_list(argument):
        return sum(argument)
    elif argument:
        return argument[0]
    else:
        return None

print(my_function([1, 2, 3]))

Default Encoding Warning¶

In a pleasingly symmetric fashion, we’ll close with another smallish feature. The default encoding used by open() and io.TextIOWrapper is depends on both the current platform and the locale currently configured on that platform. However, since so many systems are configured to use UTF-8 now, it’s becoming increasingly common for code to omit any encoding when opening a file and assume UTF-8 will be used — this can create subtle bugs later when the code is run in different environments.

There is a new EncodingWarning which can be emitted whenever the default locale-specified encoding is used to try to detect such issues. Because of the speculative nature of these bugs, the warning isn’t enabled by default and must be explicitly requested with -X warn_default_encoding, or by setting environment variable PYTHONWARNDEFAULTENCODING to a non-empty string.

$ python3.10 -X warn_default_encoding
Python 3.10.6 (main, Aug 30 2022, 05:12:36) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> fd = open("/tmp/testfile", "r")
<stdin>:1: EncodingWarning: 'encoding' argument not specified

So that’s another one to remember to enable during your unit test runs.

Smaller Changes¶

And the usual list of smaller enhancements which are noteworthy but brief.

New strict Parameter to zip()

If you set this to True, it raises ValueError unless all of the supplied iterables are the same length.

For Builtins Use __index__() Not __int__()

In cases where an int is required, previously the __int__() dunder method was used. However, this can result in loss of precision — for example, in (3.6).__int__(). Now the __index__() method is used, which is only defined for objects which are intrinsically integers, not for objects for which an integer approximation may be desirable.

aiter() And anext() Added

These are the asyncronous equivalents of the existing iter() and next() builtin functions.

SyntaxError Attributes

Instances of SyntaxError now have end_lineno and end_offset attributes, in line with the improvements to the display of these errors discussed earlier in the article. These may be None if not determined in a given case.

Conclusion¶

I would say nothing hugely earth-shattering in this set, but that’s not unreasonable given the scope of the pattern matching feature we looked at in the previous article. The improvements to the syntax errors is very welcome, however, and I’ve already noticed I get annoyed when I have to go back and work on code running under earlier Python versions because everything takes just that little bit longer to track down and fix.

The type hinting improvements are good to see, although I do get the impression we’re getting increasingly esoteric in these. However, given the heavily dynamic nature of Python, where objects of all sorts can be passed around freely, a good deal of flexibility is going to be required to express all reasonable cases.

Next time we’ll run through the improvements to the library modules, as has become my habit. Having taken a brief peek it doesn’t look like there’s a massive amount there, but I’m sure there will be some gems waiting to be discovered, as always.