☑ What’s New in Python 3.11 - Improved Modules III

Continuing from the previous article, let’s plunge on to the remaining changes to the standard library.

Concurrent Execution — threading¶

We kick off with a really small change in the threading module, similar to the changes to time.sleep() in the previous article.

On Unix, a variant of sem_timedwait() called sem_clockwait() has been added to glibc in version 2.30. As with the original, it decrements the specified semaphore and returns, unless it can’t do so within the specified timeout, in which case it returns an error. The difference is that sem_clockwait() allows the specific clock to be specified.

In Python 3.11, threading.Lock.acquire() has been updated to use sem_clockwait(), if it’s available, to use time.CLOCK_MONOTONIC instead of the default time.CLOCK_REALTIME. As with the changes in time.sleep(), this avoids timeout issues if the system clock is changed whilst such a function is waiting.

Networking and IPC¶

Continuing the theme of recent releases, asyncio has quite a basket of changes in this release which add new features, as well as convenient interfaces to existing functionality and more flexibility. There’s also a couple of minor changes to the socket module.

asyncio¶

In keeping with recent releases, there are quite a few improvements to asyncio, which we’ll run through in the sections below.

TaskGroup¶

First up we have a feature which I believe was inspired from Trio, which is an async framework which aims to offer more simplicity than things like asyncio and Twisted without sacrificing capability¹. Trio offers a class called Nursery which is something that can watch over your child tasks², and as of this release asyncio now offers a similar class called TaskGroup.

The TaskGroup class is intended as a more friendly alternative to calling asyncio.create_task() and asyncio.gather() directly. It acts as an async context manager, which you create first and then use its create_task() method to add tasks to it. When the context exits, it waits for all tasks added during the intervening block.

There’s a simple example below which illustrates how it’s used.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

import asyncio

async def timer(msg, interval):
    for i in range(3):
        await asyncio.sleep(interval)
        print(msg)

async def group_demo():
    async with asyncio.TaskGroup() as tasks:
        print("Adding one")
        tasks.create_task(timer("One", 0.5))
        print("Adding two")
        tasks.create_task(timer("Two", 0.8))
        print("Adding three")
        tasks.create_task(timer("Three", 1.2))
    print("All done")

asyncio.run(group_demo())

For reference, an equivalent group_demo() function using asyncio.gather() directly is shown below.

 7
 8
 9
10
11
12
13
14
15
16
17
18

⋮
async def group_demo():
    aws = []
    print("Adding one")
    aws.append(timer("One", 0.5))
    print("Adding two")
    aws.append(timer("Two", 0.8))
    print("Adding three")
    aws.append(timer("Three", 1.2))
    await asyncio.gather(*aws)
    print("All done")
⋮

Timeout Context¶

In the same way that the TaskGroup context manager is a more pleasant and flexible interface around create_task() and gather(), there’s also a new asyncio.timeout() context manager which is a more flexible version of asyncio.wait_for().

As a reminder, wait_for() allows you to wait for Task completion with a timeout — if the timeout expires before the Task completes, cancel() is called on it and the wait_for() function raises TimeoutError to the caller.

The new timeout() context manager is similar, except that it can wrap a whole block, which can be running any task when the timeout expires. When that happens the behaviour is similar — the currently executing task is cancelled with cancel(), and the CancelledError thus raised is caught by timeout() and converted to TimeoutError which is raised to the caller.

There’s also a timeout_at(), which accepts an absolute time rather than a relative delay as the point to trigger the timeout. The actual object returned in both cases is an instances of asyncio.Timeout, and this also offers a few useful methods:

when()

Returns the current deadline, or None if there isn’t one set.

reschedule()

Updates the current timeout — note that this always expects an absolute time, regardless of whether you originally created this instance with timeout() or timeout_at().

expired()

Returns True if the timeout has expired already.

The excerpt below demonstrates this all together — have a quick read through and I’ll put some notes at the end if it’s still a little unclear.

>>> import asyncio
>>> async def slow_count(limit):
...     for n in range(limit):
...         yield n
...         await asyncio.sleep(n)
...
>>> async def count_with_timeout():
...     try:
...         async with asyncio.timeout(9) as timer:
...             async for n in slow_count(10):
...                 print(f"Got value {n}")
...                 timer.reschedule(timer.when() + 1)
...                 print(f"Now timer ends at {timer.when()}")
...             print("Done with loop")
...         print("Done with timer")
...     except TimeoutError:
...         print("Function timed out")
...     print("Exiting function")
...
>>> asyncio.run(count_with_timeout())
Got value 0
Now timer ends at 113045.671532505
Got value 1
Now timer ends at 113046.671532505
Got value 2
Now timer ends at 113047.671532505
Got value 3
Now timer ends at 113048.671532505
Got value 4
Now timer ends at 113049.671532505
Got value 5
Now timer ends at 113050.671532505
Function timed out
Exiting function

First we define slow_count() as an async generator like range() but with increasing pauses between the items. The count_with_timeout() async function then declares a timeout, initially set to 9 seconds, and starts iterating through the generator. On each loop we add 1 second to the remaining time, but this is not enough to keep the timeout ahead of the increasing delays of the generator. On the sixth iteration of the async for loop, the timeout triggers, cancelling the running task and raising TimeoutError.

As you’d expect this exception terminates the loop early, bypassing the two intervening print() statements and jumps stright into the except clause — after this the rest of the count_with_timeout() task runs to completion.

Cancellation Count¶

If this section gets a little confusing, do go ahead and skip it — this is an obscure corner of the changes about which most people will probably never need to care.

As a side-effect of the way TaskGroup and timeout() are implemented, there are also some minor changes to Task.cancel() and related functions. Instead of simply tracking a bool, which is set by cancel(), the number of requests is now tracked, a little like a semaphore. So cancel() increments this number, and a new uncancel() method decrements it. There’s another new method cancelling() which returns the current number of pending requests.

I believe these new methods are generally intended for use by the asyncio internals rather than the developer directly, but it’s useful to know about them. Essentially since both of these cases are block-delimited, as they both use context managers, the uncancel() method offers a way to reverse the cancellation which happened as a result of the timeout or task completion.

As an illustration of why this is useful, if you look at the TaskGroup implementation, for example, you’ll see the case which handles a task terminating due to exception. In this case, all the remaining tasks are cancelled, to avoid endlessly blocking on some await within the context-managed block. Then, during the __aexit__() function, all those tasks have their cancellation reversed with uncancel(), so other code outside the context-managed block can continue to interact with them.

This also makes clear why it had to be changed from a bool to an int — this process shouldn’t interfere with any other explicit calls to cancel() in user code, which should still be honoured. With a bool any trace of these would be lost with the uncancel().

To reiterate, this is somewhat implementation black magic, but it’s implemented with public methdods of Task so I thought the semantics may be potentially interesting to note.

Barriers¶

In the threading module there’s Barrier class which, as you might reasonably surmise, implements a barrier synchronisation method. The Python version is a simple primitive which blocks threads until they reach a pre-defined quorum, at which point all threads are released simultaneously. The Barrier instance is constructed with the number of threads it expects, and each thread calls wait() on it. Once the requisite number of threads is waiting, they’re all released and their wait() function returns. You can also specify an action argument to the Barrier constructor, which is a callable that is invoked by one of threads prior to them all being released.

For example, perhaps you have a set of worker threads all controlled by some global configuration, and you need to reload that configuration — but you don’t want each thread to be in the middle of a calculation when you do. There are several ways to achieve that, but one is to just force each thread to wait in a barrier whilst the configuration update happens.

By now you’re probably wondering why I’m blathering on about the threading module in a section about asyncio. Well, the relevance is that there is now a corresponding asyncio.Barrier class which operates in more or less the same way, although it doesn’t provide the action parameter.

...
>>> import asyncio
>>> import random
>>> async def run_around(name, barrier):
...     for i in range(3):
...         await asyncio.sleep(random.randint(1, 3))
...         print(f"Task {name} waiting...")
...         await barrier.wait()
...         print(f"Task {name} released!")
...     print(f"Task {name} exiting")
...
>>> async def test_barrier():
...     async with asyncio.TaskGroup() as tasks:
...         my_barrier = asyncio.Barrier(3)
...         tasks.create_task(run_around("A", my_barrier))
...         tasks.create_task(run_around("B", my_barrier))
...         tasks.create_task(run_around("C", my_barrier))
>>> asyncio.run(test_barrier())
Task C waiting...
Task A waiting...
Task B waiting...
Task B released!
Task C released!
Task A released!
Task B waiting...
Task A waiting...
Task C waiting...
Task C released!
Task B released!
Task A released!
Task C waiting...
Task A waiting...
Task B waiting...
Task B released!
Task B exiting
Task C released!
Task C exiting
Task A released!
Task A exiting

This is fairly straightforward, but what happens if one of the tasks exits uncleanly? Since the number of tasks for which the barrier is waiting is fixed, the remainder will all end up waiting indefinitely.

The solution is to call the abort() method on the Barrier object in this case. This flags the barrier as “broken” and will cause any current or future wait() calls on the barrier to immediately raise BrokenBarrierError. There’s also a reset() method to return the barrier to a default empty state, which will remove any “broken” state, but also cause any currently blocked calls to wait() to immediately raise BrokenBarrierError.

It’s also worth being aware that if one of the waiting tasks is cancelled with its cancel() method, the wait() call will raise CancelledError for that task, and if the barrier is currently “filling” then the number of waiting tasks is decremented by one. This can also cause barriers to wait indefinitely if this leaves insufficient tasks to meet the quorum.

Smaller Changes¶

Some additional changes in this release which I didn’t feel warranted too much detail.

asyncio.Runner

The addition of asyncio.run() in Python 3.7 certainly simplified running async functions. However, the fact that every invocation to run() creates a new event loop and contextvars.Context is sometimes annoying, particularly in cases like unit tests where you may want to reuse the same one multiple times. This is now possible using asyncio.Runner() as a context manager and using its run() method instead of asyncio.run().

loop.create_connection() Exception Group

The create_connection() method of the event loop function now has a new parameter all_errors which, if True, means all exceptions raised are returned as an ExceptionGroup, as discussed a few articles back.

StreamWriter.start_tls()

The asyncio.StreamWriter now has a start_tls() method for upgrading a TCP connection to TLS and changing the transport used as appropriate.

UDP Raw Sockets

The event loop supports raw sockets at present, but lacks methods to pass UDP traffic over them. In this release that’s been remedied, and the event loop now offers methods such as sock_sendto(), sock_recvfrom() and sock_recvfrom_into().

socket¶

Briefly, there are a couple of very brief changes to the socket module.

The first one is a bit niche. The CAN (Controller Area Network), that was supported in Python 3.3 on Linux, is now also supported on NetBSD. Enough said on that.

The second change is that socket.create_connection() takes an all_errors parameter which, if True, means all exceptions raised are returned as an ExceptionGroup. This is the same change as for create_connection() method on the asyncio event loop, which I mentioned above. Although a small change, I thought it was worth highlighting as one of the places in this release where ExceptionGroup is already supported.

Development Tools — typing¶

As always there are some type hinting changes in this release. We already saw the major changes in this release in an earlier article, but there are also a slew of smaller changes which I’ll cover here.

Bottom Type: Never¶

Since Python 3.5.4 and 3.6.2, typing.NoReturn was added as an annotation to indicate that a function never returns — or to put it another way, the only way of exiting the function is via an exception. I think this slipped through the net of these articles since it was added between the annual major releases.

Since it was added, this has found another use as a bottom type — that is, an annotation that has no possible values. This has utility in cases other than the return type of functions — for example, if you’re dispatching based on the type of an argument and you want to demonstrate that you’ve checked every possible type that value can take, you can declare an else clause where the argument is given the bottom type. If the type checker finds a path to this statement with some possible value of the argument, it can raise an error.

For this purpose, as a bottom type, however, the name NoReturn isn’t ideal. As a result, typing.Never has been added in Python 3.11 for this purpose. Code should treat these as distinct annotations, and only use whichever is appropriate for their case, but in reality they’re interchangeable for the moment.

In addition to typing.Never, there’s also a new assertion typing.assert_never(). This takes a single argument which is type annotated to Never, so at type-check time this allows a type checker to validate that all possible types of a value have been branched off earlier — the annotation will only match if that line of code is not reachable for any conceivable type that the argument could have.

In addition to this type checking behaviour, at runtime the function will also raise an AssertionError if that line is actually reached. So it acts a little like assert False in this regard.

For example, consider a case where you have a function foo_int() which takes an int parameter, and you want to wrap a dispatcher around it which converts other types to int to make it more flexible. If you want to accept int and float you could write something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import typing

def foo_int(value: int) -> None:
    print("Value", value)

def foo_dispatcher(value: int|float) -> None:
    match value:
        case int():
            foo_int(value)
        case float():
            foo_int(round(value))
        case _:
            typing.assert_never(value)

foo_dispatcher(9)
foo_dispatcher(15.8)

As it stands, this is all self-consistent. But now imagine someone changes the annotation of value to int|float|str, but forgets to add a case to handle str in the match block.

 5
 6
 7
 8
 9
10
11
12
13
14

⋮
def foo_dispatcher(value: int|float|str) -> None:
    match value:
        case int():
            foo_int(value)
        case float():
            foo_int(round(value))
        case _:
            typing.assert_never(value)
⋮

At this point if you run mypy over this code, it’ll warn you of the problem:

$ mypy --python-version=3.11 assert_test.py
assert_test.py:13: error: Argument 1 to "assert_never" has incompatible type "str"; expected "NoReturn"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

Type Diagnostics: reveal_type() and assert_type()¶

The new reveal_type() function is intended to help with locating issues with type mismatches. When a type checker encounters a call to reveal_type(), it will emit a diagnostic indicating the range of types that it believes the argument can hold at that point. At runtime this call also has an effect, printing the type of the actual value at that point to stderr, and also returning its argument unchanged so that it can be used in the middle of an expression.

The assert_type() function is similar, except that it at type checking time it asserts that the type of an expression is exactly as specified — the type checker should emit an error if this is not the case. At runtime it does nothing except return the first argument unchanged.

Consider the following code, which has a few uses of both reveal_type() and assert_type().

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

from typing import Any, reveal_type, assert_type

def dispatcher(arg: Any) -> Any:
    match arg:
        case int():
            return assert_type(arg, int)
        case str():
            return assert_type(arg, str)
        case float():
            return assert_type(arg, float)
        case _:
            raise ValueError(arg)

value = eval(input("Expression: "))
reveal_type(value)
try:
    new_value = reveal_type(dispatcher(value))
    assert_type(new_value, int|str|float)
    print(f"Value is {new_value}")
except ValueError:
    print("Unsupported type")

When we check this with mypy, it emits “note” events for each of the reveal_type() calls. It also detects an error in line 18, since we’ve asserted that the return value of dispatcher() must be Union[int, str, float], but this conflicts with the type Any that mypy has derived.

$ mypy --python-version=3.11 dispatcher.py
dispatcher.py:15: note: Revealed type is "Any"
dispatcher.py:17: note: Revealed type is "Any"
dispatcher.py:18: error: Expression is of type "Any", not "Union[int, str, float]"  [assert-type]
Found 1 error in 1 file (checked 1 source file)

This is an interesting case, because it’s clear from the implementation that dispatcher() can only return those three types — anything else will cause a ValueError instead. However, mypy is basing its conclusion on the annotated return type rather than any data flow analysis.

When we execute the code, we also see the output on stderr of the reveal_type() calls.

$ python dispatcher.py
Expression: 12.3
Runtime type is 'float'
Runtime type is 'float'
Value is 12.3
$ python dispatcher.py
Expression: b"hello"
Runtime type is 'bytes'
Unsupported type

Introspecting Overloads¶

The @overload decorator has existed in typing since it was added in Python 3.5. It allows you to declare a series of different signatures for a function with no implementation, which specify the valid signatures that the type checker should use, and then a single implementation, which is the function which is used at runtime. If you need a refresher, you can see an example in the code snippet below.

In Python 3.11 what’s new is the addition of typing.get_overloads() to obtain a list of the function signatures overloaded into a specific function. It returns a list of references to the functions themselves, and you can use things like inspect.get_annotations() on them as normal.

The internal registry used to store this information does take up some additional memory, so there is also a typing.clear_overloads() function to wipe it — this will mean that get_overloads() will return no results, however.

The snippet below shows how this works.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

import inspect
import math
import typing

@typing.overload
def perm_from_100(arg: int) -> int:
    ...

@typing.overload
def perm_from_100(arg: float) -> int:
    ...

@typing.overload
def perm_from_100(arg: str) -> str:
    ...

def perm_from_100(arg):
    if isinstance(arg, str):
        return_str = True
        arg = float(arg) if "." in arg else int(arg)
    else:
        return_str = False
    ret = math.perm(100, round(arg))
    if return_str:
        return str(ret)
    return ret

for func in typing.get_overloads(perm_from_100):
    print(func.__name__, inspect.get_annotations(func))

$ python overload_example.py
perm_from_100 {'arg': <class 'int'>, 'return': <class 'int'>}
perm_from_100 {'arg': <class 'float'>, 'return': <class 'int'>}
perm_from_100 {'arg': <class 'str'>, 'return': <class 'str'>}

Smaller Changes¶

Generic TypedDict and NamedTuple

Prior to this release, TypedDict subclasses could not mix in any other base classes. This restriction has been relaxed in the case of Generic in this release, so you can now define user-specified generics with TypedDict semantics. A similar change has also been made to NamedTuple — this actually used to work but was broken by a change in 3.9, and has now been made to work once more (you can see discussion in bpo-43923).

Inherit from Any

It’s now possible to inherit from typing.Any, which used to raise an error. This may be useful in some obscure cases where you’re inheriting from a class in code which hasn’t been annotated, for example. This allows you to add annotations for cases which you want to check, but without restricting use of other functions and attributes which come from the base class. You can find discussion and examples on this thread on the typing-sig mailing list.

@final Now Sets __final__

The @typing.final decorator used to be an identity function, as it was envisioned as simply an annotation for static type checkers. However, some type checkers do some or all of their checks on runtime objects (e.g. pydantic, pyanalyze) and to support these the decorator has now been updated to set __final__ = True on decorated classes.

typing.get_type_hints() Changes

There are several smallish fixes in get_type_hints(), including loosening of runtime requirements on type annotations, support for evaluating strings as forward references and no longer adding Optional to parameters which default to None. I think they’re small and niche enough not to be worth a detailed discussion, so if you want to know more feel free to check out the GitHub issues: bpo-46644, bpo-41370, bpo-46195, bpo-46553 and bpo-46571.

Python Runtime Services¶

Finally, we complete our look at Python 3.11 by looking at some changes to Python’s runtime services.

contextlib¶

A small change in contextlib is the addition of the chdir() context manager which changes to a specified working directory, and then changes back again at the end of the block. It’s not that it’s particularly hard to do this yourself, but it’s common enough that having something in the standard library to deal with it is handy.

>>> import contextlib, os
>>> os.getcwd()
'/Users/andy'
>>> with contextlib.chdir("/etc"):
...     print(os.getcwd())
...
/private/etc
>>> os.getcwd()
'/Users/andy'

Of course, since the current working directory is an inherently global property of a process, it should go without saying that this isn’t safe for use in multithreaded or async situations, unless you know what you’re doing. A particularly subtle case would be a generator where if you yield within the context managed block then the code consuming from the generator would be impacted by the change. This sort of thing is fairly obvious when you think about it, but it would be easy to miss when different pieces of code come together.

dataclasses¶

When determining whether a value can be used as the default of a field, without using default_factory, it used to be the case that any type except subclasses of list, dict or set was allowed. As well as allowing mutable types that should have been blocked, this also prevented types like frozenset which should have been allowed.

In this release the check has been updated to allow any hashable type — i.e. any value with a __hash__() method — which seems like a much more robust check.

inspect¶

There are a number of improvements to the inspect module, functions for getting information about live objects, in this release.

getmembers_static()¶

The inspect.getmembers() function has been around since long before Python 3, and returns a list of 2-tuples of (name, value) for each member of the object. It retrieves these values using getattr(), however, which means that if an object uses properties or a __getattr__() method then this could have side-effects — this is not necessarily ideal for cases where you’re using inspect and you want the introspection to be non-invasive.

The inspect module already provides getattr_static(), added in Python 3.2, which recovers the value of an attribute directly from __dict__ on the object instead of triggering the usual getattr() mechanisms. In Python 3.11 there’s now a getmembers_static() as well, which is the same as getmembers() except that the values of each attribute are recovered using getattr_static().

Here’s a quick example of all this, using a property attribute to log access.

>>> import inspect
>>> class MyClass:
...     def __init__(self):
...         self._value = None
...     @property
...     def value(self):
...         print("Getter called")
...         return self._value
...     @value.setter
...     def value(self, new_value):
...         print("Setter called")
...         self._value = new_value
...
>>> instance = MyClass()
>>> instance.value = 123
Setter called
>>> instance.value
Getter called
123
>>> inspect.getattr_static(instance, "value")
<property object at 0x108f54770>
>>> dict(inspect.getmembers(instance))["value"]
Getter called
123
>>> dict(inspect.getmembers_static(instance))["value"]
<property object at 0x108f54770>

ismethodwrapper() and isroutine()¶

A small change to inspect adds the function ismethodwrapper() which returns True if the argument is an instance of types.MethodWrapperType.

This is probably one of those answers which leaves you with more questions, but a full explanation requires drilling into the way that Python maps dunder names like __str__ into C functions to implement them when using extension modules (i.e. those written in C or other compiled languages), and that’s something that’s well outside the scope of this article³.

For the purposes of this discussion, suffice to say that when you have an object implemented in C, such as the base class object, then the type of a dunder method like __str__() is exposed as types.WrapperDescriptorType. The type of a bound method (i.e. on a concrete instance) is exposed as types.MethodWrapperType. You can see this illustrated below.

>>> isinstance(object.__str__, types.WrapperDescriptorType)
True
>>> isinstance(object().__str__, types.MethodWrapperType)
True

This brings us back to the original point — this new method ismethodwrapper() is really just checking isinstance(obj, types.MethodWrapperType). So why do we need this?

Well, it’s part of a change to inspect.isroutine(), which previously would not return True for these bound dunder methods. In Python 3.11, however, the result is ismethodwrapper() has been added to the conditions checked by isroutine(), so now the result is correct in this case.

The console outputs below show the comparison with Python 3.10.

$ python3.10
Python 3.10.8 (main, Oct 13 2022, 10:17:43) [Clang 14.0.0 (clang-1400.0.29.102)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import inspect
>>> inspect.isroutine(object.__str__)
True
>>> inspect.isroutine(object().__str__)
False

$ python3.11
Python 3.11.1 (main, Dec 12 2022, 08:56:30) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import inspect
>>> inspect.isroutine(object.__str__)
True
>>> inspect.isroutine(object().__str__)
True

New Frame Objects¶

In a previous article on this release we discussed the addition of fine-grained error reporting, and the resultant changes to how bytecode instructions are stored. As a result of these changes, there are also some updates to the inspect module to include this information.

Functions which return information about frames continue to use the FrameInfo class, but whilst this used to be a simple namedtuple, it’s now a subclass which also adds the new positions attribute. Because it’s a subclass of the original structure, existing code which doesn’t use positions should be totally unaffected. The same change has also been made to the Traceback class. Do be aware that in both cases positions may be None if position information wasn’t provided.

These changes affect the values returned by the following functions:

• inspect.getframeinfo()
• inspect.getouterframes()
• inspect.getinnerframes()
• inspect.stack()
• inspect.trace()

Here’s a short script to illustrate this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

#         1    1    2    2    3    3
#    5    0    5    0    5    0    5
import inspect

def inner_function():
    for frame_info in inspect.stack():
        print(frame_info.positions)

def outer_function():
    inner_function()

if __name__ == "__main__":
    outer_function()

$ python frames_example.py
Positions(lineno=6, end_lineno=6, col_offset=22, end_col_offset=37)
Positions(lineno=10, end_lineno=10, col_offset=4, end_col_offset=20)
Positions(lineno=13, end_lineno=13, col_offset=4, end_col_offset=20)

As an aside, inspect.currentframe() continues to return the raw frame object, not a FrameInfo instance, so that’s unaffected by this change.

sys¶

The sys module has some changes relating to the way exceptions are represented within the interpreter, as well as a flag for the safe import path feature we covered a couple of articles back.

Exception Changes¶

A couple of updates related to a change in the way that exceptions are represented within the interpreter. Exceptions have long been represented as a 3-tuple of (type, instance, traceback) — however, this is redundant since the type and traceback fields can be obtained from the Exception instance which forms the middle item. In this release, therefore, only the instance is stored and the other fields derived from it as needed.

First up, this has a subtle impact on sys.exc_info(). There is no change to the return type of exc_info(), because of course that would break backwards compatibility unnecessarily. If the exception instance is exc, the function effectively returns (type(exc), exc, exc.__traceback__). However this does introduce a subtle change of behaviour which is that if the currently-handled exception is changed during the course of exception handling, the corresponding values returned by subsequent calls to exc_info() will also change, whereas previously they would have been preserved. The new behaviour seems more correct to me, at least, so this seems like a positive change, if an obscure one.

The second update is a new sys.exception() method which is similar to exc_info() but just returns the current exception instance, avoiding the redundancy of the 3-tuple representation. This is probably rather more intuitive to the majority of developers, anyway, and seems like a decent change. I’m not aware of any plans to deprecate exc_info(), but that seems to me the next logical step — but perhaps the small benefit and potentially high cost to the community mean that won’t happen for a long time, if at all.

Safe Import Path Flag¶

A couple of articles back in this series we looked briefly as the safe import path feature, enabled by passing -P to the interpreter or defining the PYTHONSAFEPATH environment variable.

There is a corresponding change in sys.flags to add a new safe_path attribute which reflects whether this option is active.

$ python -c 'import sys; print(sys.flags.safe_path)'
False
$ python -P -c 'import sys; print(sys.flags.safe_path)'
True
$ PYTHONSAFEPATH=1 python -c 'import sys; print(sys.flags.safe_path)'
True

sysconfig — venv Installation Schemes¶

In Python 3.10 the concept of installation schemes was added, which allows runtime introspection of the paths into which Python and its libraries are expected to be installed on a given platform. Each platform has assigned a name, which reflects the particular locations used on that platform. For example, posix_prefix is used for most Unix-like operating systems if installed globally, or posix_user if the installation is done within the user’s home directory. Similarity, nt is used for Windows and nt_user for a single user install.

The change in Python 3.11 is that three new schemes have been added to virtual environments:

• posix_venv is the scheme used by virtual environments on POSIX platforms.
• nt_venv is the scheme used by virtual environments on Windows.
• venv uses values from whichever of the above two layouts matches the current platform.

As with other installation schemes, these specify the paths used for various components. Currently Python uses eight paths:

• stdlib contains general standard library files.
• platstdlib constains platform-specific standard library files.
• platlib contains site-specific platform-specific files.
• purelib contains site-specific general files.
• include contains general header files for the Python C API.
• platinclude contains platform-specific header files for the C API.
• scripts contains executable scripts.
• data contains static data files.

Here’s an example of the new installation schemes, called from within a venv that was set up by pyenv on MacOS.

>>> from pprint import pprint
>>> import sysconfig
>>> sysconfig.get_default_scheme()
'venv'
>>> sysconfig.get_preferred_scheme("prefix")
'venv'
>>> sysconfig.get_preferred_scheme("home")
'posix_home'
>>> sysconfig.get_preferred_scheme("user")
'posix_user'
>>> pprint(sysconfig.get_paths())
{'data': '/Users/andy/.pyenv/versions/3.11.1/envs/python-3.11',
 'include': '/Users/andy/.pyenv/versions/3.11.1/include/python3.11',
 'platinclude': '/Users/andy/.pyenv/versions/3.11.1/include/python3.11',
 'platlib': '/Users/andy/.pyenv/versions/3.11.1/envs/python-3.11/lib/python3.11/site-packages',
 'platstdlib': '/Users/andy/.pyenv/versions/3.11.1/envs/python-3.11/lib/python3.11',
 'purelib': '/Users/andy/.pyenv/versions/3.11.1/envs/python-3.11/lib/python3.11/site-packages',
 'scripts': '/Users/andy/.pyenv/versions/3.11.1/envs/python-3.11/bin',
 'stdlib': '/Users/andy/.pyenv/versions/3.11.1/lib/python3.11'}
>>> pprint(sysconfig.get_paths("nt_venv"))
{'data': '/Users/andy/.pyenv/versions/3.11.1/envs/python-3.11',
 'include': '/Users/andy/.pyenv/versions/3.11.1/Include',
 'platinclude': '/Users/andy/.pyenv/versions/3.11.1/Include',
 'platlib': '/Users/andy/.pyenv/versions/3.11.1/envs/python-3.11/Lib/site-packages',
 'platstdlib': '/Users/andy/.pyenv/versions/3.11.1/envs/python-3.11/Lib',
 'purelib': '/Users/andy/.pyenv/versions/3.11.1/envs/python-3.11/Lib/site-packages',
 'scripts': '/Users/andy/.pyenv/versions/3.11.1/envs/python-3.11/Scripts',
 'stdlib': '/Users/andy/.pyenv/versions/3.11.1/Lib'}

traceback¶

There are a couple of improvements to the traceback module, to control how frames are displayed in the traceback, and also to log a TracebackException to a file.

Formatting Stack Frames¶

When dealing with stack traces in traceback, individual frames are represented by FrameSummary objects and a sequence of frames are represented by a StackSummary object. The change in this release is that there’s a new method format_frame_summary() on the StackSummary object which allows you to customise the way the stack frames are printed. This method is called once for each frame, passing in the FrameSummary object, and is expected to return the string representation of that frame, generally including a terminating newline.

To illustrate this, take a look at the code below. Initially the BriefStackSummary class isn’t used because line 9 is commented out — this is so we can see the comparison between the way Python prints the stack by default, and the way the code below formats it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

import traceback

class BriefStackSummary(traceback.StackSummary):

    def format_frame_summary(self, frame):
        return f"Stack frame: {frame.name} at {frame.filename}:{frame.lineno}\n"

# Monkey patch it in for the purposes of this example
# traceback.StackSummary = BriefStackSummary

def inner(func, count):
    if count > 0:
        return inner(func, count - 1)
    else:
        return func(True)

def outer(return_stack=False):
    if return_stack:
        return traceback.extract_stack()
    else:
        return inner(outer, 10)

stack_summary = outer()
print("".join(stack_summary.format()))

As it’s shown above, this code will show a stack backtrace using the standard formatting — this is shown below.

$ python traceback_example.py
  File "/Users/andy/traceback_example.py", line 23, in <module>
    stack_summary = outer()
  File "/Users/andy/traceback_example.py", line 21, in outer
    return inner(outer, 10)
  File "/Users/andy/traceback_example.py", line 13, in inner
    return inner(func, count - 1)
  File "/Users/andy/traceback_example.py", line 13, in inner
    return inner(func, count - 1)
  File "/Users/andy/traceback_example.py", line 13, in inner
    return inner(func, count - 1)
  [Previous line repeated 7 more times]
  File "/Users/andy/traceback_example.py", line 15, in inner
    return func(True)
  File "/Users/andy/traceback_example.py", line 19, in outer
    return traceback.extract_stack()

The main point of interest here is that the dept of recursion within inner() is truncated in the backtrace — only the first three calls are shown, followed by a message indicating how many other calls were hidden.

If we then just uncomment line 9 to monkey-patch in BriefStackSummary then we can see below how the overridden format_frame_summary() is called instead. However, the truncation of repeated recursion still happens at a higher level.

$ python traceback_example.py
Stack frame: <module> at /Users/andy/traceback_example.py:23
Stack frame: outer at /Users/andy/traceback_example.py:21
Stack frame: inner at /Users/andy/traceback_example.py:13
Stack frame: inner at /Users/andy/traceback_example.py:13
Stack frame: inner at /Users/andy/traceback_example.py:13
  [Previous line repeated 7 more times]
Stack frame: inner at /Users/andy/traceback_example.py:15
Stack frame: outer at /Users/andy/traceback_example.py:19

Logging TracebackException to a File¶

The TracebackException was added back in Python 3.5, as part of bpo-17911, as a way to capture a traceback from an exception in a lightweight manner which doesn’t involve doing formatting and other more expensive operations. This can be useful in cases where you want to capture a traceback knowing that there’s only a small chance of needing to log it, but where it’s important to do so in a small number of cases.

In this release, the TracebackException has acquired a print() method to log it to a file. By default this is standard error, but you can pass any file-like object.

>>> import traceback
>>> def inner():
...     raise Exception("Hello")
...
>>> def outer():
...     inner()
...
>>> try:
...     outer()
... except Exception as exc:
...     tb = traceback.TracebackException.from_exception(exc)
...
>>> tb.print()
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 2, in outer
  File "<stdin>", line 2, in inner
Exception: Hello

warnings — More Concise Warning Filters¶

In certain situations you may want to ignore warnings that would normally be raised by Python — a common example of this is during unit tests. If you don’t mind doing this globally, you can use warnings.simplefilter(), but if you want to do this for only a specific context, restoring the filter on exit, then you need to do something like this.

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    ...

If you’re having to repeat this for a lot of unit tests then this can be quite an irritation. As of Python 3.11, however, there’s now a more concise approach where you can pass the filter directly to catch_warnings(). Generally the action argument will be what you’ll use, but the other arguments to simplefilter() can be passed as well.

>>> import warnings
>>> warnings.warn("This is a warning")
<stdin>:1: UserWarning: This is a warning
>>> with warnings.catch_warnings(action="ignore"):
...     warnings.warn("This is another warning")

Conclusion¶

So that brings us to the end of the Python 3.11 release. Some obscure stuff in this last article, but there are some handy parts — some of the asyncio changes are good to see for those who like to avoid the overheads of true threading, the type hint diagnostics are probably handy for tracking down those niggling confusions that can arise at times, and contextlib.chdir() is one of those things that I can’t believe it’s taken this long to add.

This article doesn’t just wrap up 3.11, it also finally brings me bang up to date in the first time in this series — at least for a few months until Python 3.12 approaches its final release. I hope you’ve found these articles interesting and/or useful. Whilst I’ve enjoyed writing them, I must admit I am also looking forward to writing about some other topics — the hardest part will be having to think of ideas for them!