☑ Python 2to3: What’s New in 3.2

In this post I’m going to continue my examination of every Python 3.x release to date with a look at Python 3.2. I seem to remember this as a pretty big one, so there’s some possibility that this article will rival the first one in this series for length. In fact, it got so long that I also implemented “Table of Contents” support in my articles! So, grab yourself a coffee and snacks and let’s jump right in and see what hidden gems await us.

Command-line Arguments¶

We kick off with one of my favourite Python modules, argparse, defined in PEP 389. This is the latest in series of modules for parsing command-line arguments, which is a topic close to my heart as I’ve written a lot of command-line utilities over the years. I spent a number of those years getting increasingly frustrated with the amount of boilerplate I needed to add every time for things like validating arguments and presenting help strings.

Python’s first attempt at this was the getopt module, which was essentially just exposing the POSIX getopt() function in Python, even offering a version that’s compatible with the GNU version. This works, and it’s handy for C programmers familiar with the API, but it makes you do most of the work of validation and such. The next option was optparse, which did a lot more work for you and was very useful indeed.

Whilst optparse did a lot of work of parsing options for you (e.g. --verbose), it left any other arguments in the list for you to parse yourself. This was always slightly frustrating for me, because let’s say you expect the user to pass a list of integers, it seemed inconvenient to force them to use options for it just to take advantage of the parsing and validation the module offers. Also, more complex command-line applications like git often have subcommands which are tedious to validate by hand as well.

The argparse module is a replacement for optparse which aims to address these limitations, and I think by this point we’ve got to something pretty comprehensive. It’s usage is fairly similar to optparse, but adds enough flexibility to parse all sorts of arguments. It also can validate the types of arguments, provide command-line help automatically and allow subcommands to be validated.

The variety of options this module provides are massive, so there’s no way I’m going to attempt an exhaustive examination here. By way of illustration, I’ve implemented a very tiny subset of the git command-line as a demonstration of how subcommands work:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

import argparse
import os

# These functions would normally carry out the subcommands.
def do_status(args):
    print("Normally I'd run the status command here")

def do_log(args):
    print("Normally I'd run the log command here")

# We construct the base parser here and add global options.
parser = argparse.ArgumentParser()
parser.add_argument("--version", action="version", version="%(prog)s 2.24")
parser.add_argument("-C", action="store", dest="working_dir", metavar="<path>",
                    help="Run as if was started in PATH")
parser.add_argument("-p", "--paginate", action="store_true", dest="paginate",
                    help="Enable pagination of output")
parser.add_argument("-P", "--no-pager", action="store_false", dest="paginate",
                    help="Disable pagingation of output")
parser.set_defaults(subcommand=None, use_pager=True, working_dir=os.getcwd())

# We add a "status" subcommand with its own parser.
subparsers = parser.add_subparsers(title="Subcommands", description="Valid subcommands",
                                   help="additional help")
parser_status = subparsers.add_parser("status", help="Show working tree status")
parser_status.add_argument("-s", "--short", action="store_const", const="short",
                           dest="format", help="Use short format")
parser_status.add_argument("-z", action="store_const", const="\x00", dest="lineend",
                           help="Terminate output lines with NUL instead of LF")
parser_status.add_argument("pathspecs", metavar="<pathspec>", nargs="*",
                           help="One or more pathspecs to show")
parser_status.set_defaults(subcommand=do_status, format="long", lineend="\n")

# We add a "log" subcommand as well.
parser_log = subparsers.add_parser("log", help="Show commit logs")
parser_log.add_argument("-p", "--patch", action="store_true", dest="patch",
                        help="Generate patch")
parser_log.set_defaults(subcommand=do_log, patch=False)

# Shows how this parser could be used.
args = parser.parse_args()
if args.subcommand is None:
    print("No subcommand chosen")
    parser.print_help()
else:
    args.subcommand(args)

You can see the command-line help generated by the class below. First up, the output of running fakegit.py --help:

usage: fakegit.py [-h] [--version] [-C <path>] [-p] [-P] {status,log} ...

optional arguments:
  -h, --help      show this help message and exit
  --version       show program's version number and exit
  -C <path>       Run as if was started in PATH
  -p, --paginate  Enable pagination of output
  -P, --no-pager  Disable pagingation of output

Subcommands:
  Valid subcommands

  {status,log}    additional help
    status        Show working tree status
    log           Show commit logs

The subcommands also support their own command-line help, such as fakegit.py status --help:

usage: fakegit.py status [-h] [-s] [-z] [<pathspec> [<pathspec> ...]]

positional arguments:
  <pathspec>   One or more pathspecs to show

optional arguments:
  -h, --help   show this help message and exit
  -s, --short  Use short format
  -z           Terminate output lines with NUL instead of LF

Logging¶

The logging module has acquired the ability to be configured by passing a dict, as per PEP 391. Previously it could accept a config file in .ini format as parsed by the configparser module, but formats such as JSON and YAML are becoming more popular these days. To allow these to be used, logging has allowed a dict to be passed specifying the configuration, given that most of these formats can be trivial reconstructed into that format, a illustrated for JSON:

1
2
3
4
5

import json
import logging.config
with open("logging-config.json") as conf_fd:
    config = json.load(conf_fd)
logging.config.dictConfig(config)

When you’re packaging a decent sized application storing logging configuration in a file makes it easier to maintain the logging configuration vs. the option of hard-coding it in executable code. For example, it becomes easier to swap in a different logging configuration in different environments (e.g. pre-production and production). The fact that more popular formats can now be supported will open this flexibility to more developers.

In addition to this, the logging.basicConfig() function now has a style parameter where you can select which type of string formatting token to use for the format string itself. All of the following are equivalent:

>>> import logging
>>> logging.basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s")
>>> logging.basicConfig(style='{', format="{name} -> {levelname} {message}")
>>> logging.basicConfig(style='$', format="$name -> $levelname: $message")

Also, if a log event occurs prior to configuring logging, there is a default setup of a StreamHandler connected to sys.stderr, which displays any message of WARNING level or higher. If you need to fiddle with this handler for any reason, it’s available as logging.lastResort.

Some other smaller changes:

• Levels can now be supplied to setLevel() as strings such as INFO instead of integers like logging.INFO.
• A getChild() method on Logger instances now returns a logger with a suffix appended to the name. For example, logging.getLogger("foo").getChild("bar.baz") will return the same logger as logging.getLogger("foo.bar.baz"). This is convenient when the first level of the name is __name__, as it often is by convention, or in cases where a parent logger is passed to some code which wants to create its own child logger from it.
• The hasHandlers() method has also been added to Logger which returns True iff this logger, or a parent to which events are propagated, has at least one configured handler.
• A new logging.setLogRecordFactory() and a corresponding getLogRecordFactory() have been added to allow programmers to override log record creation process.

Concurrency¶

There are a number of changes in concurrency this release.

Futures¶

The largest change is a new concurrent.futures module in the library, specified by PEP 3148, and it’s a pretty useful one. The intention with the new concurrent namespace is to collect together high-level code for managing concurrency, but so far it’s only acquired the one futures module.

The intention here is to provide what has become a standard abstraction over concurrent operations which represents the eventual result of a concurrent operation. In the Python module, the API style is deliberately decoupled from the implementation detail of what form of concurrency is used, whether it’s a thread, another process or some RPC to another host. This is useful as it allows the style to be potentially changed later if necessary without invalidating the business logic around it.

The style is to construct an executor which is where the flavour of concurrency is selected. Currently the module supports two options, ThreadPoolExecutor and ProcessPoolExecutor. The code can then schedule jobs to the executor, which returns a Future instance which can be used to obtain the results of the operation once it’s complete.

To exercise these in a simple example I wrote a basic password cracker, something that should benefit from parallelisation. I used PBKDF2 with SHA-256 for hashing the passwords, although only with 1000 iterations¹ to keep running times reasonable on my laptop. Also, to keep things simple we assume that the password is a single dictionary word with no variations in case.

For comparison I first wrote a simple implementation which checks every word in /usr/share/dict/words with no parallelism:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import concurrent.futures
import hashlib
import sys

# The salt is normally stored alongside the password hash.
SALT = b"\xe2\x13*\xbb\x1a\xaar\t"
# This is the hash of a dictionary word.
TARGET_HASH = b"\xba<\xdfU\xc3\xdanx\x1b\x1c\xb0js\xf1\x19\xa9\xc5\xb9"\
              b"d!l\xa2\x14\x11K\x86\xac#\xc8\xc7\x8a\x91"
ITERATIONS = 1000

def calc_checksum(line):
    word = line.strip()
    return (word, hashlib.pbkdf2_hmac("sha256", word, SALT, ITERATIONS))

def main():
    with open("/usr/share/dict/words", "rb") as fd:
        for line in fd:
            check = calc_checksum(line)
            if check[1] == TARGET_HASH:
                print(line.strip())
    return 0

if __name__ == "__main__":
    sys.exit(main())

Here’s the output of time running it:

python3 crack.py  257.08s user 0.25s system 99% cpu 4:17.72 total

On my modest 2016 MacBook Pro, this took 4m 17s in total, and the CPU usage figures indicated that one core was basically maxed out, as you’d expect. Then I swapped out main() for a version that used ThreadPoolExecutor from concurrent.futures:

15
16
17
18
19
20
21
22
23
24
25

def main():
    with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
        futures = set()
        with open("/usr/share/dict/words", "rb") as dict_fd:
            for line in dict_fd:
                futures.add(executor.submit(calc_checksum, line))
        for future in concurrent.futures.as_completed(futures):
            word, check = future.result()
            if check == TARGET_HASH:
                print(word)
    return 0

After creating a ThreadPoolExecutor which can use a maximum of 8 worker threads at any time, we then need to submit jobs to the executor. We do this in a loop around reading /usr/share/dict/words, submitting each word as a job to the executor to distribute among its workers. Once all the jobs are submitted, we then wait for them to complete and harvest the results.

Again, here’s the time output:

python3 crack.py  506.42s user 2.50s system 680% cpu 1:14.83 total

With my laptop’s four cores, I’d expect this would run around four times as fast² and it more or less did, allowing for some overhead scheduling the work to the threads. The total run time was 1m 14s so a little less than the expected four times faster, but not a lot. The CPU usage was around 85% of the total of all four cores, which is again roughly what I’d expect. Running in a quarter of the time seems like a pretty good deal for only four lines of additional code!

Finally, just for fun I then swapped out ThreadPoolExecutor for ProcessPoolExecutor, which is the same but using child processes instead of threads:

16
17

    with concurrent.futures.ProcessPoolExecutor(max_workers=8) as executor:
        …

And the time output with processes:

python3 crack.py  575.08s user 15.50s system 669% cpu 1:28.15 total

I didn’t expect this to make much difference to a CPU-bound task like this, provided that the hashing routine are releasing the GIL as they’re supposed to. Indeed, it was actually somewhat slower than the threaded case, taking 1m 28s to execute in total. The total user time was higher for the same amount of work, so this definitely points to some decreased efficiency rather than just differences in background load or similar. I’m assuming that the overhead of the additional IPC and associated memory copying accounts for the increased time, but this sort of thing may well be platform-dependent.

As one final flourish, I tried to reduce the inefficiencies of the multiprocess case by batching the work into larger chunks using a recipe from the itertools documentation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

import concurrent.futures
import hashlib
import itertools
import sys

# The salt is normally stored alongside the password hash.
SALT = b'\xe2\x13*\xbb\x1a\xaar\t'
# This is the hash of a dictionary word.
TARGET_HASH = b"\xba<\xdfU\xc3\xdanx\x1b\x1c\xb0js\xf1\x19\xa9\xc5\xb9"\
              b"d!l\xa2\x14\x11K\x86\xac#\xc8\xc7\x8a\x91"
ITERATIONS = 1000

def calc_checksums(lines):
    return {
        word: hashlib.pbkdf2_hmac('sha256', word, SALT, ITERATIONS)
        for word in (line.strip() for line in lines if line is not None)
    }

def grouper(iterable, n, fillvalue=None):
    args = [iter(iterable)] * n
    return itertools.zip_longest(*args, fillvalue=fillvalue)

def main():
    with concurrent.futures.ProcessPoolExecutor(max_workers=8) as executor:
        futures = set()
        with open("/usr/share/dict/words", "rb") as dict_fd:
            for lines in grouper(dict_fd, 1000):
                futures.add(executor.submit(calc_checksums, lines))
        for future in concurrent.futures.as_completed(futures):
            results = future.result()
            for word, check in results.items():
                if check == TARGET_HASH:
                    print(word)
    return 0

if __name__ == "__main__":
    sys.exit(main())

This definitely made some difference, bringing the time down from 1m 28s to 1m 6s. The CPU usage also indicates more of the CPU time is being spent in user space, presumably due to less IPC.

python3 crack.py  509.95s user 1.20s system 764% cpu 1:06.83 total

I suspect that the multithreaded case would also benefit from some batching, but at this point I thought I’d better draw a line under it or I’d never finish this article.

Overall, I really like the concurrent.futures module, as it takes so much hassle out of processing things in parallel. There are still cases where the threading module is going to be more appropriate, such as some background thread which performs periodic actions asynchronously. But for cases where you have a specific task that you want to tackle synchronously but in parallel, this module wraps up a lot of the annoying details.

I’m excited to see what else might be added to concurrent in the future³!

Threading¶

Despite all the attention on concurrent.futures this release, the threading module has also had some attention with the addition of a new Barrier class. This is initialised with a number of threads to wait for. As individual threads call wait() on the barrier they are held up until all the required number of threads are waiting, at which point all are allowed to proceed simultaneously. This is a little like the join() method, except the threads can continue to execute after the barrier.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

import threading
import time

def wait_thread(name, barrier, delay):
    for i in range(3):
        print("{} starting {}s delay".format(name, delay))
        time.sleep(delay)
        print("{} finishing delay".format(name))
        barrier.wait()

num_threads = 5
barrier = threading.Barrier(num_threads)
threads = [
    threading.Thread(target=wait_thread, args=(str(i), barrier, (i+1) * 2))
    for i in range(num_threads)
]
print("Starting threads...")
for thread in threads:
    thread.start()
for thread in threads:
    thread.join()
print("All threads finished.")

The Barrier can also be initialised with a timeout argument. If the timeout expires before the required number of threads have called wait() then all currently waiting threads are released and a BrokenBarrierError exception is raised from all the wait() methods.

I can think of a few use-cases where this synchronisation primitive might come in handy, such as multiple threads all producing streams of output which need to be synchronised with each other so one of them doesn’t get too far ahead of the other. For example, perhaps one thread is producing chunks of audio data and another chunks of video, you could use a barrier to ensure that neither of them gets ahead of the other.

Another small but useful change in threading is that the Lock.acquire(), RLock.acquire() and Semaphore.acquire() methods can now accept a timeout, instead of only allowing a simple choice between blocking and non-blocking as before. Also there’s been a fix to allow lock acquisitions to be interrupted by signals on pthreads platforms, which means that programs that deadlock on locks can be killed by repeated SIGINT (as opposed to requiring SIGKILL as they used to sometimes).

Finally, threading.RLock has been moved from pure Python to a C implementation, which results in a 10-15x speedup using them.

GIL Overhaul¶

In another change that will impact all forms of threading in CPython, the code behind the GIL has been rewritten. The new implementation aims to offer more predictable switching intervals and reduced overhead due to lock contention.

Prior to this change, the GIL was released after a fixed number of bytecode instructions had been executed. However, this is a very crude way to measure a timeslice since the time taken to execute an instruction can vary from a few nanoseconds to much longer, since not all the expensive C functions in the library release the GIL while they operate. This can mean that scheduling between threads can be very unbalanced depending on their workload.

To replace this, the new approach releases the GIL at a fixed time interval, although the GIL is still only released at an instruction boundary. The specific interval is tunable through sys.setswitchinterval(), with the current default being 5 milliseconds. As well as being a more balanced way to share processor time among threads, this can also reduce the overhead of locks in heavily contended situations — this is because waiting for a lock which is already held by another thread can add significant overhead on some platforms (apparently OS X is particularly impacted by this).

If you want to get technical⁴, threads wishing to take the GIL first wait on a condition variable for it to be released, with a timeout equal to the switch interval. Hence, it’ll wake up either after this interval, or if the GIL is released by the holding thread if that’s earlier. At this point the requesting thread checks whether any context switches have already occurred, and if not it sets the volatile flag gil_drop_request, shared among all threads, to indicate that it’s requesting the release of the GIL. It then continues around this loop until it gets the lock, re-requesting GIL drop after a delay every time a new thread acquires it.

The holding thread, meanwhile, attempts to release the GIL when it performs blocking operations, or otherwise every time around the eval loop it checks if gil_drop_request is set and releases the GIL if so. In so doing, it wakes up any threads which are waiting on the GIL and relies on the OS to ensure fair scheduling among threads.

The advantage of this approach is that it provides an advisory cap on the amount of time a thread may hold the GIL, by delaying setting the gil_drop_request flag, but also allows the eval loop as long as it needs to finish proessing its current bytecode instruction. It also minimises overhead in the simple case when no other thread has requested the GIL.

The final change is around thread switching. Prior to Python 3.2, the GIL was released for a handful of CPU cycles to allow the OS to schedule another thread, and then it was immediately reacquired. This was efficient if the common case is that no other threads are ready to run, and meant that threads running lots of very short opcodes weren’t unduly penalised, but in some cases this delay wasn’t sufficient to trigger the OS to context switch to a different thread. This can cause particular problems with you have an I/O-bound thread competing with a CPU-intensive one — the OS will attempt to schedule the I/O-bound thread, but it will immediately attempt to acquire the GIL and be suspended again. Meanwhile, the CPU-bound thread will tend to cling to the GIL for longer than it should, leading to higher I/O latency.

To combat this, the new system forces a thread switch at the end of the fixed interval if any other threads are waiting on the GIL. The OS is still responsible for scheduling which thread, this change just ensures that it’s not the previously running thread. It does this using a last_holder shared variable which points to the last holder of the GIL. When a thread releases the GIL, it additionally checks if last_holder is its own ID and if so, it waits on a condition variable for the value to change to another thread. This can’t cause a deadlock if no other threads are waiting, because in that case gil_drop_request isn’t set and this whole operation is skipped.

Overall I’m hopeful that these changes should make a positive impact to fair scheduling in multithreaded Python applications. As much as I’m sure everyone would love to find a way to remove the GIL entirely, it doesn’t seem like that’s likely for some time to come.

Date and Time¶

There are a host of small improvements to the datetime module to blast through.

First and foremost is that there’s now a timezone type which implements the tzinfo interface and can be used in simple cases of fixed offsets from UTC (i.e. no DST adjustments or the like). This means that creating a timezone-aware datetime at a known offset from UTC is now straightforward:

>>> from datetime import datetime, timedelta, timezone
>>> # Naive datetime (no timezone attached)
>>> datetime.now()
datetime.datetime(2021, 2, 6, 15, 26, 37, 818998)
>>> # Time in UTC (happens to be my timezone also!)
>>> datetime.now(timezone.utc)
datetime.datetime(2021, 2, 6, 15, 26, 46, 488588, tzinfo=datetime.timezone.utc)
>>> # Current time in New York (UTC-5) ignoring DST
>>> datetime.now(timezone(timedelta(0, -5*3600)))
datetime.datetime(2021, 2, 6, 10, 27, 41, 764597, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=68400)))

Also, timedelta objects can now be multiplied and divided by integers or floats, as well as divided by each other to determine how many of one interval fit into the other interval. This is all fairly straightforward by converting the values to a total number of seconds to perform the operations, but it’s convenient not to have to.

>>> timedelta(1, 20*60*60) * 1.5
datetime.timedelta(days=2, seconds=64800)
>>> timedelta(8, 3600) / 4
datetime.timedelta(days=2, seconds=900)
>>> timedelta(8, 3600) / timedelta(2, 900)
4.0

If you’re using Python to store information about the Late Medieval Period then you’re in luck, as datetime.date.strftime() can now cope with dates prior to 1900. If you want to expand your research to the Dark Ages, however, you’re out of luck since it still only handles dates from 1000 onwards.

Also, use of two-digit years is being discouraged. Until now setting time.accept2dyear to True would allow you to use a 2-digit year in a time tuple and its century would be guessed. However, as of Python 3.2 using this logic will raise you a DeprecationError. Quite right too, 2-digit years are quite an anacronism these days.

String Formatting¶

The str.format() method for string formatting is now joined by str.format_map() which, as the name implies, takes a mapping type to supply arguments by name.

>>> "You must cut down the mightiest {plant} in the forest with... a {fish}!"
    .format_map({"fish": "herring", "plant": "tree"})
'You must cut down the mightiest tree in the forest with... a herring!'

As well as a standard dict instance, you can pass any dict-like object and Python has plenty of these, such as ConfigParser and the objects created by the dbm modules.

There have also been some minor changes to formatting of numeric values as strings. Prior to this release convertinig a float or complex to string form with str() would show fewer decimal places than repr(). This was because the repr() level of precision would occasionally show surprising results, and the pragmatic way to avoid this being more of an issue was to make str() round to a lower precision.

However, as discussed in the previous article, repr() was changed to always select the shortest equivalent representation for these types in Python 3.1. Hence, in Python 3.2 the str() and repr() forms of these types have been unified to the same precision.

Function Enhancements¶

There are a series of enhancements to decorators provided by the functools module, plus a change to contextlib.

Firstly, just to make the example from the previous article more pointless, there is now a functools.lru_cache() decorator which can cache the results of a function based on its parameters. If the function is called with the same parameters, a cached result will be used if present.

This is really handy to drop in to commonly-used but slow functions for a very low effort speed boost. What’s even more useful is that you can call a cache_info() method of the decorated function to get statistics about the cache. There’s also a cache_clear() method if you need to invalidate the cache, although there’s unfortunately no option to clear only selected parameters.

>>> @functools.lru_cache(maxsize=10)
... def slow_func(arg):
...   return arg + 1
...
>>> slow_func(100)
101
>>> slow_func(200)
201
>>> slow_func(100)
101
>>> slow_func.cache_info()
CacheInfo(hits=1, misses=2, maxsize=10, currsize=2)

Secondly, there have been some improvements to functools.wraps() to improve introspection, such as a __wrapped__ attribute pointing back to the original callable and copying __annotations__ across to the wrapped version, if defined.

Thirdly, a new functools.total_ordering() class decorator has been provided. This is very useful for producing classes which support all the rich comparison operators with minimal effort. If you define a class with __eq__ and __lt__ and apply the @functools.total_ordering decorator to it, all the other rich comparision operators will be synthesized.

>>> import functools
>>> @functools.total_ordering
... class MyClass:
...     def __init__(self, value):
...         self.value = value
...     def __lt__(self, other):
...         return self.value < other.value
...     def __eq__(self, other):
...         return self.value == other.value
...
>>> one = MyClass(100)
>>> two = MyClass(200)
>>> one < two
True
>>> one > two
False
>>> one == two
False
>>> one != two
True

Finally, there have been some changes which mean that the contextlib.contextmanager() decorator now results in a function which can be used both as a context manager (as previously) but now also as a function decorator. This could be pretty handy, although bear in mind if you yield a value which is normally bound in a with statement, there’s no equivalent approach for function deocorators.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import contextlib

@contextlib.contextmanager
def log_entry_exit(ident):
    print("Entering {}".format(ident))
    yield
    print("Leaving {}".format(ident))

with log_entry_exit("foo"):
    print("In context")

@log_entry_exit("my_func")
def my_func(value):
    print("Value is {}".format(value))

my_func(123)

Itertools¶

Only one improvement to itertools which is the addition of an accumulate function. However, this has the potential to be pretty handy so I’ve given it its own section.

Passed an iterable, itertools.accumulate() will return the cumulative sum of all elements so far. This works with any type that’s defined for operator +:

>>> import itertools
>>> list(itertools.accumulate([1,2,3,4,5]))
[1, 3, 6, 10, 15]
>>> list(itertools.accumulate([[1,2],[3],[4,5,6]]))
[[1, 2], [1, 2, 3], [1, 2, 3, 4, 5, 6]]

For other types, you can define any binary function to combine them:

>>> import operator
>>> list(itertools.accumulate((set((1,2,3)), set((3,4,5))),
         func=operator.or_))
[{1, 2, 3}, {1, 2, 3, 4, 5}]

And it’s also possible to start with an initial value before anything’s added by providing the initial argument.

Collections¶

The collections module has had a few improvements.

Counter¶

The collections.Counter class added in the previous release has now been extended with a subtract() method which supports negative numbers. Previously the semantics of -= as applied to a Counter would never reduce a value beyond zero — it would simply be removed from the set. This is consistent with how you’d expect a counter to work:

>>> x = Counter(a=10, b=20)
>>> x -= Counter(a=5, b=30)
>>> x
Counter({'a': 5})

However, in its initerpretation as a multiset, you might actually want values to go negative. If so, you can use the new subtract() method:

>>> x = Counter(a=10, b=20)
>>> x.subtract(Counter(a=5, b=30))
>>> x
Counter({'a': 5, 'b': -10})

OrderedDict¶

As demonstrated in the previous article, it’s a little inconvenient to move something to the end of the insertion order. That’s been addressed in this release with the OrderedDict.move_to_end() method. By default this moves the item to the last position in the ordered sequence in the same way as x[key] = x.pop(key) would but is significantly more efficient. Alternatively you can call move_to_end(key, last=False) to move it to the first position in the sequence.

Deque¶

Finally, collections.deque has two new methods, count() and reverse() which allow them to be used in more situations where code was designed to take a list.

>>> import collections
>>> x = collections.deque('antidisestablishmentarianism')
>>> x.count('i')
5
>>> x.reverse()
>>> x
deque(['m', 's', 'i', 'n', 'a', 'i', 'r', 'a', 't', 'n', 'e', 'm', 'h', 's',
'i', 'l', 'b', 'a', 't', 's', 'e', 's', 'i', 'd', 'i', 't', 'n', 'a'])

Internet Modules¶

The three modules email, mailbox and nntplib now correctly support the str and bytes types that Python 3 introduced. In particular, this means that messages in mixed encodings now work correctly. These have also necessitated a number of changes in the mailbox module, which should now work correctly.

The email module has new functions message_from_bytes() and message_from_binary_file(), and classes BytesFeedParser and BytesParser, to allow messages read or stored in the form of bytes to be parsed into model objects. Also, the get_payload() method and Generator class have been updated to properly support the Content-Transfer-Encoding header, encoding or decoding as appropriate.

Sticking with the theme of email, imaplib now supports upgrade of an existing connection to TLS using the new imaplib.IMAP4.starttls() method.

The ftplib.FTP class now supports the context manager protocol to consume socket.error exceptions which are thrown and close the connection when done. This makes it pretty handy, but due to the way that FTP opens additional sockets, you need to be careful to close all these before the context manager exits or your application will hang. Consider the following example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

from ftplib import FTP

with FTP("ftp1.at.proftpd.org") as ftp:
    ftp.login()
    print(ftp.dir())
    sock = ftp.transfercmd("RETR README.MIRRORS")
    while True:
        data = sock.recv(8192)
        if not data:
            break
        print(data)
    sock.close()

Assuming that FTP site is still up, and README.MIRRORS is still available, that should execute fine. However, if you remove that sock.close() line then you should find it just hangs up and never terminiates (perhaps until the TCP connection gets terminated due to being idle).

The socket.create_connection() function can also be used as a context manager, and swallows errors and closes the connection in the same way as the FTP class above.

The ssl module has seen some love with a host of small improvements. There’s a new SSLContext class to hold persistent connection data such as settings, certificates and private keys. This allows the settings to be reused for multiple connections, and provides a wrap_socket() method for creating a socket using the stored details.

There’s a new ssl.match_hostname() which applies RFC-specified rules for confirming that a specified certificate matches the specified hostname. The certificate specification it expects is as returned by SSLSocket.getpeercert(), but it’s not particularly hard to fake as shown in the session below.

>>> import ssl
>>> cert = {'subject': ((('commonName', '*.andy-pearce.com'),),)}
>>> ssl.match_hostname(cert, "www.andy-pearce.com")
>>> ssl.match_hostname(cert, "ftp.andy-pearce.com")
>>> ssl.match_hostname(cert, "www.andy-pearce.org")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ssl.py", line 420, in match_hostname
    raise CertificateError("hostname %r "
ssl.SSLCertVerificationError: ("hostname 'www.andy-pearce.org' doesn't match '*.andy-pearce.com'",)

This release also adds support for SNI (Server Name Indication), which is like virtual hosting but for SSL connections. This removes the longstanding issue whereby you can host as many domains on a single IP address for standard HTTP, but for SSL you needed a unique IP address for each domain. This is essentially beause the virtual hosting of websites is implemented by passing the HTTP Host header, but since the SSL connection is set up prior to sending the HTTP request (by definition!) then the only thing you have to connect to is an IP address. The remote end needs to decide what certificate to send you, and since all it has to decide that is the IP address then you can’t have different certificates for different domains on the same IP. This is problematic because the certificate needs to match the domain or the browser will reject it.

SNI handles this by extending the SSL ClientHello message to include the domain. To implement this with the ssl module in Python, you need to specify the server_hostname parameter to SSLContext.wrap_socket().

The http.client module has been updated to use the new certificate verification processes when using a HTTPSConnection. The request() method is now more flexible on sending request bodies — previously it required a file object, but now it will also accept an iterable providing that an explicit Content-Length header was sent. According to HTTP/1.1 this header shouldn’t be required, since requests can be sent using chunked encoding, which doesn’t require the length of the request body to be known up front. In practice, however, it’s common for servers not to bother supporting chunked requests, despite being mandated by the HTTP/1.1 standard. As a result, it’s sensible to regard Content-Length as mandatory for requests with a body. HTTP/2 has its own methods of streaming data anyway, so once that gains wide acceptance then chunked encoding won’t be used anyway — but given the rate of adoption so far, I wouldn’t hold your breath.

The urllib.parse module has some changes as well, with urlparse() now supporting IPv6 and urldefrag() returning a collections.namedtuple for convenience. The urlencode() function can also now accept both str and bytes for the query parameter.

Markup Languages¶

There have been some significant updates to the xml.etree.ElementTree package, including the addition of the following top-level functions:

fromstringlist()

A handy method which builds an XML document from a series of fragment strings. In partiular this means you can open a filehandle in text mode and have it parsed one line at a time, since iterating the filehandle will yield one line at a time.

tostringlist()

The opposite of fromstringlist(), generates the XML output in chunks. It doesn’t make any guarantees except that joining them all together will yield the same as generating the output as a single string, but in my experience each chunk is around 8192 bytes plus whatever takes it up to the next tag boundary.

register_namespace()

Allows you to register a namespace prefix globally, which can be useful for parsing lots of XML documents which make heavy use of namespaces.

The Element class also has a few extra methods:

Element.extend()

Appends children to the current element from a sequence, which must itself contain Element instances.

Element.iterfind()

As Element.findall() but yields elements instead of returning a list.

Element.itertext()

As Element.findtext() but iterates over all the current element and all child elements as opposed to just returning the first match.

The TreeBuilder class also has acquired the end() method to end the current element and doctype() to handle a doctype declaration.

Finally, a couple of unnecessary methods have been deprecated. Instead of getchildren() you can just use list(elem), and instead of getiterator() just use Element.iter().

Also in 3.2 there’s a new html module, but it only contains one function escape() so far which will do the obvious HTML-escaping.

>>> import html
>>> html.escape("<blink> & <marquee> tags are both deprecated")
'&lt;blink&gt; &amp; &lt;marquee&gt; tags are both deprecated'

Compression and Archiving¶

The gzip.GzipFile class now provides a peek() method which can read a number of bytes from the archive without advancing the read pointer. This can be very useful when implemented parsers which need to choose between various functions to branch into based on what’s next in the file, but which to also leave those functions to read from the file itself as a simpler interface.

The gzip module has also added the compress() and decompress() methods which simply perform in-memory compression/decompression without the need to construct a GzipFile instance. This has been a source of irritation for me in the past, so it’s great to see it finally addressed.

The zipfile module also had some improvements, with the ZipFile class now supporting use as a context manager. Also, the ZipExtFile object has had some performance improvements. This is the file-like object returned when you open a file within a ZIP archive using the ZipFile.open() method. You can also wrap it in io.BufferedReader for even better performance if you’re doing multiple smaller reads.

The tarfile module has changes, with tarfile.TarFile also supporting use as a context manager. Also, the add() method for adding files to the archive now supports a filter parameter which can modify attributes of the files as they’re added, or exclude them altogether. You pass a callable using this parameter, which is called on each file as it’s added. It’s passed a TarInfo structure which has the metainformation about the file, such as the permissions and owner. It can return a modified version of the structure (e.g. to squash all files to being owned by a specific user), or it can return None to block the file from being added.

Finally, the shutil module has also grown a couple of archive-related functions, make_archive() and unpack_archive(). These provide a convenient high-level interface to zipping up multiple files into an archive without having to mess around with the details of the individual compression modules. It also means that the format of your archives can be altered with minimal impact on your code by changing a parameter.

It supports the common archiving formats out of the box, but there’s also a register_archive_format() hook should you wish to add code to handle additional formats.

Math¶

There are some new functions in the math library, some of which look pretty handy.

isfinite()

Returns True iff the float argument is not a special value (e.g. NaN or infinity)

expm1()

Calculates $e^x-1$ for small x in a way which doesn’t result in a loss of precision that can occur when subtracting nearly equal values.

erf() and erfc()

erf() is the Guassian Error Function, which is useful for assessinig how much of an outlier a data point is against a normal distribution. The erfc() function is simply the compliment where erfc(x) == 1 - erf(x).

gamma() and lgamma()

Implements the Gamma Function, which is an extension of factorial to cover continuous and complex numbers. I suspect for almost everyone math.factorial() will be what you’re looking for. Since the value grows so quickly, larger values will yield an OverflowError. To deal with this, the lgamma() function returns the natural logarithm of the value.

Compiled Code¶

There have been a couple of changes to the way that both compiled bytecode and shared object files are stored on disk. More casual users of Python might want to skip over this section, although I would say it’s always helpful to know what’s going on under the hood, if only to help diagnose problems you might run into.

PYC Directories¶

The previous scheme of storing .pyc files in the same directory as the .py files didn’t play nicely when the same source files were being used by multiple different interpreters. The interpreter would note that the file was created by another one, and replace it with its own. As the files swap back and forth, it cancels out the benefits of caching in the first place.

As a result, the name of the interpreter is now added to the .pyc filename, and to stop these files cluttering things up too much they’ve all been moved to a __pycache__ directory.

I suspect many people will not need to care about this any further than it being another entry for the .gitignore file. However, sometimes there can be odd effects with these compiled files, so it’s worth being aware of. For example, if a module is installed and used and then deleted, it might leave the .pyc files behind, confusing programmers who were expecting an import error. If you do want to check for this, there’s a new __cached__ attribute of an imported module indicating the file that was loaded, in addition to the existing __file__ attribute which continues to refer to the source file. The imp module also has some new functions which are useful for scripts that need to correlate source and compiled files for some reason, as illustrated by the session below:

>>> import mylib
>>> print(mylib.__file__)
/tmp/mylib.py
>>> print(mylib.__cached__)
/tmp/__pycache__/mylib.cpython-32.pyc
>>> import imp
>>> imp.get_tag()
'cpython-32'
>>> imp.cache_from_source("/tmp/mylib.py")
'/tmp/__pycache__/mylib.cpython-32.pyc'
>>> imp.source_from_cache("/tmp/__pycache__/mylib.cpython-32.pyc")
'/tmp/mylib.py'

There are also some corresponding changes to the py_compile, compileall and importlib.abc modules which are a bit esoteric to cover here, the documentation has you well covered. You can also find lots of details and a beautiful module loading flowchart in PEP 3147.

Shared Objects¶

Similar changes have been implemented for shared object files. These are compiled against a specific ABI ([Application Binary Interface][abi-wikipedia]) and the ABI is sensitive to major Python version, but also the compilation flags that were used to compiled the interpreter can also affect it. As a result, being able to support the same shared object compiled against multiple ABIs is useful.

The implementation is similar to that for compiled bytecode, where .so files acquire unique filenames based on the ABI and are collected into a shared directory pyshared. The suffix for the current interpreter can be queried using sysconfig:

>>> import sysconfig
>>> sysconfig.get_config_var("SOABI")
'cpython-32m-x86_64-linux-gnu'
>>> sysconfig.get_config_var("EXT_SUFFIX")
'.cpython-32m-x86_64-linux-gnu.so'

The interpreter is cpython, 32 is the version and the letters appended indicate the compilation flags. In this example, m corresponds to pymalloc.

If you want more details, PEP 3149 has a ton of interesting info.

Syntax Changes¶

The syntax of the language has been expanded to allow deletion of a variable that are free in a nested block. If that didn’t make any sense, it’s best explained with an example. The following code was legal in Python 2.x, but would raised a SyntaxError in Python 3.0 or 3.1. In Python 3.2, however, this is once again legal.

1
2
3
4
5
6
7

def outer_function(x):
    def inner():
        # Reference to x in a nested scope.
        return x
    inner()
    # Deleting variable referenced in nested scope.
    del x

So what happens if we were to call inner() again after the del x now? We exactly the same results as if we hadn’t declared the local yet which is to get NameError with the message free variable 'x' referenced before assignment in enclosing scope. The following example may make this message clearer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

def outer_function():
    def inner():
        return x
    # print(inner()) here would raise NameError
    x = 123
    print(inner())  # Prints 123
    x = 456
    print(inner())  # Prints 456
    del x
    # print(inner()) here would raise NameError

An important example of an implicit del is at the end of an except block, so the following code would have raised a SyntaxError in Python 3.0-3.1, but is now valid again:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

import traceback

def func():
    def print_exception():
        traceback.print_exception(type(exc), exc, exc.__traceback__)
    try:
        do_something_here()
    except Exception as exc:
        print_exception()
        # There is an implicit `del exc` here

Diagnostics and Testing¶

A new ResourceWarning has been added to detect issues such as gc.garbage not being empty at interpreter shutdown, indicating finalisation problems with the code. It’s also raised if a file object is destroyed before being properly closed.

This warning is silenced by default, but can be enabled by the warnings module, or using an appropriate -W option on the command-line. The session shown below shows the warning being triggered by destroying an unclosed file object:

>>> warnings.filterwarnings("default")
>>> f = open("/etc/passwd", "rb")
>>> del f
<stdin>:1: ResourceWarning: unclosed file <_io.BufferedReader name='/etc/passwd'>

Note that as of Python 3.4 most of the cases that could cause garbage collection to fail have been resolved, but we have to pretend we don’t know that for now.

There have also been a range of improvements to the unittest module. There are two new assertions, assertWarns() and assertWarnsRegex(), to test whether code raises appropriate warnings (e.g. DeprecationWarning). Another new assertion assertCountEqual() can be used to perform an order-independent comparison of two iterables — functionally this is equivalent to feeding them both into collections.Counter() and comparing the results. There is also a new maxDiff attribute for limiting the size of diff output when logging assertion failures.

Some of the assertion names are being tidied up. Examples include assertRegex() being the new name for assertRegexpMatches() and assertTrue() replacing assert_(). The assertDictContainsSubset() assertion has also been deprecated because the arguments were in the wrong order, so it was never quite clear which argument was required to be a subset of which.

Finally, the command-line usage with python -m unittest has been made more flexible, so you can specify either module names or source file paths to indicate which tests to run. There are also additional options for python -m unittest discover for specifying which directory to search for tests, and a regex filter on the filenames to run.

Optimisations¶

Some performance tweaks are welome to see. Firstly, the peephole optimizer is now smart enough to convert set literals consisting of constants to frozenset. This makes things faster in cases like this:

1
2
3

def is_archive(path):
    _, ext = os.path.splitext(path)
    return ext.lower() in {"zip", "tgz", "gz", "tar", "bz2"}

The Timsort algorithm used by list.sort() and sorted() is now faster and uses less memory when a key function is supplied by changing the way this case is handled internally. The performance and memory consumption of json decoding is also improved, particularly in the case where the same key is used repeatedly.

A faster substring search algorithm, which is based on the Boyer-Moore-Horspool algorithm, is used for a number of methods on str, bytes and bytearray objects such as split(), rsplit(), splitlines(), rfind() and rindex().

Finally, int to str conversions now process two digits at a time to reduce the number of arithmetic operations required.

Other Changes¶

There’s a whole host of little changes which didn’t sit nicely in their own section. Strap in and prepare for the data blast!

• As part of this release PEP 3333 is included as an update to the original PEP 333 which specifies the WSGI (Web Server Gateway Interface) specification. Primarily this tightens up the specifications around request/response header and body strings with regards to the types (str vs. bytes) and encodings to use. This is important reading for anyone building web apps conforming to WSGI.
• range objects now support index() and count() methods, as well as slicing and negative indices, to make them more interoperable with list and other sequences.
• The csv module now supports a unix_dialect output mode where all fields are quoted and lines are terminated with \n. Also, csv.DictWriter has a writeheader() method which writes a row of column headers to the output file, using the key names you provided at construction.
• The tempfile module now provides a TemporaryDirectory context manager for easy cleanup of temporary directories.
• os.popen() and subprocess.Popen() can now act as context managers to automatically close any associated file descriptors.
• configparser.SafeConfigParser has been renamed to ConfigParser to replace the old unsafe one. The default settings have also been updated to make things more predictable.
• The select module has added a PIPE_BUF constant which defines the minimum number of bytes which is guaranteed not to block when a select.select() has indicated that a pipe is ready for writing.
• The callable() builtin from Python 2.x was re-added to the language, as it’s a more readable alternative to isinstance(x, collections.Callable).
• The ast module has a useful literal_eval() function which can be used to evaluate expressions more safely than the builtin eval().
• When writing __repr__() special methods, it’s easy to forget to handle the case where a container can contain a reference to itself, which easily leads to __repr__() calling itself in an endlessly recursive loop. The reprlib module now provides a recursive_repr() decorator which will detect the recursive call and add ... to the string representation instead.
• Hash values of the various different numeric types should now be equal whenever their actual values are equal.
• The hashlib module now provides the algorithms_available set which indicates the hashing algorithms available on the current platform, as well as algorithms_guaranteed which are the algorithms guaranteed to be available on all platforms.
• Some undesiriable behaviour in hasattr() has been fixed. This works by calling getattr() and checking whether an exception is thrown. This approach allows it to support the multiple ways in which an attribute may be provided, such as implementing __getattr__(). However, prior to this release hasattr() would catch any exception, which could mask genuine bugs. As of Python 3.2 it will only catch AttributeError, allowing any other exceptioni to propogate out.
• Bit of an esoteric one this, but memoryview objects now have a release() method and support use as a context manager. These objects allow a zero-copy view into any object that supports the buffer protocol, which includes the builtins bytes and bytearray. Some objects may need to allocate resources in order to provide this view, particularly those provided by C/C++ extension modules. The release() method allows these resources to be freed earlier than the memoryview object itself going out of scope.
• The internal structsequence tool has been updated so that C structures returned by the likes of os.stat() and time.gmtime() now work like namedtuple and can be used anywhere where a tuple is expected.
• There’s a -q command-line option to the interpreter to enable “quiet” mode, which suppresses the copyright and version information being displayed in interactive mode. I struggle a little to think of cases where this would matter, I’ll be honest — perhaps if you’re embedding the interpreter as a feature in a larger application?

Conclusion¶

Well now, I must admit that I did not expect that to be double the size of the post covering Python 3.0! If you’ve come here reading that whole article in one go, I must say I’m impressed. Perhaps lay off caffeine for awhile…?

Overall it feels like a really massive release, this one. Admittedly I did cover a high proportion of the details, whereas in the first article I glossed over quite a lot as some of the changes were so massive I wanted to focus on them.

Out of all that, it’s really hard to pick only a few highlights, but I’ll give it a go. As I said at the outset I love argparse — anyone who writes command-line tools and cares about their usability should save a lot of hassle with this. Also, the concurrent.futures module is great — I’ve only really started using it recently, and I love how it makes it really convenient to add parallelism in simple cases to applications where the effort might otherwise be too high to justify the effort.

The functools.lru_cache() and functools.total_ordering() decorators are both great additions because they offer significant advantages with minimal coding effort, and this is the sort of feature that a language like Python should really be focusing on. It’s never going to beat C or Rust in the performance stakes, but it has real strengths in time to market, as well as the concision and elegance of code.

It’s also great to see some updates to the suite of Internet-facing modules, as having high quality implementations of these in the standard library is another great strength of Python that needs to be maintained. SSL adding support for SNI is a key improvement that can’t come too soon, as it still seems a long way off that we’ll be saying goodbye to the limited address space of IPv4.

Finally, the GIL changes are great to see. Although we’d all love to see the GIL be deprecated entirely, this is clearly a very difficult problem or it would have been addressed by now. Until someone can come up with something clever to achieve this, at least things are significantly better than they were for multithreaded Python applications.

So there we go, my longest article yet. If you have any feedback on the amount of detail that I’m putting in (either too much or too little!) then I’d love to hear from you. I recently changed my commenting system from Disqus to Hyvor which is much more privacy-focused and doesn’t require you to register an account to comment, and also has one-click feedback buttons. I find writing these articles extremely helpful for myself anyway, but it’s always nice to know if anyone else is reading them! If you’re reading this on the front-page, you can jump to the comments section of the article view using the link at the end of the article at the bottom-right.

OK, so that’s it — before I even think of looking at the Python 3.3 release notes, I’m going to go lie down in a darkened room with a damp cloth on my forehead.

☑ Python 2to3: What’s New in 3.1

The previous post in this series tries to explain why it’s 2021 and I appear to be posting what’s “new” in a release of Python almost nine years old. In that article I trawled through what I considered the major changes included in Python 3.0, and in this one we’re moving on to release 3.1. I’m hoping I can drill into the remaining releases in a little more detail given the number of features added is a little smaller than the change from Python 2.x to Python 3.x.

And so, with as little further ado as I can manage, let’s get started.

Ordered Dictionaries¶

Over the years many people have found use-cases for a dict where the order of insertion is maintained when you iterate through the entries. If you poked around a lot of Python codebases, you’d likely find quite a few implementations of this, but thankfully in this release it was added to the standard library in the form of collections.OrderedDict as per PEP 372.

There are a few uses for such a class, one being the configparser module which was also updated to use this new class. This means that the ordering of configuration items read in from a file can be preserved on write, which could be helpful if you need to manually diff them to see the changes.

LRU Cache Using OrderedDict¶

One of the most common uses is to implement an LRU cache and I’m going to use that as an example to illustrate the behaviour of OrderedDict. For anyone who’s unaware, an LRU cache is for cases where you need associative storage with a fixed maximum size, such as caching the result of some lookup. If your lookup involves a slow remote request and you’re doing this frequently then caching the result in memory can massively improve performance. However, you don’t want to exhaust memory by caching too many values, so once the cache hits some size limit you want to throw away old entries to make room for new ones. The way you do this is by discarding the least recently used (LRU) value, which is supposed to be the least useful to cache since recent hits are generally more like to be be hit again soon in many real-world use cases.

Here’s the code, and I’ll put the discussion below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

import collections

class LRUCache:

    def __init__(self, capacity):
        self._cache = collections.OrderedDict()
        self._capacity = max(0, int(capacity))

    def __iter__(self):
        """Just iterate through keys to keep code simple."""
        return iter(self._cache)

    def _trim(self):
        """Drop items as required to cap size at capacity."""
        while (len(self._cache) > self._capacity):
            self._cache.popitem(last=False)

    def get(self, key):
        """Retrieve item and, if found, move to end of LRU list."""
        value = self._cache.pop(key)
        self._cache[key] = value
        return value

    def set(self, key, value):
        """Add or update item, moving to end of LRU list."""
        self.clear(key)
        self._cache[key] = value
        self._trim()

    def clear(self, key):
        """Remove a value from the cache."""
        self._cache.pop(key, None)

    def resize(self, new_capacity):
        """Set the new capacity then re-trim."""
        self._capacity = max(0, int(new_capacity))
        self._trim()

As you can see, all the heavy lifting is done by the OrderedDict class itself. The first thing to note is that I implemented this as a wrapper rather than subclassing OrderedDict directly. This is to avoid annoying bugs due to self-recurring calls. For example, the implementation of the pop() method refers to self[key], and as you can see the get() method above calls pop(). Had I overridden __getitem__() myself instead of making a new get() method then the underlyinig pop() would have jumped straight back into my __getitem__(), thus creating recursion loop resulting in a RecursionError exception.

The second thing to note is that to move items to the end of the LRU list in get() I’m removing the item with pop() and re-adding it again. This is the only way to shuffle an item to the end of the list in Python 3.1. Thankfully in Python 3.2 a move_to_end() method was added to perform this action more gracefully and more efficiently too, but since the conceit of this post is that Python 3.2 doesn’t exist yet I can’t possible know that unless I’m psychic¹.

The third point of interest is that I’m also calling pop() in set() (via the call to clear()). This is unnecessary if the key doesn’t already exist in the OrderedDict because a newly added key is always added at the end of the LRU list. However, if the key does exist then the value is updated but the position of the item in the list is not changed. This behaviour may be helpful in some use-cases for OrderedDict, but in our example we do want it to move, so we remove and re-add this to force it. Once again, in Python 3.2 and beyond move_to_end() would be a better approach.

OrderedDict Implementation¶

So given that OrderedDict supports everything dict does, but adds some ordering behaviour in some cases, should we always use it instead of dict?

Well, I wouldn’t, personally. For one thing, OrderedDict is implemented in pure Python in the CPython library, so its performance is probably going to be at least marginally lower than dict. Also, the memory consumption is probably significantly higher as it uses a doubly-link list to store items in access order. Since Python doesn’t have a native linked-list type it implements its own, again in pure Python, although it does at least use __slots__ to minimise the additional memory overhead of this. It also has to use weakref.proxy in a few places to avoid issues with circular references, and this is going to add a little more overhead.

All this means that although the time complexity of all the methods is no worse than dict, and the time overheads in real-world usage are going to be at least somewhat higher. More notably, the memory requirements are going to be significantly higher for larger structures. As well as the obvious hit on available memory, increasing the size of in-memory structures can reduce the locality of reference which can further impact performance by reducing the benefits of the L2 and L3 caches. That said, if you build your OrderedDict in a short amount of time and iterate through the items in the original insertion order then you might actually find your locality of reference improves, on the basis that blocks of memory allocated close together in time are more likely to be phsyically close in memory.

When all’s said and done, though, the overheads are not likely to be significant in many common use-cases and its implementation is almost certainly better (and safer) than you’ll manage yourself unless you spend a lot of time on it. Since a lot of software development tasks are optimised for time to market, these days, then it’s definitely a very useful addition to the library. I just wouldn’t advise using it to replace dict for standard uses where you don’t actually need the ordering.

New Container Where It Counts¶

The collections module is doubly blessed in this release as there’s also a collections.Counter class added for counting unique instances. I remember when I first came across this I was a little puzzled why they’d have added this when it seemed that collections.defaultdict(int) would do the job just as well. However, this class has some features which aren’t immediately obvious which set it apart.

The key point to note is that this container is less like a conventional dict and more like a multiset[multieset] in some other languages. A good example is the elements() method, which iterates through all the items as many time as they’re counted, as if it was a true multiset.

>>> y = collections.Counter()
>>> y["two"] += 2
>>> y["three"] += 3
>>> list(y)
['two', 'three']
>>> list(y.elements())
['two', 'two', 'three', 'three', 'three']

It’s also possible and add and subtract counters from each other, which has the effect of modifying the counters in the target set, removing any where the counts go to (or below) zero. Being a form of set, they also support intersection with & and union with |.

Finally, there’s also a most_common() method which returns the top N items in the set sorted in reducing order of cardinality.

All in all this is a simple-seeming container, but with some handy features which make it a useful choice in quite a few use-cases. For having started off questioning why it was even added, the more I’ve looked at this class the more I think it’s a bit of a hidden gem.

Itertools¶

The itertools module has also seen some love with a few smaller changes.

First up is the combinations_with_replacement() function which is a variant of combinations() but allowing individual elements to be repeated more than once. Bit specialised, but for the cases that’s what you need then it’ll definitely be nice not having to write it.

Next is the compress() which takes two iterables typically of the same length, and filters the first iterable to only include elements where the corresponding element in the second iterable evaluates to True.

Finally, the count() generator now has an optional step parameter which can accept multiple types of numeric interval, including those from the fractions and decimal modules. It would have been handy to also support datetime.timedelta(), but that’s probably a bit too much of a stretched overload of this iterator’s purpose.

String Format Enhancements¶

You may recall from the String Formatting section in the previous post the following snippet:

>>> # This requires locale to be set first...
>>> import locale
>>> locale.setlocale(locale.LC_ALL, '')
'en_GB.UTF-8'
>>> # Use locale-specific number separator.
>>> "{0:n}".format(1234567)
'1,234,567'

The thousands separators are quite useful, but having to ensure a locale is set may be a bit of a pain, especially in short scripts. Hence, Python 3.1 adds a new option for a non-locale-aware thousands separator, which always uses a comma every 3 digits. This can be added by putting a , prior to the precision, and can be used with any of the base-10 numeric output formats.

>>> "{0:,.4f}".format(12345678.123456)
'12,345,678.1235'

A small change, but probably quite useful for all sorts of diagnostic output where locale correctness is probably secondary to functionality.

Additionally, there’s another handy change which removes the need for explicitly numbering the arguments in format parameters if they’re referenced in the same order they’re defined. Again taking the example from the previous post, here’s the old approach and the new one compared:

>>> "My name is {1} and I'm {0} years old".format(40, "Brian")
"My name is Brian and I'm 40 years old"
>>> "My name is {} and I'm {} years old".format("Brian", 40)
"My name is Brian and I'm 40 years old"

Executable Modules¶

Directories and zip files can now include a __main__.py and have this executed if run as a script. This is particularly useful for .zip files on Unix because you can prepend the file with a shebang line which will be respected by the kernel when you attempt to execute the file (assuming the user has permission to execute that file). The zip implementation that the Python interpreter uses to look into zip files is tolerant of unknown data at the start of the file, so despite this header the Python interpreter is still able to import libraries from the zip file as normal.

The net result of all this is that you can distribute a Python application, including multiple additional modules, and have it be executable without the user needing to unpack it.

The terminal session below demonstrates this, if you’re familiar enough with Linux/Unix to follow along:

$ cat __main__.py
import mymodule
mymodule.myfunc()

$ cat mymodule.py
def myfunc():
    print("Hello, world")

$ zip /tmp/pytmp.zip *.py
  adding: __main__.py (deflated 15%)
  adding: mymodule.py (stored 0%)
$ cat > /tmp/pyexec.zip
#!/usr/bin/python3
$ cat /tmp/pytmp.zip >> /tmp/pyexec.zip
$ chmod 0755 /tmp/pyexec.zip
$ /tmp/pyexec.zip
Hello, world

The unzip utility can still deal with the archive, even helpfully pointing out the additional bytes due to the shebang line:

$ unzip -l /tmp/pyexec.zip
Archive:  /tmp/pyexec.zip
warning [/tmp/pyexec.zip]:  19 extra bytes at beginning or within zipfile
  (attempting to process anyway)
  Length      Date    Time    Name
---------  ---------- -----   ----
       34  01-29-2021 23:10   __main__.py
       41  01-29-2021 23:10   mymodule.py
---------                     -------
       75                     2 files

Context Managers¶

There are a few changes relating to context managers in this release.

A simple change has been made to the syntax of the with statement to allow multiiple context managers to be specified. Although, simple, it’s rather handy in certain use-cases. The best example of this I know is when you’re parsing data from one file into another. Previously I used to use nested with statements like this:

1
2
3
4
5
6

def remove_comments(in_path, out_path):
    with open(in_path, "r") as in_fd:
        with open(out_path, "w") as out_fd:
            for line in in_fd:
                stripped = line.split("#", 1)[0]
                out_fd.write(stripped.rstrip() + "\n")

There was a contextlib.nested() function which could be used to stack multiple context managers on one line, but it wasn’t the neatest and had some annoying quirks. Now you can simply append additional managers with a comma:

1
2
3
4

def remove_comments(in_path, out_path):
    with open(in_path, "r") as in_fd, open(out_path, "w" as out_fd):
        for line in in_fd:
            …

Beautiful. I’m in serious danger of breaking into a James Blunt song here, so let’s move on quickly — neither of us wants to risk that.

The use of the with statement has also been expanded a little more wiith gzip.GzipFile and bz2.BZ2File now supporting the context manager protocol. This change is more like addressing a historical ommission, and frankly I’m surprised it’s taken this long.

Unit Testing¶

The unittest module got some enhancements in 3.1. The first of these is that it supports skipping tests or classes of tests based on arbitrary criteria, and it also allows tests to be flagged as an expected failure, which means it won’t count as a failure for the purposes of determining whether the test suite passed. Both of these handy features are implemented using decorators.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

import unittest

class TestLumberjack:

    @unittest.skipIf(sys.version_info.major < 3, "Felling not supported with 2.x")
    def test_cut_down_trees(self):
        …

    @unittest.skip("Lunch was moved out of scope for Q2")
    def test_eat_my_lunch(self):
        …

    # There's some sort of intermittent constipation bug, so we'll mark
    # this as a known failure until we have time to investigate.
    @unittest.expectedFailure()
    def test_go_to_the_lavatory(self):
        …

Next up, the useful assertRaises() can now be used as a context manager. This makes it a lot easier to check whether a block of code throws a particular expected exception, particularly if that block of code is a little convoluted.

1
2
3
4
5

class TestGenerators(unittest.TestCase):

    def test_stop_iteration(self):
        with self.assertRaises(StopIteration):
            next(i for i in ())

Finally, a set of additional assertions have been added whose primary value is typically the diagnostic detail they provide on failures without the programmer having to take any extra steps. Here are some examples:

• assertSetEqual()
• assertDictEqual(), assertDictContainsSubset()
• assertListEqual(), assertTupleEqual(), assertSequenceEqual()
• assertRaisesRegexp()
• assertIsNone(), assertIsNotNone()

Numeric Types¶

There are also a collection of small changes in assorted numeric types, including int, float and decimal.Decimal.

First up, int objects gained a bit_length() method to indicate the minimum number of bits required to store the binary representation of the number. Seems a little esoteric this one, but could be handy for serialisation code that’s picking from multiple different integer representations to use, or that sort of thing.

>>> (255).bit_length()
8
>>> (2**63).bit_length()
64
>>> (2**63-1).bit_length()
63
>>> (5**5**5).bit_length()
7257

Sticking with int the behaviour of round() has been updated. Previously this function would always return a float regardless of the type it was passed as an input. As of Python 3.1, however, if round() is passed an int then it returns an int instead.

Moving on to float, the string representation created by repr() in certain cases has been made more intuitive by using an alternative algorithm by David Gay. It’s outside the scope of this article to go into the full details, as floating point is hard, but suffice to say that repr() of certain values of float should now seem a bit more intuitive.

One minor wrinkle here is that the change in representation is going to change what the interactive interpreter shows you by default, so could break any docstrings you have using doctest becaues the float values shown may now be shorter.

Continuing the theme of rational numbers, the decimal.Decimal class can now be constructed from a float. It’s worth noting that it can have some surprising results due to the inability of binary floating point to represent the same numbers precisely as decimal floats, but as long as you round off to a sensible number of significant digits then you shouldn’t run into too many surprises.

Pickling¶

The pickle module has had a couple of small changes. Firstly, it’s been updated for better interoperability with Python 2.x where in many cases the same objects exist but with different names. For example, __builtin__.set in Python 2.x exists as builtins.set in Python 3.x. This unfortunately breaks compatibility with Python 3.0, but only if protocol version 2 is used — if you can assume Python 3.x only then use version 3 and it’ll work fine across all Python 3.x versions (but not Python 2.x). Protocol version 3 should work fine with all subsequent versions.

In other pickle news, functools.partial objects can now be serialised with the pickle module. This is still subject to the standard limitation of pickle that functions are sent by fully qualified name only, and rely on the deserialisation code to have the same function available during deserialisation.

Other Changes¶

There are a few other small points which I didn’t think deserved their own section.

• The old string.maketrans() has been deprecated since, rather confusingly for a function in the string module, it required bytes or bytesarray objects instead of str. To avoid this confusion, str, bytes and bytesarray now all have their own maketrans() and translate() methods. This is a somewhat obscure function unless you’re doing lots of character substitutions in your code, but for people who use it this definitely feels more consistent with the way other methods on these classes are accessed.
• collections.namedtuple now offers a rename parameter which, if true, causes invalid field names to be replaced with numeric positional names such as _0 or _1. This is useful where you don’t have control of the names and they may not be valid Python identifiers, such as constructing a namedtuple from the result column names in an SQL query where one of the columns is called COUNT(*).
• The NullHandler class was added to the logging module, which is a handler which always throws away its log messages. It’s useful for applications which don’t use logging, but use libraries which are written without logginig support. Unless you configure a handler, you get irritating “No handlers could be found…” warning messages. Libraries can also add a NullHandler, to avoid applications having to worry about this. Indeed, as explainied in the logging tutorial, this is the only handler that a library should add, since handling of log events should be left to the application to configure.
• The sys.version_info structure is now a namedtuple, so its attributes can more helpfully be accessed by name.
• IPv6 support added to nntplib and imaplib.
• The importlib library was added, which is a portable pure Python implementation of the import statement. This is hoped to add transparency to the import process.
• A number of optimisations have been made, including:
  • The new io library was previously pure python but has now been re-written in pure C, for a 2-20x speed up.
  • The json module has a C extension to improve performance.
  • Enabling the --with-computed-gotos compile flag gives speedups of up to 20% on the bytecode evaluation loop, depending on the platform, which is always welcome.
  • Decoding of UTF-8, UTF-16 and Latin-1 is 2-4x faster.
  • int was previously stored in base 2¹⁵, but now on 64-bit platforms 2³⁰ which significantly improves performance of integer arithmetic on them.

Conclusion¶

It’s been particularly interesting for me going through these changes so long after they were first released, because I’m realising how much I missed. Many of the smaller changes aren’t particularly impactful — I can’t see int.bit_length() is likely to become my most-used function, for example — but things like multiple context managers using a single with statement, and the non-locale-dependent thousands separator are handy little details to have up your sleeve. The versatility of collections.Counter as a multiset also makes it well worth keeping it in mind for when you need it.

Overall another great batch of improvements, and it’s nice to see performance enhancements because Python 3.x as a whole was initially a bit of a jump down from Python 2.x from a performance standpoint. Also, I’m mildly surprised at the extent of the changes, as in my head 3.1 was a fairly minor release to tidy up a few small points. I’m now looking forward to going through the remainder of the 3.x releases to see what other gems I’ve been missing out on.

☑ Python 2to3: What’s New in 3.0

I’ve always had a fondness for Python, since I started using it back in around 2005 or so. The only scripting I’d done before that was Perl and PHP, which had always felt totally unsuited to anything beyond the most trivial scripts¹. Python always felt like a breath of fresh air: the minimal core language harked back to the compactness of C, the first language I learned after BASIC, and compared to anything else at the time the standard library felt less like batteries included and more like a sizeable power station.

In those heady days I kept up with the releases keenly, checking out every new feature as it was added. I remember the slightly giddy glee as I built lazily evaluated lists with generator comprehensions, and even trivial features like adding timeouts to various blocking operations gave me a little thrill. Perhaps I didn’t have enough going on in my life…

Alas, the release of Python 3.0 coincided with me having less and less time to devote to such detailed following, and before you know it it’s well over a decade later and there’s a mountain of new features with which I don’t feel intimately familiar. Of course, I’ve accumulated some bits and pieces of knowledge and experience on some of the more major aspects as I’ve gone along, but there’s nothing quite like a detailed trawl through the full release notes to pick up on handy tricks and features one might have missed.

This is the first in a series of articles where I’ll attempt to (very belatedly!) catch myself up on the latest and greatest. I’m going to start by examining all the major new features in Python 3.0 in this article, and then take a new release in each subsequent one until I’m all caught up with 3.9². I may not go through every little change, especially in the standard library, but I’ll try to cover all the changes that noteworthy.

Since the potential scope of this article is all of the changes from Python 2.6 to 3.0, it’s probably going to be the one that’s most likely to make me wish I’d broken it up into smaller pieces. So let’s get going before I change my mind, and dive into python 3!

print is a Function¶

We’ll start with one of the most straight-forward changes: the keyword print became the functioni print(). This was at the same time both pleasing, as it was one less irksome special case to worry about; but also slightly vexing, as my muscle memory took quite some time to adjust. Still, it was absolutely the right thing to do, and I for one certainly haven’t missed the inconsistent use of the >> operator for redirecting print output.

Iterators Replace Lists¶

On a rather more substantial note, many of the functions that used to return lists now return iterators. This addressed some rather ugly duplication such as having both range() and xrange(), and dict having both items() and iteritems(). In most cases you want an iterator anyway, so you don’t have to buffer up potentially large lists in memory, and if you really do want a real list you can just pass the iterator to the list() constructor. If you’re doing that just to sort it, though, then sorted() probably does what you want.

So far so simple, but there’s some interesting detail here for dict which may not be immediately apparently. The three methods dict.keys(), dict.values() and dict.items() don’t actually return just simple iterators but instead return views. Unlike a generator these can be iterated repeatedly, and furthermore they remain linked with the original dict such that updates to the original will immediately be reflected in the view. That said, they do suffer the usual limitation that the dict can’t change size while it’s being iterated or it’ll raise a RuntimeError.

The functions map() and filter() were both updated to return iterators instead of lists, although many of the cases where you might be tempted to use these are more readably implemented as a list comprehension or generator expression instead.

Finally, xrange() was renamed to range() to replace the original list version, and zip() also returns an iterator.

These changes mean that it only takes a little care and you can structure your code as almost entirely lazily-evaluated iterators to build up some pretty complex processing chains with a minimum of complexity in the code. I was a big fan of all these changes.

Type-safe Comparisons¶

One change that has a bit more potential to break things was that the ordering comparison operators <, <=, > and >= will now raise TypeError where there is no obvious natural order. For example, 3 > None will error out, as will 2 < "3". This means that heterogenous lists can’t necessarily be sorted, as not all the elements are likely to be comparable to each other. That said, it’s hard to think of any such cases which aren’t the result of poor design somewhere.

A related change that took me a little while to adjust to was the loss of the cmp parameter to sorted() and list.sort(). Instead the key parameter can be used to supply a function to convert the value to a “sort equivalent” value. In general any sensible cases are quite easy to convert between these two, but it took a bit of a shift in thinking at times.

Both cmp() and __cmp__() also vanished (well, were deprecated at least). The former was mostly only useful in building comparison methods to pass as the cmp parameter to sorting functions anyway, so with these removed the need for cmp() was basically gone.

The loss of the __cmp__() method was a bit more irritating as implementing the rich comparison operators such as __lt__() and __ge__() gets a little tedious if you want to implement several classes which are support all six such methods. These days the functools.total_ordering decorator makes this rather less cumbersome, but that wasn’t added until Python 3.2 so let’s not get ahead of ourselves. At least != returns the inverse of == by default, unless the latter returns NotImplemented³.

Whilst I understand why __cmp__() was deprecated in favour of the rich comparison operators, as it’s important to support partially ordered types for some cases, this is one case where I feel a little conflicted as __cmp__() was genuinely useful for the common case of fully ordered classes.

Unified Integers¶

A number of changes impacted the int and long types which are worth being familiar with as arithmetic is the bread and butter of so much code.

Firstly, the long type is no more, as it’s been rolled into a unified int type whose backend storage is seamlessly converted as necessary. This was very pleasant as it always felt oddly low-level for Python to have exposed C’s confusingly inconsistent int size across platforms. In the same vein, sys.maxint was removed as there is effectively no longer a limit to the value of an int. However, it’s still sometimes useful to have a value larger than the size of any container and sys.maxsize was retained for this. Generally I’d say having None removes most of the need for this, but it’s there if you find it useful.

New Literals and Comprehensions¶

Several types have new ways to specify literals or other expressions.

The new literals for octal (e.g. 0o644) and binary (e.g. 0b11001) were added in Python 2.6, but now the old form of octal literals (e.g. 0644) is no longer supported. There’s a new bin() function to convert an integer to a binary string, similar to oct() for octal and hex() and hexadecimal.

There’s also a new more concise format for set literals which is {1, 2, 3, 4}. Note, however, that {} is still an empty dict so you’ll need to use set() for that case. This can also be used as set comprehension, as in {i**2 for i in range(10) if i % 2 == 0}.

In a similar vein there’s also a format for dict comprehensions, so you can do things like this totally not in any way contrived example:

1
2
3

def line_lookup_table(filename):
    with open(filename, "r") as fd
        return {num: line for num, line in enumerate(fd.readlines())}

Strings and Unicode¶

Now we’re getting to what is, in my opinion, one of the most impactful changes in Python 3.

In Python 2, there were two types: str represented a sequence of bytes, and unicode represented a series of Unicode code points, which was independent of any particular encoding. Since data invariably comes into an application as bytes, there was always a bit of a dance where everything coming in should immediately have been decoded to unicode, using some out-of-band mechanism to determine the correct encoding to use; and all data being sent out had to be re-encoded, again using some particular encoding as required.

Whilst this all seems simple enough, in practice it lead to an awful lot of bugs. Typically programmers learned the str type and didn’t know anything about unicode until later, after the use of str was baked into all sorts of awkward parts of their system. Even if the programmer was knowledgeable enough to know to convert to unicode everywhere, there’s often not enough information available to select the best encoding both on input and output. The bugs created by this sort of issue often wouldn’t manifest until much later, as an application was exposed to users from other countries, leading to lots of highly painful bugs found in production setups.

Python 3 can’t solve all of these issues, but what it does do is force the programmer to deal with them rather more explicitly. This is done by renaming the unicode type to str, and storing bytes in a new bytes type which is also immutable. The key point is that, unlike in Python 2, you can’t mix these types. Previously things would all work for simple ASCII cases, where Python 2 would translate between str and unicode as required. This is where the “found in production” bugs creep in, unless you’re rigorous in your test cases. In Python 3, however, if you attempt to mix these types you’ll invariably get an exception. This forces the programmer to do explicit conversions when reading or writing byte-oriented storage (by which I mean, pretty much all storage).

Along with this change, u"..." literals ceased to be valid, as there is no longer a unicode type, and standard "..." literals are now interpreted as Unicode str types. To specify bytes explicitly, a new b"..." literal was added. The old basestring, which used to be a base class for both str and unicode, has been removed as it no longer makes sense to treat str and bytes interchangably with the new cleaner distinction between them.

The upshot is that all code now needs to be written to be totally explicit about whether it’s dealing with text, which has no definite size in bytes, or bytes, which is just a sequence of 8-bit values that has no specific interpretation as text. It’s important to remember that this isn’t a problem that Python 3 has created for programmers, it’s simply one that everyone should have always dealt with but has managed to avoid early on, only to cause undue pain later. Trust me, I speak from bitter experience of retrofitting i18n to a fairly large system, it’s not something you’ll thank your past self for.

One last note on all this is that there’s also a bytearray builtin type which is a mutable version of the bytes type. This was actually added in Python 2.6, so is strictly outside the scope of this article, but it’s given new relevance with the bytes type added in Python 3.0 and also the semantics deserve some clarification.

Essentially, if you index this type, you’ll get an int in the range 0-255, and if you want to set something you’ll need to pass one. Unlike Python 2.x, you can’t pass a str or bytes for this purpose any more⁴. However, if you want to extend it then you’ll need to pass a bytes or another bytearray. You can’t pass a str, since this isn’t composed of bytes but a set of unicode code points; you’d need to choose an encoding and convert to a bytes first.

>>> x = bytearray()
>>> x += "Knights who say"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't concat str to bytearray
>>> x += b"Knights who say"
>>> x.append(b"N")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'bytes' object cannot be interpreted as an integer
>>> x.append(32)
>>> x.append(78)
>>> x.append(105)
>>> x
bytearray(b"Knights who say Ni")
>>> x[2]
105

Unicode and File Content¶

All these changes to support unicode are particularly relevant when dealing with files. The first thing to note is that the mode in which you open your file is now more relevant than it used to be on some platforms.

To recap, open() takes a filename as its first parameter and a mode as its second. Everyone is probably used to specifying this as r to read, w to write (creating a new file or truncating any existing file to zero length) or a to append (open for write but don’t truncate). You may or may not be aware there’s also an x mode, which creates a new file and opens for writing, raising FileExistsError if that file already exists. On top of any of these you can add + to open for both reading and writing, with other behaviours such as creating or truncating the file remaining the same. Finally you can add a b to open the file in binary mode, or a t for text mode, although since this is the default most people omit it.

In python 2 the difference between binary and text was essentially whether line ending conversion was done on platforms which used CRLF conventions (i.e. Windows) and Unix users typically didn’t need to worry about the distinction. In Python 3 the difference is more pronounced: if you open a file in text mode then expect to use str objects for read and write, whereas in binary mode you need to use bytes object. As per earlier discussion these are not interchangable, you must remain consistent with the mode you’ve used or expect exceptions.

Opening a file in binary mode is straightforward, as you’d expect. After all, an 8-bit value is an 8-bit value anywhere in the world. The interesting cases come with text mode — since Python is always dealing with bytes when it talkes to the OS, it always needs some sort of encoding to do this transation in both directions. It’s important to bear in mind this is true regardless of whether you’ve supplied such an encoding.

Ideally you know what encoding to choose, either because it’s been supplied with the text out-of-band (e.g. in a HTTP header), or because the file you’re reading it supposed to follow some pre-defined standard which fixes the encoding. In this case, you can supply the encoding parameter to open() and all is good. If you don’t, the system default encoding will be used as per locale.getpreferredencoding(), which is likely to work in many cases but definitely cannot be relied upon.

One other option is to use the same encoding detection that Python itself uses to read source files via tokenize.detect_encoding()⁵. This will look for a BOM or special cookies as defined in PEP 263. That said, in my experience it’s pretty rare for content to contain such helpful markers on many platforms.

A final note on a feature which is actually present already in Python 2.6 but I don’t know how many people are aware of it: there’s a newline parameter to control newline translation for files opened in text mode. This defaults to None which enables universal newlines mode, which translates system-specific line-endings⁶ into \n on input and does the reverse translation on output. This means that any of the builtin functions that deal with lines will respect any of the possible line endings, and is a sensible default. However, there may be times you need to generate files which may be read on platforms other than your own, and in these cases you have a few other choices.

If you pass newline="" to open() then universal newline mode is enabled to detect newlines on input, but the line endings will be passed to you without translation. Passing this value on output will disable any translation of line endings during write. Passing any of \n, \r or \r\n as the value of newlines will treat that sequence as the only valid line-ending character to respect for reading this file and it will be translated to and from \n if required on input and output.

Unicode Errors¶

All this said, it seems like we need to make sure our code handles encoding/decoding errors gracefully if we care about our application’s stability. If you read or write bytes, there’s no way the content can be invalid, but if you’re reading in (say) UTF-8 then there are byte sequences which are simply invalid. Unless you want any user submitting content to be able to crash your application with an unhandled exception, you’d better do something about it.

So what to do? The first option is to simply make sure you handle UnicodeEncodeError and UnicodeDecodeError gracefully anywhere you’re doing the conversion, either explicitly or implicitly. These are both usefully subclasses of UnicodeError, itself a subclass of ValueError, so there’s several layers of granularity you can use.

This policy of raising exceptions on encoding/decoding errors can be changed, however, by supplying the errors parameter to open() and some of the other functions which interact with the codecs module. By default this is strict, which means to raise the exceptions, but in Python 3.0 you can also supply any of the following⁷:

strict

The default, raises exceptions on any error.

ignore

Ignore any bad data and just keep on encoding/decoding without further action.

replace

Replace the bad data by some appropriate marker in the output. On encoding Python uses a ? to replace bad characters and the official U+FFFD character on decoding.

xmlcharreplace

Only for encoding, replace bad data with the appropriate XML character reference.

backslashreplace

Replace bad data with the corresponding backslash escape sequences.

Later versions of Python add a couple more options, which I’ll try and remember to discuss in the appropriate article, but the full list for any version can be found in the documentation for the codecs module.

Unicode and Filenames¶

Truly the Unicode cup runneth over in Python 3, there are yet a few more wrinkles to iron out. All the discussion so far as talked about file content, but files also have names and these are strings — how does Unicode affect these?

In Python 3 these are generally treated as str so that filenames with arbitrary Unicode characters are permitted. However, this can cause problems on platforms where filenames are instead arbitrary byte strings, and so there may not be a valid translation to Unicode. As a result, many of the APIs that accept filenames as str will also accept them as bytes, and sometimes this can change their behaviour. For example, os.listdir() normally returns a list of str, but if you pass a bytes parameter then you’ll get a list of bytes instead. Some functions also have bytes alternatives, such as os.gwtcwdb() which is like os.getcwd() in every respect except that it returns bytes instead of str.

All this sounds rather tedious, I’ll be honest, and just treating everything as Unicode sounds much more pleasant to me. My strong advice to anyone is to keep control of the filenames you need to deal with so that you never run into these issues. If you really need to support user-specified filenames (e.g. you’re building your own cloud storage offering) then store these as metadata in some database and generate the filename yourself as (e.g.) the SHA-256 of the file content.

It’s worth mentioning that this issue doesn’t necessarily just occur with filenames, but also things like os.environ and sys.argv. In general I suspect that there’s not a lot to be done about these cases except fail early and obviously so the user can take corrective action before they’ve wasted too much time.

Function Parameter Changes¶

There were some changes to the way function parameters and return values are specified.

The first of these is annotations, specified in PEP 3107. These don’t make any functional change at runtime, but these annotations can be used by third party tools to perform type-checking or other functions. The annotations can be any Python expression, such as strings for documentation:

1
2
3

def my_function(filename: "The input filename",
                encoding: "Treat `filename` as this encoding",
                block_size: "Read `block_size` bytes per read"):

Or it could be types, for tools that do data flow analysis and type checking:

def my_function(filename: str, encoding: str, block_size: int):

We’ll come back to this feature more in the discussion of Python 3.5 where the typing module was added.

As well as annotations, there were some changes in PEP 3102 to allow keyword arguments to be specified after varargs style. Imagine you want to write a function that takes a variable number of positional parameters, but you also want to allow keyword arguments to specify optional flags. In Python 2 your only option was this:

1
2

def my_function(*args, **kwargs):
    # Now I have to check for invalid in args manually. Sigh.

Python 3 introduced a small syntax tweak to allow you to do this:

1
2

def my_function(*args, option=None, another_option=False):
    # Thanks, Python 3!

This effectively makes the parameters after the varargs keyword-only. But what if you want to do this without actually allowing varargs positional arguments? You can do this with a bare *, which won’t accept any parameters like *args, but will still flag any remaining arguments as keyword-only:

1
2

def my_function(x, y, z, *, coord_system="cartesian"):
    # No more accidental requests in 4+ dimensions.

Exception Changes¶

There are a number of changes to tidy up the use of exceptions in Python 3, make life easier for everyone.

Firstly, it’s not mandatory that exception classes are derived (directly or indirectly) from BaseException. This was always a good idea, it just wasn’t mandatory until now. That said, you almost certainly want to derive from Exception instead, as BaseException is generally reserved for things that you actually want to bypass your handles and proceed to the outer scope — SystemExit, KeyboardInterrupt and GeneratorExit. Believe me, you don’t want the pain of tracking down a bug where you’ve accidentally caught GeneratorExit because you forgot it was implemented with an exception.

As per PEP 3109, exceptions must be constructed as other classes, so use raise Exception(args) instead of the old raise Exception, args which is no longer supported.

In a similar vein, PEP 3110 updated the syntax for catching exceptions so that except MyException, exc is no longer valid, the cleaner syntax except MyException as exc must now be used. Slightly more subtly, the scope of the variable to which the exception is bound is now limited to the handler itself.

On a more fundamental level, PEP 3134 adds exception chaining support. This generally occurs when an exception is raised in the handler of a previous exception. In Python 2 the current exception being handled was effectively a global singleton, so the original exception information was lost to be replaced by the second one. This was pretty annoying, since generally the traceback from the original exception is going to be the one that helps you track down the bug. A common case of this is where library owners “helpfully” wrap system exceptions in their own errors; this is great for encapsulation outside the library, but makes it really painful to track down bugs in the library itself.

With exception chaining, however, no exceptions are lost. Instead, the original exception is saved under the __context__ attribute of the new exception. This can occur several times in a row, hence the term “chaining”. As well as being available to application code via __context__, the default exception logging also does a good job of presenting the full context:

>>> try:
...     raise Exception("one")
... except Exception as exc:
...     try:
...         raise Exception("two")
...     except Exception as exc:
...         raise Exception("three")
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
Exception: one

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
Exception: two

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
Exception: three

As well as the implicit chaining of raising exceptions in a handler, it’s also possible to explicitly chain exceptions with raise NewException() from exc. This is broadly similar, but stores the original exception under the __cause__ attribute instead of __context__. This also subtly changes the output in the default handler:

>>> try:
...     raise Exception("one")
... except Exception as exc:
...     raise Exception("two") from exc
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
Exception: one

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
Exception: two

The same PEP also adds a __traceback__ attribute of exception objects for better encapsulation. This is cleaner than having to dig it out of sys.exc_info(), especially where you now have multiple exceptions floating around at the same time.

String Formatting¶

Python 3.0 brings a new approach to string formatting which is described in PEP 3101. Even if you’ve never used Python 3 you may well already be familiar with it since it was actually added in Python 2.6. I’ll give it a brief run through here, though, since it’s mentioned in the Python 3.0 release notes.

One point I found amusing whilst going back through the old release notes was the certainty with which the % operator was going to be deprecated in Python 3.1 and removed shortly after. Now we’re well over a decade and getting on for ten releases later and we still don’t seem to be any closer to removing it. I can understand why; I’m sure there’s a large amount of code which would need to be painstakingly updated, and it’s not something that lends itself to reliable automatic conversion.

In any case, the approach isn’t too dissimilar to the old printf() syntax, but it’s more flexible. The simplest form is actually quite similar to the printf() version except instead of %s and the like then {n} is used, where n is the number of the argument to use, and instead of the % operator the arguments are passed to the str.format() method. More readably, names can be used instead of numbers along with keyword parameters to format().

Both of the following will produce the same output:

>>> "My name is {1} and I'm {0} years old".format(40, "Brian")
"My name is Brian and I'm 40 years old"
>>> "My name is {name} and I'm {age} years old".format(name="Brian", age=40)
"My name is Brian and I'm 40 years old"

On top of the basic field references, two explicit conversions are recognised, where !s can be appended to convert the value with str() and !r can be appended to convert the value with repr(). This overrides the default formatting for the type, and !r is useful for (e.g.) diagnostic output.

The final item is a formatting specifier, which is of the form [[fill]align][sign][#][0][minimumwidth][.precision][type]. Instead of duplicating the documentation by taking you through all the options, especially when you may well already be familiar, I’ll limit myself to some examples:

>>> # Exponent notation with precision 3.
>>> "{0:.3e}".format(123456789)
'1.235e+08'
>>> # General float format, min. length 8, precision 4.
>>> "{0:8.4}".format(12.345678)
'   12.35'
>>> # Hex format, centred in 10 chars using '\' as padding.
>>> "{0:\^+#10x}".format(1023)
'\\\\+0x3ff\\\\'
>>> # Format float as percentage, min. length 7, precision 2.
>>> "{0:7.2%}".format(0.123456789)
' 12.35%'
>>> # This requires locale to be set first...
>>> import locale
>>> locale.setlocale(locale.LC_ALL, '')
'en_GB.UTF-8'
>>> # Use locale-specific number separator.
>>> "{0:n}".format(1234567)
'1,234,567'

User-defined types can define a __format__() method to override the formatting for them, in a similar way to the existing __str__() and __repr__() special methods. There are also ways to override the formattinig more globally, but that’s a little esoteric and outside the scope of this already rather too long article — let’s move on.

Operators and Special Methods¶

There have been some other changes to operators and special methods.

The special methods for slices __getslice__(), __setslice__() and __delslice__() have been removed, which is a pleasant simplification of what’s becoming quite a massive set of special methods. Instead the standard __getitem__(), __setitem__() and __delitem__() methods will be passed a slice object containing whichever of start, stop and step size have been specified.

The next() method has been renamed to __next__() for consistency. A new builtin next() has been added to call this method in the same way that iter() already calls __iter__().

The __oct__() and __hex__() special methods have been removed in favour of just supplying __index__() which returns an integer used to populate the results of bin(), oct() and hex().

The __members__ and __methods__ attributes are no longer supported, and attributes of the form func_X have been renamed to __X__ for consistency and to avoid polluting the user-definable namespace. This specifically refers to __closure__, __code__, __defaults__, __dict__, __doc__, __globals__ and __name__.

Finally, __nonzero__ is now __bool__.

This means the full list of special methods is now a little more manageable in Python 3.0 with a little less duplication. Still pretty busy, but that’s probably inevitably given the degree of flexibility Python offers:

• Initialisation and destruction: __new__(), __init__() and __del__().
• Conversion to string form: __repr__(), __str__() and __format__().
• Rich comparisons: __lt__(), __le__(), __eq__(), __ne__(), __gt__() and __ge__().
• Hashable itemas: __hash__().
• Truth testing: __bool__().
• Attribute access: __getattr__(), __getattribute__(), __setattr__(), __delattr__() and __dir__().
• Descriptor (e.g. propterties) access: __get__(), __set__() and __delete__().
• Attribute storage: __slots__()
• Emulating callables: __call__()
• Emulating containers: __len__(), __getitem__(), __setitem__(), __delitem__(), __iter__(), __reversed__() and __contains__().
• Unary arithmetic: __neg__(), __pos__(), __abs__() and __invert__().
• Built-in conversions: __complex__(), __int__(), __float__() and __round__().
• Conversion to offset: __index__().
• Context managers: __enter__() and __exit__().
• Arithmetic operations: __add__(), __sub__(), __mul__(), __truediv__(), __floordiv__(), __mod__(), __divmod__(), __pow__(), __lshift__(), __rshift__(), __and__(), __xor__() and __or__().
• Reversed arithmetic: __radd__(), __rsub__(), __rmul__(), __rtruediv__(), __rfloordiv__(), __rmod__(), __rdivmod__(), __rpow__(), __rlshift__(), __rrshift__(), __rand__(), __rxor__() and __ror__().
• In-place update: __iadd__(), __isub__(), __imul__(), __itruediv__(), __ifloordiv__(), __imod__(), __ipow__(), __ilshift__(), __irshift__(), __iand__(), __ixor__() and __ior__().

I realise that calling 80 methods “manageable” may seem a little far-fetched, but at least there’s been some progress.

Other Changes¶

Here’s a few more things that I wanted to mention, but didn’t seem to fit neatly into their own category.

Firstly, there’s a new scope. Variables are still bound in local scope by default, and global still binds them to the global scope. Python 3 additionally adds the nonlocal keyword as specified in PEP 3104. This means that a nested function can declare a variable as referring to an outer scope without being constrained to only the global scope. This finally brings proper nested scopes to Python as many other langauges already enjoy (C, JavaScript, Ruby, etc.).

There’s also a neat change to unpacking iterables, so you can embed a “rest” argument when unpacking which consumes any remaining arguments. This can be at the start, middle or end of the lvalue list:

>>> (first, second, *rest, last) = range(6)
>>> print(first, second, rest, last, sep='\n')
0
1
[2, 3, 4]
5

There’s a new version of super() which can be invoked without arguments in a regular instance method inside a class definition, and it will automatically select the correct class and instance to call into. The behaviour of super() with arguments supplied is unchanged.

Sticking with classes, if you’re a fan of metaclasses there’s a cleaner syntax for specifying them. Instead of the Python 2 version:

1
2
3

class MyClass:
    __metaclass__ = MyMetaClass
    …

… you must now use the more concise and consistent Python 3 version:

1
2

class MyClass(metaclass=MyMetaClass):
    …

The builtin raw_input() has been renamed input() to replace the original. This was always a dangerously tempting function for Python newbies who didn’t understand the stability and security implications of evaluating arbitrary user input in Python. If you really want the old behaviour, you can still eval(input()) (but you probably don’t).

There’s also been some tidying up of builtins:

• intern() is now sys.intern().
• reload() is now imp.reload().
• Removed apply(), instead of apply(func, args) use func(*args).
• Removed callable(), use hasattr(func, "__call__").
• Removed coerce(), no longer required now old-style classes are gone.
• Removed execfile(), instead of execfile(filename) use exec(open(filename).read()).
• Removed file, now you must use open().
• Removed reduce(), use functools.reduce() if you must, but it’s probably more readable just to use an explicit loop.
• Removed dict.has_key(), just use the in operator.

Module Moves¶

Some of the standard library moved around in Python 3, although there’s nothing contraversial and I’m mostly just mentioning it for completeness. I’m not going to try to pick through everything but some examples that jumped out at me:

• The StringIO and cStringIO modules are gone, replaced by io.StringIO and io.BytesIO for text and data respectively.
• The md5 module is gone, but hashlib has all the functionality you need from it.
• The bsddb3 package was removed from the standard library, to ease maintenance burden, but it’s still available externally.
• Some modules have been renamed for consistency: ConfigParser is now configparser, Queue is queue, SocketServer is socketserver and so on.
• Some modules with a C implementation variant were rolled into their pure Python module, which selects it as an implementation detail instead of forcing applications to choose. For example, cPickle should never be used directly, pickle will choose the best available implementation at import time.
• Many modules have been grouped to keep things organised. Some examples:
• http now contains submodules httplib, BaseHTTPServer, CGIHTTPServer, SimpleHTTPServer, Cookie and cookielib.
• urllib now contains submodules urllib, urllib2, urlparse and robotparse.
• xmlrpc now contains submodules xmlrpclib, DocXMLRPCServer and SimpleXMLRPCServer.
• The sets module is gone, as it’s no longer needed given that set() and frozenset() are builtins.
• The string.letters, string.lowercase and string.uppercase are all gone along with their locale-specific behaviour, to be replaced by the more consistently defined string.ascii_letters, string.ascii_lowercase and string.ascii_uppercase.
• __builtin__ has been renamed to builtins.

Conclusion¶

If you’ve waded through all the above and made it down here, I salute your tenacity; and if you’ve just skipped down here in case there were some closing comments, I can’t really blame you in the slightest — it’s been a bit of a long slog, this one.

Overall, I feel the changes in Python 3.0 were very positive indeed. They took some great opportunities to tidy up some dirty corners and the switch to Unicode by default, whilst quite a pain for existing code, is the right decision overall. It would be nicer if the whole world could just settle on a single encoding and be done with it, but that’s probably somewhat outside the remit of the Python community.

There are a few aspects I’m more ambivalent about. The new string formatting approach is more flexible (although I little less performant, I’m given to understand) than the old % formatting. However, it also lacks some of the convenience for simpler cases, and the fact that the logging library still uses % formatting under the hood is a bit inconsistent. The reason is that changing this would break the existing API and lots of code out there. At time of writing, I believe there’s still no entirely satisfactory solution to this issue.

The loss of __cmp__() is also a bit of a blow, but I’m certainly not going to lose sleep over it. I can see that the new approach is more intuitive, it’s just also a bit more verbose and cumbersome in some cases.

I’m also glad I decided to run through this, beacuse I discovered quite a few things of which I was previously unaware, such as set comprehensions and overriding the Unicode error handling strategy. This is great because, given the long time that Python 3 has already been with us, it initially felt like reviewing the changes in Python 3.0 would be a bit of a waste of time.

I’m hoping the remaining articles in this series will be a little shorter and rather heavier on interesting new features and lighter on just tidying things up.

☑ Uncovering Rust: Types and Matching

There are a few features you expect from any mainstream imperative programming language. One of them is some support for basic builtin types, such as integers and floats. Another is some sort of structured data type, where you can assign values to named fields. Yet another is some sort of vector, array or list for sequences of values.

We’re going to start this post by looking at how these standard features manifest in Rust. Some of this will be quite familiar to programmers from C++ and similar languages, but there are a few surprises along the way and my main aim is to discuss those.

Scalar Types¶

Rust has builtin scalar types for integers, floats, booleans and characters.

Due to Rust’s low-level nature, you generally have to be explicit about the sizes of these. There are integral types for 8-, 16-, 32-, 64- and 128-bit values, both signed and unsigned. For example i32 is a signed 32-bit integer, u128 is an unsigned 128-bit integer. There are also architecture-dependent types isize and usize which use the native word size of the machine. These are typically used for array offsets. Floats can be f32 for single-precision and f64 for double.

One point that’s worth noting here is that Rust is a strongly typed language and won’t generally perform implicit casts for you, even for numeric types. For example, you can’t assign or compare integers with floats, or even integers of different sizes without doing an explicit conversion. This keeps costs explicit, but it does mean programmers need to consider their types carefully; but that’s no bad thing in my humble opinion.

Specifically on the topic of integers it’s also worth noting that Rust will panic (terminate the execution) if you overflow your integer size, but only in a debug build. If you compile a release build, the overflow is instead allowed to wrap around. However, the clear intention is that programmers shouldn’t be relying on such tricks to write safe and portable code.

Types of bool can be true or false. Even Rust hasn’t managed to introduce anything surprising or unconventional about booleans! One point of interest is that the expression in an if statement has to be a bool. Once again there are no implicit conversions, and there is no assumption of equivalence between, say, false and 0 as there is in C++.

The final type char has a slight surprise waiting for us, which is that it has a size of four bytes and can represent any Unicode code point. It’s great to see Unicode support front and centre in the language like this, hopefully making it very difficult for people who want to assume that the world is ASCII. Those of you familiar with Unicode may also know that the concept of what constitutes a “character” may surprise those who are used to working only with ASCII, so there could be puzzled programmers out there at times. But we live in a globalised world now and there’s no long any excuse for any self-respecting programmer to write ASCII-first code.

Arrays¶

Rust arrays are homogeneous (each array contains values of only one type) and are of a fixed-size, which must be known at compile time. They are always stored on the stack. Rust does provide a more dynamic Vec type which uses the heap and allows resizing, but I’m not going to discuss that here.

In the interests of safety, Rust requires that every element of an array be initialise when constructed. Because of this, it’s usually not required to specify a type, but of course there is a syntax for doing so. It’s also possible to initialise every item to the same value using a shorthand. These are all illustrated in the example below.

1
2
3
4

// These two are equivalent, due to type inference.
let numbers1 = [9, 9, 9, 9, 9];
let numbers2: [i32; 5] = [9, 9, 9, 9, 9];
let numbers3 = [9; 5];  // Repeated value shorthand.

Although the size of the array must be known at compile-time, of course the compiler can’t police your accesses to the array. For example, you may access an item based on user input. Rust does do bounds-checking at runtime, however, Discussion of how to handle runtime errors like this is a topic for another time, but the default action will be to terminate the executable immediately.

Structures and Tuples¶

The basic mechanics of structs in Rust work quite analogously to those in C++, aside from some minor syntactic differences. Here’s a definition to illustrate:

1
2
3
4
5
6
7

struct Contact {
    first_name: String,
    last_name: String,
    email: String,
    age: u8,
    business: bool,
}

To create an instance of a struct the syntax is similar except providing values instead of types after the colons. After creation the dot notation to read and assign struct fields will also be familiar to both C++ and Python programmers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

fn main() {
    let mut contact1 = Contact {
        first_name: String::from("John"),
        last_name: String::from("Doe"),
        email: String::from("jdoe@example.com"),
        age: 21,
        business: false,
    };
    println!("Contact name is {} {}",
             contact1.first_name, contact1.last_name);
    contact1.first_name = String::from("Jane");
    println!("Contact name is {} {}",
             contact1.first_name, contact1.last_name);
}

Note that to assign to first_name we had to make contact1 mutable and that this mutability applies to the entire structure, not to each field. No surprises for C++ programmers there either.

Now there are a couple more unique features that are worth mentioning. The first of them comes when creating constructor methods. Let’s say we want to avoid having to set the business field, so we wrap it up in a function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

fn new_business_contact(first_name: String,
                        last_name: String,
                        email: String,
                        age: u8)
                        -> Contact {
    Contact {
        first_name: first_name,
        last_name: last_name,
        email: email,
        age: age,
        business: true
    }
}

However, it’s a bit tedious repeating all those field names in the body. Well, if the function parameters happen to match the field names you can use a shorthand for this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

fn new_business_contact(first_name: String,
                        last_name: String,
                        email: String,
                        age: u8)
                        -> Contact {
    Contact {
        first_name,
        last_name,
        email,
        age,
        business: true
    }
}

Another convenient syntactic trick is the struct update syntax, which can be used to create a copy of another struct with some changes:

1
2
3
4
5
6
7
8
9

let contact1 = Contact {
    …
};
…
let contact2 = Contact {
    first_name: String::from("John"),
    last_name: String::from("Smith"),
    ..contact1
};

This will duplicate all fields not explicitly changed. There can be a sting in this particular tail, though, due to the ownership rules. In this example, the String value from contact1.email will be moved into contact2.email and so the first instance will no longer be valid after this point.

Finally in this section I’ll briefly talk about tuples. I’m talking about them here rather than along with other compound types because I feel they work in a very similar way to structs, just without the field names. They have a fixed size defined when they are created and this cannot change, as with an array. Unlike an array, however, they are heterogeneous: they can contain multiple different types.

One thing that might surprise Python programmers in particular, however, is that the elements of a tuple are accessed using dot notation in the same way as a struct. In a way you can think of it as a struct where the names of the fields are just automatically chosen as base-zero integers.

1
2
3
4
5
6

fn main() {
    let tup = (123, 4.56, "hello");
    println!("{} {} {}", tup.0, tup.1, tup.2);
    // Can also include explicit types for the tuple fields.
    let tup_copy: (u32, f64, String) = tup;
}

If you want to share the definition of a tuple around in the same way as for a struct but you don’t want to give the fields names, you can use a tuple struct to do that:

1
2
3
4
5
6

struct Colour(u8, u8, u8);

fn main() {
    let purple = Colour(255, 0, 255);
    println!("R={} G={}, B={}", purple.0, purple.1, purple.2);
}

In all honesty I’m not entirely sure how useful that’ll be, but time will tell.

The final note here is that structs can also hold references, although none of the examples here utilised that. However, doing so means exercising a little more care because the original value can’t go out of scope any time before any structs with references to it. This is a topic for a future discussion on lifetimes.

Enumerations¶

Continuing the theme of data types that C++ offers, Rust also has enumerations, hereafter referred to as enums. Beyond the name the similarity gets very loose, however. In C++ enums are essentially a way to add textual aliases to integral values; there’s a bit of syntactic sugar to treat them as regular values, but you don’t have to dip your toes too far under the water to get them bitten by an integer.

In Rust, however, they have features that are more like a union in C++, although unlike a union they don’t rely on the programmer to know which variant is in use at any given time.

You can use them very much like a regular enum. The values defined within the enum are scoped within the namespace of the enumeration name¹.

1
2
3
4
5
6
7
8
9

enum ContactType {
    Personal,
    Colleague,
    Vendor,
    Customer,
}

let contact1_type = ContactType::Personal;
let contact2_type = ContactType::Vendor;

However, much more powerfully than this these variants can also have data values associated with them, and each variant can be associated with its own data type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

// We reference contacts by their email address except for
// colleagues, where we use employee number; and vendors,
// where we use supplier ID, which consists of three numbers.
enum ContactType {
    Personal(String),
    Colleague(u64),
    Vendor(u32, u32, u32),
    Customer(String)
}

let customer = ContactType::Customer("andy@example.com");
let colleague = ContactType::Colleague(229382);
let supplier = ContactType::Vendor(23, 223, 4);

This construct is great for implementing the sort of code where you need to branch differently based on the underlying type of something. I can just hear the voices of all the object-orientation purists declaring that polymorphism is the correct solution to this problem: that everything should be exposed as an abstract method in the base class that all the derived classes implement. I wouldn’t say I disagree necessarily, but I would also say that this isn’t a clean fit in every case and polymorphism isn’t the one-size-fits-all solution as which it has on occasion been presented.

Rust implements some types of polymorphism and features such as traits are a useful alternative to inheritance for code reuse, as we’ll see in a later post. But since Rust doesn’t implement true inheritance, more properly called subtype polymorphism, then I suspect this flexibility of enumerations is more important in Rust than it would be in C++.

A little further down we’ll see how to use the match operator to do this sort of switching in an elegant way, but first we’ll see one example of a pre-defined enum in Rust that’s particularly widely used.

Option¶

It’s a very common case that a function needs to return a value in the happy case or raise some sort of error in the less happy case. Different languages have different mechanisms for this, one of the more common in modern languages being to raise exceptions. This is particularly common in Python, where exceptions are used for a large proportion of the functionality, but it’s also quite normal in C++ where the function of the destructors and the stack unwinding process are both heavily oriented around making this a fairly safe process.

Despite its extensive support for exceptions, however, C++ is still a bit of a hybrid and it has a number of cases where its APIs still use the other primary method of returning errors, via the return value. A good example of this is the std::string::find() method which searches for a substring within the parent string. This clearly has two different classes of result: either the string is found, in which case the offset within the parent string is returned; or the string is not found, in which case the method returns the magic std::string::npos value. In other cases functions can return either a pointer for the happy case or a NULL in case of error.

Rust does not support exceptions. This is for a number of reasons, partly related to the overhead of raising exceptions and also the fact that return values make it easier for the compiler to force the programmer to handle all error cases that a function can return.

To implement these error returns in Rust, therefore, is where the Option enum comes in useful. It’s defined something like this:

1
2
3
4

enum Option<T> {
    Some(T).
    None,
}

This enum is capable of storing some type T which is a template type (generics will be discussed properly in a later post), or the single value None. This allows a function to return any value type it wishes, but also leave open the possibility of returning None for an error.

That’s about all there is to say about Option, and we’ll see the idiomatic way to use it in the next section.

Matching¶

The final thing I’m going to talk about is the match flow control operator. This is conceptually similar to the switch statement in C++, but it’s got rather more cleverness up its sleeves.

The first thing to note about match is that unlike switch in C++ it is an expression instead of a statement. One aspect of Rust I haven’t talked about yet is that expressions may contain statements, however, so this isn’t a major obstacle. But it does mean that it’s fairly easy to use simple match expressions in assignments or as return values:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

enum Direction {
    North,
    South,
    East,
    West,
}

fn get_bearing(d: Direction) -> u16 {
    match d {
        Direction::North => 0,
        Direction::East => 90,
        Direction::South => 180,
        Direction::West => 270,
    }
}

The match expression has multiple “arms” which have a pattern and a result expression. To do more than just return a value from the expression, we can wrap it in braces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

fn get_bearing(d: Direction) -> u16 {
    match d {
        Direction::North => 0,
        Direction::East => {
            println!("East is East");
            90
        },
        Direction::South => {
            println!("Due South");
            180
        },
        Direction::West => {
            println!("Go West");
            270
        },
    }
}

We can use the patterns to do more than just match specific values, though. Taking the Option type from earlier, we can use it to extract the return values from functions whilst still ensuring we handle all the error cases.

For example, the String::find() method searches for a substring and returns an Option<usze> which is None if the value wasn’t found or the offset within the string if it was found. We can use this to, say, extract the domain part from an email address:

1
2
3
4
5
6

fn get_domain(email: &String) -> &str {
    match email.find('@') {
        None => "",
        Some(x) => &email[x+1..],
    }
}

This function takes a String reference and returns a string slice representing the domain part of the email, unless the email address doesn’t contain an @ character in which case we return an empty string. I’m not going to say that the semantics of an empty string are ideal in this case, but it’s just an example.

As another example we could write a function to display the contact details for the ContactType defined earlier:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

enum ContactType {
    Personal(String),
    Colleague(u64),
    Vendor(u32, u32, u32),
    Customer(String)
}

fn show_contact(contact: ContactType) {
    match contact {
        ContactType::Personal(email) => {
            println!("Personal: {}", email);
        },
        ContactType::Colleague(employee_number) => {
            println!("Colleague: {}", employee_number);
        },
        ContactType::Vendor(id1, id2, id3) => {
            println!("Vendor: {}-{}-{}", id1, id2, id3);
        },
        ContactType::Customer(email) => {
            println!("Customer: {}", email);
        },
    }
}

One aspect of match statements that isn’t immediately obvious is that they are required to be exhaustive. So, if you don’t handle every time enum value, for example, then you’ll get a compile error. This is what makes things like the Option example particularly safe as it forces handling of all errors, which is generally regarded as a good practice if you’re writing robust code. This also makes perfect sense if you consider that match is an expression: if you assign the result to a variable, say, then then compiler needs something to assign and if you hit a case that your match doesn’t handle then what’s the compiler going to do?

Of course if we’re using match for something other than an enum then handling every value would be pretty tedious. For these cases we can use the pattern _ as the default match. The example below also shows how we can match multiple patterns using | as a separator:

1
2
3
4
5
6

fn is_perfect(n: u32) -> bool {
    match n {
        6 | 28 | 496 | 8128 | 33_550_336 => true,
        _ => false
    }
}

Here we’re meeting the needs of match by covering every single case. If we removed that final default arm, the compiler wouldn’t let us get away with it:

error[E0004]: non-exhaustive patterns: `0u32..=5u32`,
`7u32..=27u32`, `29u32..=495u32` and 3 more not covered
  --> src/main.rs:10:11
   |
10 |     match n {
   |           ^ patterns `0u32..=5u32`, `7u32..=27u32`,
`29u32..=495u32` and 3 more not covered
   |
   = help: ensure that all possible cases are being handled,
possibly by adding wildcards or more match arms

But what if we really wanted to only handle a single case? It would be pretty dull if we had to have a default arm in a match then check for that value being returned and ignore it.

Let’s take the get_domain() example from earlier. Let’s say that if you find a domain, you want to use it; but if not, you have some more complicated logic to invoke to infer the domain by looking at the username. You could handle that by doing something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

fn get_domain(email: &String) -> &str {
    let ret = match email.find('@') {
        None => "",
        Some(x) => &email[x+1..],
    };
    if ret != "" {
        ret;
    } else {
        // More complex logic goes here...
    }
}

But that’s a little clunky. Rust has a special syntax called if let for handling just a single case like this:

1
2
3
4
5
6
7

fn get_domain(email: &String) -> &str {
    if let Some(x) = email.find('@') {
        &email[x+1..];
    } else {
        // More complex logic goes here...
    }
}

I only recently came across this syntax and my opinions are honestly a little mixed. Whilst I find the match statements comprehensible and intuitive, this odd combination of if and let just seems unusual to me. Mind you, I suspect it’s a common enough case to be useful.

So that’s a whirlwind tour of match and Rust’s pattern-matching. It’s important to note that this is a much more powerful feature than I’ve managed to express here as we’ve only really discussed matching by literals and by enum type. In general patterns can be used in fairly creative ways to extract fields from values at the same time as matching literals, and they can even have conditional expressions added, which Rust calls match guards. These are illustrated in the (rather contrived!) example below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

struct Colour {
    red: u8,
    green: u8,
    blue: u8
}

fn classify_colour(c: Colour) {
    match c {
        Colour {red: 0, green: 0, blue: 0} => {
            println!("Black");
        },
        Colour {red: 255, green: 255, blue: 255} => {
            println!("White");
        },
        Colour {red: r, green: 0, blue: 0} => {
            println!("Red {}", r);
        },
        Colour {red: 0, green: g, blue: 0} => {
            println!("Green {}", g);
        },
        Colour {red: 0, green: 0, blue: b} => {
            println!("Blue {}", b);
        },
        Colour {red: r, green: g, blue: 0} => {
            println!("Brown {} {}", r, g);
        },
        Colour {red: r, green: 0, blue: b} => {
            println!("Purple {} {}", r, b);
        },
        Colour {red: r, green: g, blue: b} if r == b && r == g => {
            println!("Grey {}", r);
        }
        Colour {red: r, green: g, blue: b} => {
            println!("Mixed colour {}, {}, {}", r, g, b);
        }
    }
}

Hopefully most things there are fairly self-explanatory and in any case it’s just intended as an illustration of the sorts of facilities that are available. It’s also worth mentioning that the compiler does give you some help to detect if you’re masking patterns with earlier ones, but it doesn’t appear to be perfect. For example, if I moved the first two matches to the end of the list, they’re both correctly flagged as unreachable. However, if I move the pattern for white after the pattern for grey it didn’t generate a warning; I’m guessing the job of determining reachability around match guards is just too difficult to do reliably.

Conclusions¶

Rust’s type system certainly offers some powerful flexibility, and the pattern matching looks like a fantastic feature for pulling apart structures and matching special cases within them. The specific Option enum also looks like quite a pleasant way to implement the “value or error” case given that Rust doesn’t offer exceptions for this purpose.

My main reservation around these features is that there’s an awful lot of syntax building up here, and it’s a fine line between a good amount of expressive power and edging into Perl’s “there’s too many ways to do it” philosophy. The if let syntax in particular seems possibly excessive to me. But I’m certainly reserving judgement on that for now until I’ve had some more experience with the language.

☑ Uncovering Rust: References and Ownership

Over the last few years I’ve become more aware of the Rust programming langauge. Slightly more than a decade old, it has consistently topped the Stack Overflow Developer Survey in the most loved langauge category for the last four years, so there’s clearly a decent core of very keen developers using it. It aims to offer performance on a par with C++ whilst considerably improving on the safety of the language, so as a long-time C++ programmer who’s all too aware of its potential for painfully opaque bugs, I thought it was definitely worth checking what Rust brings to the table.

As the first article in what I hope will become a reasonable series, I should briefly point out what these articles are not. They are certainly not meant to be a detailed discussion of Rust’s history or design principles, nor a tutorial. The official documentation and other sources already do a great job of those things.

Instead, this series is a hopefully interesting tour of some of the aspects of the language that set it apart, enough to get a flavour of it and perhaps decide if you’re interested in looking further yourself. I’m specifically going to be comparing the language to C++ and perhaps occasionally Python as the two languages with which I’m currently most familiar.

Mutability¶

Before I get going on the topic of this post, I feel it’s important to clarify one perhaps surprising detail of Rust to help understand the code examples below, and it is this: all variables are immutable by default. It’s possible to declare any variable mutable by prefixing with the mut keyword.

I could imagine some people considering this is a minor syntactic issue as it just means what would be const in C++ is non-mut in Rust, and non-const in C++ is mut in Rust. So why mention it? Well, mostly to help people understand the code examples a little easier; whilst it’s debatably not a fundamental issue, it’s also not something that’s necessarily self-evident from the syntax either.

Also, I think it’s a nice little preview of the way the language pushes you towards one of its primary goals: safety. If you forget the modifier things default to the most restrictive situation, and the compiler will prod you to add the modifier explicitly if that’s what you want. But if it isn’t what you want, you get the hint to fix a potential bug. Immutable values typically also make it much easier to take advantage of concurrency safely, but that’s a topic for a future post.

Ownership¶

Since one of the touted features of the language is safety around memory allocation, I’m going to start off outlining how ownership works in Rust.

Ownership is a concept that’s stressed many times during the Rust documentation, although in my view it’s pretty fundamental to truly understanding any language. Manipulating variables in memory is the bulk of what software does most of the time and errors around ownership are some of the most common sources of bugs across multiple langauges.

In general “owning” a value in this context means that a piece of code has a responsibility to manage the memory associated with that value. This isn’t about mutability or any other concept people might feasibly regard as forms of ownership.

Just to be clear, I’m going to skip discussion of stack-allocated variables here. Management of data on the stack is generally similar in all mainstream imperative languages and generally falls out of the language scoping rules quite neatly, so I’m going to focus this discussion on the more interesting and variable topic of managing heap allocations.

In C++ ownership is a nebulous concept and left for the programmer to define. The language provides the facility to allocate memory and it’s up to the programmer to decide when it’s safe to free it. Techniques such as RAII allow a heap allocation to be tied to a particular scope, either on the stack or linked with an owning class, but this must be manually implemented by the programmer. It’s quite easy to neglect this in some case or other, and since it’s aggressively optional then the compiler isn’t going to help you police yourself. As a result, memory mismangement is a very common class of bugs in C++ code.

Higher-level languages tend to utilise different forms of garbage collection to avoid exposing the programmer to these issues. Python’s reference counting is a simple concept and covers most cases gracefully, although it adds peformance overhead to many operations in the language and cyclic references complicate matters such that additional garbage collection algorithems are still required. Languages like Java with tracing garbage collectors impose less performance penalty on access than reference counting, but may be prone to spikes of sudden load when a garbage sweep is done. These systems are also often more complex to implement, especially as in the real world they’re often a hybrid of multiple techniques. This isn’t necessarily a direct concern for the programmer, as someone else has done all the hard work of implementing the algorithm, but it does inch up the risk of hitting unpredictable pathalogical performance behaviour. These can be the sort of intermittent bugs that we all love to hate to investigate.

All this said, Rust takes a simpler approach, which I suppose you could think of as what’s left of reference counting after a particularly aggressive assult from Ockham’s Razor.

Rust enforces three simple rules of ownership:

1. Each value has a variable which is the owner.
2. Each value has exactly one owner at a time.
3. When the owner goes out of scope the value is dropped¹.

I’m not going to go into detail on the scoping rules of Rust right now, although there are some interesting details that I’ll probably cover in another post. For now suffice to say that Rust is lexically scoped in a very similar way to C++ where variables are in scope from their definition until the end of the block in which they’re defined².

This means, therefore, that because a value has only a single owner, and because the scope of that owner is well-defined and must always exit at some point, there is no possible way for the value to not be dropped and its memory leaked. Hence achieving the promised memory safety with some very simple rules that can be validated at compile-time.

So there you go, you assign a variable and the value will be valid until such point as that variable goes out of scope. What could be simpler?

1
2
3
4
5
6
7
8
9

// Start of block.
{
    …
    // String value springs into existence.
    let my_value = String::from("hello, world");
    println!("Value: {}", my_value);
    …
}
// End of block, my_value out of scope, value dropped.

Moving right along¶

Well of course it’s not quite that simple. For example, what happens if we assign the value to another variable? I mean, that’s a pretty simple case. How hard can it be to figure out what this code will print?

1
2
3
4
5

fn main() {
    let my_value = String::from("hello, world");
    let another_value = my_value;
    println!("Values: {} {}", my_value, another_value);
}

The answer is: slightly harder than you might imagine. In fact the code above won’t even compile:

   Compiling sample v0.1.0 (/Users/apearce16/src/local/rust-tutorial/sample)
error[E0382]: borrow of moved value: `my_value`
 --> src/main.rs:4:31
  |
2 |     let my_value = String::from("hello, world");
  |         -------- move occurs because `my_value` has type
`std::string::String`, which does not implement the `Copy` trait
3 |     let another_value = my_value;
  |                         -------- value moved here
4 |     println!("Values: {} {}", my_value, another_value);
  |                               ^^^^^^^^ value borrowed here after move

  error: aborting due to previous error

This is because Rust implements move semantics by default on assignment. So what’s really happening in the code above is that a string value is created and ownership is assigned to the my_value variable. Then this is assigned to another_value which results in ownership being transferred to the another_value variable. At this point the my_value variable is still in scope, but it’s no longer valid.

The compiler is pretty comprehensive in explaining what’s going on here, the value is moved in the second line and then the invalidated my_value is referenced in the third line, which is what triggers the error.

This may seem unintuitive to some people, but before making any judgements you should consider the alternatives. Firstly, Rust could abandon its simple ownership rules and allow arbitrary aliasing like in C++. Except that would mean either exposing manual memory management or replacing it with a more expensive garbage collector, both of which compromise on the goals of safety and performance respectively.

Secondly, Rust could perform a deep copy of the data on the assignment, so duplicating the value and ending up with two variables each with its own copy. This is workable, but defeats the goal of performance as memory copying is pretty slow if you end up doing an awful lot of it. It also violates a basic programmer expectation that a simple action like assignment should not be expensive.

And so we’re left with the move semantics defined above. It’s worth noting, however, that this doesn’t apply to all types. Some are defined as being safe to copy: generally the simple scalar types such as integers, floats, booleans, and so on. The key property of these which make them safe is that they’re stored entirely on the stack, there’s no associated heap allocation to handle. It’s also possible to declare that new types are safe to copy by adding the Copy trait, but traits are definitely a topic for a later post.

It’s also worth noting that these move semantics are not as restrictive as they might seem due to the existence of references, which I’ll talk about later in this post. First, though, it’s interesting to look at how these semantics work with functions.

Onwnership in and out of functions¶

The ownership rules within a scope are now clear, but what about passing values into functions? In C++, for example, arguments are passed by value which means that the function essentially operates on a copy. If this value happens to be a pointer or reference then of course the original value may be modified, but as mentioned above we’re deferring discussion of references in Rust for a moment.

Argument passing would appear to suffer the same issues as the assignment example above, in that we don’t want to perform a deep copy, but neither do we want to complicate the ownership rules. So it’s probably little surprise that argument passing into functions also passes ownership in the same way as the assignment.

This code snippet will fail to compile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

fn main() {
    let s = String::from("hello");
    my_function(s);
    // Oops, s isn't valid here any more!
    println!("Value of s: {}", s);
}

fn my_function(arg: String) {
    // Ownership passes to the 'arg' parameter.
    println!("Now I own {}", s);
    // Here 'arg' goes out of scope and the String is dropped.
}

Although this may seem superficially surprising, when you really think about it argument passing is just a fancy form of assignment into a form of nested scope, so it shouldn’t be a surprise that it follows the same semantics.

The same logic applies to function return values, and this is where things could get slightly surprising for C++ programmers who are used to returning pointers or references to stack values being a tremendous source of bugs; and returning non-referential values as a cause of potentially expensive copy operations.

In C++ when the function call ends, any pointer or reference to anything on its stack that is passed to the caller will now be invalid. These can be some pretty nasty bugs, particuarly for less experienced programmers. It doesn’t help that the compiler doesn’t stop you doing this, and also that these situations often give the appearance of working correctly initially, since the stack frame of the function has often not been reused yet so the pointer still seems to point to valid data immediately after the call returns. This clearly harms the safety of the code.

If the programmer decides to resolve this issue by returning a complex class directly by value instead of by pointer or reference, then this generally entails default construction of an instance in the caller, then execution of the function and then assignment of the returned value to the instance in the caller which might involve some expensive copying. This potentially harms the performance of the code.

I’m deliberately glossing over some subtleties here around returning temporary objects, return value optimisation and move semantics in C++ which are all well outside the scope of this post on Rust. But even though solutions to these issues exist, they require significant knowledge and experience on the part of the programmer to take advantage of correctly, particularly for user-defined classes.

In Rust things are simpler: you can return a local value and ownership passes to the caller in the obvious manner.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

fn main() {
    let my_value = create();
    // At this point 'my_value' owns a String.
    println!("Now I own {}", my_value);

    let another_value = transform(my_value);
    // At this point 'another_value' owns a string,
    // but 'my_value' is now invalid.
    println!("Now I own {}", another_value);
}

fn create() -> String {
    let new_str = String::from("hello, world");
    // Ownership will pass to the caller.
    new_str
}

fn transform(mut arg: String) -> String {
    // We've delcared the argument mutable, which is OK
    // since ownership has passed to us. We append some
    // text to it and then return it, whereupon ownership
    // passes back to the caller.
    arg.push_str("!!!");
    arg
}

For anyone puzzled by the bare expressions at the end of the functions on lines 15 and 24, suffice to say for now this is an idiomatic way to return a value in Rust. The language does have a return statement, but a bare expression also works in some cases. I’ll discuss this more in a later post.

So in the case of return values, the move semantics of ownership in Rust turn out to be pretty useful: the ownership passes to the caller safely and with no need for expensive copying, since somewhere under the hood it’s just a transfer of some reference to a value on the heap. Since the rules apply everywhere it all feels quite consistent and logical.

But as logical as it is, it may seem awfully inconvenient. There are many cases we want a value to persist after it has been operated on by a function. It would be annoying to have to deep-copy an object every time, or to constantly have to return the argument to the caller as in the example above.

Fortunately Rust provides references to resolve this inconvenience.

References¶

In Rust references provide a way to refer to a value without actually taking ownership of it. The example below demonstrates the syntax, which is quite reminiscent of C++:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

fn main() {
    let my_string = String::from("one two three");
    let num_words = count_words(&my_string);
    // 'my_string' is still valid here.
    println!("'{}' has {} words", my_string, num_words);
}

// I'm sure there are more elegant ways to implement
// this function, this is just for illustrating the point.
fn count_words(s: &String) -> usize {
    let mut words = 0;
    let mut in_word = false;
    for c in s.chars() {
        if c.is_alphanumeric() {
            if !in_word {
                words += 1;
                in_word = true;
            }
        } else {
            in_word = false;
        }
    }
    words
}