This is part 1 of the “Python 2to3” series.
I was slow to make the transition from Python 2 to 3 in the first place, and I never felt like I kept up properly with the new features. So I’m going to aim to do a series of articles looking at a different Python version in each and go through the new features added and catch myself up properly. This one addresses features added in Python 3.0 beyond those already in 2.6.
I’ve always had a fondness for Python, since I started using it back in around 2005 or so. The only scripting I’d done before that was Perl and PHP, which had always felt totally unsuited to anything beyond the most trivial scripts1. Python always felt like a breath of fresh air: the minimal core language harked back to the compactness of C, the first language I learned after BASIC, and compared to anything else at the time the standard library felt less like batteries included and more like a sizeable power station.
In those heady days I kept up with the releases keenly, checking out every new feature as it was added. I remember the slightly giddy glee as I built lazily evaluated lists with generator comprehensions, and even trivial features like adding timeouts to various blocking operations gave me a little thrill. Perhaps I didn’t have enough going on in my life…
Alas, the release of Python 3.0 coincided with me having less and less time to devote to such detailed following, and before you know it it’s getting steadily closer to two decades later and there’s a mountain of new features with which I don’t feel intimiately familiar. Of course, I’ve accumulated some bits and pieces of knowledge and experience on some of the more major aspects as I’ve gone along, but there’s nothing quite like a detailed trawl through the full release notes to pick up on handy tricks and features one might have missed.
This is the first in a series of articles where I’ll attempt to (very belatedly!) catch myself up on the latest and greatest. I’m going to start by examining all the major new features in Pyhon 3.0 in this article, and then take a new release in each subsequent one until I’m all caught up with 3.92. I’m not going to go through every little change, especially in the standard library, but I’ll try to pick out what I feel to be the highlights.
Since the potential scope of this article is all of the changes from Python 2.6 to 3.0, it’s probably going to be the one that’s most likely to make me wish I’d broken it up into smaller pieces. So let’s get going before I change my mind, and dive into python 3!
We’ll start with one of the most straight-forward changes: the keyword print became the functioni print(). This was at the same time both pleasing, as it was one less irksome special case to worry about; but also slightly vexing, as my muscle memory took quite some time to adjust. Still, it was absolutely the right thing to do, and I for one certainly haven’t missed the inconsistent use of the >> operator for redirecting print output.
On a rather more substantial note, many of the functions that used to return lists now return iterators. This addressed some rather ugly duplication such as having both range() and xrange(), and dict having both items() and iteritems(). In most cases you want an iterator anyway, so you don’t have to buffer up potentially large lists in memory, and if you really do want a real list you can just pass the iterator to the list() constructor. If you’re doing that just to sort it, though, then sorted() probably does what you want.
So far so simple, but there’s some interesting detail here for dict which may not be immediately apparently. The three methods dict.keys(), dict.values() and dict.items() don’t actually return just simple iterators but instead return views. Unlike a generator these can be iterated repeatedly, and furthermore they remain linked with the original dict such that updates to the original will immediately be reflected in the view. That said, they do suffer the usual limitation that the dict can’t change size while it’s being iterated or it’ll raise a RuntimeError.
The functions map() and filter() were both updated to return iterators instead of lists, although many of the cases where you might be tempted to use these are more readably implemented as a list comprehension or generator expression instead.
Finally, xrange() was renamed to range() to replace the original list version, and zip() also returns an iterator.
These changes mean that it only takes a little care and you can structure your code as almost entirely lazily-evaluated iterators to build up some pretty complex processing chains with a minimum of complexity in the code. I was a big fan of all these changes.
One change that has a bit more potential to break things was that the ordering comparison operators <, <=, > and >= will now raise TypeError where there is no obvious natural order. For example, 3 > None will error out, as will 2 < "3". This means that heterogenous lists can’t necessarily be sorted, as not all the elements are likely to be comparable to each other. That said, it’s hard to think of any such cases which aren’t the result of poor design somewhere.
A related change that took me a little while to adjust to was the loss of the cmp parameter to sorted() and list.sort(). Instead the key parameter can be used to supply a function to convert the value to a “sort equivalent” value. In general any sensible cases are quite easy to convert between these two, but it took a bit of a shift in thinking at times.
Both cmp() and __cmp__() also vanished (well, were deprecated at least). The former was mostly only useful in building comparison methods to pass as the cmp parameter to sorting functions anyway, so with these removed the need for cmp() was basically gone.
The loss of the __cmp__() method was a bit more irritating as implementing the rich comparison operators such as __lt__() and __ge__() gets a little tedious if you want to implement several classes which are support all six such methods. These days the functools.total_ordering decorator makes this rather less cumbersome, but that wasn’t added until Python 3.2 so let’s not get ahead of ourselves. At least != returns the inverse of == by default, unless the latter returns NotImplemented3.
Whilst I understand why __cmp__() was deprecated in favour of the rich comparison operators, as it’s important to support partially ordered types for some cases, this is one case where I feel a little conflicted as __cmp__() was genuinely useful for the common case of fully ordered classes.
A number of changes impacted the int and long types which are worth being familiar with as arithmetic is the bread and butter of so much code.
Firstly, the long type is no more, as it’s been rolled into a unified int type whose backend storage is seamlessly converted as necessary. This was very pleasant as it always felt oddly low-level for Python to have exposed C’s confusingly inconsistent int size across platforms. In the same vein, sys.maxint was removed as there is effectively no longer a limit to the value of an int. However, it’s still sometimes useful to have a value larger than the size of any container and sys.maxsize was retained for this. Generally I’d say having None removes most of the need for this, but it’s there if you find it useful.
Several types have new ways to specify literals or other expressions.
The new literals for octal (e.g. 0o644) and binary (e.g. 0b11001) were added in Python 2.6, but now the old form of octal literals (e.g. 0644) is no longer supported. There’s a new bin() function to convert an integer to a binary string, similar to oct() for octal and hex() and hexadecimal.
There’s also a new more concise format for set literals which is {1, 2, 3, 4}. Note, however, that {} is still an empty dict so you’ll need to use set() for that case. This can also be used as set comprehension, as in {i**2 for i in range(10) if i % 2 == 0}.
In a similar vein there’s also a format for dict comprehensions, so you can do things like this totally not in any way contrived example:
def line_lookup_table(filename): with open(filename, "r") as fd return {num: line for num, line in enumerate(fd.readlines())}
Now we’re getting to what is, in my opinion, one of the most impactful changes in Python 3.
In Python 2, there were two types: str represented a sequence of bytes, and unicode represented a series of Unicode code points, which was independent of any particular encoding. Since data invariably comes into an application as bytes, there was always a bit of a dance where everything coming in should immediately have been decoded to unicode, using some out-of-band mechanism to determine the correct encoding to use; and all data being sent out had to be re-encoded, again using some particular encoding as required.
Whilst this all seems simple enough, in practice it lead to an awful lot of bugs. Typically programmers learned the str type and didn’t know anything about unicode until later, after the use of str was baked into all sorts of awkward parts of their system. Even if the programmer was knowledgeable enough to know to convert to unicode everywhere, there’s often not enough information available to select the best encoding both on input and output. The bugs created by this sort of issue often wouldn’t manifest until much later, as an application was exposed to users from other countries, leading to lots of highly painful bugs found in production setups.
Python 3 can’t solve all of these issues, but what it does do is force the programmer to deal with them rather more explicitly. This is done by renaming the unicode type to str, and storing bytes in a new bytes type which is also immutable. The key point is that, unlike in Python 2, you can’t mix these types. Previously things would all work for simple ASCII cases, where Python 2 would translate between str and unicode as required. This is where the “found in production” bugs creep in, unless you’re rigorous in your test cases. In Python 3, however, if you attempt to mix these types you’ll invariably get an exception. This forces the programmer to do explicit conversions when reading or writing byte-oriented storage (by which I mean, pretty much all storage).
Along with this change, u"..." literals ceased to be valid, as there is no longer a unicode type, and standard "..." literals are now interpreted as Unicode str types. To specify bytes explicitly, a new b"..." literal was added. The old basestring, which used to be a base class for both str and unicode, has been removed as it no longer makes sense to treat str and bytes interchangably with the new cleaner distinction between them.
The upshot is that all code now needs to be written to be totally explicit about whether it’s dealing with text, which has no definite size in bytes, or bytes, which is just a sequence of 8-bit values that has no specific interpretation as text. It’s important to remember that this isn’t a problem that Python 3 has created for programmers, it’s simply one that everyone should have always dealt with but has managed to avoid early on, only to cause undue pain later. Trust me, I speak from bitter experience of retrofitting i18n to a fairly large system, it’s not something you’ll thank your past self for.
One last note on all this is that there’s also a bytearray builtin type which is a mutable version of the bytes type. This was actually added in Python 2.6, so is strictly outside the scope of this article, but it’s given new relevance with the bytes type added in Python 3.0 and also the semantics deserve some clarification.
Essentially, if you index this type, you’ll get an int in the range 0-255, and if you want to set something you’ll need to pass one. Unlike Python 2.x, you can’t pass a str or bytes for this purpose any more4. However, if you want to extend it then you’ll need to pass a bytes or another bytearray. You can’t pass a str, since this isn’t composed of bytes but a set of unicode code points; you’d need to choose an encoding and convert to a bytes first.
>>> x = bytearray() >>> x += "Knights who say" Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: can't concat str to bytearray >>> x += b"Knights who say" >>> x.append(b"N") Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'bytes' object cannot be interpreted as an integer >>> x.append(32) >>> x.append(78) >>> x.append(105) >>> x bytearray(b"Knights who say Ni") >>> x[2] 105
All these changes to support unicode are particularly relevant when dealing with files. The first thing to note is that the mode in which you open your file is now more relevant than it used to be on some platforms.
To recap, open() takes a filename as its first parameter and a mode as its second. Everyone is probably used to specifying this as r to read, w to write (creating a new file or truncating any existing file to zero length) or a to append (open for write but don’t truncate). You may or may not be aware there’s also an x mode, which creates a new file and opens for writing, raising FileExistsError if that file already exists. On top of any of these you can add + to open for both reading and writing, with other behaviours such as creating or truncating the file remaining the same. Finally you can add a b to open the file in binary mode, or a t for text mode, although since this is the default most people omit it.
In python 2 the difference between binary and text was essentially whether line ending conversion was done on platforms which used CRLF conventions (i.e. Windows) and Unix users typically didn’t need to worry about the distinction. In Python 3 the difference is more pronounced: if you open a file in text mode then expect to use str objects for read and write, whereas in binary mode you need to use bytes object. As per earlier discussion these are not interchangable, you must remain consistent with the mode you’ve used or expect exceptions.
Opening a file in binary mode is straightforward, as you’d expect. After all, an 8-bit value is an 8-bit value anywhere in the world. The interesting cases come with text mode — since Python is always dealing with bytes when it talkes to the OS, it always needs some sort of encoding to do this transation in both directions. It’s important to bear in mind this is true regardless of whether you’ve supplied such an encoding.
Ideally you know what encoding to choose, either because it’s been supplied with the text out-of-band (e.g. in a HTTP header), or because the file you’re reading it supposed to follow some pre-defined standard which fixes the encoding. In this case, you can supply the encoding parameter to open() and all is good. If you don’t, the system default encoding will be used as per locale.getpreferredencoding(), which is likely to work in many cases but definitely cannot be relied upon.
One other option is to use the same encoding detection that Python itself uses to read source files via tokenize.detect_encoding()5. This will look for a BOM or special cookies as defined in PEP 263. That said, in my experience it’s pretty rare for content to contain such helpful markers on many platforms.
A final note on a feature which is actually present already in Python 2.6 but I don’t know how many people are aware of it: there’s a newline parameter to control newline translation for files opened in text mode. This defaults to None which enables universal newlines mode, which translates system-specific line-endings6 into \n on input and does the reverse translation on output. This means that any of the builtin functions that deal with lines will respect any of the possible line endings, and is a sensible default. However, there may be times you need to generate files which may be read on platforms other than your own, and in these cases you have a few other choices.
If you pass newline="" to open() then universal newline mode is enabled to detect newlines on input, but the line endings will be passed to you without translation. Passing this value on output will disable any translation of line endings during write. Passing any of \n, \r or \r\n as the value of newlines will treat that sequence as the only valid line-ending character to respect for reading this file and it will be translated to and from \n if required on input and output.
All this said, it seems like we need to make sure our code handles encoding/decoding errors gracefully if we care about our application’s stability. If you read or write bytes, there’s no way the content can be invalid, but if you’re reading in (say) UTF-8 then there are byte sequences which are simply invalid. Unless you want any user submitting content to be able to crash your application with an unhandled exception, you’d better do something about it.
So what to do? The first option is to simply make sure you handle UnicodeEncodeError and UnicodeDecodeError gracefully anywhere you’re doing the conversion, either explicitly or implicitly. These are both usefully subclasses of UnicodeError, itself a subclass of ValueError, so there’s several layers of granularity you can use.
This policy of raising exceptions on encoding/decoding errors can be changed, however, by supplying the errors parameter to open() and some of the other functions which interact with the codecs module. By default this is strict, which means to raise the exceptions, but in Python 3.0 you can also supply any of the following7:
strictignorereplace? to replace bad characters and the official U+FFFD character on decoding.xmlcharreplacebackslashreplaceLater versions of Python add a couple more options, which I’ll try and remember to discuss in the appropriate article, but the full list for any version can be found in the documentation for the codecs module.
Truly the Unicode cup runneth over in Python 3, there are yet a few more wrinkles to iron out. All the discussion so far as talked about file content, but files also have names and these are strings — how does Unicode affect these?
In Python 3 these are generally treated as str so that filenames with arbitrary Unicode characters are permitted. However, this can cause problems on platforms where filenames are instead arbitrary byte strings, and so there may not be a valid translation to Unicode. As a result, many of the APIs that accept filenames as str will also accept them as bytes, and sometimes this can change their behaviour. For example, os.listdir() normally returns a list of str, but if you pass a bytes parameter then you’ll get a list of bytes instead. Some functions also have bytes alternatives, such as os.gwtcwdb() which is like os.getcwd() in every respect except that it returns bytes instead of str.
All this sounds rather tedious, I’ll be honest, and just treating everything as Unicode sounds much more pleasant to me. My strong advice to anyone is to keep control of the filenames you need to deal with so that you never run into these issues. If you really need to support user-specified filenames (e.g. you’re building your own cloud storage offering) then store these as metadata in some database and generate the filename yourself as (e.g.) the SHA-256 of the file content.
It’s worth mentioning that this issue doesn’t necessarily just occur with filenames, but also things like os.environ and sys.argv. In general I suspect that there’s not a lot to be done about these cases except fail early and obviously so the user can take corrective action before they’ve wasted too much time.
There were some changes to the way function parameters and return values are specified.
The first of these is annotations, specified in PEP 3107. These don’t make any functional change at runtime, but these annotations can be used by third party tools to perform type-checking or other functions. The annotations can be any Python expression, such as strings for documentation:
def my_function(filename: "The input filename", encoding: "Treat `filename` as this encoding", block_size: "Read `block_size` bytes per read"):
Or it could be types, for tools that do data flow analysis and type checking:
def my_function(filename: str, encoding: str, block_size: int):
We’ll come back to this feature more in the discussion of Python 3.5 where the typing module was added.
As well as annotations, there were some changes in PEP 3102 to allow keyword arguments to be specified after varargs style. Imagine you want to write a function that takes a variable number of positional parameters, but you also want to allow keyword arguments to specify optional flags. In Python 2 your only option was this:
def my_function(*args, **kwargs): # Now I have to check for invalid in args manually. Sigh.
Python 3 introduced a small syntax tweak to allow you to do this:
def my_function(*args, option=None, another_option=False): # Thanks, Python 3!
This effectively makes the parameters after the varargs keyword-only. But what if you want to do this without actually allowing varargs positional arguments? You can do this with a bare *, which won’t accept any parameters like *args, but will still flag any remaining arguments as keyword-only:
def my_function(x, y, z, *, coord_system="cartesian"): # No more accidental requests in 4+ dimensions.
There are a number of changes to tidy up the use of exceptions in Python 3, make life easier for everyone.
Firstly, it’s not mandatory that exception classes are derived (directly or indirectly) from BaseException. This was always a good idea, it just wasn’t mandatory until now. That said, you almost certainly want to derive from Exception instead, as BaseException is generally reserved for things that you actually want to bypass your handles and proceed to the outer scope — SystemExit, KeyboardInterrupt and GeneratorExit. Believe me, you don’t want the pain of tracking down a bug where you’ve accidentally caught GeneratorExit because you forgot it was implemented with an exception.
As per PEP 3109, exceptions must be constructed as other classes, so use raise Exception(args) instead of the old raise Exception, args which is no longer supported.
In a similar vein, PEP 3110 updated the syntax for catching exceptions so that except MyException, exc is no longer valid, the cleaner syntax except MyException as exc must now be used. Slightly more subtly, the scope of the variable to which the exception is bound is now limited to the handler itself.
On a more fundamental level, PEP 3134 adds exception chaining support. This generally occurs when an exception is raised in the handler of a previous exception. In Python 2 the current exception being handled was effectively a global singleton, so the original exception information was lost to be replaced by the second one. This was pretty annoying, since generally the traceback from the original exception is going to be the one that helps you track down the bug. A common case of this is where library owners “helpfully” wrap system exceptions in their own errors; this is great for encapsulation outside the library, but makes it really painful to track down bugs in the library itself.
With exception chaining, however, no exceptions are lost. Instead, the original exception is saved under the __context__ attribute of the new exception. This can occur several times in a row, hence the term “chaining”. As well as being available to application code via __context__, the default exception logging also does a good job of presenting the full context:
>>> try: ... raise Exception("one") ... except Exception as exc: ... try: ... raise Exception("two") ... except Exception as exc: ... raise Exception("three") ... Traceback (most recent call last): File "<stdin>", line 2, in <module> Exception: one During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 5, in <module> Exception: two During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 7, in <module> Exception: three
As well as the implicit chaining of raising exceptions in a handler, it’s also possible to explicitly chain exceptions with raise NewException() from exc. This is broadly similar, but stores the original exception under the __cause__ attribute instead of __context__. This also subtly changes the output in the default handler:
>>> try: ... raise Exception("one") ... except Exception as exc: ... raise Exception("two") from exc ... Traceback (most recent call last): File "<stdin>", line 2, in <module> Exception: one The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<stdin>", line 4, in <module> Exception: two
The same PEP also adds a __traceback__ attribute of exception objects for better encapsulation. This is cleaner than having to dig it out of sys.exc_info(), especially where you now have multiple exceptions floating around at the same time.
Python 3.0 brings a new approach to string formatting which is described in PEP 3101. Even if you’ve never used Python 3 you may well already be familiar with it since it was actually added in Python 2.6. I’ll give it a brief run through here, though, since it’s mentioned in the Python 3.0 release notes.
One point I found amusing whilst going back through the old release notes was the certainty with which the % operator was going to be deprecated in Python 3.1 and removed shortly after. Now we’re well over a decade and getting on for ten releases later and we still don’t seem to be any closer to removing it. I can understand why; I’m sure there’s a large amount of code which would need to be painstakingly updated, and it’s not something that lends itself to reliable automatic conversion.
In any case, the approach isn’t too dissimilar to the old printf() syntax, but it’s more flexible. The simplest form is actually quite similar to the printf() version except instead of %s and the like then {} are used, and instead of the % operator the arguments are passed to the str.format() method. It’s also possible, and good practice, to specifically refer to the argument number within the braces, which allows the order of items within the string to differ from that in the argument list. Even more readably, names can be used along with keyword parameters to format().
All of the following will produce the same output:
"My name is {} and I'm {} years old".format("Brian", 40) "My name is {1} and I'm {0} years old".format(40, "Brian") "My name is {name} and I'm {age} years old".format(name="Brian", age=40)
On top of the basic field references, two explicit conversions are recognised, where !s can be appended to convert the value with str() and !r can be appended to convert the value with repr(). This overrides the default formatting for the type, and !r is useful for (e.g.) diagnostic output.
The final item is a formatting specifier, which is of the form [[fill]align][sign][#][0][minimumwidth][.precision][type]. Instead of duplicating the documentation by taking you through all the options, especially when you may well already be familiar, I’ll limit myself to some examples:
>>> # Exponent notation with precision 3. >>> "{:.3e}".format(123456789) '1.235e+08' >>> # General float format, min. length 8, precision 4. >>> "{:8.4}".format(12.345678) ' 12.35' >>> # Hex format, centred in 10 chars using '\' as padding. >>> "{:\^+#10x}".format(1023) '\\\\+0x3ff\\\\' >>> # Format float as percentage, min. length 7, precision 2. >>> "{:7.2%}".format(0.123456789) ' 12.35%' >>> # This requires locale to be set first... >>> import locale >>> locale.setlocale(locale.LC_ALL, '') 'en_GB.UTF-8' >>> # Use locale-specific number separator. >>> "{:#n}".format(1234567) '1,234,567'
User-defined types can define a __format__() method to override the formatting for them, in a similar way to the existing __str__() and __repr__() special methods. There are also ways to override the formattinig more globally, but that’s a little esoteric and outside the scope of this already rather too long article — let’s move on.
There have been some other changes to operators and special methods.
The special methods for slices __getslice__(), __setslice__() and __delslice__() have been removed, which is a pleasant simplification of what’s becoming quite a massive set of special methods. Instead the standard __getitem__(), __setitem__() and __delitem__() methods will be passed a slice object containing whichever of start, stop and step size have been specified.
The next() method has been renamed to __next__() for consistency. A new builtin next() has been added to call this method in the same way that iter() already calls __iter__().
The __oct__() and __hex__() special methods have been removed in favour of just supplying __index__() which returns an integer used to populate the results of bin(), oct() and hex().
The __members__ and __methods__ attributes are no longer supported, and attributes of the form func_X have been renamed to __X__ for consistency and to avoid polluting the user-definable namespace. This specifically refers to __closure__, __code__, __defaults__, __dict__, __doc__, __globals__ and __name__.
Finally, __nonzero__ is now __bool__.
This means the full list of special methods is now a little more manageable in Python 3.0 with a little less duplication. Still pretty busy, but that’s probably inevitably given the degree of flexibility Python offers:
__new__(), __init__() and __del__().__repr__(), __str__() and __format__().__lt__(), __le__(), __eq__(), __ne__(), __gt__() and __ge__().__hash__().__bool__().__getattr__(), __getattribute__(), __setattr__(), __delattr__() and __dir__().__get__(), __set__() and __delete__().__slots__()__call__()__len__(), __getitem__(), __setitem__(), __delitem__(), __iter__(), __reversed__() and __contains__().__neg__(), __pos__(), __abs__() and __invert__().__complex__(), __int__(), __float__() and __round__().__index__().__enter__() and __exit__().__add__(), __sub__(), __mul__(), __truediv__(), __floordiv__(), __mod__(), __divmod__(), __pow__(), __lshift__(), __rshift__(), __and__(), __xor__() and __or__().__radd__(), __rsub__(), __rmul__(), __rtruediv__(), __rfloordiv__(), __rmod__(), __rdivmod__(), __rpow__(), __rlshift__(), __rrshift__(), __rand__(), __rxor__() and __ror__().__iadd__(), __isub__(), __imul__(), __itruediv__(), __ifloordiv__(), __imod__(), __ipow__(), __ilshift__(), __irshift__(), __iand__(), __ixor__() and __ior__().I realise that calling 80 methods “manageable” may seem a little far-fetched, but at least there’s been some progress.
Here’s a few more things that I wanted to mention, but didn’t seem to fit neatly into their own category.
Firstly, there’s a new scope. Variables are still bound in local scope by default, and global still binds them to the global scope. Python 3 additionally adds the nonlocal keyword as specified in PEP 3104. This means that a nested function can declare a variable as referring to an outer scope without being constrained to only the global scope. This finally brings proper nested scopes to Python as many other langauges already enjoy (C, JavaScript, Ruby, etc.).
There’s also a neat change to unpacking iterables, so you can embed a “rest” argument when unpacking which consumes any remaining arguments. This can be at the start, middle or end of the lvalue list:
(first, second, *rest, last) = range(6) # first=0, second=1, rest=[2,3,4], last=5
There’s a new version of super() which can be invoked without arguments in a regular instance method inside a class definition, and it will automatically select the correct class and instance to call into. The behaviour of super() with arguments supplied is unchanged.
Sticking with classes, if you’re a fan of metaclasses there’s a cleaner syntax for specifying them. Instead of the Python 2 version:
class MyClass: __metaclass__ = MyMetaClass …
… you must now use the more concise and consistent Python 3 version:
class MyClass(metaclass=MyMetaClass): …
The builtin raw_input() has been renamed input() to replace the original. This was always a dangerously tempting function for Python newbies who didn’t understand the stability and security implications of evaluating arbitrary user input in Python. If you really want the old behaviour, you can still eval(input()) (but you probably don’t).
There’s also been some tidying up of builtins:
intern() is now sys.intern().reload() is now imp.reload().apply(), instead of apply(func, args) use func(*args).callable(), use hasattr(func, "__call__").coerce(), no longer required now old-style classes are gone.execfile(), instead of execfile(filename) use exec(open(filename).read()).file, now you must use open().reduce(), use functools.reduce() if you must, but it’s probably more readable just to use an explicit loop.dict.has_key(), just use the in operator.Some of the standard library moved around in Python 3, although there’s nothing contraversial and I’m mostly just mentioning it for completeness. I’m not going to try to pick through everything but some examples that jumped out at me:
StringIO and cStringIO modules are gone, replaced by io.StringIO and io.BytesIO for text and data respectively.md5 module is gone, but hashlib has all the functionality you need from it.bsddb3 package was removed from the standard library, to ease maintenance burden, but it’s still available externally.ConfigParser is now configparser, Queue is queue, SocketServer is socketserver and so on.cPickle should never be used directly, pickle will choose the best available implementation at import time.http now contains submodules httplib, BaseHTTPServer, CGIHTTPServer, SimpleHTTPServer, Cookie and cookielib.urllib now contains submodules urllib, urllib2, urlparse and robotparse.xmlrpc now contains submodules xmlrpclib, DocXMLRPCServer and SimpleXMLRPCServer.sets module is gone, as it’s no longer needed given that set() and frozenset() are builtins.string.letters, string.lowercase and string.uppercase are all gone along with their locale-specific behaviour, to be replaced by the more consistently defined string.ascii_letters, string.ascii_lowercase and string.ascii_uppercase.__builtin__ has been renamed to builtins.If you’ve waded through all the above and made it down here, I salute your tenacity; and if you’ve just skipped down here in case there were some closing comments, I can’t really blame you in the slightest — it’s been a bit of a long slog, this one.
Overall, I feel the changes in Python 3.0 were very positive indeed. They took some great opportunities to tidy up some dirty corners and the switch to Unicode by default, whilst quite a pain for existing code, is the right decision overall. It would be nicer if the whole world could just settle on a single encoding and be done with it, but that’s probably somewhat outside the remit of the Python community.
There are a few aspects I’m more ambivalent about. The new string formatting approach is more flexible (although I little less performant, I’m given to understand) than the old % formatting. However, it also lacks some of the convenience for simpler cases, and the fact that the logging library still uses % formatting under the hood is a bit inconsistent. The reason is that changing this would break the existing API and lots of code out there. At time of writing, I believe there’s still no entirely satisfactory solution to this issue.
The loss of __cmp__() is also a bit of a blow, but I’m certainly not going to lose sleep over it. I can see that the new approach is more intuitive, it’s just also a bit more verbose and cumbersome in some cases.
I’m also glad I decided to run through this, beacuse I discovered quite a few things of which I was previously unaware, such as set comprehensions and overriding the Unicode error handling strategy. This is great because, given the long time that Python 3 has already been with us, it initially felt like reviewing the changes in Python 3.0 would be a bit of a waste of time.
I’m hoping the remaining articles in this series will be a little shorter and rather heavier on interesting new features and lighter on just tidying things up.
An opinion, it must be said, of which I’ve not been meaningfully disabused in the intervening years. ↩
Or based on my historical posting frequency, 3.10 might even have been released by that point! ↩
The NotImplemented built-in constant, used only for the rich comparison methods, shouldn’t be confused with the NotImplementedError exception, which is similar in purpose but used in other contexts. ↩
Although if you don’t mind a little hoop-jumping, and you really have a pressing desire to set a character using a bytes object you could do so using slice notation. So whilst x[1] = b"a" might not work, x[1:2] = b"a" should do. ↩
From Python 3.2 onwards there’s a more convenient tokenize.open() which you should generally use instead of directly calling detect_encoding() yourself, but we’ll cover that in a couple of articles. ↩
In case you’re unaware, different operating systems use either \n (e.g. Unix), \r\n (e.g. Windows) or \r (e.g. MacOS prior to OS X) as a line separator. You can check your OS-specific value with os.linesep. ↩
Note that only strict and ignore apply to all encodings, the remainder only apply to text encodings; that is, those that translate between str and bytes. There are also a scattering of special encodings that are str to str or bytes to bytes, but I’m not going to discuss them further in this article further as they’re a little esoteric. ↩
This is part 2 of the “Uncovering Rust” series which started with Uncovering Rust: References and Ownership.
Rust is fairly new multi-paradigm system programming language that claims to offer both high performance and strong safety guarantees, particularly around concurrency and memory allocation. As I play with the language a little, I’m using this series of blog posts to discuss some of its more unique features as I come across them. This one discusses Rust’s data types and powerful match operator.
There are a few features you expect from any mainstream imperative programming language. One of them is some support for basic builtin types, such as integers and floats. Another is some sort of structured data type, where you can assign values to named fields. Yet another is some sort of vector, array or list for sequences of values.
We’re going to start this post by looking at how these standard features manifest in Rust. Some of this will be quite familiar to programmers from C++ and similar languages, but there are a few surprises along the way and my main aim is to discuss those.
Rust has builtin scalar types for integers, floats, booleans and characters.
Due to Rust’s low-level nature, you generally have to be explicit about the sizes of
these. There are integral types for 8-, 16-, 32-, 64- and 128-bit values, both
signed and unsigned. For example i32 is a signed 32-bit integer, u128 is an
unsigned 128-bit integer. There are also architecture-dependent types isize
and usize which use the native word size of the machine. These are typically used
for array offsets. Floats can be f32 for single-precision and f64 for double.
One point that’s worth noting here is that Rust is a strongly typed language and won’t generally perform implicit casts for you, even for numeric types. For example, you can’t assign or compare integers with floats, or even integers of different sizes without doing an explicit conversion. This keeps costs explicit, but it does mean programmers need to consider their types carefully; but that’s no bad thing in my humble opinion.
Specifically on the topic of integers it’s also worth noting that Rust will panic (terminate the execution) if you overflow your integer size, but only in a debug build. If you compile a release build, the overflow is instead allowed to wrap around. However, the clear intention is that programmers shouldn’t be relying on such tricks to write safe and portable code.
Types of bool can be true or false. Even Rust hasn’t managed to introduce
anything surprising or unconventional about booleans! One point of
interest is that the expression in an if statement has to be a bool. Once
again there are no implicit conversions, and there is no assumption of equivalence
between, say, false and 0 as there is in C++.
The final type char has a slight surprise waiting for us, which is that it
has a size of four bytes and can represent any Unicode code point. It’s great to see
Unicode support front and centre in the language like this, hopefully making
it very difficult for people who want to assume that the world is ASCII. Those
of you familiar with Unicode may also know that the concept of what constitutes
a “character” may surprise those who are used to working only with ASCII, so
there could be puzzled programmers out there at times. But we live in a globalised
world now and there’s no long any excuse for any self-respecting programmer to write
ASCII-first code.
Rust arrays are homogeneous (each array contains values of only one type) and
are of a fixed-size, which must be known at compile time. They are always
stored on the stack. Rust does provide a more dynamic Vec type which uses
the heap and allows resizing, but I’m not going to discuss that here.
In the interests of safety, Rust requires that every element of an array be initialise when constructed. Because of this, it’s usually not required to specify a type, but of course there is a syntax for doing so. It’s also possible to initialise every item to the same value using a shorthand. These are all illustrated in the example below.
1 2 3 4 | // These two are equivalent, due to type inference. let numbers1 = [9, 9, 9, 9, 9]; let numbers2: [i32; 5] = [9, 9, 9, 9, 9]; let numbers3 = [9; 5]; // Repeated value shorthand. |
Although the size of the array must be known at compile-time, of course the compiler can’t police your accesses to the array. For example, you may access an item based on user input. Rust does do bounds-checking at runtime, however, Discussion of how to handle runtime errors like this is a topic for another time, but the default action will be to terminate the executable immediately.
The basic mechanics of structs in Rust work quite analogously to those in C++, aside from some minor syntactic differences. Here’s a definition to illustrate:
1 2 3 4 5 6 7 | struct Contact { first_name: String, last_name: String, email: String, age: u8, business: bool, } |
To create an instance of a struct the syntax is similar except providing values instead of types after the colons. After creation the dot notation to read and assign struct fields will also be familiar to both C++ and Python programmers:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | fn main() { let mut contact1 = Contact { first_name: String::from("John"), last_name: String::from("Doe"), email: String::from("jdoe@example.com"), age: 21, business: false, }; println!("Contact name is {} {}", contact1.first_name, contact1.last_name); contact1.first_name = String::from("Jane"); println!("Contact name is {} {}", contact1.first_name, contact1.last_name); } |
Note that to assign to first_name we had to make contact1 mutable and that
this mutability applies to the entire structure, not to each field. No surprises
for C++ programmers there either.
Now there are a couple more unique features that are worth mentioning. The first
of them comes when creating constructor methods. Let’s say we want to avoid
having to set the business field, so we wrap it up in a function:
1 2 3 4 5 6 7 8 9 10 11 12 13 | fn new_business_contact(first_name: String, last_name: String, email: String, age: u8) -> Contact { Contact { first_name: first_name, last_name: last_name, email: email, age: age, business: true } } |
However, it’s a bit tedious repeating all those field names in the body. Well, if the function parameters happen to match the field names you can use a shorthand for this:
1 2 3 4 5 6 7 8 9 10 11 12 13 | fn new_business_contact(first_name: String, last_name: String, email: String, age: u8) -> Contact { Contact { first_name, last_name, email, age, business: true } } |
Another convenient syntactic trick is the struct update syntax, which can be used to create a copy of another struct with some changes:
1 2 3 4 5 6 7 8 9 | let contact1 = Contact { … }; … let contact2 = Contact { first_name: String::from("John"), last_name: String::from("Smith"), ..contact1 }; |
This will duplicate all fields not explicitly changed. There can be a sting
in this particular tail, though, due to the ownership rules. In this example,
the String value from contact1.email will be moved into contact2.email
and so the first instance will no longer be valid after this point.
Finally in this section I’ll briefly talk about tuples. I’m talking about them here rather than along with other compound types because I feel they work in a very similar way to structs, just without the field names. They have a fixed size defined when they are created and this cannot change, as with an array. Unlike an array, however, they are heterogeneous: they can contain multiple different types.
One thing that might surprise Python programmers in particular, however, is that the elements of a tuple are accessed using dot notation in the same way as a struct. In a way you can think of it as a struct where the names of the fields are just automatically chosen as base-zero integers.
1 2 3 4 5 6 | fn main() { let tup = (123, 4.56, "hello"); println!("{} {} {}", tup.0, tup.1, tup.2); // Can also include explicit types for the tuple fields. let tup_copy: (u32, f64, String) = tup; } |
If you want to share the definition of a tuple around in the same way as for a struct but you don’t want to give the fields names, you can use a tuple struct to do that:
1 2 3 4 5 6 | struct Colour(u8, u8, u8); fn main() { let purple = Colour(255, 0, 255); println!("R={} G={}, B={}", purple.0, purple.1, purple.2); } |
In all honesty I’m not entirely sure how useful that’ll be, but time will tell.
The final note here is that structs can also hold references, although none of the examples here utilised that. However, doing so means exercising a little more care because the original value can’t go out of scope any time before any structs with references to it. This is a topic for a future discussion on lifetimes.
Continuing the theme of data types that C++ offers, Rust also has enumerations, hereafter referred to as enums. Beyond the name the similarity gets very loose, however. In C++ enums are essentially a way to add textual aliases to integral values; there’s a bit of syntactic sugar to treat them as regular values, but you don’t have to dip your toes too far under the water to get them bitten by an integer.
In Rust, however, they have features that are more like a union in C++, although unlike a union they don’t rely on the programmer to know which variant is in use at any given time.
You can use them very much like a regular enum. The values defined within the enum are scoped within the namespace of the enumeration name1.
1 2 3 4 5 6 7 8 9 | enum ContactType { Personal, Colleague, Vendor, Customer, } let contact1_type = ContactType::Personal; let contact2_type = ContactType::Vendor; |
However, much more powerfully than this these variants can also have data values associated with them, and each variant can be associated with its own data type.
1 2 3 4 5 6 7 8 9 10 11 12 13 | // We reference contacts by their email address except for // colleagues, where we use employee number; and vendors, // where we use supplier ID, which consists of three numbers. enum ContactType { Personal(String), Colleague(u64), Vendor(u32, u32, u32), Customer(String) } let customer = ContactType::Customer("andy@example.com"); let colleague = ContactType::Colleague(229382); let supplier = ContactType::Vendor(23, 223, 4); |
This construct is great for implementing the sort of code where you need to branch differently based on the underlying type of something. I can just hear the voices of all the object-orientation purists declaring that polymorphism is the correct solution to this problem: that everything should be exposed as an abstract method in the base class that all the derived classes implement. I wouldn’t say I disagree necessarily, but I would also say that this isn’t a clean fit in every case and polymorphism isn’t the one-size-fits-all solution as which it has on occasion been presented.
Rust implements some types of polymorphism and features such as traits are a useful alternative to inheritance for code reuse, as we’ll see in a later post. But since Rust doesn’t implement true inheritance, more properly called subtype polymorphism, then I suspect this flexibility of enumerations is more important in Rust than it would be in C++.
A little further down we’ll see how to use the match operator to do this sort of switching in an elegant way, but first we’ll see one example of a pre-defined enum in Rust that’s particularly widely used.
It’s a very common case that a function needs to return a value in the happy case or raise some sort of error in the less happy case. Different languages have different mechanisms for this, one of the more common in modern languages being to raise exceptions. This is particularly common in Python, where exceptions are used for a large proportion of the functionality, but it’s also quite normal in C++ where the function of the destructors and the stack unwinding process are both heavily oriented around making this a fairly safe process.
Despite its extensive support for exceptions, however, C++ is still a bit of a
hybrid and it has a number of
cases where its APIs still use the other primary method of returning errors,
via the return value. A good example of this is the
std::string::find() method which searches for a
substring within the parent string. This clearly has two different classes
of result: either the string is found, in which case the offset within the
parent string is returned; or the string is not found, in which case the
method returns the magic std::string::npos value. In other cases functions
can return either a pointer for the happy case or a NULL in case of error.
Rust does not support exceptions. This is for a number of reasons, partly related to the overhead of raising exceptions and also the fact that return values make it easier for the compiler to force the programmer to handle all error cases that a function can return.
To implement these error returns in Rust, therefore, is where the Option enum
comes in useful. It’s defined something like this:
1 2 3 4 | enum Option<T> { Some(T). None, } |
This enum is capable of storing some type T which is a template type
(generics will be discussed properly in a later post), or the single
value None. This allows a function to return any value type it wishes,
but also leave open the possibility of returning None for an error.
That’s about all there is to say about Option, and we’ll see the
idiomatic way to use it in the next section.
The final thing I’m going to talk about is the match flow control
operator. This is conceptually similar to the switch statement
in C++, but it’s got rather more cleverness up its sleeves.
The first thing to note about match is that unlike switch in C++
it is an expression instead of a statement. One aspect of Rust I haven’t
talked about yet is that expressions may contain statements, however,
so this isn’t a major obstacle. But it does mean that it’s fairly easy
to use simple match expressions in assignments or as return values:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | enum Direction { North, South, East, West, } fn get_bearing(d: Direction) -> u16 { match d { Direction::North => 0, Direction::East => 90, Direction::South => 180, Direction::West => 270, } } |
The match expression has multiple “arms” which have a pattern and a
result expression. To do more than just return a value from the expression,
we can wrap it in braces:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | fn get_bearing(d: Direction) -> u16 { match d { Direction::North => 0, Direction::East => { println!("East is East"); 90 }, Direction::South => { println!("Due South"); 180 }, Direction::West => { println!("Go West"); 270 }, } } |
We can use the patterns to do more than just match specific values, though.
Taking the Option type from earlier, we can use it to extract the return values
from functions whilst still ensuring we handle all the error cases.
For example, the String::find() method searches for a substring and returns
an Option<usze> which is None if the value wasn’t found or the offset within
the string if it was found. We can use this to, say, extract the domain part
from an email address:
1 2 3 4 5 6 | fn get_domain(email: &String) -> &str { match email.find('@') { None => "", Some(x) => &email[x+1..], } } |
This function takes a String reference and returns a string slice representing
the domain part of the email, unless the email address doesn’t contain an @
character in which case we return an empty string. I’m not going to say that
the semantics of an empty string are ideal in this case, but it’s just an example.
As another example we could write a function to display the contact details for
the ContactType defined earlier:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | enum ContactType { Personal(String), Colleague(u64), Vendor(u32, u32, u32), Customer(String) } fn show_contact(contact: ContactType) { match contact { ContactType::Personal(email) => { println!("Personal: {}", email); }, ContactType::Colleague(employee_number) => { println!("Colleague: {}", employee_number); }, ContactType::Vendor(id1, id2, id3) => { println!("Vendor: {}-{}-{}", id1, id2, id3); }, ContactType::Customer(email) => { println!("Customer: {}", email); }, } } |
One aspect of match statements that isn’t immediately obvious is that they
are required to be exhaustive. So, if you don’t handle every time enum
value, for example, then you’ll get a compile error. This is what makes
things like the Option example particularly safe as it forces handling
of all errors, which is generally regarded as a good practice if you’re
writing robust code. This also makes perfect sense if you consider that
match is an expression: if you assign the result to a variable, say, then
then compiler needs something to assign and if you hit a case that your
match doesn’t handle then what’s the compiler going to do?
Of course if we’re using match for something other than an enum then handling
every value would be pretty tedious. For these cases we can use the pattern
_ as the default match. The example below also shows how we can match multiple
patterns using | as a separator:
1 2 3 4 5 6 | fn is_perfect(n: u32) -> bool { match n { 6 | 28 | 496 | 8128 | 33_550_336 => true, _ => false } } |
Here we’re meeting the needs of match by covering every single case. If we
removed that final default arm, the compiler wouldn’t let us get away with it:
error[E0004]: non-exhaustive patterns: `0u32..=5u32`,
`7u32..=27u32`, `29u32..=495u32` and 3 more not covered
--> src/main.rs:10:11
|
10 | match n {
| ^ patterns `0u32..=5u32`, `7u32..=27u32`,
`29u32..=495u32` and 3 more not covered
|
= help: ensure that all possible cases are being handled,
possibly by adding wildcards or more match arms
But what if we really wanted to only handle a single case? It would be pretty
dull if we had to have a default arm in a match then check for that value
being returned and ignore it.
Let’s take the get_domain() example from earlier. Let’s say that if you
find a domain, you want to use it; but if not, you have some more complicated
logic to invoke to infer the domain by looking at the username. You could
handle that by doing something like this:
1 2 3 4 5 6 7 8 9 10 11 | fn get_domain(email: &String) -> &str { let ret = match email.find('@') { None => "", Some(x) => &email[x+1..], }; if ret != "" { ret; } else { // More complex logic goes here... } } |
But that’s a little clunky. Rust has a special syntax called if let
for handling just a single case like this:
1 2 3 4 5 6 7 | fn get_domain(email: &String) -> &str { if let Some(x) = email.find('@') { &email[x+1..]; } else { // More complex logic goes here... } } |
I only recently came across this syntax and my opinions are honestly a
little mixed. Whilst I find the match statements comprehensible and
intuitive, this odd combination of if and let just seems unusual to me.
Mind you, I suspect it’s a common enough case to be useful.
So that’s a whirlwind tour of match and Rust’s pattern-matching.
It’s important to note that this is a much more powerful feature than I’ve
managed to express here as we’ve only really discussed matching by literals
and by enum type. In general patterns can be used in fairly creative ways
to extract fields from values at the same time as matching literals, and they
can even have conditional expressions added, which Rust calls match guards.
These are illustrated in the (rather contrived!) example below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | struct Colour { red: u8, green: u8, blue: u8 } fn classify_colour(c: Colour) { match c { Colour {red: 0, green: 0, blue: 0} => { println!("Black"); }, Colour {red: 255, green: 255, blue: 255} => { println!("White"); }, Colour {red: r, green: 0, blue: 0} => { println!("Red {}", r); }, Colour {red: 0, green: g, blue: 0} => { println!("Green {}", g); }, Colour {red: 0, green: 0, blue: b} => { println!("Blue {}", b); }, Colour {red: r, green: g, blue: 0} => { println!("Brown {} {}", r, g); }, Colour {red: r, green: 0, blue: b} => { println!("Purple {} {}", r, b); }, Colour {red: r, green: g, blue: b} if r == b && r == g => { println!("Grey {}", r); } Colour {red: r, green: g, blue: b} => { println!("Mixed colour {}, {}, {}", r, g, b); } } } |
Hopefully most things there are fairly self-explanatory and in any case it’s just intended as an illustration of the sorts of facilities that are available. It’s also worth mentioning that the compiler does give you some help to detect if you’re masking patterns with earlier ones, but it doesn’t appear to be perfect. For example, if I moved the first two matches to the end of the list, they’re both correctly flagged as unreachable. However, if I move the pattern for white after the pattern for grey it didn’t generate a warning; I’m guessing the job of determining reachability around match guards is just too difficult to do reliably.
Rust’s type system certainly offers some powerful flexibility, and the
pattern matching looks like a fantastic feature for pulling apart
structures and matching special cases within them. The specific
Option enum also looks like quite a pleasant way to implement the
“value or error” case given that Rust doesn’t offer exceptions for
this purpose.
My main reservation around these features is that there’s an awful
lot of syntax building up here, and it’s a fine line between a good
amount of expressive power and edging into Perl’s “there’s too many
ways to do it” philosophy. The if let syntax in particular seems
possibly excessive to me. But I’m certainly reserving judgement on
that for now until I’ve had some more experience with the language.
For anyone familiar with C++11, this is what you get when you declare a C++ enum with enum class MyEnum { … }. ↩
This is part 1 of the “Uncovering Rust” series.
Rust is fairly new multi-paradigm system programmating langauge that claims to offer both high performance and strong safety guarantees, particularly around concurrency and memory allocation. As I play with the language a little, I’m using this series of blog posts to discuss some of its more unique features as I come across them. This one talks about Rust’s ownership model.
Over the last few years I’ve become more aware of the Rust programming langauge. Slightly more than a decade old, it has consistently topped the Stack Overflow Developer Survey in the most loved langauge category for the last four years, so there’s clearly a decent core of very keen developers using it. It aims to offer performance on a par with C++ whilst considerably improving on the safety of the language, so as a long-time C++ programmer who’s all too aware of its potential for painfully opaque bugs, I thought it was definitely worth checking what Rust brings to the table.
As the first article in what I hope will become a reasonable series, I should briefly point out what these articles are not. They are certainly not meant to be a detailed discussion of Rust’s history or design principles, nor a tutorial. The official documentation and other sources already do a great job of those things.
Instead, this series is a hopefully interesting tour of some of the aspects of the language that set it apart, enough to get a flavour of it and perhaps decide if you’re interested in looking further yourself. I’m specifically going to be comparing the language to C++ and perhaps occasionally Python as the two languages with which I’m currently most familiar.
Before I get going on the topic of this post, I feel it’s important to clarify
one perhaps surprising detail of Rust to help understand the code examples below,
and it is this: all variables are immutable by default. It’s possible to declare
any variable mutable by prefixing with the mut keyword.
I could imagine some people considering this is a minor syntactic issue as it
just means what would be const in C++ is non-mut in Rust, and non-const in
C++ is mut in Rust. So why mention it? Well, mostly to help people understand
the code examples a little easier; whilst it’s debatably not a fundamental issue,
it’s also not something that’s necessarily self-evident from the syntax either.
Also, I think it’s a nice little preview of the way the language pushes you towards one of its primary goals: safety. If you forget the modifier things default to the most restrictive situation, and the compiler will prod you to add the modifier explicitly if that’s what you want. But if it isn’t what you want, you get the hint to fix a potential bug. Immutable values typically also make it much easier to take advantage of concurrency safely, but that’s a topic for a future post.
Since one of the touted features of the language is safety around memory allocation, I’m going to start off outlining how ownership works in Rust.
Ownership is a concept that’s stressed many times during the Rust documentation, although in my view it’s pretty fundamental to truly understanding any language. Manipulating variables in memory is the bulk of what software does most of the time and errors around ownership are some of the most common sources of bugs across multiple langauges.
In general “owning” a value in this context means that a piece of code has a responsibility to manage the memory associated with that value. This isn’t about mutability or any other concept people might feasibly regard as forms of ownership.
Just to be clear, I’m going to skip discussion of stack-allocated variables here. Management of data on the stack is generally similar in all mainstream imperative languages and generally falls out of the language scoping rules quite neatly, so I’m going to focus this discussion on the more interesting and variable topic of managing heap allocations.
In C++ ownership is a nebulous concept and left for the programmer to define. The language provides the facility to allocate memory and it’s up to the programmer to decide when it’s safe to free it. Techniques such as RAII allow a heap allocation to be tied to a particular scope, either on the stack or linked with an owning class, but this must be manually implemented by the programmer. It’s quite easy to neglect this in some case or other, and since it’s aggressively optional then the compiler isn’t going to help you police yourself. As a result, memory mismangement is a very common class of bugs in C++ code.
Higher-level languages tend to utilise different forms of garbage collection to avoid exposing the programmer to these issues. Python’s reference counting is a simple concept and covers most cases gracefully, although it adds peformance overhead to many operations in the language and cyclic references complicate matters such that additional garbage collection algorithems are still required. Languages like Java with tracing garbage collectors impose less performance penalty on access than reference counting, but may be prone to spikes of sudden load when a garbage sweep is done. These systems are also often more complex to implement, especially as in the real world they’re often a hybrid of multiple techniques. This isn’t necessarily a direct concern for the programmer, as someone else has done all the hard work of implementing the algorithm, but it does inch up the risk of hitting unpredictable pathalogical performance behaviour. These can be the sort of intermittent bugs that we all love to hate to investigate.
All this said, Rust takes a simpler approach, which I suppose you could think of as what’s left of reference counting after a particularly aggressive assult from Ockham’s Razor.
Rust enforces three simple rules of ownership:
I’m not going to go into detail on the scoping rules of Rust right now, although there are some interesting details that I’ll probably cover in another post. For now suffice to say that Rust is lexically scoped in a very similar way to C++ where variables are in scope from their definition until the end of the block in which they’re defined2.
This means, therefore, that because a value has only a single owner, and because the scope of that owner is well-defined and must always exit at some point, there is no possible way for the value to not be dropped and its memory leaked. Hence achieving the promised memory safety with some very simple rules that can be validated at compile-time.
So there you go, you assign a variable and the value will be valid until such point as that variable goes out of scope. What could be simpler?
1 2 3 4 5 6 7 8 9 | // Start of block. { … // String value springs into existence. let my_value = String::from("hello, world"); println!("Value: {}", my_value); … } // End of block, my_value out of scope, value dropped. |
Well of course it’s not quite that simple. For example, what happens if we assign the value to another variable? I mean, that’s a pretty simple case. How hard can it be to figure out what this code will print?
1 2 3 4 5 | fn main() { let my_value = String::from("hello, world"); let another_value = my_value; println!("Values: {} {}", my_value, another_value); } |
The answer is: slightly harder than you might imagine. In fact the code above won’t even compile:
Compiling sample v0.1.0 (/Users/apearce16/src/local/rust-tutorial/sample)
error[E0382]: borrow of moved value: `my_value`
--> src/main.rs:4:31
|
2 | let my_value = String::from("hello, world");
| -------- move occurs because `my_value` has type
`std::string::String`, which does not implement the `Copy` trait
3 | let another_value = my_value;
| -------- value moved here
4 | println!("Values: {} {}", my_value, another_value);
| ^^^^^^^^ value borrowed here after move
error: aborting due to previous error
This is because Rust implements
move semantics by default on assignment. So what’s really happening in the code
above is that a string value is created and ownership is assigned to the my_value
variable. Then this is assigned to another_value which results in ownership being
transferred to the another_value variable. At this point the my_value variable
is still in scope, but it’s no longer valid.
The compiler is pretty comprehensive in explaining what’s going on here, the value is
moved in the second line and then the invalidated my_value is referenced in the third
line, which is what triggers the error.
This may seem unintuitive to some people, but before making any judgements you should consider the alternatives. Firstly, Rust could abandon its simple ownership rules and allow arbitrary aliasing like in C++. Except that would mean either exposing manual memory management or replacing it with a more expensive garbage collector, both of which compromise on the goals of safety and performance respectively.
Secondly, Rust could perform a deep copy of the data on the assignment, so duplicating the value and ending up with two variables each with its own copy. This is workable, but defeats the goal of performance as memory copying is pretty slow if you end up doing an awful lot of it. It also violates a basic programmer expectation that a simple action like assignment should not be expensive.
And so we’re left with the move semantics defined above. It’s worth noting, however,
that this doesn’t apply to all types. Some are defined as being safe to copy: generally
the simple scalar types such as integers, floats, booleans, and so on. The key property
of these which make them safe is that they’re stored entirely on the stack, there’s
no associated heap allocation to handle.
It’s also possible to declare that new types are safe to copy by adding the Copy
trait, but traits are definitely a topic for a later post.
It’s also worth noting that these move semantics are not as restrictive as they might seem due to the existence of references, which I’ll talk about later in this post. First, though, it’s interesting to look at how these semantics work with functions.
The ownership rules within a scope are now clear, but what about passing values into functions? In C++, for example, arguments are passed by value which means that the function essentially operates on a copy. If this value happens to be a pointer or reference then of course the original value may be modified, but as mentioned above we’re deferring discussion of references in Rust for a moment.
Argument passing would appear to suffer the same issues as the assignment example above, in that we don’t want to perform a deep copy, but neither do we want to complicate the ownership rules. So it’s probably little surprise that argument passing into functions also passes ownership in the same way as the assignment.
This code snippet will fail to compile:
1 2 3 4 5 6 7 8 9 10 11 12 | fn main() { let s = String::from("hello"); my_function(s); // Oops, s isn't valid here any more! println!("Value of s: {}", s); } fn my_function(arg: String) { // Ownership passes to the 'arg' parameter. println!("Now I own {}", s); // Here 'arg' goes out of scope and the String is dropped. } |
Although this may seem superficially surprising, when you really think about it argument passing is just a fancy form of assignment into a form of nested scope, so it shouldn’t be a surprise that it follows the same semantics.
The same logic applies to function return values, and this is where things could get slightly surprising for C++ programmers who are used to returning pointers or references to stack values being a tremendous source of bugs; and returning non-referential values as a cause of potentially expensive copy operations.
In C++ when the function call ends, any pointer or reference to anything on its stack that is passed to the caller will now be invalid. These can be some pretty nasty bugs, particuarly for less experienced programmers. It doesn’t help that the compiler doesn’t stop you doing this, and also that these situations often give the appearance of working correctly initially, since the stack frame of the function has often not been reused yet so the pointer still seems to point to valid data immediately after the call returns. This clearly harms the safety of the code.
If the programmer decides to resolve this issue by returning a complex class directly by value instead of by pointer or reference, then this generally entails default construction of an instance in the caller, then execution of the function and then assignment of the returned value to the instance in the caller which might involve some expensive copying. This potentially harms the performance of the code.
I’m deliberately glossing over some subtleties here around returning temporary objects, return value optimisation and move semantics in C++ which are all well outside the scope of this post on Rust. But even though solutions to these issues exist, they require significant knowledge and experience on the part of the programmer to take advantage of correctly, particularly for user-defined classes.
In Rust things are simpler: you can return a local value and ownership passes to the caller in the obvious manner.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | fn main() { let my_value = create(); // At this point 'my_value' owns a String. println!("Now I own {}", my_value); let another_value = transform(my_value); // At this point 'another_value' owns a string, // but 'my_value' is now invalid. println!("Now I own {}", another_value); } fn create() -> String { let new_str = String::from("hello, world"); // Ownership will pass to the caller. new_str } fn transform(mut arg: String) -> String { // We've delcared the argument mutable, which is OK // since ownership has passed to us. We append some // text to it and then return it, whereupon ownership // passes back to the caller. arg.push_str("!!!"); arg } |
For anyone puzzled by the bare expressions at the end of the functions on lines 15 and
24, suffice to say for now this is an idiomatic way to return a value in Rust. The
language does have a return statement, but a bare expression also works in some
cases. I’ll discuss this more in a later post.
So in the case of return values, the move semantics of ownership in Rust turn out to be pretty useful: the ownership passes to the caller safely and with no need for expensive copying, since somewhere under the hood it’s just a transfer of some reference to a value on the heap. Since the rules apply everywhere it all feels quite consistent and logical.
But as logical as it is, it may seem awfully inconvenient. There are many cases we want a value to persist after it has been operated on by a function. It would be annoying to have to deep-copy an object every time, or to constantly have to return the argument to the caller as in the example above.
Fortunately Rust provides references to resolve this inconvenience.
In Rust references provide a way to refer to a value without actually taking ownership of it. The example below demonstrates the syntax, which is quite reminiscent of C++:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | fn main() { let my_string = String::from("one two three"); let num_words = count_words(&my_string); // 'my_string' is still valid here. println!("'{}' has {} words", my_string, num_words); } // I'm sure there are more elegant ways to implement // this function, this is just for illustrating the point. fn count_words(s: &String) -> usize { let mut words = 0; let mut in_word = false; for c in s.chars() { if c.is_alphanumeric() { if !in_word { words += 1; in_word = true; } } else { in_word = false; } } words } |
The code example above shows a value being passed by immutable reference. Note that
the function signature needs to be updated to take a reference &String, but the
caller must also explicitly declare the parameter to be a reference with &my_string.
This is unlike in C++ where there’s no explicit hint to someone reading the code in
the caller that a value might be passed by reference. For immutable references (or
const refs in C++ parlance) this isn’t a big deal, but I’ve always felt that it’s
always important to know for sure whether a function might modify one of its parameters
in-place, and in C++ you have to go check the function signature every time to tell
whether this is the case. This has always been one of my biggest annoyances with C++
syntax and it’s great to see it’s been addressed in Rust.
Taking a reference is rather quaintly known as borrowing in Rust. You can take as many references to a value as you like as long as they’re immutable.
1 2 3 4 5 6 | fn main() { let mut my_value = String::from("hello, world"); let ref1 = &my_value; let ref2 = &my_value; let ref3 = &my_value; } |
Of course, attempting to modify the value through any of these references will result in a compile error, since they’re immutable. As you’d expect it’s also possible to take mutable references:
1 2 3 4 5 6 7 8 9 | fn main() { let mut my_value = String::from("world"); prefix_hello(&mut my_value); println!("New value: {}", my_value); } fn prefix_hello(arg: &mut String) { arg.insert_str(0, "hello "); } |
This example also illustrates that it’s once again clear in the context of the caller that it’s specifically a mutable reference that’s being passed.
This all seems great, but there’s a couple of restrictions I haven’t mentioned yet. Firstly, it’s only valid to have a single mutable reference to a value at once. If you try to create more than one you’ll get an error at compile-time. Secondly, you can’t have both immutable and a mutable reference valid at the same time, which would also be a compile-time error.
The logic behind this is around safety when values are used concurrently. These rules do a good job of ruling out race conditions, as it’s not possible to multiple references to the same object unless they’re all immutable, and if the data doesn’t change then there can’t be a race. It’s essentially a multiple readers/single writer lock.
The compiler also protects you against creating dangling references, such as returning a reference to a stack function. That will fail to compile3.
Whilst I’m talking about references anyway, it’s worth briefly mentioning slices. These are like references, but they only refer to a subset of a collection.
1 2 3 4 5 6 | fn main() { let my_value = String::from("hello there, world"); // String slice 'there'. let there = &my_value[6..11]; println!("<<{}>>", there); } |
The example above shows a use for an immutable string slice. Actually you may not realised it but you’ve seen one of those earlier in this post: all string literals are in fact immutable string slices.
As with slices in most languages the syntax is a half-open interval where the first index is inclusive, the second exclusive. It’s also possible to have slices of other collections that are contiguous and it’s possible to have mutable slices as well.
1 2 3 4 5 6 7 | fn main() { let mut my_list = [1,2,3,4,5]; let slice = &mut my_list[1..3]; slice[1] = 99; // [1, 2, 99, 4, 5] println!("{:?}", my_list); } |
As far as I’ve been able to tell so far, however, it doesn’t seem to be possible to assign to the entirity of a mutable slice to replace it. I can understand several reasons why this might not be a good idea to implement, not least of which that it can change the size of the slice and hence necessitate moving items around in memory that aren’t even part of the slice (if you assign something of a different length). But I thought it was worth noting.
In this post I’ve summarised what I know so far about ownership and references in Rust and generally I think it’s shaping up to be a pretty sensible language. Of course it’s hard to say until you’ve put it to some serious use4, but I can see that there are good justifications for the quirks that I’ve discovered so far, bearing in mind the overarching goals of the language.
The ownership rules seem simple enough to keep in mind in practice, and it remains to be seen whether they will make writing non-trivial code more cumbersome than it needs to be. I like the explicit reference syntax in the caller and whilst the move semantics might seem odd at first, I think they’re simple and consistent enough to get used to pretty quickly. The fact that the compiler catches so many errors should be particularly helpful, especially as I’ve found its output format to be pleasantly detailed and particularly helpful compared to many other languages.
What you would call memory being freed in C++ is referred to as a value being dropped in Rust. The meaning is more or less the same. ↩
Spoiler alert: the scope of a variable in Rust actually extends to the last place in the block where it is referenced, not necessarily to the end of the block, but that doesn’t materially alter the discussion of ownership. ↩
Unless you specify the value has a static lifetime but I’ll talk about lifetimes another time. ↩
I came across Perl in 1999 and thought it was a pretty cool from learning it right up until I had to try to fix bugs in the first large project I wrote in it, so it just goes to show that first impressions of programming languages are hardly infallible. ↩