☑ Brexit, or Brexit Not — There Is No Try

I voted against Brexit as I feel the UK is significantly better within the EU. However, the looming uncertainty over whether the UK will follow through is much worse than either option.

On Thursday 23 June the United Kingdom held a referendum to decide whether to remain within the European Union, of which it has been a member since 1973. The vote was to leave by a majority of 52% with a fairly substantial turnout of almost 72%. Not the largest majority, but a difference of over a million people can’t be called ambiguous.

So that was it then — we were out. Time to start comparing people’s plans for making it happen to decide which was the best.

Except, of course, it turned out nobody really had any plans. The result seemed to have been a bit of a shock to everyone, including all the politicians who were campaigning for it. Nobody really seemed to know what to do next. Disappointing, but hardly surprising — we’re a rather impulsive nation, always jumping into things without really figuring out what our end game should be. Just look at the shambles that followed the Iraq war.

Fortunately for the Brexiteers there was a bit of a distraction in the form of David Cameron’s resignation — having campaigned to remain within the EU he felt that remaining as leader was untenable. Well, let’s face it, that’s probably disingenuous — what he most likely really felt was he didn’t want to go down in history as the Prime Minister who took the country out of the EU, just in case (as many people think quite likely) it’s a bit of a disaster, quite possibly resented by generations to come.

This triggered an immediately leadership contest within the Tory party which drew all eyes for a time, until former Home Secretary Theresa May was left as the only candidate and assumed leadership of the party. At this point everyone’s attention seems to be meandering its way back to thoughts of Brexit and all the questions it raises.

And a lot of questions there certainly are. There are immigration questions, NHS questions, questions for the Bank of England, questions for EU migrants, questions for Northern Ireland, profound questions for Scotland1 questions for David Davis, and even a whopping great 94 questions on climate and energy policy, which frankly I think is rather hypocritical — they know full well that nobody has any use for so many questions and most of them will end up on landfill.

To my mind, however, there’s still one question that supercedes all these when talking about Brexit — namely, will Br actually exit?

You’d think this was a done deal — I mean, we had a referendum on it and everything. Usually clears these things right up. But in this case, even well over a month after the vote, there’s still talk about whether we’re going to go through with it.

Apparently the legal situation seems quite muddy but there are possible grounds for a second referendum — although Theresa May is on record as rejecting that possibility. I must say I can see her point — to reject the clearly stated opinion of the British public would need some pretty robust justification and “the leave campaigners lied through their teeth” probably doesn’t really cut it. It’s not like people aren’t used to dealing with politicians being economical with the truth in general elections.

Then we hear that the House of Lords might try to block the progress of Brexit — or at least delay it. Once again, it’s not yet at all clear to what extent this will happen; and if it happens, how effective it will be; and if it’s effective, how easily the government can bypass it. For example, the government could try to force it through with the Parliament Act.

What this all adds up to is very little clarity right now. We have a flurry of mixed messages coming out of government where they tell us that the one thing they are 100% certain of is that they’re definitely going to leave the EU, but not only can’t they give us a plan, they can’t even give us a rough approximation of when they’ll have a plan; we have a motley crew of different groups clamouring for increasingly deperate ways to delay, defer or cancel the whole thing, but very little certainty on whether they even have the theoretical grounds to do so let alone the public support to push it through; and we have an increasingly grumpy EU who are telling us that if we’re really going to leave then we should jolly well get on with it and don’t let Article 50 hit us in the bum on the way out.

Meanwhile the rest of the world doesn’t seem to know what to make of it, so it’s not clear that we’ve seen much of the possible impact, even assuming we do go ahead. But to think there hasn’t been any impact is misleading — even when things are uncertain we’ve already seen negative impacts on academics, education and morale in the public sector. Let’s be clear here, it hasn’t happened yet and it isn’t even a certainty that it will, and we’re already seeing a torrent of negative sentiment.

To be fair, though, we haven’t yet really had the chance to see any possible positive aspects of the decision filtering through. In fact, we probably won’t see any of those until the decision is finalised — or at least until Article 50 is triggered and there’s a deadline to chivvy everyone along.

That’s a big problem.

I think that the longer this “will they/won’t they” period of uncertainty carries on, the more we’ll start to see these negative impacts. Nobody wants to bank on the unlikely event that the UK will change course and remain in the EU, but neither can anyone count on the fact that we won’t. We’re stuck in an increasingly acrimonious relationship that we can’t quite bring ourselves to end yet. If they could find an actor with a sufficient lack of charisma to play Nigel Farage, they could turn it into a low budget BBC Three sitcom.

Don’t get me wrong, I voted firmly to remain in this EU. But whatever we do, I feel like we, as a nation — and by that I mean they as a government that we, as a nation, were daft enough to elect2 — need to make a decision and act on it. This wasteland of uncertainty is worse than either option, and doesn’t benefit anyone except the lawyers and the journalists — frankly they can both find more worthwhile ways to earn their keep.

So come on Theresa, stop messing about. Stick on a Spotify3 playlist called something like “100 Best Break Up Songs”, mutter some consoling nonsense to yourself about how there are plenty more nation states in the sea and pick up the phone. Then we can get on with making the best of wherever we find ourselves.


  1. Although they’re asked by Michael Gove so I dont know if they count — given his behaviour during the Tory leadership election I’m not sure he’s been allowed off the naughty step yet. 

  2. In the interests of balance I should point out that, in my opinion, more or less every party this country elected since 1950 has been a daft decision. Probably before that, too, but my history gets a little too rusty to be certain. The main problem is that the people elected have an unpleasant tendency to be politicians, and if there’s one group of people to whom the business of politics should never be entrusted, it’s politicians. 

  3. Assuming Spotify, being Swedish, are still allowed? 

Thu 04 Aug 2016 at 07:45PM by Andy Pearce in Politics tagged with uk, brexit and politics  |  See comments

☑ Responsive Design

My website now looks hopefully very slightly less terrible on mobile devices, and I learned a few things getting there.

This website is more or less wholly my own work. I use a content generator to create the HTML from Markdown source files, but the credit (or blame) for the layout and design lies with me alone1.

Actually, I should hastily add that the one exception to this is that the current colour scheme comes straight from Ethan Schoover’s excellent Solarized palette. But that’s not really germane to the topic of this post, which I note with dismay I haven’t yet broached despite this already being the end of the second paragraph.

Now I’m not going to try and claim that this site is brilliantly designed — I’m a software engineer not a graphic designer. But I do strongly believe that developers should strive to be generalists where practical and this site is my opportunity to tinker. Most recently my tinkering has been to try to improve the experience on mobile devices, and I thought I’d share my meagre discoveries for anyone else who’s looking for a some basic discussion on the topic to get them started.

The first thing I’d like to make clear is that this stuff is surprisingly simple. I’ve known for awhile that the experience of reading this site on my iPhone was not a hugely pleasant one compared to the desktop, but I’d put off dealing with it on the assumption that fixing it would require all sorts of hacky Javascript or browser-specific hacks — it turns out that it doesn’t, really.

However, there are some gotchas and quirks due to the way that font rendering works and other odd issues. Without further ado let’s jump head-first into them.

Any viewport in a storm

No matter how simple your site layout or styling, you might find that it doesn’t render optimally on mobile devices. One reason for this is that mobile browsers render to a much larger resolution than the actual screen, and then they zoom font sizes up to remain legible on the screen.

The reason they do this is because they’ve grown up having to deal with all sorts of awful web designers to whom it never occurred that someone might want to view their page at anything less than a maximised web browser on a 1600x1200 display. As a result they use all sorts of absolute positioning and pixel widths, and these cope horribly when rendered on a low resolution screen. Thus the browsers pretend that their screen is high resolution and render accordingly — but to keep the text readable they have to boost the font sizes. The net result is typically something that’s not necessarily all that pretty, but is often surprisingly usable.

This is quite annoying if you’ve gone to the trouble of making sure your site renders correctly at lower resolutions, however, and that’s pretty much the topic of this post. So how do you go about disabling this behaviour?

The short answer is to include a tag like this:

<meta name="viewport" content="initial-scale=1.0" />

The slightly longer answer is that the viewport meta tag specifies the viewport area to which you expect the browser to render. If your layout assumes a certain minimal resolution for desirable results then you can specify that width in pixels like this:

<meta name="viewport" content="width=500" />

This instructs the browser to render to a viewport of width 500 pixels and then perform any scaling required to fill the screen with this. It’s also possible to specify a height attribute but most web layouts don’t make too many assumptions about viewport height.

The first line I showed above didn’t set width, however, but instead the initial-scale. As you might expect, this instructs the browser to render the the full width of the page on the pixel width of the window without applying any scaling. Note that this is probably the only value you’ll need if you’re confident you’ve designed your styles to adapt well to all device screens.

You can also use this tag to constrain user scaling to prevent the user zooming in or out, but personally I find this a rather gross affront to accessibility so I’m not going to discuss it further — you should never presume to know how your users want to access your site better than your users, in my opinion.

When is a pixel not a pixel?

This is all fine and dandy, but pixels aren’t everything. If I render to a viewport of 1000 pixels the resultant physical size is significantly different on a desktop monitor than on a retina display iPhone.

The correct solution to this would probably be some sort of way to query the physical size or dot pitch of the display, but that would require all web designers to do The Right Thing™ and that’s always been a bit too much to ask.

Instead, therefore, browser vendors have invented a layer of “virtual pixels” which is designed to be approximately the same physical size on all devices. When you specify a size in pixels anywhere within your CSS then these “virtual pixels” are used — the browser will have some mapping to real physical pixels according to the screen size and resolution, and also no doubt to some extent the whims of the browser writers and consistency with other browsers.

The CSS specifications even have a definition of a reference pixel which links pixel measurements to physical distances — broadly speaking it says that a pixel means a pixel on a 96 DPI standard monitor. I wouldn’t want to make any assumptions about this, however — I’m guessing these things still vary from browser to browser and it’ll be some time, if ever, before everyone converges on the same standard.

As a result of all this you might find your page being stretched around despite your best efforts — there doesn’t seem to be a good way around this except have a fluid layout and use high resolution images in case they’re ever stretched (few things look more shoddy than a pixelated image).

If you want more details, Mozilla have a great article about the whole thing. You can also read all about the reference-pixel and this slightly old article about CSS units is also very useful.

Responsive design

Now our site renders at the correct size and scale on the device. Great. If your layout is anything like mine, however, now you’re feeling the loss of all those grotty scaling hacks and things don’t look all that good at all. What you need to do is restyle your site so it degrades gracefully at smaller screen sizes.

As much as possible I really recommend doing this through fluid layouts — the browsers are very good at laying out text and inline elements in an appropriate manner for a given width so where possible just use that. However, there are things that aren’t possible with this approach — for example, if you’re using a mobile device to view this site, or you shrink your browser window enough, you’ll see that the side bar disappears and jumps to the top to free up some additional width to stop the text getting too squashed. There’s no way that a browser will be able to make this sort of decision automatically, it needs a little help from the designer.

This sort of thing can be achieved with media queries.

These offer a way to conditionally include bits of CSS based on your current media type2 and resolution. They don’t make it any easier to design a good mobile layout — making pretty websites is left as an exercise for the reader — but they do at least give you a simple way to swap between the styles you’ve created for different viewports.

There are two ways to use them — you can put conditional blocks in your CSS:

@media screen and (max-width: 450px) {
    body {
        font-size:0.9em
    }
}

Or alternatively you can switch to an entirely different stylesheet in your HTML:

<link rel="stylesheet" media="screen and (max-width: 750px)"
      href="/styles/mobile.css" type="text/css" />

I use both of these approaches on my site — I find it easier to write a wholly different stylesheet for the mobile layout since it has so many small changes3, and then I make a few small tweaks within the CSS itself for more fine-grained viewport widths within that range.

For example, switching to the mobile stylesheet on my site converts the sidebar sections to inline-block and renders them above the content. Within that same stylesheet, however, there are some small tweaks to make these render centered and spaced out where there’s room to do so, but left-aligned for narrow displays where they’re likely to flow on to multiple lines.

However, it’s important to note that you could use either approach equally well on its own — I don’t believe there’s anything that can be achieved with one but not the other.

As an aside if you’re tempted to go for the <link> approach to save network traffic on the assumption that the browser will only fetch the sheets it needs, think again. The browser can’t really have any special foresight about how you’re going to resize the viewport so it fetches all CSS resources. The media queries then just control which rules get applied.

You can find a few more details in this Stack Overflow question. It does turn out that some browsers will avoid downloading image files they don’t need, but that would presumably apply regardless of whether the media query was located in a <link> tag or in a CSS file.

Font scaling

One additional issue to be aware of when designing mobile layouts is that if you render to the device width you can get odd results when you turn a mobile device into landscape orientation. Intuitively I’d expect a wider, shorter viewport with text rendered to the same font size, but in actual fact what you can end up with is a zoomed page instead. I think what’s going on here is that the scale factor is determined for portrait orientation and then applied in both cases, but I must admit I’m not confident I fully appreciate the details.

Whatever the cause, one fix appears to be to disable the mobile browser font size adjustments — this should no longer be required, after all, because the layout is now designed to work equally well in any viewport.

Because this is a somewhat new property and not necessarily supported in its standard form by all browsers it’s wise to specify all the browser-specific versions as well:

-webkit-text-size-adjust: 100%;
-moz-text-size-adjust: 100%;
-ms-text-size-adjust: 100%;
text-size-adjust: 100%;

You might have better luck playing around with different values of this setting than I did — I must confess I didn’t experiment much once I realised that disabling the adjustment seemed to fix my issues.

Hard copy

The discussion so far has been all about screens, primarily — the same media selectors can also be used to specify an alternative stylesheet for printing.

<link rel="stylesheet" media="print"
      href="/styles/print.css" type="text/css" />

This all works in exactly the same way except the sorts of things you do in these stylesheets is likely quite different. For example, I force all my colours to greyscale4 and remove the sidebar and other navigation elements entirely. I also remove decorations around links since these of course have no significance in a printed document.

If you want to remove something from the flow of the document you can do this in CSS by setting display to none:

@media print {
    div.sidebar {
        display: none;
    }
}

You might wonder how this differs from setting visibility: hidden — the difference is that display: none removes the element from the document completely, changing the layout; whereas visibility: hidden still reserves space for the element in the layout, it just doesn’t actually render it.

If you want to test out the print stylesheet of this or any site without wasting paper, you can do so with Google Chrome Developer Tools5. Open up the developer tools with the More Tools » Developer Tools menu option and then click the vertical ellipsis () and select More Tools » Rendering Settings to show a new window. Now you can tick the Emulate Media option and choose Print from the dropdown.

Layout in CSS

One important thing to note about all the techniques discussed on this page is that they only allow you to swap in or out CSS in response to viewport width — the HTML document structure itself is unchanged. This isn’t generally much of a limitation in today’s browsers since modern CSS offers a huge degree of flexibility, but it’s certainly something to bear in mind when you’re writing your HTML. In general the closer you are to a clean structural HTML document where CSS supplies all of the layout controls, the easier you’re likely to find adapting your site for multiple devices.

If you really need to redirect users to a different page based on width then of course it’s possible with Javascript, but this is a pretty ugly solution — it’s the sort of thing that leads to a world of pain where you have a version of your site for every device under the sun. If you’re the sort of masochist to whom that appeals, go right ahead.

Responsive vs. adaptive design

One final point that I should mention is that there are two schools of thought about laying things out for multiple devices — these are responsive and adaptive design, although it’s important to note that they’re not actually mutually exclusive.

Responsive design
This describes layouts that change continuously with the width of the viewport as it changes.
Adaptive design
This describes layouts that have several distinct options and “snap” between them as the viewport changes.

Responsive layouts are generally regarded as more graceful, I think, but adaptive layouts may be easier for more complex sites where it’s easier to implement a test a small number of distinct options. Personally I’ve used aspects of both, but I think I’d be comfortable describing my design as responsive overall since I try to use the full viewport width where I can, except in very wide cases where overlong text harms readability.

This is probably better illustrated than explained so I suggest checking out this set of GIFs that demonstrate the differences.

Conclusions

Overall I found the whole process of making my site more mobile-friendly very painless. I have quite a simple layout (quite intentionally) which made it a lot less hassle than I can imagine a lot of image-heavy sites might find. Frankly, though, that’s the modern web for you — bitmap images are so passé, and for good reason.


  1. Anyone who’s been curious enough to poke around in the HTML or CSS (as if anyone would do that…) might notice some references to Sphinx in the naming. This isn’t because I’ve pilfered anything from the Python documentation generator, but simply that this website theme started life as an ill-fated attempt to re-style Sphinx generated documentation — I soon realised, however, it was significantly deficient in every aspect to others such as the Read The Docs style and gave up on that idea and used it solely for this site. 

  2. Where media type is essentially either screen or print in the vast majority of cases. 

  3. I do factor out some commonality on the backend with sass, however, so arguably I’d save a little network bandwidth by putting those into a common CSS file and using only the CSS-style media queries. However, I feel that such minute fiddling would be somehwat against the spirit of the title of this blog.  

  4. This might be a little irksome to someone with a colour printer but due to the way I’ve factored out my palette choices from the rest of the CSS it makes life a good deal easier for me. For example, if I change my colour scheme to light-on-dark then there’s no guarantee that the text colours will render legibly in the dark-on-light world of hard copy, whereas greyscale should always be consistently readable on any printer. 

  5. Other browsers are available — some of them may even have equivalent functionality. As well as a browser, Google are good enough to provide you a search engine to find out just as easily as I could. 

Tue 02 Aug 2016 at 08:12AM by Andy Pearce in Software tagged with web  |  See comments

☑ The State of Python Coroutines: Python 3.5

This is part 4 of the “State of Python Coroutines” series which started with The State of Python Coroutines: yield from.

I recently spotted that Python 3.5 has added yet more features to make coroutines more straightforward to implement and use. Since I’m well behind the curve I thought I’d bring myself back up to date over a series of blog posts, each going over some functionality added in successive Python versions — this one covers additional syntax that was added in Python 3.5.

In the previous post in this series I went over an example of using coroutines to handle IO with asyncio and how it compared with the same example implemented using callbacks. This almost brings us up to date with coroutines in Python but there’s one more change yet to discuss — Python 3.5 contains some new keywords to make defining and using coroutines more convenient.

As usual for a Python release, 3.5 contains quite a few changes but probably the biggest, and certainly the most relevant to this article, are those proposed by PEP-492. These changes aim to raise coroutines from something that’s supported by libraries to the status of a core language feature supported by proper reserved keywords.

Sounds great — let’s run through the new features.

Declaring and awaiting coroutines

To declare a coroutine the syntax is the same as a normal function but where async def is used instead of the def keyword. This serves approximately the same function as the @asyncio.coroutine decorator did previously — indeed, I believe one purpose of the decorator, aside from documentation purposes, was to allow async def routines to be called. Since coroutines are now a language mechanism and shouldn’t be intrinsically tied to a specific library, there’s now also a new decorator @types.coroutine that can be used for this purpose.

Previously coroutines were essentially a special case of generators — it’s important to note that this is no longer the case, they are a wholly separate language construct. They do still use the generator mechanisms under the hood, but my understanding is that’s primarily an implementation detail with which programmers shouldn’t need to concern themselves most of the time.

The distinction between a function and a generator is whether the yield keyword appears in its body, but the distinction between a function and a coroutine is whether it’s delcared with async def. If you try to use yield in a coroutine declared with async def you’ll get SyntaxError (i.e. a routine cannot be both a generator and a coroutine).

So far so simple, but coroutines aren’t particularly useful until they can yield control to other code — that’s more or less the whole point. With generator-based coroutines this was achieved with yield from and with new syntax it’s achieved with the await keyword. This can be used to wait for the result from any object which is awaitable

An awaitable object is one of:

  • A coroutine, as declared with async def.
  • A coroutine-compatible generator (i.e. decorated with @types.coroutine).
  • Any object that implements an appropriate __await__() method.
  • Objects defined in C/C++ extensions with a tp_as_async.am_await method — this is more or less equivalent to __await__() in pure Python objects.

The last option is perhaps simpler than it sounds — any object wishes to be awaitable needs to return an interator from its __await__() method. This iterator is used to implement the funamental wait operation — the iterator’s __next__() method is invoked and the value it yields is used as the value of the await expression.

It’s important to note that this definition of awaitable is what’s required of the argument to await, but the same conditions don’t apply to yield from. There are some things that both will accept (i.e. coroutines) but await won’t accept generic generators and yield from won’t accept the other forms of awaitable (e.g. an object with __await__()).

It’s also equally important to note that a coroutine defined with async def can’t every directly return control to the event loop — there simply isn’t the machinery to do so. Typically this isn’t much of a problem since most of the time you’ll be using asyncio functions to do this, such as asyncio.sleep() — however, if you wanted to implement something like asyncio.sleep() yourself then as far as I can tell you could only do so with generator-based coroutines.

OK, so let me be pedantic and contradict myself for a moment — you can indeed implement something like asyncio.sleep() yourself. Indeed, here’s a simple implementation:

1
2
3
4
5
async def my_sleep(delay, result=None):
    loop = asyncio.get_event_loop()
    future = loop.create_future()
    loop.call_later(delay, future.set_result, result)
    return (await future)

This has a lot of deficiencies as it doesn’t handle being cancelled or other corner cases, but you get the idea. However the key point here is that this depends on asyncio.Future and if you go look at the implementation for that then you’ll see that __await__() is just an alias for __iter__() and that method uses yield to return control to the event loop. As I said earlier, it’s all built on generators under the hood, and since yield isn’t permitted in an async def coroutine, there’s no way to achieve that (at least as far as I can tell).

In general, however, the amount of times you would be returning control to the event loop is very low — the vast majority of cases where you’re likely to do that are for a fixed delay or for IO and asyncio already has you covered in both cases.

One final note is that there’s also an abstract base class for awaitable objects in case you ever need to test the “awaitability” of something you’re passed.

Coroutines example

As a quick example of await in action consider the script below which is used to ping several hosts in parallel to determine whether they’re alive. This example is quite contrived, but it illustrates the new syntax — it’s also an example of how to use the asyncio subprocess support.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
import asyncio
import os
import sys


PING_PATH = "/sbin/ping"

async def ping(server, results):
    with open(os.devnull, "w") as fd:
        # -c1 -> perform a single ping request only
        # -t3 -> timeout of three seconds on response
        # -q  -> generate less output
        proc = await asyncio.create_subprocess_exec(
                PING_PATH, '-c1', '-q', '-t3', server, stdout=fd)

        # Wait for the ping process to exit and check exit code
        returncode = await proc.wait()
        results[server] = not bool(returncode)


async def progress_ticker(results, num_servers):
    while len(results) < num_servers:
        waiting = num_servers - len(results)
        msg = "Waiting for {0} response(s)".format(waiting)
        sys.stderr.write(msg)
        sys.stderr.flush()
        await asyncio.sleep(0.5)
        sys.stderr.write("\r" + " "*len(msg) + "\r")


def main(argv):
    results = {}
    tasks = [ping(server, results) for server in argv[1:]]
    tasks.append(progress_ticker(results, len(tasks)))
    loop = asyncio.get_event_loop()
    loop.run_until_complete(asyncio.wait(tasks))
    loop.close()
    for server, pingable in sorted(results.items()):
        status = "alive" if pingable else "dead"
        print("{0} is {1}".format(server, status))


if __name__ == "__main__":
sys.exit(main(sys.argv))

One point that’s worth noting is that since we’re using coroutines as opposed to threads to achieve concurrency within the script1, we can safely access the results dictionary without any form of locking and be confident that only one coroutine will be accessing it at any one time.

Asynchronous context manager and iterators

As well as the simple await demonstrated above there’s also a new syntax for allowing context managers to be used in coroutines.

The issue with a standard context manager is that the __enter__() and __exit__() methods could take some time or perform blocking operations - how then can a coroutine use them whilst still yielding to the event loop during these operations?

The answer is support for asynchronous context managers. These work in a vary similar manner but provide two new methods __aenter__() and __aexit__() — these are called instead of the regular versions when the caller invokes async with instead of the plain with statement. In both cases they are expected to return an awaitable object that does the actual work.

These are a natural extension to the syntax already described and allow coroutines to make use of any constructions which may perform blocking IO in their enter/exit routines — this could be database connections, distributed locks, socket connections, etc.

Another natural extension are asynchronous iterators. In this case objects that wish to be iterable implement an __aiter__() method which returns an asynchronous iterator which implements an __anext__() method. These two are directly analogous to __iter__() and __next__() for standard iterators, the difference being that __anext__() must return an awaitable object to obtain the value instead of the value directly.

Note that in Python 3.5.x prior to 3.5.2 the __aiter__() method was also expected to return an awaitable, but this changed in 3.5.2 so that it should return the iterator object directly. This makes it a little fiddly to write compatible code because earlier versions still expect an awaitable, but I strongly recommend writing code which caters for the later versions — the Python documentation has a workaround if necessary.

To wrap up this section let’s see an example of async for — with apologies in advance to anyone who cares even the slightest bit about the correctness of HTTP implementations I present a HTTP version of the cat utility.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
import asyncio
import os
import sys
import urllib.parse


class HTTPCat:

    def __init__(self, urls):
        self.urls = urls
        self.url_reader = None

    class URLReader:

        def __init__(self, url):
            self.parsed_url = urllib.parse.urlparse(url)

        async def connect(self):
            port = 443 if self.parsed_url.scheme == "https" else 80
            connect = asyncio.open_connection(
                    self.parsed_url.netloc, port, ssl=(port==443))
            self.reader, writer = await connect
            query = ('GET {path} HTTP/1.0\r\n'
                     'Host: {host}\r\n'
                     '\r\n').format(path=self.parsed_url.path, 
                                    host=self.parsed_url.netloc)
            writer.write(query.encode('latin-1'))
            while True:
                line = await self.reader.readline()
                if not line.strip():
                    break

        async def readline(self):
            line = await self.reader.readline()
            return line.decode('latin1')


    def __aiter__(self):
        return self

    async def __anext__(self):
        while True:
            if self.url_reader is None:
                if not self.urls:
                    raise StopAsyncIteration
                self.url_reader = self.URLReader(self.urls.pop(0))
                await self.url_reader.connect()
            line = await self.url_reader.readline()
            if line:
                return line
            self.url_reader = None


async def http_cat(urls):
    async for line in HTTPCat(urls):
        print("Line: {0}".format(line.rstrip()))


def main(argv):
    loop = asyncio.get_event_loop()
    loop.run_until_complete(http_cat(argv[1:]))
    loop.close()

if __name__ == "__main__":
sys.exit(main(sys.argv))

This is a heavily over-simplified example with many shortcomings (e.g. it doesn’t even support redirections or chunked encoding) but it shows how the __aiter__() and __anext__() methods can be used to wrap up operations which may block for significant periods.

One nice property of this construction is that lines of output will flow down as soon as they arrive from the socket — many HTTP clients seem to want to block until the whole document is retrieved and return it as a string. This is terribly inconvenient if you’re fetching a file of many GB.

Coroutines make streaming the document back in chunks a much more natural affair, however, and I really like the ease of use for the client. Of course, in reality you’d use a library like aiohttp to avoid messing around with HTTP yourself.

Conclusions

That’s the end of this sequence of articles and we’re brought about bang up to date. Overall I really like the fact that the Python developers have focused on making coroutines a proper first-class concept within the language. The implementation is somewhat different to other languages, which often seem to try to hide the coroutines themselves and offer only futures as the language interface, but I do like knowing when my context switches are constrained to be — especially if I’m relying on this mechanism to avoid locking that would otherwise be required.

The syntax is nice and the paradigm is pleasant to work with — but are there any downsides? Well, because the implementation is based on generators under the hood I do have my concerns around raw performance. One of the benefits of asynchonrous IO should really be the performance boost and scalability vs. threads for dominantly IO-bound applications — while the scalability is probably there, I’m a little unconvinced about the performance for real-world cases.

I hunted around for some proper benchmarks but they see few and far between. There’s this page which has a useful collection of links, although it hasn’t been updated for almost a year — I guess things are unlikely to have moved on significantly in that time. From looking over these results it’s clear that asyncio and aiohttp aren’t the cutting edge of performance, but then again they’re not terrible either.

When all’s said and done, if performance is the all-consuming overriding concern then you’re unlikely to be using Python anyway. If it’s important enough to warrant an impact on readability then you might want to at least investigate threads or gevent before making a decision. But if you’ve got what I would regard as a pretty typical set of concerns, where your readablity and maintainability are the top priority, even though you don’t want performance to suffer too much, then take a serious look at coroutines — with a bit of practice I think you might learn to love them.

Or maybe at least dislike them less than the other options.


  1. I’m ignoring the fact that we’re also using subprocesses for concurrency in this example since it’s just an implementation detail of this particular case and not relevant to the point of safe access to data structures within the script. 

Wed 13 Jul 2016 at 07:00PM by Andy Pearce in Software tagged with python and coroutines  |  See comments

☑ The State of Python Coroutines: asyncio - Callbacks vs. Coroutines

This is part 3 of the “State of Python Coroutines” series which started with The State of Python Coroutines: yield from.

I recently spotted that Python 3.5 has added yet more features to make coroutines more straightforward to implement and use. Since I’m well behind the curve I thought I’d bring myself back up to date over a series of blog posts, each going over some functionality added in successive Python versions — this one covers more of the asyncio module that was added in Python 3.4.

In the preceding post in this series I introduced the asyncio module and its utility as an event loop for coroutines. However, this isn’t the only use of the module — its primary purpose is to act as an event loop for various forms of I/O such as network sockets and pipes to child processes. In this post, then, I’d like to compare the two main approaches to doing this: using callbacks and using coroutines.

A brief digression: handling multiple connections

Anyone that’s done a decent amount of non-blocking I/O can probably skim or skip this section — for anyone who’s not come across this problem in their coding experience, this might be useful.

There are quite a few occasions where you end up needing to handle multiple I/O streams simultaneously. An obvious one is something like a webserver, where you want to handle multiple network connections concurrently. There are other examples, though — one thing that crops up quite often for me is managing multiple child processes, where I want to stream output from them as soon as it’s generated. Another possibility is where you’re making multiple HTTP requests that you want to be fetched in parallel.

In all these cases you want your application to respond immediately to input received on any stream, but at the same time it’s clear you need to block and wait for input — endlessly looping and polling each stream would be a massive waste of system resources. Typically there are two main approaches to this: threads and non-blocking I/O1.

These days threads seem to be the more popular solution — each I/O stream has a new thread allocated for it and the stack of this thread encapsulates its complete state. This makes it easy for programmers who aren’t used to dealing with event loops — they can continue to write simple sequential code that uses standard blocking I/O calls to yield as required. It has some downsides, however — cooperating with other threads requires the overhead of synchronisation and if the turnover of connections is high (consider, say, a busy DNS server) then it’s slightly wasteful to be continually creating and destroying thread stacks. If you want to solve the C10k problem, for example, I think you’d struggle to do it using a thread per connection.

The other alternative is to use a single thread and have it wait for activity on any stream, then process that input and go back to sleep again until another stream is ready. This is typically simpler in some ways — for example, you don’t need any locking between connections because you’re only processing one at any given time. It’s also perfectly performant in cases where you expect to be primarily IO-bound (i.e. handling connections won’t require significant CPU time) — indeed, depending on how the data structures associated with your connections are allocated this approach could improve performance by avoiding false sharing issues.

The downside to this method is that it’s a rather less intuitive for many programmers. In general you’d like to write some straight-line code to handle a single connection, then have some magical means to extend that to multiple connections in parallel — that’s the lure of threading. But there is a way we can achieve, to some extent, the best of both worlds (spoiler alert: it’s coroutines).

The mainstays for implementing non-blocking I/O loops in the Unix world have long been select(), introduced by BSD in 1983, and the slightly later poll(), added to System V in 1986. There are some minor differences but in both cases the model is very similar:

  • Register a list of file descriptors to watch for activity.
  • Call the function to wait for activity on any of them.
  • Examine the returned value to discover which descriptors are active and process them.
  • Loop around to the beginning and wait again.

This is often known as the event loop — it’s a loop, it handles events. Implementing an event loop is quite straightforward, but the downside is that the programmer essentially has to find their own way to maintain the state associated with each connection. This often isn’t too tricky, but sometimes when the connection handling is very context-dependent it can make the code rather hard to follow. If often feels like scrabbling to implement some half-arsed version of closures and it would preferable to let language designers worry about that sort of thing.

The rest of this article will focus on how we can use asyncio to stop worrying so much about some of these details and write more natural code whilst still getting the benefits of the non-blocking I/O approach.

asyncio with callbacks

One problem with using the likes of select() is that it can encourage you to drive all your coding from one big loop. Without a bit of work, this tends to run counter to the design principle of separating concerns, so we’d like to move as much as possible out of this big loop. Ideally we’d also like to abstract it, implement in a library somewhere and get benefits of reusing well-tested code. This is partiularly important for event loops where the potential for serious issues (such as getting into a busy loop) is rather higher than in a lot of areas of code.

The most common way to hook into a generic event look is with callbacks. The application registers callback functions which are to be invoked when particular events occur, and then the application jumps into a wait function whose purpose is to simply loop until there are events and invoke the appropriate callbacks.

It’s unsurprising, then, that asyncio is designed to support the callback approach. To illustrate this I’ve turned to my usual example of a chat server — this is a really simple daemon that waits for socket connections (e.g. using netcat or telnet) then prompts for a username and allows connected users to talk to each other.

This implementation is, of course, exceedingly basic — it’s meant to be an example, not a fully-featured application. Here’s the code, I’ll touch on the highlights afterwards.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
import asyncio
import sys

class ChatServer:

    class ChatProtocol(asyncio.Protocol):

        def __init__(self, chat_server):
            self.chat_server = chat_server
            self.username = None
            self.buffer = ""
            self.transport = None

        def connection_made(self, transport):
            # Callback: when connection is established, pass in transport.
            self.transport = transport
            welcome = "Welcome to " + self.chat_server.server_name
            self.send_msg(welcome + "\nUsername: ")

        def data_received(self, data):
            # Callback: whenever data is received - not necessarily buffered.
            data = data.decode("utf-8")
            self.buffer += data
            self.handle_lines()

        def connection_lost(self, exc):
            # Callback: client disconnected.
            if self.username is not None:
                self.chat_server.remove_user(self.username)

        def send_msg(self, msg):
            self.transport.write(msg.encode("utf-8"))

        def handle_lines(self):
            while "\n" in self.buffer:
                line, self.buffer = self.buffer.split("\n", 1)
                if self.username is None:
                    if self.chat_server.add_user(line, self.transport):
                        self.username = line
                    else:
                        self.send_msg("Sorry, that name is taken\nUsername: ")
                else:
                    self.chat_server.user_message(self.username, line)


    def __init__(self, server_name, port, loop):
        self.server_name = server_name
        self.connections = {}
        self.server = loop.create_server(
                lambda: self.ChatProtocol(self),
                host="", port=port)

    def broadcast(self, message):
        for transport in self.connections.values():
            transport.write((message + "\n").encode("utf-8"))

    def add_user(self, username, transport):
        if username in self.connections:
            return False
        self.connections[username] = transport
        self.broadcast("User " + username + " joined the room")
        return True

    def remove_user(self, username):
        del self.connections[username]
        self.broadcast("User " + username + " left the room")

    def get_users(self):
        return self.connections.keys()

    def user_message(self, username, msg):
        self.broadcast(username + ": " + msg)


def main(argv):

    loop = asyncio.get_event_loop()
    chat_server = ChatServer("Test Server", 4455, loop)
    loop.run_until_complete(chat_server.server)
    try:
        loop.run_forever()
    finally:
        loop.close()


if __name__ == "__main__":
sys.exit(main(sys.argv))

The ChatServer class provides the main functionality of the application, tracking the users that are connected and providing methods to send messages. The interaction with asycio, however, is provided by the nested ChatProtocol class. To explain what this is doing, I’ll summarise a little terminology.

The asyncio module splits IO handling into two areas of responsibility — transports take care of getting raw bytes from one place to another and protocols are responsible for interpreting those bytes into some more meaningful form. In the case of a HTTP request, for example, the transport would read and write from the TCP socket and the protocol would marshal up the request and parse the response to exract the headers and body.

This is something asyncio took from the Twisted networking framework and it’s one of the aspects I really appreciate. All too many HTTP client libraries, for example, jumble up the transport and protocol handling into one big mess such that changing one aspect but still making use of the rest is far too difficult.

The transports that asyncio provides cover TCP, UDP, SSL and pipes to a subprocess, which means that most people won’t need to roll their own. The interesting part, then, is asycio.Protocol and that’s what ChatProtocol implements in the example above.

The first thing that happens is that the main() function instantiates the event loop — this occurs before anything else as it’s required for all the other operations. We then create a ChatServer instance whose constructor calls create_server() on the event loop. This opens a listening TCP socket on the specified port2 and takes a protocol factory as a parameter. Every time there is a connection on the listening socket, the factory will be used to manufacture a protocol instance to handle it.

The main loop then calls run_until_complete() passing the server that was returned by create_server() — this will block until the listening socket is fully open and ready to accept connections. This probably isn’t really required because the next thing it does is then call run_forever() which causes the event loop to process IO endlessly until explicitly terminated.

The meat of the application is then how ChatProtocol is implemented. This implements several callback methods which are invoked by the asyncio framework in response to different events:

  • A ChatProtocol instance is constructed in response to an incoming connection on the listening socket. No parameters are passed by asyncio — because the protocol needs an instance to the ChatServer instance this is passed via a closure by the lambda in the create_server() call.
  • Once the connection is ready, the connection_made() method is invoked which passes the transport that asyncio has allocated for the connection. This allows the protocol to store a reference to it for future writes, and also to trigger any actions required on a new connection — in this example, prompting the user for a username.
  • As data is received on the socket, data_received() is invoked to pass this to the protocol. In our example we only want line-oriented data (we don’t want to send a message to the chat room until the user presses return) so we buffer up data in a string and then process any complete lines found in it. Note that we also should take care of character encoding here — in our simplistic example we blindly assume UTF-8.
  • When we want to send data back to the user we invoke the write() method of the transport. Again, the transport expects raw bytes so we handle encoding to UTF-8 ourselves.
  • Finally, when the user terminates their connection then our connection_lost() method is invoked — in our example we use this to remove the user from the chatroom. Note that this is subtly different to the eof_received() callback which represents TCP half-close (i.e. the remote end called shutdown() with SHUT_WR) — this is important if you want to support protocols that indicate the end of a request in this manner.

That’s about all there is to it — with this in mind, the rest of example should be quite straightforward to follow. The only other aspect to mention is that once the loop has been terminated, we go ahead and call its close() method — this clears out any queued data, closes listening sockets, etc.

asyncio with coroutines

Since we’ve seen how to implement the chat server with callbacks, I think it’s high time we got back to the theme of this post and now compare that with an implementation of the same server with coroutines. In usual fashion, let’s jump in and look at the code first:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
import asyncio
import sys

class ChatServer:

    def __init__(self, server_name, port, loop):
        self.server_name = server_name
        self.connections = {}
        self.server = loop.run_until_complete(
                asyncio.start_server(
                    self.accept_connection, "", port, loop=loop))

    def broadcast(self, message):
        for reader, writer in self.connections.values():
            writer.write((message + "\n").encode("utf-8"))

    @asyncio.coroutine
    def prompt_username(self, reader, writer):
        while True:
            writer.write("Enter username: ".encode("utf-8"))
            data = (yield from reader.readline()).decode("utf-8")
            if not data:
                return None
            username = data.strip()
            if username and username not in self.connections:
                self.connections[username] = (reader, writer)
                return username
            writer.write("Sorry, that username is taken.\n".encode("utf-8"))

    @asyncio.coroutine
    def handle_connection(self, username, reader):
        while True:
            data = (yield from reader.readline()).decode("utf-8")
            if not data:
                del self.connections[username]
                return None
            self.broadcast(username + ": " + data.strip())

    @asyncio.coroutine
    def accept_connection(self, reader, writer):
        writer.write(("Welcome to " + self.server_name + "\n").encode("utf-8"))
        username = (yield from self.prompt_username(reader, writer))
        if username is not None:
            self.broadcast("User %r has joined the room" % (username,))
            yield from self.handle_connection(username, reader)
            self.broadcast("User %r has left the room" % (username,))
        yield from writer.drain()


def main(argv):

    loop = asyncio.get_event_loop()
    server = ChatServer("Test Server", 4455, loop)
    try:
        loop.run_forever()
    finally:
        loop.close()


if __name__ == "__main__":
sys.exit(main(sys.argv))

As you can see, this version is written in quite a different style to the callback variant. This is because it’s using the streams API which is essentially a set of wrappers around the callbacks version which adapts them for use with a coroutines.

To use this API we call start_server() instead of create_server() — this wrapper changes the way the supplied callback is invoked and instead passes it two streams: StreamReader and StreamWriter instances. These represent the input and output sides of the socket, but importantly they’re also coroutines so that we can delegate to them with yield from.

On the subject of coroutines, you’ll notice that some of the methods have an @asyncio.coroutine decorator — this serves a practical function in Python 3.5 in that it enables you to delegate to the new style of coroutine that it defines. Pre-3.5 it’s therefore useful for future compatibility, but also serves as documentation that this method is being treated as a coroutine. You should always use it to decorate your coroutines, but this isn’t enforced anywhere.

Back to the code. Our accept_connection() method is the callback that we provided to the start_server() method and the lifetime of this method call is the same as the lifetime of the connection. We could implement the handling of a connection in a strictly linear fashion within this method — such is the flexibility of coroutines — but of course being good little software engineers we like to break things out into smaller functions.

In this case I’ve chosen to use a separate coroutine to handle prompting the user for their username, so accept_connection() delegates to prompt_username() with this line:

username = (yield from self.prompt_username(reader, writer))

Once delegated, this coroutine takes control for as long as it takes to obtain a unique username and then returns this value to the caller. It also handles storing the username and the writer in the connections member of the class — this is used by the broadcast() method to send messages to all users in the room.

The handle_connection() method is also implemented in quite a straightforward fashion, reading input and broadcasting it until it detects that the connection has been closed by an empty read. At this point it removes the user from the connections dictionary and returns control to accept_connection(). We finally call writer.drain() to send any last buffered output — this is rather pointless if the user’s connection was cut, but could still serve a purpose if they only half-closed or if the server is shutting down instead. After this we simply return and everything is cleaned up for us.

How does this version compare, then? It’s a little shorter for one thing — OK, that’s a little facile, what else? We’ve managed to lose the nested class, which seems to simplify the job somewhat — there’s less confusion about the division of responsibilities. We also don’t need to worry so much about where we store things — there’s no transport that we have to squirrel away somewhere while we wait for further callbacks. The reader and writer streams as just passed naturally through the callchain in an intuitive manner. Finally, we don’t have to engage in any messy buffering of data to obtain line-oriented input — the reader stream handles all that for us.

Conclusions

That about wraps it up for this post. Hopefully it’s been an interesting comparison — I know that I certainly feel like I understand the various layers of asyncio a little better having gone through this exercise.

It takes a bit of a shift in one’s thinking to use coroutine approach, and I think it’s helpful to have a bit of a handle on both mechanisms to better understand what’s going on under the hood, but overall the more I use the coroutine style for IO the more I like it. It feels like a good compromise between the intuitive straight-line approach of the thread-per-connection approach and the lock-free simplicity of non-blocking IO with callbacks.

In the next post I’m going to look at the new syntax for coroutines introduced in Python 3.5, which was the inspiration for writing this series of posts in the first place.


  1. Some people use the term asynchronous IO for what I’m discussing here, which is certainly the more general term, but I prefer to avoid it due to risk of confusion with the POSIX asynchronous IO interface. 

  2. In this example we use a hard-coded port of 4455 for simplicity. 

Tue 05 Jul 2016 at 07:45AM by Andy Pearce in Software tagged with python and coroutines  |  See comments

☑ The State of Python Coroutines: Introducing asyncio

This is part 2 of the “State of Python Coroutines” series which started with The State of Python Coroutines: yield from.

I recently spotted that Python 3.5 has added yet more features to make coroutines more straightforward to implement and use. Since I’m well behind the curve I thought I’d bring myself back up to date over a series of blog posts, each going over some functionality added in successive Python versions — this one covers parts of the asyncio module that was added in Python 3.4.

In the previous post I discussed the state of coroutines in Python 2.x and then the yield from enhancement added in Python 3.3. Since that release there’s been a succession of improvements for coroutines and in this post I’m going to discuss those that were added as part of the asyncio module.

It’s a pretty large module and covers quite a wide variety of functionality, so covering all that with in-depth discussion and examples is outside the scope of this series of articles. I’ll try to touch on the finer points, however — in this article I’ll discuss the elements that are relevant to coroutines directly and then in the following post I’ll talk about the IO aspects.

History of asyncio

Python 2 programmers may recall the venerable asyncore module, which was added way back in the prehistory of Python 1.5.2. Its purpose was to assist in writing endpoints that handle IO from sources such as sockets asynchronously. To create clients you derive your own class from asyncore.dispatcher and override methods to handle events.

This was a helpful module for basic use-cases but it wasn’t particularly flexible if what you wanted didn’t quite match its structure. Generally I found I just ended up rolling my own polling loop based on things from the select module as I needed them (although if I were using Python 3.4 or above then I’d prefer the selectors module).

If you’re wondering why talk of an old asynchronous IO module is relevant to a series on coroutines, bear with me.

The limitations of asyncore were well understood and several third party libraries sprang up as alternatives, one of the most popular being Twisted. However, it was always a little annoying that such a common use-case wasn’t well catered for within the standard library.

Back in 2011 PEP 3153 was created to address this deficiency. It didn’t really have a concrete proposal, however, it just defined the requirements — Guido addressed this in 2012 with PEP 3156 and the fledgling asyncio library was born.

The library went through some iterations under the codename Tulip and a couple of years later it was included in the standard library of Python 3.4. This was on a provisional basis — this means that it’s there, it’s not going away, but the core developers reserve the right to make incompatible changes prior to it being finalised.

OK, still not seeing the link with coroutines? Well, as well as handling IO asynchronously, asyncio also has a handy event loop for scheduling coroutines. This is because the entire library is designed for use in two different ways depending on your preferences — either a more traditional callback-based scheme, where callbacks are invoked on events; or with a set of coroutines which can each block until there’s IO activity for them to process. Even if you don’t need to do IO, the coroutine scheduler is a useful piece that you don’t need to build yourself.

asyncio as a scheduler

At this point it would be helpful to consider a quick example of what asyncio can do on the scheduling front without worrying too much about IO — we’ll cover that in the next post.

In the example below, therefore, I’ve implemented something like logrotate — mine is extremely simple1 and doesn’t run off a configuration file, of course, because it’s just for demonstration purposes.

First here’s the code — see if you can work out what it does, then I’ll explain the finer points below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
import asyncio
import datetime
import errno
import os
import sys

def rotate_file(path, n_versions):
    """Create .1 .2 .3 etc. copies of the specified file."""

    if not os.path.exists(path):
        return
    for i in range(n_versions, 1, -1):
        old_path = "{0}.{1}".format(path, i - 1)
        if os.path.exists(old_path):
            os.rename(old_path, "{0}.{1}".format(path, i))
    os.rename(path, path + ".1")


@asyncio.coroutine
def rotate_by_interval(path, keep_versions, rotate_secs):
    """Rotate file every N seconds."""

    while True:
        yield from asyncio.sleep(rotate_secs)
        rotate_file(path, keep_versions)


@asyncio.coroutine
def rotate_daily(path, keep_versions):
    """Rotate file every midnight."""

    while True:
        now = datetime.datetime.now()
        last_midnight = now.replace(hour=0, minute=0, second=0)
        next_midnight = last_midnight + datetime.timedelta(1)
        yield from asyncio.sleep((next_midnight - now).total_seconds())
        rotate_file(path, keep_versions)


@asyncio.coroutine
def rotate_by_size(path, keep_versions, max_size, check_interval_secs):
    """Rotate file when it exceeds N bytes checking every M seconds."""

    while True:
        yield from asyncio.sleep(check_interval_secs)
        try:
            file_size = os.stat(path).st_size
            if file_size > max_size:
                rotate_file(path, keep_versions)
        except OSError as exc:
            if exc.errno != errno.ENOENT:
                raise


def main(argv):

    loop = asyncio.get_event_loop()
    # Would normally read this from a configuration file.
    rotate1 = loop.create_task(rotate_by_interval("/tmp/file1", 3, 30))
    rotate2 = loop.create_task(rotate_by_interval("/tmp/file2", 5, 20))
    rotate3 = loop.create_task(rotate_by_size("/tmp/file3", 3, 1024, 60))
    rotate4 = loop.create_task(rotate_daily("/tmp/file4", 5))
    loop.run_forever()


if __name__ == "__main__":
    sys.exit(main(sys.argv))

Each file rotation policy that I’ve implemented is its own coroutine. Each one operates independently of the others and the underlying rotate_file() function is just to refactor out the common task of actually rotating the files. In this case they all delegate their waiting to the asyncio.sleep() function as a convenience, but it would be equally possible to write a coroutine which does something more clever, like hook into inotify, for example.

You can see that main() just creates a bunch of tasks and plugs them into an event loop, then asyncio takes care of the scheduling. This script is designed to run until terminated so it uses the simple run_forever() method of the loop, but there are also methods to run until a particular coroutine completes or just wait for one or more specific futures.

Under the hood the @asyncio.coroutine decorator marks the function as a coroutine such that asyncio.iscoroutinefunction() returns True — this may be required for disambiguation in parts of asyncio where the code needs to handle coroutines differently from regular callback functions. The create_task() call then wraps the coroutine instance in a Task class — Task is a subclass of Future and this is where the coroutine and callback worlds meet.

An asyncio.Future represents the future result of an asynchronous process. Completion callbacks can be registered with it using the add_done_callback(). When the asynchronous result is ready then it’s passed to the Future with the set_result() method — at this point any registered completion callbacks are invoked. It’s easy to see, then, how the Task class is a simple wrapper which waits for the result of its wrapped coroutine to be ready and passes it to the parent Future class for invocation of callbacks. In this way, the coroutine and callback worlds can coexist quite happily — in fact in many ways the coroutine interface is a layer implemented on top of the callbacks. It’s a pretty crucial layer in making the whole thing cleaner and more manageable for the programmer, however.

The part that links it all together is the event loop, which asyncio just gives you for free. There are a few details I’ve glossed over, however, since it’s not too important for a basic understanding. One thing to be aware of is that there are currently two event loop implementations — most people will be using SelectorEventLoop, but on Windows there’s also the ProactorEventLoop which uses different underlying primitives and has different tradeoffs.

This scheduling may all seem simplistic, and it’s true that in this example asyncio isn’t doing anything hugely difficult. But building your own event loop isn’t quite as trivial as it sounds — there are quite a few gotchas that can trip you up and leave your code locked up or sleeping forever. This is particularly acute when you introduce IO into the equation, where there are some slightly surprising edge cases that people often miss such as handling sockets which have performed a remote shutdown. Also, this approach is quite modular and manages to produce single-threaded code where different asynchronous operations interoperate with little or no awareness of each other. This can also be achieved with threading, of course, but this way we don’t need locks and we can more or less rule out issues such as race conditions and deadlocks.

That wraps it up for this article. I’ll cover the IO aspects of ascynio in my next post, covering and comparing both the callback and coroutine based approaches to using it. This is particularly important because one area where coroutines really shine (vs threads) is where your application is primarily IO-bound and so there’s no need to explode over multiple cores.


  1. In just one example of many issues, for extra credit2 you might like to consider whay happens to the rotate_daily() implementation when it spans a DST change. 

  2. Where the only credit to which I’m referring are SmugPoints(tm): a currency that sadly only really has any traction inside the privacy of your own skull. 

Thu 16 Jun 2016 at 08:29AM by Andy Pearce in Software tagged with python and coroutines  |  See comments

☑ The State of Python Coroutines: yield from

This is part 1 of the “State of Python Coroutines” series.

I recently spotted that Python 3.5 has added yet more features to make coroutines more straightforward to implement and use. Since I’m well behind the curve I thought I’d bring myself back up to date over a series of blog posts, each going over some functionality added in successive Python versions — this one covers the facilities up to and including the yield from syntax added in Python 3.3.

I’ve always thought that coroutines are an underused paradigm.

Multithreading is great for easily expanding single threaded approaches to make better use of modern hardware with minimal changes; multiprocess is great for enforcement of interfaces and also extending across multiple machines. In both cases, however, the premise is on performance at the expense of simplicity.

To my mind, coroutines offer the flip side of the coin — perhaps performance isn’t critical, but your approach is just more naturally expressed as a series of cooperative processes. You don’t want to wade through a sea of memory barriers to implement such things, you just want to divide up your responsibilities and let the data flow through.

In this short series of posts I’m going to explore what facilities we have available for implementing coroutines in Python 3, and in the process catch myself up developments in that area.

Coroutines in Python 2

Before looking at Python 3 it’s worth having a quick refresher on the options for implementing coroutines in Python 2, not least because many programmers will still be constrained to use this version in many commercial environments.

The genesis of coroutines was when generators were added to the language in Python 2.2 — these are essentially lazily-evaluated lists. One defines what looks like a normal function but instead of a return statement yield is used. This has the effect of emitting a value from your generator but — and this is crucial — it also suspends execution of your generator in place and returns the flow of execution back to the calling code. This continues until the caller requests the next value from the generator at which point it resumes execution just after the yield statement.

For a real-world example consider the following implementation of the sieve of Eratosthenes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# This is Python 2 code, since this section discusses Python 2.
# For Python 3 replace range(...) with list(range(...)) and
# replace xrange(...) with range(...).
def primes(limit):
    "Yield all primes <= limit."

    sqrt_limit = int(round(limit**0.5))
    limit += 1
    sieve = range(limit)
    sieve[1] = 0
    for i in xrange(2, sqrt_limit + 1):
        if sieve[i]:
            sieve[i*i:limit:i] = [0] * len(xrange(i*i, limit, i))
            yield i
    for i in xrange(sqrt_limit + 1, limit):
        if sieve[i]:
            yield i

Generators are, of course, fantasically useful on their own. In terms of coroutines, however, they’re only half the story — they can yield outputs but they can only take their initial inputs, they can’t be updated during their execution.

To address this Python 2.5 extended generators in several ways which allow them to be turned into general purpose generators. A quick summary of these enhancements is:

  • yield, which was previously a statement, was redefined to be an expression.
  • Added a send() method to inject inputs during execution.
  • Added a throw() method to inject exceptions.
  • Added a close() method to allow the caller to terminate a generator early.

There are a few other tweaks, but those are the main points. The net result of these changes is that one could now write a generator where new values can be injected, via the send() method, and these are returned within the generator as the value of the yield expression.

As a simple example of this, consider the code below which implements a coroutine that accepts a number as a parameter and returns back the average of all the numbers up to that point.

1
2
3
4
5
6
7
import itertools

def averager():
    sum = float((yield))
    counter = itertools.count(start=1)
    while True:
        sum += (yield sum / next(counter))

Python 3.3 adds “yield from”

The conversion of generators to true coroutines was the final development in this story in Python 2 and development of the language long ago moved to Python 3. In this vein there was another advancement of coroutines added in Python 3.3 which was the yield from construction.

This stemmed from the observation that it was quite cumbersome to refactor generators into several smaller units. The complication is that a generator can only yield to its immediate caller — if you want to split generators up for reasons of code reuse and modularity, the calling generator would have to manually iterate the sub-generator and re-yield all the results. This is tedious and inefficient.

The solution was to add a yield from statement to delegate control entirely to another generator. The subgenerator is run to completion, with results being passed directly to the original caller without involvement from the calling generator. In the case of coroutines, sent values and thrown exceptions are also propogated directly to the currently executing subgenerator.

At its simplest this allows a more natural way to express solutions where generators are delegated. For a really simple example, compare these two sample1 implementations of itertools.chain():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Implementation in pre-3.3 Python
def chain(*generators):
    for generator in generators:
        for item in generator:
            yield item

# Implementation in post-3.3 Python
def chain(*generators):
    for generator in generators:
        yield from generator

Right now, of course, this looks somewhat handy but a fairly minor improvement. But when you consider general coroutines, it becomes a great mechanism for transferring control. I think of them a bit like a state machine where each state can have its own coroutine, so the concerns are kept separate, and where the whole thing just flows data through only as fast as required by the caller.

I’ve illustrated this below by writing a fairly simple parser for expressions in Polish Notation — this is just like Reverse Polish Notation only backwards. Or perhaps I mean forwards. Well, whichever way round it is, it really lends itself to simple parsing because the operators precede their arguments which keeps the state machine nice and simple. As long as the arity of the operators is fixed, no brackets are required for an unambiguous representation.

First let’s see the code, then I’ll discuss its operation below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
import math
import operator

# Subgenerator for unary operators.
def parse_unary_operator(op):
    return op((yield from parse_argument((yield))))

# Subgenerator for binary operators.
def parse_binary_operator(op):
    values = []
    for i in (1, 2):
        values.append((yield from parse_argument((yield))))
    return op(*values)

OPERATORS = {
    'sqrt': (parse_unary_operator, math.sqrt),
    '~': (parse_unary_operator, operator.invert),
    '+': (parse_binary_operator, operator.add),
    '-': (parse_binary_operator, operator.sub),
    '*': (parse_binary_operator, operator.mul),
    '/': (parse_binary_operator, operator.truediv)
}

# Detect whether argument is an operator or number - for
# operators we delegate to the appropriate subgenerator.
def parse_argument(token):
    subgen, op = OPERATORS.get(token, (None, None))
    if subgen is None:
        return float(token)
    else:
        return (yield from subgen(op))

# Parent generator - send() tokens into this.    
def parse_expression():
    result = None
    while True:
        token = (yield result)
        result = yield from parse_argument(token)

The main entrypoint is the parse_expression() generator. In this case it’s necessary to have a single parent because we want the behaviour of the top-level expressions to be fundamentally different — in this case, we want it to yield the result of the expression, whereas intermediate values are instead consumed internally within the set of generators and not exposed to calling code.

We use the parse_argument() generator to calculate the result of an expression and return it — it can use a return value since it’s called as a subgenerator of parse_expression() (and others). This determines whether each token is an operator or numeric literal — in the latter case it just returns the literal as a float. In the former case it delegates to a subgenerator based on the operator type — here I just have unary and binary operators as simple illustrative cases. Note that one could easily implement an operator of variable arity here, however, since the delegate generator makes its own decision of when to relinquish control back to the caller — this is an important property when modularising code.

Hopefully this example is otherwise quite clear — the parse_expression() generator simply loops and yields the values of all the top-level expressions that it encounters. Note that because there’s no filtering of the results by the calling generator (since it’s just delegating) then it will yield lots of None results as it consumes inputs until the result of a top-level expression can be yielded — it’ll be up to the calling code to ignore these. This is just a consequence of the way send() on a generator always yields a value even if there isn’t a meaningful value.

The only other slight wrinkle is that you might see some excessive bracketing around the yield operators — this is typically a good idea. PEP 342 describes the parsing rules, but if you just remember to always bracket the expression then that’s one less thing to worry about.

One thing that’s worth noting is that this particular example is quite wasteful for deeply nested expressions in the same way that recursive functions can be. This is because it constructs two new generators for each nested expression — one for parse_argument() and one for whichever operator-specific subgenerator this delegates to. Whether this is acceptable depends on your use-cases and the extent which you want to trade off the code expressiveness against space and time complexity.

Below is an example of how you might use parse_expression():

1
2
3
4
5
6
7
8
9
def parse_pn_string(expr_str):
    parser = parse_expression()
    next(parser)
    for result in (parser.send(i) for i in expr_str.split()):
        if result is not None:
            yield result

results = parse_pn_string("* 2 9 * + 2 - sqrt 25 1 - 9 6")
print("\n".join(str(i) for i in results))

Here I’ve defined a convenience wrapper generator which accepts the expression as a whitespace-delimited string and strips out the intermediate None values that are yielded. If you run that you should see there’s two top-level expressions which yield the same result.

Coming up

That wraps it up for this post — I hope it’s been a useful summary of where things stand in terms of coroutines as far as Python 3.3. In future posts I’ll discuss the asyncio library that was added in Python 3.4, and the additional async keyword that was added in Python 3.5.


  1. Neither of these are anything to do with the official Python library, of course — they’re just implementations off the top of my head. I chose itertools.chain() purely because it’s very simple. 

Fri 10 Jun 2016 at 07:58AM by Andy Pearce in Software tagged with python and coroutines  |  See comments

☑ Fighting Fonts on Mobile

I recently ran into some odd font sizing issues when viewing my website on my iPhone and discovered a few interesting tidbits about rendering on mobile browsers along the way.

Recently I found the time to finally get around to some tweaks to my website theme that I’d been meaning to make for some time — primarily these were related to syntax highlighting and code snippets.

Handling code snippets in HTML has some surprising subtleties, one of them being the overflow behaviour. As much as you try to wrap everything at a sensible number of columns, there will always be times when you absolutely need to represent a long line and it’s always rather unsatisfactory to have to explain that the line breaks were just added for readability. As a result, the styling needs a graceful fallback for these inevitable cases.

I considered using white-space: pre-wrap, which acts like <pre> except that it wraps text on overflow as well as explicit line breaks. One complication, however, is that I sometimes use line numbering for my longer code snippets:

1
2
3
This isn't actually very long.
So the line numbers are rather pointless.
But you get the idea.

To facilitate easy cut and paste this is actually a HTML table with a single row and the line numbers and file content in their own cells, contained within nested <pre> tags1. This is handy, but it does lead to some odd formatting issues, since there’s nothing explicitly aligning the individual lines in the two elements except for the fact that they’re using a consistent font and line width.

One issue I’ve had in the past, for example, is when I used bold text for some of the syntax highlighting styles — I found that some browsers would adjust the line height when this happened such that the rows no longer quite lined up after that point. I tried various line-height fixes with limited success, but eventually it was just easiest to avoid bold text in code snippets.

Another issue concerns overflows — if you wrap text in the file content as I suggested earlier then you’d need to also somehow arrange a gap (or repeat) in the line numbering or all the line numbers will be off by one for each wrapped line. There’s no practical way to arrange this except by perhaps putting each code row in its own table row and modifying third party code extensively to do that just didn’t appeal for a quick fix.

Instead, therefore, I opted for using overflow: auto, which inserts scrollbars as required, combined with judicious max-width: 100% here and there. I was pleasantly surprised to see this works on any sensible browser2.

However, when I tested the site on my iPhone I discovered a new and puzzling issue: the overflow scrolling was working fine, but for some reason when the text overflowed the font size of the line numbers was oddly reduced and hence no longer aligned with the code.

I realised fairly early on that this was some odd resizing issue due to the content being too large, and hence presumably an attempt to fit more text into place getting confused with some of the CSS rules — but no amount of font-size, width, height or anything else seemed to fix it, even with !important littered all over the place.

This threatened to be one of those annoying issues that can be very tough to track down — but luckily for me I stumbled upon the solution fairly quickly. As it turns out, it’s all about how browsers on mobile devices render pages.

The issue with smartphone browsers is that most web content authors didn’t design their content and/or styles to cope with such a small viewport. All kinds of web pages don’t render correctly — the kind of hacks that are required to get cross-browser compatibility almost certainly don’t help, either. As a result of this the browsers use a bit of a trick to get things looking a little more natural.

What they do is render the page to a larger viewport (e.g. 1000 pixels wide) and then scale the page down to fit on the screen. This allows the styles to render more plausibly on most pages, but it does tend to make the text awfully hard to read. This would entail scrolling the screen left/right to read a full line of text which would be a trifle vexing to say the least.

To get around this issue the browsers inflate the text — they scale the font size up so that it becomes legible once again, whilst still leaving all the pixel-oriented sizes intact.

As it turns out, this was happening with my code snippets — but for some reason a different inflation ratio was applied to the line numbers element than the code. Once I knew that this was the issue it was quite easy to fix3 by adding -webkit-text-size-adjust: 100% to my assorted <pre> elements — apparently this did the trick as it now seems to work. Check out that linked page because it’s got a lot of additional useful details which isn’t Mozilla-specific.

There you are, then — who knew rendering pages on mobile was such a subtle business? Actually, given the usual state of all but the simplest CSS tasks I’m quite surprised it was as straightforward as it was.


  1. I can’t really take either credit or blame for this, it’s part of the functionality provided by the Pygments package that I use for syntax highlighting. The documentation indicates that it’s optional, but I think it’s useful. 

  2. sensible, adj.: Any popular web browser which isn’t IE

  3. Perhaps not fix optimally, but at least fix. Work around. Hack. Whatever you want to call it. 

Thu 08 Oct 2015 at 08:10PM by Andy Pearce in Software tagged with fonts and web  |  See comments

☑ C++ Streaming Pains

C++ streams have their advantages, but in some ways they’re a downright pain.

Some time ago I wrote a post called The Dark Arts of C++ Streams. The name was a reference to the fact that, as a C programmer who later learned C++, I’d always regarded them with some suspicion and disdain as a much more cumbersome alternative to the likes of printf(). Of course, streams are much more suited to C++ where behaviours may be extended and overridden as needed to support user-specified types, and the output system needs to support that — but for the common cases of printing formatted output of strings, integers and floats, printf() just seemed much more convenient.

Since writing that post I’ve become rather more familiar with streams, and even grown fond of some of their quirks — their flexibility is useful for things like debugging and their comparative verbosity has lost much of its sting. There are still some things that bug me about streams, however — mostly a set of related traps that are easy to fall into and could, to my mind at least, have easily been avoided.

The main issue I have rests on the statefulness of the formatting. Let’s say you want to output a floating point value to 3 decimal places, and then another to 5 decimal places. Each time you need to set the precision on the stream and not on the value being streamed out. This is possible with explicit methods on the stream:

std::cout.setf(std::ios::fixed);
std::cout.precision(3);
std::cout << 123.456789 << std::endl;  // 123.456

However, this is rather awkward — one of the big syntactic advantages of streams is the way that they can be chained. Taking the previous example of two floats, you’d like to be able to do something like this:

std::cout << first << " and " << second << std::endl;

Indeed you can do that — but now add some formatting using the method-based approach earlier and it’s absolutely horrible:

std::cout.setf(std::ios::fixed);
std::cout.precision(3);
std::cout << first << " and ";
std::cout.precision(5);
std::cout << second << std::endl;

Of course, as anyone who’s done this sort of formatting will probably know, it is possible to do the whole thing inline:

std::cout << std::fixed
          << std::setprecision(3) << first
          << " and "
          << std::setprecision(5) << second
          << std::endl;

That’s conceptually a lot nicer, but the spanner in the works is the fact that these functions are just a fancy shortcut for calling the methods on the stream itself. In other words the formatting changes they make are not scoped in any way — they persist on the stream until explicitly replaced.

This could lead to all sorts of nasty consequences. Imagine code like this:

// Add a trace comment to the output file in a debug build.
void traceFloat(std::ostream& out, std::string name, double value)
{
#ifdef DEBUG
    out << "# The value of '" << value << "' is "
        << std::fixed << std::setprecision(16) << value
        << std::endl;
#endif /* ifdef DEBUG */
}

// ... Later on in the source file...

void myFunction()
{
    double foo_value = getFooValue();
    // ...
    myFile.precision(3);
    traceFloat(myFile, "foo", foo_value);
    myFile << "foo:" << foo_value << "\n";
    // ...
}

Here we have some routine which is generating some sort of formatted file, but in a debug build (only) it also adds some comments for diagnostics. Except in this particular case, the act of adding the diagnostic actually changes the output format of the file — the use of setprecision() in the traceFloat() function overrides the earlier setting of precision() within myFunction(). This is something that’s just begging to cause those annoying problems that only materialise in a production environment.

This is a fairly contrived example, but this sort of action at a distance is really poor practice. Of course all of this is documented, but an important property of a programming language (and I always include standard libraries in that term) is how easily programmers can avoid or detect both current and potential future mistakes.

All this said, that’s the way it works and it’s unlikely to change in any fundamental manner — so is there a way to use streams safely? Well, as is so often the case in C++ RAII comes to the rescue.

It’s possible to capture the current state of the stream and restore it later, which is probably best practice for anyone using streams for any purpose where the format is critical.

It’s not hard to build a class which uses an RAII approach to save and later restore the current state of the stream:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
class StreamStateGuard
{
  public:
    StreamStateGuard(std::ios_base& stream) : stream_(stream)
    {
        captureStreamState();
    }

    ~StreamStateGuard()
    {
        restoreStreamState();
    }

    void captureStreamState()
    {
        precision_ = stream_.precision();
        width_ = stream_.width();
        flags_ = stream_.flags();
    }

    void restoreStreamState()
    {
        stream_.precision(precision_);
        stream_.width(width_);
        stream_.flags(flags_);
    }

  private:
    std::ios_base& stream_;
    std::streamsize precision_;
    std::streamsize width_;
    std::ios_base::fmtflags flags_;
};

This is for illustrative purposes and I haven’t tested it extensively, so there may be additional state to save I’ve forgotten or exceptions I haven’t handled, but it’s pretty simple. If you’re lucky enough to be able to use Boost in your C++ environment then it already has classes to do all this for you.

Let’s say you’re writing some code to output formatted values in some defined text file format — you could write a function to output a row which saves the state at the start of the function and restores it at the end. However, wouldn’t it be nice if each field could do that itself? That would keep things modular, and it’s something we can do by writing new IO manipulators. I should mention here that this is something I’ve done little of, so there may be more elegant approaches, but the code below seems to work:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
class MyClass
{
  public:
    MyClass(double value1, double value2)
            : value1_(value1), value2_(value2)
    { }

    void writeValues(std::ostream& out);

  private:
    class outputField
    {
      public:
        outputField(double value, int decimal_places, int width)
                : value_(value),
                  decimal_places_(decimal_places),
                  width_(width)
        { }
        std::ostream& operator()(std::ostream& out) const;

      private:
        double value_;
        int decimal_places_;
        int width_;
    };

    friend std::ostream& operator<<(
            std::ostream& out,
            const MyClass::outputField& item);

    double value1_;
    double value2_;
};

std::ostream& operator<<(std::ostream& out, const MyClass::outputField& item)
{
    return item(out);
}

std::ostream& MyClass::outputField::operator()(std::ostream& out) const
{
    StreamStateGuard streamStateGuard(out);

    return out << std::fixed << std::noshowpoint
               << std::setprecision(decimal_places_)
               << std::setw(width_)
               << value_;
}

void MyClass::writeValues(std::ostream& output)
{
    output << "MyClass|"
           << outputField(value1_, 2, 8) << "|"
           << outputField(value2_, 6, 12) << "|\n";
}

The sample MyClass has only two double values to output, but this example could easily be extended to more or less any types.

These sorts of techniques start to illustrate what I’m increasingly learning as the best practices to keep a programmer sane when using C++ IO streams. I still have a soft spot for the old stdio.h functions, but I suppose all of us have to be dragged into the future at some point — at least now I’ve got a few crude tools to help me survive there.

Thu 20 Aug 2015 at 01:55PM by Andy Pearce in Software tagged with c++ and c++-stl  |  See comments

Page 1 of 6   |   Page 2 →   |   Page 6 ⇒