Asyncio basics in python

Python 3.5 brought with it asyncio. An event loop based paradigm previously available as a library but now it is built in as a standard library. There are other async libraries out there, but I am going to focus on the one built in to Python.

Creating an event loop

1
2
3
4
5
import asyncio

loop = asyncio.get_event_loop()
loop.run_until_complete(some_func())

This is the basic example to get you up and running. This is the core of what is happening. Starting an event loop and running some function on top of it. Some frameworks abstract this from you and handle it on the bootstrap of the application layer (aiohttp, for example).

The async function

1
2
async def some_func():
    pass

Just like defining a regular function, except we add the

1
async
keyword. This not only let’s you know the function is asynchrounous, but it also let’s the interpreter know. The interpreter wraps up the function inside a
1
coroutine
and then is handle in various ways. This is the crux of what this post is about.

In our first example we started an event loop and ran our function inside

1
run_until_complete()
That will run your function until all synchrounous and non-synchrounous calls are complete. I like to think of this step as an instantiation of our asynchrounous paradigm.

The real magic -
1
await

Cool. We have our code starting to come together. Let’s add a new async function and rename

1
some_func
to
1
run
since that is all we are going to have it do – run our async logic.

1
2
3
4
5
6
7
8
9
10
import asyncio

async def speak():
    print('Hey!')

async def run():
    await speak()

loop = asyncio.get_event_loop()
loop.run_until_complete(run())

You see how we have

1
await
before we call
1
speak()
? Coroutines are generators in asyncio, so we could handle them like we do generators. However python 3.5 gives us
1
await
as syntactic sugar around
1
yield from
, which should make it feel more like other languages async syntax. (btw, if you are not familiar with Python generators, that’s fine. You don’t need them, but for sure read about them later)

What happens if you leave out

1
await
? There are times, which I will go over, that do not require you to immediately
1
await
your coroutine, but at some point before the execution of your code is complete, you will need to handle your coroutine.

Starting a function and getting back to it later.

The first thing you might need an async pattern for is starting a job in the background, do other tasks in the front and then coming back to your original task.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import asyncio

async def speak():
    print('C')
    await asyncio.sleep(3)
    return 'D'

async def run():
    will_speak = speak()
    print('A')
    print('B')
    print(await will_speak)

loop = asyncio.get_event_loop()
loop.run_until_complete(run())

If you run this you will see the results:

1
2
3
4
A
B
C
D

That’s what we wanted right? Close. If you remember, we wanted to start our async job

1
speak()
and when it was done, return the results. So the first thing that should have been printed to our terminal is
1
C
not
1
A
. While the above is a good pattern if you just want to define your job(s) and then return to them later to actually perform the execution, this does not perform our task while we move on to do other things. To do that you actually need to use
1
asyncio.ensure_future
.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import asyncio

async def speak():
    print('C')
    await asyncio.sleep(3)
    return 'D'

async def run():
    will_speak = asyncio.ensure_future(speak())
    await asyncio.sleep(1)
    print('A')
    print('B')
    print(await will_speak)

loop = asyncio.get_event_loop()
loop.run_until_complete(run())

Note: I added a 1 second sleep after we did

1
ensure_future()
That’s because
1
will_speak
is now running in parallel with the rest of the function, and you cannot guarantee that C will print before A and B, so putting a sleep in there ensures the speak() function can finish it’s task first. This is just for the example and not necessarily necessary (that’s fun to say)

Cool. We have a function that branches off and does it’s own thing while we continue on, and when we need it’s results we will have them.

Order mucking

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import asyncio


async def meow(number):
    print(f'starting {number}')
    await asyncio.sleep(1)
    print(f'stopping {number}')


async def run():
    # notice how we assign each coroutine to a variable
    a = meow(1)
    b = meow(2)
    c = meow(3)
    d = meow(4)
    e = meow(5)

    # then we can decide which order we wish our coroutines to execute
    await c
    await a
    await e
    await b
    await d


loop = asyncio.get_event_loop()
loop.run_until_complete(run())

Let’s start looking at how our coroutines are handled. In the above example, I am showing that you can define your work in any order. Then await for them out of order. Or out of order and then back in order. Or any combination. Great. They all run sequentially though in the order they are awaited. What about if we wanted all those to run at the same time. Well, we could

1
ensure_future
all the things (which is just fine). OR we can show you another way.

Parallel runnings.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import asyncio


async def meow(number):
    print(f'starting {number}')
    await asyncio.sleep(1)
    print(f'stopping {number}')


async def run():
    f = []
    for x in range(1, 6):
        f.append(meow(x))
    await asyncio.wait(f)

loop = asyncio.get_event_loop()
loop.run_until_complete(run())

Here we are creating a list and then adding all of our items to said list. In this case, we are just iterating over a

1
range()
and appending those coroutines to that list. Then with
1
asyncio.wait()
we can wait for all those coroutines to complete. (Don’t forget to
1
await
the
1
asyncio.wait()
, a common mistake.)

One thing you should notice, in Python 3.5, 3.6 and 3.7 they all work out of order. I’m not sure why, or if/when they will resolve this. If order is important, wrap your coroutine in

1
ensure_future
. Free beer/tacos for anyone that can tell me why and can meet me in Salt Lake City.

Example of order keeping parallelism.

1
2
3
4
5
6
async def run():
    # We create an empty list to store our coroutines in
    f = []
    for x in range(1, 6):
        f.append(asyncio.ensure_future(meow(x)))
    await asyncio.wait(f)

Handling async function results

Our previous example handles the parallelism beautifully (albeit out of order), but what if you want to do something with the results of those functions, a la “scatter-gather”?

GATHER!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import asyncio

async def meow(number):
    await asyncio.sleep(1)
    return number

async def run():
    f = []
    for x in range(1, 6):
        f.append(asyncio.ensure_future(meow(x)))
    x = await asyncio.gather(*f)
    print(x)

loop = asyncio.get_event_loop()
loop.run_until_complete(run())

# $>: [1, 2, 3, 4, 5]

That it? What else you got?

A couple things come to mind. Look at this example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import asyncio

async def new_future(n):
    print('future', n)
    await asyncio.sleep(3)
    print('future done', n)
    return n

async def run():
    results = asyncio.ensure_future(new_future(1))
    print(results) # <Task> object returned
    print(await results) # `1` is returned
    print(results) # <Task> is returned again
    print(await results) # `1` is returned again

    results = new_future(2)
    print(results) # coroutine returned
    print(await results) # `2` is returned
    print(results) # coroutine returned again
    print(await results) # RuntimeError: cannopy reuse aalready awaited coroutine


loop = asyncio.get_event_loop()
loop.run_until_complete(run())

1
ensure_future
wraps up our coroutine in a
1
Task
. A task is more roubust than a coroutine. First a task is executed immediately when it is created, and then stores the results inside of it’s object. Then, any time in the future if you call
1
await
on your future, you can retrieve the result, without having to re-execute the item again. A coroutine dies as soon as it is used up, and will raise an exception.

And lastly, just have fun with it. You can do all sorts of crazy stuff with asyncio. Execute things in weird orders, or whatever makes sense for your application.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import asyncio


async def meow(number):
    print(f'starting {number}')
    await asyncio.sleep(1)
    print(f'stopping {number}')


async def run():
    f = []
    for x in range(1, 6):
        f.append(meow(x))

    for x, y in enumerate(f):
        if x % 2 == 0:
            await y
    for x, y in enumerate(f):
        if x % 2 != 0:
            await y

loop = asyncio.get_event_loop()
loop.run_until_complete(run())

And as always.

The default installer that comes with Laravel is great, but unfortunately that means you have to install dependencies of PHP and Laravel/Lumen first, before you can get started working on your application. This is not longer the case, so long as you use Docker. And you should.

All you need to do is:

Laravel

1
docker run -v `pwd`:/var/www alairock/laravel 'laravel new site-name'

Lumen

1
docker run -v `pwd`:/var/www alairock/lumen 'laravel new service-name'

Bonus Round!

These containers aren’t just good for installing laravel apps. You can also use them for development. (Remember, you need to add

1
--host=0.0.0.0
in order to make sure that external connections to the artisan server are allowed)

1
2
$ cd site-name # or service-name if using lumen
$ docker run -v `pwd`:/var/www -p "8080:8000" alairock/lumen 'php artisan serve --host=0.0.0.0'

Just don’t forget to link to your database, nginx, or whatever other containers you are needing to integrate with.

Lessons learned going from PHP to Python.

Classes

While Python does have and use classes, it’s not always required to use them. You can think of files as classes in most cases.

File: cat.py

1
2
def meow():
	print "meow"

File: speak.py

1
2
3
import cat

cat.meow()

Although, if you want to use classes you can. There are some interesting differences to remember.

1
2
3
4
5
6
class Cat: # Class names are StudlyCased
	def purr(self): #functions are typically lowercase, need self  if you aren't passing params
		print "purr"

var kitten = Cat() # To instantiate use parens () "new" is not a thing in python
kitten.purr()

What is

1
self
? Check out this answer here

What about

1
construction
?

PHP:

1
2
3
4
<?
	class Kitten() {
		public function __construct() {}
	}

Python:

1
2
3
class Kitten:
	__init__():
		# construction stuff here

Switches

PHP:

1
2
3
4
5
6
7
8
9
10
11
12
<?
switch ($i) {
	case 0:
		echo "i equals 0";
		break;
	case 1:
		echo "i equals 1";
		break;
	default:
		echo "default";
		break;
}

Python:

1
2
3
4
5
6
7
i = 1
if (i == 0):
	print "One"
elif (i == 1):
	print "Two"
else:
	print "default"

Notice in Python it’s just an if-elseif-else statement. In Python there is no switch statement. A switch is just shorthand for conditionals, so it’s not a huge loss if you really think about it.

Notice

This article is a work in progress. More to come. If you notice any issues or typos, please leave it in the comments and I will happily fix it.

Just upgraded to a 2015 Macbook Pro and one of the primary hardware update is the trackpad. The trackpad is not a physical button, like previous models, but static, and has force tracking technology underneath.

This hasn’t really changed the experience for me day-to-day but I did find a really cool trick with it while playing with a Midi Python library. When you open Logic and open up the virtual keyboard, you can use your keyboard or mouse to activate a note and play a sound. With the new trackpad, it adds another element to this keyboard, aftertouch.

Aftertouch is after the note has been turned on, but before it has been turned off. In midi, this can be used to control things like vibrato, volume, pitch, and any combination of things. Logic has integrated the force sensor of the trackpad into the virtual keyboard, allowing you to incorporate aftertouch activities in the software. Pretty neat, if you ask me.

tldr;
location ^~ /blog {
    proxy_set_header Host $http_host;
    proxy_set_header X-NginX-Proxy true;
    proxy_redirect off;
    proxy_pass http://wordPress-3211562.us-west-2.elb.amazonaws.com/blog;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    server_name_in_redirect off;
    port_in_redirect off;
}

Laravel Cashier lets you have these fancy-shmancy PDF downloads of invoices.

This is awesome, except it seems that in some cases the permissions are not correct... or fall out of correctness. Chmod'ing will fix this.

https://github.com/laravel/cashier/issues/74 (Link no longer working since Taylor took down the issues  *sheds tear*)

cd [app_path]/vendor/laravel/cashier/src/Laravel/Cashier/bin/linux-x86_64
chmod a+x phantomjs

Note: That path above may not necessarily be in linux-x86_64 as it will depend on your system. Be sure to verify which system you are running and update this path accordingly.

I just updated SourceTree, and what did I find? They updated their logo! But the funny thing about it is their striking similarity to the icon I designed in March, 6 months before they released theirs.

 

Their icon in March
sourcetree_icon

My icon designed in March
SourceTree Desktop Icon

Their icon, released yesterday
SourceTreeNewIcon-300x300

 

 

The similarity is striking. But alas, I don't care. It's just cool to see I was on the right path. :)

(Also, as of writing this post, if you did a google search for "sourcetree app icon", my icon is the first or second result)

After doing the Atom Editor Desktop Icon, I kinda felt that the source tree one was pretty out dated as well. I flattened it up.

SourceTree Desktop Icon

 

You can download it HERE.