Asyncio basics in python
Python 3.5 brought with it asyncio. An event loop based paradigm previously available as a library but now it is built in as a standard library. There are other async libraries out there, but I am going to focus on the one built in to Python.
Creating an event loop
1
2
3
4
5
import asyncio
loop = asyncio.get_event_loop()
loop.run_until_complete(some_func())
This is the basic example to get you up and running. This is the core of what is happening. Starting an event loop and running some function on top of it. Some frameworks abstract this from you and handle it on the bootstrap of the application layer (aiohttp, for example).
The async function
1
2
async def some_func():
pass
Just like defining a regular function, except we add the 1
async
1
coroutine
In our first example we started an event loop and ran our function inside 1
run_until_complete()
The real magic - 1
await
1 | await |
Cool. We have our code starting to come together. Let’s add a new async function and rename 1
some_func
1
run
1
2
3
4
5
6
7
8
9
10
import asyncio
async def speak():
print('Hey!')
async def run():
await speak()
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
You see how we have 1
await
1
speak()
1
await
1
yield from
What happens if you leave out 1
await
1
await
Starting a function and getting back to it later.
The first thing you might need an async pattern for is starting a job in the background, do other tasks in the front and then coming back to your original task.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import asyncio
async def speak():
print('C')
await asyncio.sleep(3)
return 'D'
async def run():
will_speak = speak()
print('A')
print('B')
print(await will_speak)
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
If you run this you will see the results:
1
2
3
4
A
B
C
D
That’s what we wanted right? Close. If you remember, we wanted to start our async job 1
speak()
1
C
1
A
1
asyncio.ensure_future
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import asyncio
async def speak():
print('C')
await asyncio.sleep(3)
return 'D'
async def run():
will_speak = asyncio.ensure_future(speak())
await asyncio.sleep(1)
print('A')
print('B')
print(await will_speak)
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
Note: I added a 1 second sleep after we did 1
ensure_future()
1
will_speak
Cool. We have a function that branches off and does it’s own thing while we continue on, and when we need it’s results we will have them.
Order mucking
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import asyncio
async def meow(number):
print(f'starting {number}')
await asyncio.sleep(1)
print(f'stopping {number}')
async def run():
# notice how we assign each coroutine to a variable
a = meow(1)
b = meow(2)
c = meow(3)
d = meow(4)
e = meow(5)
# then we can decide which order we wish our coroutines to execute
await c
await a
await e
await b
await d
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
Let’s start looking at how our coroutines are handled. In the above example, I am showing that you can define your work in any order. Then await for them out of order. Or out of order and then back in order. Or any combination. Great. They all run sequentially though in the order they are awaited. What about if we wanted all those to run at the same time. Well, we could 1
ensure_future
Parallel runnings.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import asyncio
async def meow(number):
print(f'starting {number}')
await asyncio.sleep(1)
print(f'stopping {number}')
async def run():
f = []
for x in range(1, 6):
f.append(meow(x))
await asyncio.wait(f)
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
Here we are creating a list and then adding all of our items to said list. In this case, we are just iterating over a 1
range()
1
asyncio.wait()
1
await
1
asyncio.wait()
One thing you should notice, in Python 3.5, 3.6 and 3.7 they all work out of order. I’m not sure why, or if/when they will resolve this. If order is important, wrap your coroutine in 1
ensure_future
Example of order keeping parallelism.
1
2
3
4
5
6
async def run():
# We create an empty list to store our coroutines in
f = []
for x in range(1, 6):
f.append(asyncio.ensure_future(meow(x)))
await asyncio.wait(f)
Handling async function results
Our previous example handles the parallelism beautifully (albeit out of order), but what if you want to do something with the results of those functions, a la “scatter-gather”?
GATHER!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import asyncio
async def meow(number):
await asyncio.sleep(1)
return number
async def run():
f = []
for x in range(1, 6):
f.append(asyncio.ensure_future(meow(x)))
x = await asyncio.gather(*f)
print(x)
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
# $>: [1, 2, 3, 4, 5]
That it? What else you got?
A couple things come to mind. Look at this example.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import asyncio
async def new_future(n):
print('future', n)
await asyncio.sleep(3)
print('future done', n)
return n
async def run():
results = asyncio.ensure_future(new_future(1))
print(results) # <Task> object returned
print(await results) # `1` is returned
print(results) # <Task> is returned again
print(await results) # `1` is returned again
results = new_future(2)
print(results) # coroutine returned
print(await results) # `2` is returned
print(results) # coroutine returned again
print(await results) # RuntimeError: cannopy reuse aalready awaited coroutine
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
1
ensure_future
1
Task
1
await
And lastly, just have fun with it. You can do all sorts of crazy stuff with asyncio. Execute things in weird orders, or whatever makes sense for your application.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import asyncio
async def meow(number):
print(f'starting {number}')
await asyncio.sleep(1)
print(f'stopping {number}')
async def run():
f = []
for x in range(1, 6):
f.append(meow(x))
for x, y in enumerate(f):
if x % 2 == 0:
await y
for x, y in enumerate(f):
if x % 2 != 0:
await y
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
And as always.
