Concurrent Web Requests with asyncio

NicePython

— 1 min read

Here's something more advanced: Coroutine-based concurrency with asyncio.

Python's asyncio module provides a powerful framework for writing asynchronous code, allowing you to perform I/O-bound tasks concurrently without the complexity of traditional multithreading or multiprocessing.

Let's demonstrate this with an advanced example of how you can perform multiple HTTP requests concurrently using asyncio and the aiohttp library. This setup allows you to download data from multiple URLs at the same time without blocking.

You'll need to install the aiohttp library, which you can do via pip:

pip install aiohttp

Here’s how the code looks:

import asyncio
import aiohttp

async def fetch_url(session, url):
    async with session.get(url) as response:
        content = await response.text()
        print(f"Downloaded {len(content)} characters from {url}")

async def main(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_url(session, url) for url in urls]
        await asyncio.gather(*tasks)

# Example usage:
urls = [
    "https://www.example.com",
    "https://www.python.org",
    "https://www.asyncio.org"
]

# Run the asyncio event loop
asyncio.run(main(urls))

Explanation:

  1. async def: Defines an asynchronous function that can use await to perform non-blocking I/O.
  2. aiohttp.ClientSession: Creates an HTTP session to manage connections and reuse them across requests, increasing efficiency.
  3. asyncio.gather(*tasks): Runs multiple tasks concurrently. Here, it sends multiple requests at the same time instead of one by one, resulting in faster execution when dealing with multiple URLs.
  4. await: Pauses the coroutine until the non-blocking operation completes, preventing the blocking of the entire program.

This approach allows handling high-concurrency tasks, such as scraping websites or querying APIs, with minimal blocking, making it much more efficient for I/O-bound operations compared to traditional synchronous programming.

If you're working with heavy I/O tasks, asyncio and aiohttp will significantly improve performance.

Read more