Make asynchronous HTTP requests using asyncio and aiohttp
Quick Answer
Use `aiohttp.ClientSession` and `async with` to make non-blocking HTTP requests; `asyncio.gather()` fetches multiple URLs concurrently without blocking.
1import asyncio
2import aiohttp
3from typing import List
4
5# Single async request
6async def fetch_url(session: aiohttp.ClientSession, url: str) -> str:
7 async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as response:
8 return await response.text()
9
10# Fetch multiple URLs concurrently
11async def fetch_multiple(urls: List[str]) -> List[str]:
12 async with aiohttp.ClientSession() as session:
13 tasks = [fetch_url(session, url) for url in urls]
14 return await asyncio.gather(*tasks)
15
16# Usage
17url_list = [
18 'https://api.github.com/users/github',
19 'https://api.github.com/users/google',
20 'https://api.github.com/users/facebook'
21]
22
23results = asyncio.run(fetch_multiple(url_list))
24print(results)
25
26# Parse JSON from async request
27async def fetch_json(url: str) -> dict:
28 async with aiohttp.ClientSession() as session:
29 async with session.get(url) as response:
30 return await response.json()
31
32# Batch requests with limit
33async def batch_fetch(urls: List[str], limit: int = 5):
34 semaphore = asyncio.Semaphore(limit)
35 async def bounded_fetch(url):
36 async with semaphore:
37 async with aiohttp.ClientSession() as session:
38 async with session.get(url) as response:
39 return await response.json()
40 return await asyncio.gather(*[bounded_fetch(url) for url in urls])Synchronous HTTP requests block the event loop — if you need to fetch 100 URLs, a normal loop takes ~100x the single request time. Async/await and `aiohttp` let you fetch many URLs concurrently, completing in roughly the time of the slowest single request. Essential for high-performance data pipelines.
Default is 100 connections per host. Use `TCPConnector(limit=50)` to configure. Most APIs have rate limits, so check their docs.
asyncio is single-threaded with cooperative multitasking (tasks yield control). Threading is OS-scheduled but has GIL overhead and race condition complexity. Use asyncio for I/O-heavy tasks.
This free python code snippet for async http request is production-ready and copy-paste friendly. Whether you are building a web app, API, or frontend interface, this intermediate-level example will help you implement async http request quickly and correctly.
All snippets in the Snippetly library follow python best practices and are tested for real-world use. You can adapt this code to work with React, Vue, Node.js, or any project that uses python.
Organise your team's code snippets with Snippetly. Share knowledge and boost productivity across your organisation.