11.3 HTTP Requests with requests in Python

NicePython

— 5 min read

The requests library is one of the most popular and user-friendly Python libraries for making HTTP requests. It simplifies sending requests to web servers and handling the responses. Whether you're accessing APIs, scraping websites, or automating web tasks, requests is a powerful tool that helps you interact with HTTP-based services effortlessly.

In this section, we’ll cover how to use the requests library for making various types of HTTP requests, handling responses, working with APIs, and managing request headers, parameters, and data.

11.3.1 Installing the requests Library

Before you can start using the requests library, you need to install it using pip. Run the following command in your terminal:

pip install requests

Once installed, you can import it into your Python scripts.

11.3.2 Making a Basic HTTP GET Request

The most common type of HTTP request is a GET request, which is used to retrieve data from a server. With the requests library, you can make a GET request with a single function call.

Example: Making a Basic GET Request

import requests

# Make a GET request to the GitHub API
response = requests.get('https://api.github.com')

# Print the response status code
print(response.status_code)  # Output: 200 (if successful)

# Print the response content (JSON data)
print(response.json())

In this example:

  • requests.get() sends a GET request to the specified URL (https://api.github.com).
  • The response.status_code attribute shows the status of the request (e.g., 200 means success).
  • The response.json() method is used to convert the response data (which is in JSON format) into a Python dictionary.

11.3.3 HTTP Methods: GET, POST, PUT, DELETE

The requests library supports all major HTTP methods. Some common methods are:

  • GET: Used to retrieve data from a server.
  • POST: Used to send data to a server (often used for submitting forms or uploading data).
  • PUT: Used to update or replace data on the server.
  • DELETE: Used to delete data from the server.

Example: Making a POST Request

You can send data to a server using the POST method. The data parameter in requests.post() allows you to send form-encoded data, while json can be used to send JSON data.

import requests

# Define the URL and data to send
url = 'https://httpbin.org/post'
data = {'username': 'john', 'password': 'mypassword'}

# Send a POST request
response = requests.post(url, data=data)

# Print the response
print(response.text)  # The server will return the posted data

In this example:

  • requests.post() sends the data (data={'username': 'john', 'password': 'mypassword'}) to the server at https://httpbin.org/post.
  • The response contains the data that was posted, which you can access using response.text.

11.3.4 Handling Query Parameters

When making a GET request, you often need to send query parameters in the URL. Query parameters are used to pass data to the server via the URL (e.g., https://example.com/api?search=python&limit=10).

With the requests library, you can easily pass query parameters using the params argument.

Example: Sending Query Parameters in a GET Request

import requests

# Define the base URL and query parameters
url = 'https://httpbin.org/get'
params = {'search': 'python', 'limit': 10}

# Send a GET request with query parameters
response = requests.get(url, params=params)

# Print the URL that was requested
print(response.url)  # Output: https://httpbin.org/get?search=python&limit=10

# Print the response data
print(response.json())

In this example:

  • params={'search': 'python', 'limit': 10} passes the query parameters as a dictionary.
  • The resulting URL includes the parameters (e.g., https://httpbin.org/get?search=python&limit=10).

11.3.5 Working with JSON Data

The requests library makes it easy to work with JSON data. When receiving a response in JSON format, you can use the response.json() method to parse the JSON data into a Python dictionary. Similarly, you can send JSON data using the json parameter in POST requests.

Example: Sending JSON Data in a POST Request

import requests

# Define the URL and JSON data
url = 'https://httpbin.org/post'
json_data = {'name': 'Alice', 'age': 30}

# Send a POST request with JSON data
response = requests.post(url, json=json_data)

# Print the response (contains the sent JSON data)
print(response.json())

In this example:

  • The json parameter in requests.post() sends the data as JSON.
  • The server responds with the data, which is printed using response.json().

11.3.6 Handling Request Headers

HTTP headers allow the client and server to pass additional information along with the request or response. You can specify custom headers using the headers parameter in a request.

Example: Sending Custom Headers

import requests

# Define the URL and headers
url = 'https://httpbin.org/headers'
headers = {
    'User-Agent': 'my-app/1.0',
    'Authorization': 'Bearer my_token'
}

# Send a GET request with custom headers
response = requests.get(url, headers=headers)

# Print the response (contains the sent headers)
print(response.json())

In this example:

  • The headers dictionary contains custom headers, such as a User-Agent and Authorization token.
  • The request sends the custom headers, which are included in the server's response.

11.3.7 Handling Status Codes

The status code of an HTTP response indicates the result of the request. Some common status codes include:

  • 200: OK (the request was successful).
  • 404: Not Found (the requested resource could not be found).
  • 500: Internal Server Error (the server encountered an error).

You can check the response status code using response.status_code.

Example: Checking Status Codes

import requests

# Send a GET request to an invalid URL
response = requests.get('https://httpbin.org/status/404')

# Check the status code
if response.status_code == 200:
    print('Success!')
elif response.status_code == 404:
    print('Not Found.')
else:
    print(f'Error: {response.status_code}')

In this example:

  • The status code 404 indicates that the resource was not found, and the program prints "Not Found."

11.3.8 Handling Timeouts

If a request takes too long to get a response, you can specify a timeout to prevent the program from waiting indefinitely.

Example: Setting a Timeout

import requests

# Send a GET request with a timeout of 5 seconds
try:
    response = requests.get('https://httpbin.org/delay/10', timeout=5)
    print(response.text)
except requests.Timeout:
    print('The request timed out.')

In this example:

  • timeout=5 limits the time the program will wait for a response. If the request takes longer than 5 seconds, it raises a Timeout exception.

11.3.9 Error Handling with requests

The requests library provides built-in error handling for network issues, timeouts, and invalid responses. You can use try-except blocks to catch exceptions such as:

  • requests.exceptions.Timeout: Raised when a request times out.
  • requests.exceptions.ConnectionError: Raised for network-related errors.
  • requests.exceptions.HTTPError: Raised for HTTP error responses (e.g., 404, 500).

Example: Handling Errors with requests

import requests

url = 'https://invalid-url.com'

try:
    response = requests.get(url)
    response.raise_for_status()  # Raise an error for bad status codes (4xx/5xx)
except requests.exceptions.HTTPError as http_err:
    print(f'HTTP error occurred: {http_err}')
except requests.exceptions.ConnectionError as conn_err:
    print(f'Connection error occurred: {conn_err}')
except requests.exceptions.Timeout as timeout_err:
    print(f'Timeout error occurred: {timeout_err}')
except requests.exceptions.RequestException as req_err:
    print(f'An error occurred: {req_err}')

In this example:

  • The program catches different types of exceptions and prints an error message accordingly.

11.3.10 Summary

The requests library simplifies the process of making HTTP requests in Python. With it, you can easily send GET and POST requests, handle headers and query parameters, work with JSON data, and manage errors and timeouts. Whether you're working with APIs or performing web scraping, requests provides a clean and intuitive interface for interacting with web resources.

Key Features:

  • GET and POST requests: Retrieve or send data to a server.
  • Working with JSON: Parse JSON responses and send JSON payloads.
  • Headers and Parameters: Customize HTTP headers and URL query parameters.
  • Error Handling: Handle network errors, timeouts, and HTTP errors.
  • Timeouts: Prevent requests from waiting indefinitely by setting a timeout.

By mastering requests, you can efficiently interact with web APIs and integrate HTTP-based services into your Python applications.

Read more