Lesson 64 of 70 – Python Requests Module

91%

Python Requests Module

The Python requests module is a popular library used to send HTTP requests to web servers and APIs. It makes it easy to communicate with websites and web services from Python programs.

Note: The requests library is not part of Python's standard library. Install it using pip install requests before using it.

What is the Requests Module?

The Requests library allows Python programs to send HTTP requests such as GET, POST, PUT, PATCH and DELETE.

It is commonly used when working with websites, REST APIs and web services.

import requests

response = requests.get(
    "https://example.com"
)

print(response.status_code)

Installing Requests

Open Command Prompt or Terminal and run:

pip install requests

You can also use:

python -m pip install requests

Tip: Using python -m pip helps ensure that pip is associated with the Python interpreter you intend to use.

Importing Requests

After installation, import the module using:

import requests

You can then use methods such as get(), post(), put(), patch() and delete().

GET Request

The get() method sends an HTTP GET request.

import requests

response = requests.get(
    "https://example.com"
)

print(response.status_code)

A GET request is commonly used to retrieve information.

Response Status Code

The status_code property gives the HTTP status code returned by the server.

import requests

response = requests.get(
    "https://example.com"
)

print(response.status_code)

A successful request commonly returns status code 200, although the appropriate success code depends on the operation.

Reading Response Text

The text property returns the response body as text.

import requests

response = requests.get(
    "https://example.com"
)

print(response.text)

This is useful when you need to read HTML, plain text or another textual response.

Response Content

The content property returns the response body as bytes. This can be useful when working with binary data such as images or files.

import requests

response = requests.get(
    "https://example.com/image.jpg"
)

data = response.content

print(len(data))

Reading JSON Response

If an API returns JSON, the json() method can decode the JSON response into Python data.

import requests

response = requests.get(
    "https://api.example.com/users"
)

data = response.json()

print(data)

The returned Python value may be a dictionary, list or another JSON compatible data structure.

GET Request with Parameters

Query parameters can be passed using the params argument.

import requests

params = {
    "city": "Patna",
    "country": "India"
}

response = requests.get(
    "https://api.example.com/weather",
    params=params
)

print(response.url)

Requests will encode the parameters into the URL.

POST Request

The post() method is commonly used to submit data to a server.

import requests

data = {
    "name": "Rahul",
    "course": "Python"
}

response = requests.post(
    "https://api.example.com/students",
    json=data
)

print(response.status_code)

Sending Form Data

The data argument can be used to send form-style data.

import requests

data = {
    "username": "rahul",
    "course": "python"
}

response = requests.post(
    "https://example.com/login",
    data=data
)

print(response.status_code)

Sending JSON Data

The json argument can be used to send a Python dictionary as a JSON request body.

import requests

student = {
    "name": "Amit",
    "age": 21,
    "course": "Python"
}

response = requests.post(
    "https://api.example.com/students",
    json=student
)

print(response.status_code)

PUT Request

The put() method is commonly used to replace or update a resource.

import requests

data = {
    "name": "Amit",
    "course": "Python Full Stack"
}

response = requests.put(
    "https://api.example.com/students/1",
    json=data
)

print(response.status_code)

PATCH Request

The patch() method is commonly used when only part of a resource needs to be updated.

import requests

data = {
    "course": "Python"
}

response = requests.patch(
    "https://api.example.com/students/1",
    json=data
)

print(response.status_code)

DELETE Request

The delete() method sends an HTTP DELETE request.

import requests

response = requests.delete(
    "https://api.example.com/students/1"
)

print(response.status_code)

Request Headers

Headers provide additional information about an HTTP request.

import requests

headers = {
    "Accept": "application/json"
}

response = requests.get(
    "https://api.example.com/students",
    headers=headers
)

print(response.status_code)

Authentication Headers

Many APIs require authentication. A token can often be sent through an HTTP header.

import requests

headers = {
    "Authorization": "Bearer YOUR_TOKEN"
}

response = requests.get(
    "https://api.example.com/profile",
    headers=headers
)

print(response.status_code)

Security Tip: Never expose real API keys, passwords or private tokens in public source code.

Request Timeout

A timeout specifies how long Requests should wait for a response before raising a timeout-related exception.

import requests

response = requests.get(
    "https://example.com",
    timeout=10
)

print(response.status_code)

A timeout is useful because network requests should not be allowed to wait indefinitely.

raise_for_status()

The raise_for_status() method raises an HTTP error exception when the response contains an unsuccessful HTTP status code.

import requests

response = requests.get(
    "https://example.com"
)

response.raise_for_status()

print("Request successful")

Handling Requests Exceptions

Network problems and HTTP errors should be handled using exception handling.

import requests

try:

    response = requests.get(
        "https://example.com",
        timeout=10
    )

    response.raise_for_status()

    print(response.text)

except requests.exceptions.RequestException as error:

    print("Request failed:", error)

Common Requests Exceptions

The Requests library provides exception classes for different types of request problems.

RequestException – base class for request-related exceptions
Timeout – request took too long
ConnectionError – connection problem
HTTPError – HTTP error response
TooManyRedirects – too many redirects

try:

    response = requests.get(
        "https://example.com",
        timeout=5
    )

    response.raise_for_status()

except requests.exceptions.Timeout:

    print("Request timed out")

except requests.exceptions.RequestException as error:

    print("Request error:", error)

Response Headers

The headers property provides access to response headers.

import requests

response = requests.get(
    "https://example.com"
)

print(response.headers)

A particular header can be retrieved using get().

content_type = response.headers.get(
    "Content-Type"
)

print(content_type)

Response URL

The url property shows the final URL associated with the response.

import requests

response = requests.get(
    "https://example.com"
)

print(response.url)

This can be especially useful when query parameters or redirects are involved.

Checking Content Type

The response's Content-Type header can help identify the type of data returned by the server.

import requests

response = requests.get(
    "https://example.com"
)

content_type = response.headers.get(
    "Content-Type"
)

print(content_type)

Downloading a File

The Requests library can also be used to download files.

import requests

url = "https://example.com/file.pdf"

response = requests.get(
    url,
    timeout=20
)

response.raise_for_status()

with open("file.pdf", "wb") as file:
    file.write(response.content)

print("File downloaded")

The wb mode writes binary data to the file.

Streaming a Large Download

For large files, streaming can help avoid loading the entire response into memory at once.

import requests

url = "https://example.com/large-file.zip"

with requests.get(
    url,
    stream=True,
    timeout=30
) as response:

    response.raise_for_status()

    with open("large-file.zip", "wb") as file:

        for chunk in response.iter_content(
            chunk_size=8192
        ):

            if chunk:
                file.write(chunk)

Session Object

A Session object can be used when making multiple requests to the same service. It can persist certain settings such as headers and cookies across requests.

import requests

session = requests.Session()

session.headers.update({
    "Accept": "application/json"
})

response = session.get(
    "https://example.com"
)

print(response.status_code)

session.close()

Using a Session with with

A Session can also be used as a context manager.

import requests

with requests.Session() as session:

    response = session.get(
        "https://example.com"
    )

    print(response.status_code)

Cookies are small pieces of data that a server can send to a client. Requests can send and receive cookies.

import requests

response = requests.get(
    "https://example.com"
)

print(response.cookies)

Cookies can also be supplied when making a request.

cookies = {
    "user": "rahul"
}

response = requests.get(
    "https://example.com",
    cookies=cookies
)

Common Mistakes

Forgetting to install the Requests package.
Using an incorrect URL.
Using the wrong HTTP method.
Ignoring the response status code.
Not setting a timeout.
Not handling network exceptions.
Trying to decode a non-JSON response as JSON.
Exposing API keys or tokens in source code.
Downloading large files without considering memory usage.

Complete API Request Example

import requests

url = "https://api.example.com/students"

try:

    response = requests.get(
        url,
        timeout=10
    )

    response.raise_for_status()

    data = response.json()

    for student in data:
        print(student)

except requests.exceptions.Timeout:

    print("The request timed out.")

except requests.exceptions.RequestException as error:

    print("Request failed:", error)

Key Points

The Requests library is used to send HTTP requests from Python.
Install it using pip install requests.
requests.get() retrieves data.
requests.post() sends data to a server.
requests.put() commonly replaces or updates a resource.
requests.patch() commonly performs a partial update.
requests.delete() sends a DELETE request.
response.text returns response text.
response.content returns response content as bytes.
response.json() decodes a JSON response.
response.status_code gives the HTTP status code.
params can be used for query parameters.
headers can be used for request headers.
timeout helps prevent requests from waiting indefinitely.
raise_for_status() raises an HTTP error for unsuccessful responses.
Exception handling makes network programs more reliable.
Sessions can be useful when making multiple requests.

🧠 Quick Quiz

Question: Which Requests method is commonly used to retrieve data from a web server?

requests.get()
requests.post()
requests.delete()
requests.patch()

← Previous Next →

Python Tutorial

Python Requests Module

🧠 Quick Quiz