How To Retry Failed Python Requests

Errors during HTTP requests are something you’ll inevitably run into when scraping. In this article, I’ll share the main reasons requests fail, how to handle those failures, and how to implement retries when needed. Since Requests is the go-to library for HTTP in Python, most of the examples will focus on using it. 

Common Reasons for Failed Requests  

The try…except block is a useful tool when handling errors in Python. We’ll organize exceptions into their respective sections, but for now, let’s focus on the try block:

import requests


url = "https://example.com"


try:
    response = requests.get(url)  
    response.raise_for_status() 

If you’re looking to catch all request errors, no matter the cause, you can use this snippet:

except requests.exceptions.RequestException as e:
    print(f"Request error: {e}")

Since we’ll be building on this script later, I won’t repeat this part every single time. It’s just here to set the foundation.

Timeout Errors  

Let’s start with one of the most common issues: the timeout error. A timeout happens when the server doesn’t respond within the specified time limit. This could be due to server overload, network problems, or just a slow response time.

The catch is that the requests library doesn’t enforce a default limit, so if you don’t manually set a timeout, your request could theoretically hang forever. In reality, though, something will eventually interrupt your request, whether it’s the server, a router, or some other network component.

To avoid this, it’s a good idea to set a timeout explicitly, like this:

    response = requests.get(url, timeout=5)

Now, what if you actually want to handle a timeout error? You can use a try-except block to catch it:

except requests.exceptions.Timeout:
    print("Timeout error")

Instead of logging the error, you could take action, like retrying the request. That way, your script doesn’t give up at the first sign of trouble. I’ll go over setting up retries later in this article.

HTTP Status Error Codes

Working with APIs or scraping data invariably involves encountering unsuccessful HTTP requests. Whenever you send a request, you’ll get a status code back. This status code could be a successful one, a redirect, or an error. The codes worth handling are in the table below:

While the table lists all possible status codes, I’ll focus on the ones that indicate errors. Specifically, the 4xx and 5xx codes, as well as timeouts. Thankfully, Python’s requests library allows us to handle these issues gracefully using exceptions. Here’s an example of how you can process errors in your code:

except requests.exceptions.HTTPError as http_err:
	#Here will be code

Note that the requests.exceptions.HTTPError exception only gets raised for status codes 400 and above because we’re calling response.raise_for_status() in the example.

403 Forbidden

A 403 error means the server is blocking your request. This usually happens because of missing permissions, incorrect authentication, or absent API keys or tokens. If you’re confident that none of these apply, try adding some headers to your request. Servers often expect headers like User-Agent, Authorization, or Referer.

Here’s an example:

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36'
}
response = requests.get(url, headers=headers)

Check out this table if you’re looking for up-to-date User-Agent strings.

Want to handle this error programmatically? Add an exception block like this:

except requests.exceptions.HTTPError as http_err:
    if http_err.response.status_code == 403:
        print("Error. 403 Forbidden")

Correct headers and authentication fix the 403s you caused. The ones served by bot management persist no matter what the request looks like, and the measurement later in this article shows how common those are.

429 Too Many Requests

Now the 429 error. Anyone running a scraper meets this one sooner or later. It means the requests went out faster than the server allows.

Here’s how to get around it:

1. Add delays between your requests.
2. Use proxies to distribute the requests.

Rotating the exit address is the other half of the fix, and proxies with Python’s requests covers the setup. For the delay approach you need the time library:

import time
time.sleep(5) 

To catch and handle this error, you can reuse the same code snippet as before:

except requests.exceptions.HTTPError as http_err:
    if http_err.response.status_code == 429:
        print("Error. 429 Too Many Requests.")

Avoid sending too many requests from the same IP address, and you probably won’t have to deal with this error at all.

500 Internal Server Error

The 500 error is a server-side problem, meaning it’s not your fault. It usually indicates that the server failed to process your request due to some internal issue. Unfortunately, there’s not much you can do here except wait and try again later, or contact the website’s support team if it’s urgent.

To handle this error, use the following code:

except requests.exceptions.HTTPError as http_err:
    if http_err.response.status_code == 500:
        print("500 Internal Server Error.")

Keep in mind that this error is beyond your control to predict or correct.

502 Bad Gateway

A 502 error happens when there’s a communication issue between a server and a gateway or proxy. It’s usually a server problem, but if you’re using proxy, you might want to double-check them.

You can catch this error using:

except requests.exceptions.HTTPError as http_err:
    if http_err.response.status_code == 502:
        print("502 Bad Gateway.")

If you’re feeling proactive, verify your proxy settings and any intermediary servers you’re working with. But in most cases, the issue lies with the target server.

503 Service Unavailable

A 503 error means the server is either overloaded or temporarily down for maintenance. These errors are often short-lived, so the best solution is to wait and try again after a delay.

Here’s how you can handle it:

except requests.exceptions.HTTPError as http_err:
    if http_err.response.status_code == 503:
        print("503 Service Unavailable.")

For peace of mind, you might also try visiting the page in your browser to confirm its status.

504 Gateway Timeout

The 504 error indicates that a gateway couldn’t get a timely response from another server. Network issues or a slow backend server can cause this. Unfortunately, your only options here are to check your network or wait and retry the failed requests later.

Catch this error with:

except requests.exceptions.HTTPError as http_err:
    if http_err.response.status_code == 504:
        print("504 Gateway Timeout")

Instead of logging the error, I recommend building a requests retry mechanism. We’ll explore that in detail in the next section.

Configuring Retries in Python Requests  

Using retries in Python requests can be super helpful, especially when dealing with unreliable network conditions. But let’s be real: too many retries can backfire. It can cause unnecessary delays or even get your client blocked by the server. That’s why it’s crucial not to configure retries simply but also to set a sensible limit on how many times your Python requests retry.

The ideal retry limit depends on the type of request and the server you’re targeting. Generally, it’s advisable to adhere to a three to five retry limit before ceasing altogether or introducing a longer pause between attempts. I’ll break down some key aspects of implementing retries effectively.

Setting Timeout Values in Retry Logic

Timeouts came up earlier, and they’re the first thing to set. Otherwise, a request might hang indefinitely, leaving your program stuck. In the earlier section, I showed you how to set a basic timeout limit, but you can make things more efficient by splitting the timeout into connection time and read time.

Here’s an example:

response = requests.get(url, timeout=(3, 10))

The two values here are:

1. 3 seconds for the connection timeout.
2. 10 seconds for the read timeout.

This approach gives you better control over how your requests behave and is just good practice overall. Both numbers also shape the retry logic below, because every retry attempt waits out the same timeout before it counts as a failure.

Handling Status Codes and Exceptions

Previously, I walked you through handling errors individually. This time, let’s look at a broader example that shows how to manage all status codes in one go.

Like in life, when writing scripts, it’s a good idea to anticipate errors and plan for them. For instance, you might want to specify fallback actions for unexpected errors like this:

except requests.exceptions.HTTPError as http_err:
    if http_err.response.status_code == 403:
        print("403 Forbidden.")
    elif http_err.response.status_code == 429:
        print("429 Too Many Requests.")
    elif http_err.response.status_code == 500:
        print("500 Internal Server Error.")
    elif http_err.response.status_code == 502:
        print("502 Bad Gateway.")
    elif http_err.response.status_code == 503:
        print("503 Service Unavailable.")
    elif http_err.response.status_code == 504:
        print("504 Gateway Timeout.")
    else:
        print(f"HTTP error: {http_err.response.status_code}")


except requests.exceptions.RequestException as e:
    print(f"Request error: {e}")

That said, not all errors warrant retrying failed requests. Some can’t be “fixed” by simply reconnecting. This is especially true for 4xx errors, which typically result from something on the client side.

For example, if you’re getting a 403 (Forbidden), it’s worth double-checking your headers before retrying. For a 429 (Too Many Requests), you might need to wait or use a proxy to rotate your IP address. 

Avoiding Infinite Retry Loops

Retry loops are helpful but can be dangerous if not handled properly. Infinite requests retry loops can cause resource leaks, hammer servers, and even get your IP banned. That’s why it’s critical to define clear exit conditions for retries. For example, you might want to stop retrying after reaching a specific status code or exceeding a time limit.

Here’s how you can implement that:

MAX_RETRIES = 5
TIMEOUT = 5
for attempt in range(1, MAX_RETRIES + 1):
    try:
        response = requests.get(url, timeout=TIMEOUT)
        print("Request succeeded:", response.status_code)
        break
    except requests.exceptions.RequestException as e:
        print(f"Connection attempt {attempt} failed: {e}")
        if attempt == MAX_RETRIES:
            raise
        time.sleep(2)

If you skip this step, retry loops can lead to more than just a temporary server block. They can overload your own network, leaving you unable to even Google your way out of trouble. So, be sure not to skip this step.

Exponential Backoff with Tenacity

Backoff strategies help prevent server overload, especially when dealing with frequent requests. A common choice is exponential backoff, where each retry is delayed by an increasing amount of time (e.g., 1 second, 2 seconds, 4 seconds, and so on).

You can implement this manually using time.sleep(), but honestly, it’s much easier to use a library like Tenacity.

Here’s an example of how to implement backoff using Tenacity: 

from tenacity import retry, stop_after_attempt, wait_exponential
import requests


@retry(stop=stop_after_attempt(5), wait=wait_exponential(multiplier=1, min=2, max=10))
def make_request():
    response = requests.get("https://example.com", timeout=5)
    response.raise_for_status()
    print(response.status_code)
    return response


try:
    make_request()
except Exception as e:
    print("Request failed after retries:", e)

The raise_for_status() call inside the function matters, because Tenacity retries only exceptions, and without it a 500 response returns normally and never triggers a retry. This is far simpler than coding delays manually. If you want an even shorter path, the retry-requests package on PyPI wraps the same urllib3 setup in two lines.

Session Retries with urllib3 Retry and HTTPAdapter

Alright, here’s the simplest way to combine the timeout, the status handling, and the backoff in one place. To make this work, we’ll need urllib3, which already ships with requests. Here’s the complete code:

from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
import requests


retry_strategy = Retry(
    total=5,  
    status_forcelist=[429, 500, 502, 503, 504], 
    backoff_factor=1  
)
adapter = HTTPAdapter(max_retries=retry_strategy)


session = requests.Session()
session.mount("https://", adapter)
session.mount("http://", adapter)


response = session.get("https://example.com")
print(response.status_code)

But if you’re like me and prefer to break things down step by step, let’s go through each part together.

First, we need to import the required libraries. The Retry class comes from urllib3, which requests installs as its own dependency (older snippets import it through requests.packages.urllib3, a legacy alias that still works but adds nothing):

from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
import requests

Next, we’ll set up the retry logic:

retry_strategy = Retry(
    total=5,  
    status_forcelist=[429, 500, 502, 503, 504], 
    backoff_factor=1  
) 

Here, we specify the total number of retries (total = 5), define which status codes should trigger a retry, and set an exponential backoff factor. If you’re unfamiliar with backoff, it’s basically a delay that grows in a geometric progression. The first retry fires immediately, so with a factor of 1 the delays run 0, 2, 4, 8, then 16 seconds, capped by backoff_max (120 seconds by default).

After that, we’ll create an adapter that applies our retry strategy:

adapter = HTTPAdapter(max_retries=retry_strategy)

Now, let’s set up a session object to manage the connection throughout all retries:

session = requests.Session()
session.mount("https://", adapter)
session.mount("http://", adapter)

Finally, we make the actual request:

response = session.get("https://example.com")

A session reuses the same connection across all the retries. Without it, a new connection would be created for each request, which slows things down and puts extra load on the server. The manual retry loop from earlier works without a session, it just pays that reconnect cost on every attempt.

Which Codes Belong in status_forcelist

Retry only what can change on its own. Server-side errors (500, 502, 503, 504), the rate limit (429), and the request timeout (408) can all clear between attempts, so they belong in status_forcelist. A 403 does not. The server refused the request based on what it saw (headers, IP address, authentication), and an identical request a few seconds later gets the same answer. If your 403 comes from missing headers, fix the headers rather than repeating the request.

allowed_methods, backoff_jitter, and Retry-After

urllib3 2.x has a few more Retry parameters to set, documented in the urllib3 Retry reference:

retry_strategy = Retry(
    total=5,
    status_forcelist=[408, 429, 500, 502, 503, 504],
    backoff_factor=1,
    backoff_jitter=0.5,
    allowed_methods=["GET", "HEAD", "OPTIONS"],
)

allowed_methods narrows retries to methods that are safe to repeat. The default already excludes POST but keeps PUT and DELETE, which change state, and passing allowed_methods=None retries everything including POST, so a retried order or comment lands twice. An explicit short list removes both surprises. backoff_jitter (new in urllib3 2.x) adds a random slice of up to half a second to each delay, so a fleet of clients that failed together does not retry in one synchronized wave. respect_retry_after_header defaults to True. Keep the default. When a 429 or 503 response carries a Retry-After header, the client waits exactly as long as the server asks instead of guessing with backoff.

urllib3 2.x removed method_whitelist. Passing it raises a TypeError, so snippets written for 1.x need allowed_methods instead.

Writing a try except Retry Loop

Now that the fundamentals are covered, let’s attempt to write our own retry logic. Trust me, it’s not as scary as it sounds. Here’s a final version of the script we’ll be building:

import time
import requests
 
def make_request_with_retry(url, retries=5, backoff_factor=2, timeout=5):
    for attempt in range(1, retries + 1):
        try:
            response = requests.get(url, timeout=timeout)
            response.raise_for_status()
            return response
        except requests.exceptions.RequestException as e:
            if attempt == retries:
                raise
            wait_time = backoff_factor ** attempt
            print(f"Error: {e}. Try {attempt}/{retries}. Retry after {wait_time} sec...")
            time.sleep(wait_time)
 
url = "https://example.com"
try:
    response = make_request_with_retry(url)
    print(response.text)
except requests.exceptions.RequestException as e:
    print(f"Error: {e}")

The function runs the request, doubles the wait after every failed attempt (backoff_factor ** attempt gives 2, 4, 8, 16 seconds), and re-raises the last error once the attempts run out, so the caller decides what happens next. Even though this code performs the same job as the previous one, it’s usually better to rely on ready-made solutions like urllib3 or Tenacity since they handle edge cases better.

Do Retries Actually Rescue Failed Requests

Mostly no, at least on first contact. Across 600 requests, retries rescued exactly one of the 40 failures they exist for. The measurement: 200 domains from a Tranco-sourced list, a homepage GET with a browser user agent and a 15-second timeout, three separate passes over the same list. Every retryable failure (a network exception, a timeout, a 429, or a 5xx) was retried up to three times with 1, 2, and 4 second delays. Responses with a 4xx code other than 429 were recorded as terminal and not retried, and no 408 appeared in the sample. Requests ran under a second apart, with the passes minutes apart.

389 of the 600 requests succeeded on the first try, and 171 returned a terminal 4xx (153 of them a 403). That left 40 failures of the classes retry logic exists for, and the one rescue was a timeout that succeeded on the first attempt after the failure.

The same run as a funnel, which is the shape that shows how little of the total a retry ever touches:

Those 40 failures came from 14 domains, and 13 of the 14 failed the same way in all three passes. A domain that times out keeps timing out, a dead TLS setup stays dead, and none of the 429s cleared within the seven-second backoff window, and the same domains returned 429 again on the next pass minutes later, which points at bot management rather than short-window rate limits.

Keep retry budgets modest. On first-contact requests across many hosts, almost everything that fails is in a state a second identical request does not change, so three to five retries with backoff cost little and catch the rare transient blip, while anything past that spends time on domains that will not answer today. For those, the useful retry is a scheduled re-run of the whole job rather than another attempt seconds later. In-process retries do help in one other setup, burst traffic against a single API that rate-limits you per minute. A short-window 429 clears quickly, and respect_retry_after_header already handles the wait. Our sample measured first contact from one machine, so treat the numbers as an open-web baseline rather than a verdict on API clients.

Conclusion  

The measurement above sets the budget. One rescue in 40 retryable failures means a retry is worth having and not worth spending much on, so cap the attempts low and put the effort into telling a retryable failure from a terminal one.

Endless retry attempts can easily create loops that hammer the network, bogging down performance. Worse, if you don’t configure retries thoughtfully, you might find yourself in a situation where your machine slows down significantly, the network becomes overloaded, and ultimately, you still receive no results. Frustrating.

That’s why it’s so crucial to think not just about how your program should handle retries when things go wrong, but also about when it should stop trying altogether. Setting clear limits on the number of retries, backoff timing, or the conditions under which retries happen can save you a lot of headaches. Trust me, a little upfront planning makes a significant difference!