Being a polite scraper: rate limiting, retries, and sessions

The crawler from part 4 works — and fires requests as fast as Python and the network allow, with no recovery if one fails. Neither is fine on a real site. This part fixes both: slowing down on purpose, and surviving a bad response instead of crashing the whole crawl.

Slowing down between requests

import time

DELAY_SECONDS = 1.0

while url and pages_visited < max_pages:
    response = requests.get(url, headers=HEADERS, timeout=5)
    # ... parse, extend all_quotes, get next url ...
    pages_visited += 1
    time.sleep(DELAY_SECONDS)

A one-second pause between requests is close to invisible to you and meaningfully lighter on the target server, which — unlike the API-shaped services requests usually talks to — often isn't provisioned to handle a script hitting it as fast as your network allows. If the site's robots.txt specified a Crawl-delay, that number is a floor for DELAY_SECONDS, not a ceiling.

Reusing a Session across the whole crawl

session = requests.Session()
session.headers.update(HEADERS)

response = session.get(url, timeout=5)

Every requests.get() call so far has opened its own connection. Across a crawl making dozens of requests to the same host, a Session reuses the underlying TCP connection instead — and setting session.headers.update(HEADERS) once means the User-Agent from part 2 no longer needs to be passed to every individual call. Swap every requests.get(...) in the crawler for session.get(...) and pass the same session through to get_author_bio too.

Retrying a request that fails

def fetch_with_retry(session: requests.Session, url: str, max_retries: int = 3) -> requests.Response | None:
    for attempt in range(1, max_retries + 1):
        try:
            response = session.get(url, timeout=5)
            response.raise_for_status()
            return response
        except requests.exceptions.RequestException as err:
            if attempt == max_retries:
                print(f"Giving up on {url} after {max_retries} attempts: {err}")
                return None
            wait = 2 ** attempt
            print(f"Attempt {attempt} failed for {url}, retrying in {wait}s")
            time.sleep(wait)
    return None

One dropped connection or one slow response timing out shouldn't end a crawl that's ninety pages in. fetch_with_retry wraps a request in up to max_retries attempts, with the wait between attempts doubling each time (2 ** attempt: 2s, 4s, 8s) — long enough that a genuinely overloaded server gets breathing room, short enough that a one-off blip doesn't stall the run for minutes. Returning None after the final failure, rather than raising, lets the crawl loop decide to skip that one page and keep going instead of losing everything collected so far.

Everything's fetched, parsed, crawled, rate-limited, and resilient to failure now. The last part saves it somewhere useful, and assembles the whole series into one finished script.