Fetching pages reliably with requests

With a target picked and the project set up, the next step is getting a page's HTML into Python. This part covers requests as it applies specifically to this project — for the fuller reference on timeouts, sessions, and exception handling, see the standalone requests tutorial; this one stays focused on what the scraper actually needs.

A first request to the practice site

import requests

response = requests.get("https://quotes.toscrape.com/", timeout=5)
response.raise_for_status()
print(response.status_code)
print(len(response.text), "characters of HTML")

timeout=5 and raise_for_status() aren't optional extras here — a scraper that crawls dozens or hundreds of pages unattended is exactly the code most likely to hang on a slow response or silently process a 500 error page as if it were real data.

Setting a real User-Agent

HEADERS = {
    "User-Agent": "quote-scraper-tutorial/1.0 (learning project; contact: you@example.com)"
}

response = requests.get("https://quotes.toscrape.com/", headers=HEADERS, timeout=5)

Every request carries a User-Agent header identifying what's making it. requests' default is literally python-requests/2.x — some sites block that specific string outright, since it's the clearest possible signal of an unattended script. A descriptive custom User-Agent that names what the script is (and ideally how to reach you) is the responsible version of fixing that: it identifies your scraper honestly instead of pretending to be a browser. Save it as a constant once, like above, and reuse it on every request this project makes.

What the response actually contains

response = requests.get("https://quotes.toscrape.com/", headers=HEADERS, timeout=5)
html = response.text

response.text is the raw HTML the server sent — the same markup a browser's "View Source" shows, before any JavaScript on the page runs. That distinction matters: if the data you need only appears after the page runs JavaScript (an infinite-scroll feed, a price that loads in after the initial render), requests alone won't see it — that needs a browser automation tool instead, which is out of scope for this series. quotes.toscrape.com renders its quotes directly in the HTML, so requests sees everything.

Next: turning that raw HTML string into the actual quote data.