Planning a scraper: picking a target, and reading its robots.txt

Web scraping is really four separate problems wearing one name: fetching a page, parsing it into structured data, crawling to more pages, and storing what you found — reliably enough to run more than once. This series builds all four, one part at a time, into a real scraper for quotes.toscrape.com, a site built specifically for scraping practice. By the end you'll have a script that walks every page, follows each author's bio, and saves the result to CSV and JSON.

What you'll build

A single Python script that:

• Fetches every page of quotes from the site, following pagination until there's no "next" page left
• Follows each unique author's page once for their bio, without re-fetching the same author twice
• Retries a request that fails instead of crashing the whole run
• Saves the results — quote, author, tags — to both quotes.csv and quotes.json

Each part below builds one piece. By part 6 they're combined into the finished script.

Pick a target you're actually allowed to scrape

Before anything else: check if the site has a public API first. An API is a target that wants to be queried programmatically — stable field names, no HTML to parse, no risk of breaking when someone redesigns the page. Scraping is the fallback for when no API exists.

When there's no API, quotes.toscrape.com is what this series uses throughout — it's maintained by Zyte (the company behind the Scrapy framework) explicitly as a sandbox for learning to scrape, with a stable, simple structure and no rate limits designed to catch you out. Practicing here first means the mistakes you make are cheap ones.

For a real target, two things to check before writing any code:

• Terms of Service — some sites explicitly prohibit automated access. That's not a technical obstacle requests can solve around; it's a "don't."
• Is the data behind a login, or personal/sensitive? Public product listings and public blog posts are a different situation than scraping profile data or anything gated behind authentication.

Reading robots.txt before you write a line of code

Most sites publish a robots.txt at their root that tells automated clients which paths they'd rather bots stay out of:

curl https://example.com/robots.txt

User-agent: *
Disallow: /admin/
Disallow: /search
Crawl-delay: 5

Sitemap: https://example.com/sitemap.xml

User-agent: * means "these rules apply to every bot" (a specific bot name instead of * scopes the rules to just that one). Disallow paths are the ones the site is asking automated clients to skip — robots.txt is a request, not a technical lock, but ignoring it is exactly the kind of thing that gets an IP range blocked. Crawl-delay, when present, is the minimum seconds the site is asking you to wait between requests — treat it as a floor, not a suggestion to hit exactly.

Setting up the project

mkdir quote-scraper && cd quote-scraper
python3 -m venv venv
source venv/bin/activate
pip install requests beautifulsoup4 lxml

Three packages cover this entire series: requests to fetch pages, beautifulsoup4 to parse them, and lxml as the fast HTML parser BeautifulSoup delegates to. Freeze them once the project works:

pip freeze > requirements.txt

Next up: actually fetching a page — and the one header most tutorials skip that gets real scrapers blocked on real sites.