Saving the data and shipping the finished scraper
Everything up to here has produced a Python list of dicts in memory — useful for one interactive run, gone the moment the script exits. This part saves it to disk in two formats, then combines all five previous parts into one script you can actually run.
Writing results to CSV
import csv
def save_to_csv(quotes: list[dict], path: str = "quotes.csv") -> None:
fieldnames = ["text", "author", "author_url", "tags"]
with open(path, "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
for quote in quotes:
writer.writerow({**quote, "tags": ", ".join(quote["tags"])})csv.DictWriter writes one row per dict, matched to fieldnames by key. The tags field is a Python list, which CSV — a plain-text, one-value-per-cell format — has no native way to represent, so ", ".join(quote["tags"]) flattens it to a single comma-separated string before writing. newline="" on the open() call isn't optional on Windows: without it, Python's own newline translation and the csv module's both run, doubling every line break in the output file.
Writing results to JSON, too
import json
def save_to_json(quotes: list[dict], path: str = "quotes.json") -> None:
with open(path, "w", encoding="utf-8") as f:
json.dump(quotes, f, indent=2, ensure_ascii=False)CSV is what opens cleanly in a spreadsheet; JSON is what another script reads back in with the list-of-tags structure intact instead of flattened into a string. Saving both takes one extra function and covers both audiences. ensure_ascii=False keeps accented characters and non-Latin scripts in author names written as themselves instead of \uXXXX escape sequences.
Once the data's actually landed in a real CSV file, pandas is the natural next step for doing anything beyond just opening it — filtering, grouping, or actually analyzing what a scraper like this one collected.
The finished scraper
import csv
import json
import time
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
HEADERS = {
"User-Agent": "quote-scraper-tutorial/1.0 (learning project; contact: you@example.com)"
}
START_URL = "https://quotes.toscrape.com/"
DELAY_SECONDS = 1.0
def parse_quotes_page(html: str) -> list[dict]:
soup = BeautifulSoup(html, "lxml")
quotes = []
for quote_div in soup.select(".quote"):
text_tag = quote_div.select_one(".text")
author_tag = quote_div.select_one(".author")
link_tag = quote_div.select_one("a")
if not text_tag or not author_tag:
continue
quotes.append({
"text": text_tag.text,
"author": author_tag.text,
"author_url": link_tag["href"] if link_tag else None,
"tags": [t.text for t in quote_div.select(".tags .tag")],
})
return quotes
def get_next_page_url(soup: BeautifulSoup, base_url: str) -> str | None:
next_link = soup.select_one("li.next a")
return urljoin(base_url, next_link["href"]) if next_link else None
def fetch_with_retry(session: requests.Session, url: str, max_retries: int = 3):
for attempt in range(1, max_retries + 1):
try:
response = session.get(url, timeout=5)
response.raise_for_status()
return response
except requests.exceptions.RequestException as err:
if attempt == max_retries:
print(f"Giving up on {url}: {err}")
return None
time.sleep(2 ** attempt)
return None
def crawl_all_pages(session: requests.Session, start_url: str, max_pages: int = 20) -> list[dict]:
all_quotes, url, pages_visited = [], start_url, 0
while url and pages_visited < max_pages:
response = fetch_with_retry(session, url)
if not response:
break
soup = BeautifulSoup(response.text, "lxml")
all_quotes.extend(parse_quotes_page(response.text))
url = get_next_page_url(soup, url)
pages_visited += 1
time.sleep(DELAY_SECONDS)
return all_quotes
def save_to_csv(quotes: list[dict], path: str = "quotes.csv") -> None:
with open(path, "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["text", "author", "author_url", "tags"])
writer.writeheader()
for q in quotes:
writer.writerow({**q, "tags": ", ".join(q["tags"])})
def save_to_json(quotes: list[dict], path: str = "quotes.json") -> None:
with open(path, "w", encoding="utf-8") as f:
json.dump(quotes, f, indent=2, ensure_ascii=False)
if __name__ == "__main__":
session = requests.Session()
session.headers.update(HEADERS)
quotes = crawl_all_pages(session, START_URL)
print(f"Scraped {len(quotes)} quotes")
save_to_csv(quotes)
save_to_json(quotes)
print("Saved to quotes.csv and quotes.json")Every function here is one from an earlier part in this series, unchanged — author bio fetching is left out of this final assembly to keep it runnable as one focused script, but get_author_bio from part 4 drops in the same way: call it inside the loop in crawl_all_pages with a seen_authors dict threaded through, and add the bio fields to each quote's dict before it gets appended.
What's next: pages that need a real browser
This series covers what requests + BeautifulSoup can reach: any page whose data is present in the HTML the server sends back. Plenty of real sites load their actual content after the page loads, via JavaScript — infinite-scroll feeds, prices injected client-side, content behind a "load more" button that fires an API call. requests never runs that JavaScript, so it never sees what it produces.
That's a different tool: a browser automation library like Playwright or Selenium, which drives a real (or headless) browser that renders JavaScript exactly like a person's browser would, then lets you read the resulting page. Everything from this series — parsing, pagination, politeness, retries — still applies once you're past that; only the fetching step changes. It's the natural next tutorial from here, and a good one to look for if the page you actually need doesn't show its data to curl.