Parsing the page with BeautifulSoup

The previous part got a page's raw HTML into a string. This one turns that string into actual data — one Python dict per quote. For the fuller method reference (.find() vs .select(), handling missing elements safely), see the standalone BeautifulSoup tutorial; this part applies it directly to the project.

Inspecting the page structure

Before writing a selector, look at what you're selecting. Open the target page in a browser, right-click a quote, and choose "Inspect" — every mainstream browser has this. quotes.toscrape.com marks up each quote consistently:

<div class="quote">
  <span class="text">"The world as we have created it is a process of our thinking."</span>
  <span>
    by <small class="author">Albert Einstein</small>
    <a href="/author/Albert-Einstein/">(about)</a>
  </span>
  <div class="tags">
    <a class="tag" href="/tag/change/page/1/">change</a>
    <a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>
  </div>
</div>

Every quote on the page follows this shape: a .quote container, a .text span, an .author name with a link to their bio page, and a .tags block with one .tag link per tag.

Extracting one quote's data

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")
quote_div = soup.select_one(".quote")

text = quote_div.select_one(".text").text
author = quote_div.select_one(".author").text
author_url = quote_div.select_one("a")["href"]
tags = [tag.text for tag in quote_div.select(".tags .tag")]

print(text, "—", author, tags)

.select_one(".text") and .select_one(".author") grab the first match inside quote_div specifically — scoping the search to one quote's container instead of the whole page is what keeps this correct once there's more than one quote on the page. The tags line is a list comprehension over every .tag link inside .tags, since there's usually more than one.

Turning the whole page into a list of dicts

def parse_quotes_page(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "lxml")
    quotes = []
    for quote_div in soup.select(".quote"):
        text_tag = quote_div.select_one(".text")
        author_tag = quote_div.select_one(".author")
        link_tag = quote_div.select_one("a")
        if not text_tag or not author_tag:
            continue
        quotes.append({
            "text": text_tag.text,
            "author": author_tag.text,
            "author_url": link_tag["href"] if link_tag else None,
            "tags": [t.text for t in quote_div.select(".tags .tag")],
        })
    return quotes

soup.select(".quote") — plural — returns every quote container on the page, and the loop repeats the single-quote extraction above for each one. The if not text_tag or not author_tag: continue guard skips a malformed entry instead of crashing the whole page's worth of quotes over one bad one — the same defensive pattern the BeautifulSoup tutorial covers in more depth.

Next: this only handles one page. The real site has ten — part 4 covers following pagination to get all of them.