BeautifulSoup basics: pulling structured data out of messy HTML

BeautifulSoup turns a blob of HTML text into a tree you can search — by tag, by class, by CSS selector — without writing a single regular expression. It doesn't fetch pages itself; pair it with requests for that. This is the reference for the methods you'll actually reach for and the one mistake that crashes almost every first scraping script. For a complete, real project built around these methods — pagination, politeness, retries, exporting results — see the full web scraping series instead; this page covers the library itself, not the project.

Installing it, and choosing a parser

pip install beautifulsoup4 lxml

beautifulsoup4 is the library itself; lxml is a fast HTML parser it delegates to. BeautifulSoup ships a slower pure-Python parser too (html.parser, built into the standard library, no separate install), but lxml is worth installing from the start for two concrete reasons: it's measurably faster on large pages, and it's more forgiving of the broken, unclosed-tag HTML real websites actually serve — html.parser more often raises or silently mis-parses a page with a genuinely malformed tag, where lxml recovers and keeps going. The one case for sticking with html.parser: an environment where installing a package with a C extension (which lxml has) isn't possible, like some restricted serverless runtimes.

soup = BeautifulSoup(html, "lxml")           # fast, forgiving — the default choice
soup = BeautifulSoup(html, "html.parser")    # no install needed, slower, less forgiving

Parsing HTML and finding your first element

from bs4 import BeautifulSoup

html = """
<div class="product">
  <h2 class="name">Wireless Mouse</h2>
  <span class="price">$24.99</span>
</div>
"""

soup = BeautifulSoup(html, "lxml")
name = soup.find("h2", class_="name")
print(name.text)  # Wireless Mouse

soup.find(tag, class_=...) returns the first matching element as a Tag object, or None if nothing matches. class_ has a trailing underscore because class is a reserved word in Python — a detail that trips up most people the first time they type it.

Finding every match, not just the first

products = soup.find_all("div", class_="product")
for product in products:
    print(product.find("span", class_="price").text)

find_all() returns a list of every matching Tag, in document order. It's the one to reach for whenever a page has a repeating pattern — search results, product cards, table rows — rather than calling find() in a loop and hoping the page only has one of something.

CSS selectors for anything more specific

prices = soup.select("div.product > span.price")
featured = soup.select_one("#featured .name")

select() and select_one() accept real CSS selector syntax — descendant combinators, #id, .class, [attr=value], :nth-child(). For anything more specific than "one tag with one class," a CSS selector is usually shorter and more precise than chaining find() calls, and it's the same syntax you'd use in browser devtools to confirm the selector matches before writing any Python.

The error that crashes almost every first script

# Crashes with AttributeError if .find() returned None
price = soup.find("span", class_="price").text

# Survives a missing element instead
price_tag = soup.find("span", class_="price")
price = price_tag.text if price_tag else None

find() and select_one() return None when nothing matches — they don't raise an exception. Calling .text directly on that None raises AttributeError: 'NoneType' object has no attribute 'text', and on a real website that happens constantly: a listing without a sale badge, a profile with no bio, one page in a hundred that's structured slightly differently. Check for None before reading .text or .get() on anything that isn't guaranteed to exist on every page.

Getting attribute values, not just text

link = soup.find("a", class_="product-link")
url = link["href"]          # raises KeyError if the attribute is missing
url = link.get("href")      # returns None instead

Square-bracket access (tag["href"]) works like a dict and raises KeyError if that attribute isn't present on the tag. tag.get("href") returns None instead, the same safer pattern as find() returning None — useful when scraping images or links where an alt or href attribute is sometimes just missing.

Putting it together with requests

import requests
from bs4 import BeautifulSoup

response = requests.get("https://example.com/products", timeout=5)
response.raise_for_status()

soup = BeautifulSoup(response.text, "lxml")
for card in soup.select(".product"):
    name_tag = card.select_one(".name")
    price_tag = card.select_one(".price")
    if name_tag and price_tag:
        print(name_tag.text.strip(), price_tag.text.strip())

response.text is the raw HTML string requests downloaded — that's what BeautifulSoup parses. raise_for_status() runs first so a 404 or 500 page (which is still valid HTML, just not the page you wanted) fails loudly instead of silently parsing into an empty result set. .strip() on the text trims the whitespace and newlines real HTML is usually indented with, which .text includes as-is.

What production-ready scraping code looks like

import requests
from bs4 import BeautifulSoup

def scrape_products(url: str) -> list[dict]:
    response = requests.get(url, timeout=5)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "lxml")
    results = []
    for card in soup.select(".product"):
        name_tag = card.select_one(".name")
        price_tag = card.select_one(".price")
        if not name_tag or not price_tag:
            continue
        results.append({
            "name": name_tag.text.strip(),
            "price": price_tag.text.strip(),
        })
    return results

A timeout so a slow page can't hang the run, raise_for_status() so a broken request fails clearly instead of parsing garbage, and a None check on every element pulled from the page so one inconsistent card skips itself instead of taking down the loop. That combination — not a longer list of BeautifulSoup methods — is what separates a scraper that works once from one that keeps working the next hundred times it runs.

FAQ

Should I learn this page or the full web scraping series first? This page if the goal is understanding BeautifulSoup's methods in isolation — for a URL, a quick data pull, or using it inside a larger project. The web scraping series if the goal is a complete, real scraper handling pagination, rate limiting, and exporting results, since it builds on these same methods across a full project.

Does select() replace find() and find_all() entirely? Not entirely — select() is usually shorter for anything involving a specific combination of tag, class, and nesting, but find()/find_all() read more clearly for a simple single-attribute match, and mixing both in the same script based on what's clearest for each specific case is normal.

Why does my selector work in browser DevTools but return nothing in BeautifulSoup? Usually because the browser is showing HTML modified by JavaScript after the page loaded, while requests only ever sees the original HTML the server sent. If DevTools' "View Page Source" (not the regular Elements panel) doesn't show the element either, it was added by JavaScript and BeautifulSoup can't reach it — that's a signal to reach for a browser automation tool instead.

Is lxml required, or can I always use html.parser? html.parser works for any well-formed HTML — the difference only shows up on malformed pages or at scale where lxml's speed matters. For a one-off script html.parser is perfectly fine; for anything scraping many pages, lxml's speed and error tolerance are worth the one extra install.