Crawling multiple pages: pagination and following links

One page's worth of quotes is a fraction of the site. This part follows pagination to every page, then follows each author's link out to a second kind of page entirely — the two crawling patterns almost every scraper needs.

Finding the "next page" link

<nav>
  <ul class="pager">
    <li class="next">
      <a href="/page/2/">Next <span aria-hidden="true">→</span></a>
    </li>
  </ul>
</nav>

The last page has no .next element at all — that absence is the signal to stop:

def get_next_page_url(soup: BeautifulSoup, base_url: str) -> str | None:
    next_link = soup.select_one("li.next a")
    if not next_link:
        return None
    return urljoin(base_url, next_link["href"])

urljoin(base_url, next_link["href"]) turns the relative /page/2/ from the href into a full https://quotes.toscrape.com/page/2/ — always run a link you scraped through urljoin before requesting it, since most sites use relative paths and a raw requests.get("/page/2/") isn't a valid request on its own.

Crawling every page

from urllib.parse import urljoin

def crawl_all_pages(start_url: str, max_pages: int = 20) -> list[dict]:
    all_quotes = []
    url = start_url
    pages_visited = 0

    while url and pages_visited < max_pages:
        response = requests.get(url, headers=HEADERS, timeout=5)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "lxml")

        all_quotes.extend(parse_quotes_page(response.text))
        url = get_next_page_url(soup, url)
        pages_visited += 1

    return all_quotes

max_pages is a hard safety cap, not a guess at the real page count — if pagination logic has a bug, or a site's "next" link ever points back at a page already visited, this is what stops the loop from running until you notice and kill it manually. pages_visited < max_pages is checked on every iteration alongside url being non-None, so either condition ends the crawl.

Following links to a second kind of page

Each quote links to its author's bio page — a different template with different data (birth date, birthplace, a longer bio). Crawling to it is the same requests.get + BeautifulSoup pattern, just aimed at a different URL, with one addition: the same author quotes several times shouldn't trigger the same fetch several times.

def get_author_bio(author_url: str, seen_authors: dict) -> dict:
    if author_url in seen_authors:
        return seen_authors[author_url]

    response = requests.get(author_url, headers=HEADERS, timeout=5)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "lxml")

    bio = {
        "born": soup.select_one(".author-born-date").text,
        "born_location": soup.select_one(".author-born-location").text,
        "description": soup.select_one(".author-description").text.strip(),
    }
    seen_authors[author_url] = bio
    return bio

seen_authors is a dict keyed by URL, passed in and mutated across the whole crawl — checking it first turns "fetch this author's page again for the fifth quote of theirs" into a dict lookup instead of a fifth identical HTTP request. On a site with a handful of quotes per author, this alone can cut total requests by more than half.

Next: this crawls fast and eagerly — part 5 slows it down deliberately, and makes it survive a request that fails partway through.