Web scraping with an offline fixture

Understand the concept

Separate retrieval from extraction

Web scraping often combines obtaining a document and extracting information from it. These are separate responsibilities. A network request can fail, a server can return unexpected content, and the document structure can change. An extractor should therefore be testable against saved HTML without requiring a live website for every test.

Today you will parse an offline fixture with Beautiful Soup. This makes the input inspectable, repeatable, and independent of connectivity. The fixture is intentionally small: three study-session cards, including one incomplete card. Success means reporting both usable records and the reason one card could not become a record.

The exercise does not fetch a website. If you later add retrieval, use a source you are authorized to access, follow its access conditions, bound requests, and handle response status and content before parsing. A parser does not establish permission or make dynamic browser content appear in a response. Some websites build their visible page with JavaScript after the original HTML arrives.

Select within the correct container

Beautiful Soup is a third-party package. Install it in your chosen Python environment with python -m pip install beautifulsoup4==4.14.3; the import name is bs4. The examples here were checked with that version and Python's built-in html.parser backend. Different parser backends can produce different trees for malformed markup, so select the parser explicitly.

select returns matches for a CSS selector. select_one returns the first match or None. First select each session container, then select its topic and duration inside that container. Selecting the first duration from the entire page for every card would silently assign the same duration to unrelated sessions.

get_text(strip=True) extracts text with surrounding whitespace stripped from its text fragments. An element's get method retrieves an attribute, returning None when it is absent unless a default is provided. A missing element, a missing attribute, and a valid text value of zero are distinct cases. Keep them distinct in your audit.

Extraction still needs a data contract

HTML parsing identifies structure; it does not prove that extracted text is a valid integer or a unique ID. Our fixture uses a data-minutes attribute so the displayed label is not part of the numeric conversion contract. We require a nonempty ID, a nonempty topic, and whole minutes from 0 through 240. Duplicate IDs are rejected after the first accepted record.

Record the card number and reason for an exclusion. Do not return zero when duration is missing: that would manufacture a measurement. Reconcile the number of selected cards with the number accepted plus the number rejected. This invariant exposes silent omissions and turns a scraping exercise into an auditable data pipeline.

Worked example

from bs4 import BeautifulSoup

html = '''<main>
<article class="session" data-id="S01"><h2>Functions</h2><span data-minutes="30">30 min</span></article>
<article class="session" data-id="S02"><h2>Review</h2><span data-minutes="0">0 min</span></article>
<article class="session" data-id="S03"><h2>Files</h2></article>
</main>'''
soup = BeautifulSoup(html, "html.parser")
cards = soup.select("article.session")
assert len(cards) == 3
for card in cards:
    title = card.select_one("h2")
    duration = card.select_one("[data-minutes]")
    print(card.get("data-id"), title.get_text(strip=True),
          None if duration is None else duration.get("data-minutes"))

The worked example exposes raw values rather than hiding problems. You can now decide the policy at the validation step. Notice that select_one is scoped to card; the extraction does not depend on the topic or duration occupying a particular global position.

Guided practice

1. Predict: replace a card's h2 with h3. Determine which selector no longer matches and what select_one returns. Handle the result before calling get_text.
2. Modify: add unrelated header text and another duration outside the cards. The extracted records should remain unchanged because their selectors are scoped.
3. Reconstruct: create four cards with one duplicate ID and one invalid numeric value. Write down expected accepted and rejected counts before running the extractor.

Inspect saved fixtures whenever a selector starts returning no matches. A zero-record result can mean a genuinely empty page or a broken selector; your production contract must distinguish those possibilities. Today an empty document is permitted and returns two empty lists, which keeps the function easy to test.

Daily challenge

Implement extract_sessions(html) returning (records, rejected). Records contain id, topic, and minutes. Rejected entries contain card number and reason. Accept only ASCII digit duration text whose numeric value is at most 240. Keep the first valid occurrence of an ID and reject subsequent valid duplicates. A rejected card does not reserve its ID. Preserve source order and reconcile counts.

Build your fixture before implementing the parser. That forces you to define what a passing extraction looks like instead of adjusting expectations to whatever the first selector happens to return.

Reference solution

from bs4 import BeautifulSoup

def extract_sessions(html):
    cards = BeautifulSoup(html, "html.parser").select("article.session")
    records, rejected, seen = [], [], set()
    for number, card in enumerate(cards, start=1):
        session_id = (card.get("data-id") or "").strip()
        heading = card.select_one("h2")
        duration = card.select_one("[data-minutes]")
        topic = "" if heading is None else heading.get_text(strip=True)
        text = None if duration is None else duration.get("data-minutes")
        numeric = "0" if text is None else (text.lstrip("0") or "0")
        reason = None
        if not session_id or not topic:
            reason = "missing ID or topic"
        elif text is None:
            reason = "missing duration"
        elif not text or any(ch not in "0123456789" for ch in text):
            reason = "duration must be ASCII whole minutes"
        elif len(numeric) > 3 or int(numeric) > 240:
            reason = "duration exceeds 240"
        elif session_id in seen:
            reason = "duplicate accepted ID"
        if reason:
            rejected.append({"card": number, "reason": reason})
        else:
            seen.add(session_id)
            records.append({"id": session_id, "topic": topic, "minutes": int(numeric)})
    assert len(cards) == len(records) + len(rejected)
    return records, rejected

html = '''<article class="session" data-id="S01"><h2>Functions</h2><span data-minutes="30"></span></article>
<article class="session" data-id="S02"><h2>Review</h2><span data-minutes="0"></span></article>
<article class="session" data-id="S03"><h2>Files</h2></article>
<article class="session" data-id="S01"><h2>Duplicate</h2><span data-minutes="10"></span></article>
<article class="session" data-id="S04"><h2>Invalid</h2><span data-minutes="3.5"></span></article>'''
records, rejected = extract_sessions(html)
assert records == [{"id": "S01", "topic": "Functions", "minutes": 30},
                   {"id": "S02", "topic": "Review", "minutes": 0}]
assert [item["card"] for item in rejected] == [3, 4, 5]
assert extract_sessions("<main></main>") == ([], [])
large = '<article class="session" data-id="L"><h2>Long</h2><span data-minutes="' + "9" * 5000 + '"></span></article>'
assert len(extract_sessions(large)[1]) == 1
zeros = '<article class="session" data-id="Z"><h2>Zero</h2><span data-minutes="' + "0" * 5000 + '"></span></article>'
assert extract_sessions(zeros)[0][0]["minutes"] == 0
print(records)
print(rejected)

Debug and explain

If every card receives one duration, look for a selector called on soup instead of card. If an AttributeError appears, check for None before reading an element. If zero disappears, check whether you confused the string value with a missing value. Explain which failures belong to retrieval, tree selection, numeric validation, and duplicate handling.

Review and next step

Keep both HTML fixtures and expected extraction results. Record the parser backend with your tests so future changes are interpretable. Day 27 combines validated records, stable IDs, and JSON persistence into a complete to-do project.