Four out of thirty

What happened and why it matters

I wrote something to read a shop's catalogue and build a product grid from it. It ran, it produced clean output, nothing went wrong, and it found four products. The shop has thirty.

The code was not broken. It was pointed at the wrong page, and the wrong page looked exactly like the right one.

Why the page did not contain the page

The site is the modern kind. The shop's computer sends a shell and a small featured selection, and the rest arrives afterwards when the page runs its own code inside your browser. Asking for the address directly gives you only what was sent, which was four items. A browser would then have filled in the other twenty six. My program never ran that step, because fetching an address is not the same as being a browser, so it took the four and reported success.

The failure mode is the thing to carry away

The specific cause is interesting, but what matters is the shape: a result that is plausible, well formed, consistent with itself, and simply incomplete. There is nothing to catch. No error appears. The only thing that exposes it is knowing, independently, roughly how many things there should be.

Where the catalogue really lived

The paginated listing turned out to be a dead end, because its page control was built in a way the shop's computer never honours, so every variation of the address returned the same first batch.

What did work were the category pages. Each of those was sent complete. The combination of all of them, with duplicates removed, was the catalogue.

Then I did the thing I should always do and had not. I counted it a second, independent way, and then a third. The listing's own page control implied a total. The site's own machine readable index implied another. When all three agreed, I believed the number. They did agree, and they disagreed with my first answer by twenty three products.

How to do it yourself

How to find where a page's data really comes from, combine partial sources into a whole, remove duplicates safely, and check a count before trusting it. Everything here only reads. This is about reading public pages you are entitled to read, politely.

1. Work out the expected count from outside your own program

# acceptable sources for "how many should there be"
#   a management screen you own
#   a count the site states itself
#   the site's machine readable index
# write the number down BEFORE you look at your own output

2. Compare what was sent with what a browser shows

# what was actually sent
curl -s '<listing-address>' > sent.html
grep -c '<the-marker-for-one-product>' sent.html

# then count what you see in a real browser on the same page

3. Find the real source in the browser's own tools, not by guessing

# open the browser's developer tools, reload, watch the requests

# and check the obvious indexes while you are there
curl -s '<site>/sitemap.xml' | grep -c '<loc>'

4. Test the page control properly before trusting it

# do these return DIFFERENT items, or the same batch every time?
curl -s '<listing>?page=2'    | grep -c '<product-marker>'
curl -s '<listing>?offset=24' | grep -c '<product-marker>'

5. Combine the partial sources, removing duplicates on a real identifier

found = {}
for source in [listing] + category_pages:
    for item in parse(fetch(source)):
        found[item.id] = item      # keyed on id, so dupes collapse

# log the walk
#   /listing        24 items, 24 new, running 24
#   /category/one    1 item,   0 new, running 24
#   /category/two   21 items,  2 new, running 26

6. Check the count two independent ways, then a third

# A. everything you combined, duplicates removed
# B. what the page control implies (batches x items per batch)
# C. the machine readable index, plus anything it is missing
# all three agree -> trust it

7. Put a guard on the count, and make it refuse

if new_count < stored_count and not override_requested:
    refuse()     # keep the good data, fail, and print both numbers

Related reading: the price a program reports is not always the price is the same distrust pointed at a single field instead of a whole page.