The price in the data was not the price
What you will learn
- Why a field called price is not always the price
- How to decide which source of a number counts as the truth, before you write anything
- How to make a disagreement about money stop a build instead of filling a log
I was building a product grid for a shop. The platform holding the products offers a proper way for programs to ask about them, and one of the things it reports is called price. I took the price from the price field, which is the single most reasonable thing a person can do, and it was wrong on a third of the catalogue.
Not slightly wrong. Wrong in the direction that matters: too low. One item would have been advertised at thirteen seventy five while the checkout charged eighteen sixty. Nine out of twenty seven products were affected.
The field was real and the number in it was real. It was the cost before the shop adds its own pricing on top. There was a separate cost field sitting right next to it, which is what made me look twice. If price is what the customer pays, why does this also carry a cost, and why is the gap between them not the gap I expected? I opened the public page for that product, and the public page disagreed with the program.
A wholesale invoice and a shelf label both state a price for the same tin. Both are correct. Only one of them is what the person at the till pays.
What saves you is narrower than general suspicion
It is not about distrusting these systems in general. It is one specific habit: when money is involved, take the number from the surface the customer actually sees, and treat anything else as a hint.
The public page is not more trustworthy in any technical sense. It is simply the thing that is true by definition, because it is what the buyer reads and what the checkout honours.
These systems are authoritative about their own model of the world. They are not automatically authoritative about your question. The field was not lying. I had asked it something it was never answering.
What I did with the two numbers
I kept both. The customer facing price is what gets published, and the other number sits beside it in the data as evidence, so the next person who wonders why they differ can see that somebody already checked.
Only one of them is what the person at the till pays.
How to cross check a catalogue against the live shop, and the general pattern for checking data that looks authoritative. This applies to any connection where the number has consequences.
1. Write down which surface is the truth, before writing any code
What this does: names the winning source for every field, one sentence each.
Why it comes first: it takes two minutes and it is the decision that prevents this whole class of problem.
# sources.md
# price -> the public product page. the customer pays this.
# title -> the platform. it is the official name.
# link -> the platform, but only the field that is a real web address
# image -> the platform, highest quality available
# stock -> ? decide, and say whyWhere a field exists in two places, name the winner on purpose rather than taking whichever is convenient on the day.
2. Read one whole record, with your eyes
What this does: prints a single complete product record, formatted, so you can look at every field it carries.
What to look for: on every field about money, ask whether there is another field nearby that could also be called a price. If there is, you do not yet know which one you want.
# print one record and actually read it
curl -s -H "Authorization: Bearer $TOKEN" <the-address> |
python3 -m json.tool | head -603. Compare against the public page for a few products, by hand
What this does: picks three products, opens their public pages, and compares the numbers.
Why by hand: five minutes of looking is what caught mine, and no amount of code would have raised a question.
What to do with a difference: if they disagree on even one product, stop and find out why before building anything on top.
# what the platform reports
# product A: price field = .......
# what the public page shows
curl -s '<public-product-page>' | grep -oE '<whatever marks the price there>'4. Take the real number from the page you are already fetching
What this does: reads the price out of the same response you were already collecting to check that the link works.
Why it is free: one request, two results. The correct number costs you nothing extra.
Record which source each number came from. If one fell back to the platform figure, you know to treat it as unverified, and you can audit later without working it all out again.
status, body = fetch(product_page)
if status != 200: leave_this_product_out()
price = price_from_the_page(body) or fall_back_to_platform_price()
# and store which one won
# price_source: "shop page" | "platform fallback"5. Keep the disagreement in the data, not in your memory
What this does: stores both numbers side by side.
Why: a future reader who finds them different has the evidence that it was deliberate, rather than discovering a mystery.
# per product
# price : what we publish (from the shop page)
# other_price : what the platform reported
# price_source : which surface won6. Make a mismatch stop the build, not fill a log
What this does: checks the two sources against each other on every single run, and fails outright on any difference.
Why not a warning: money is the category where a warning nobody reads is the same as having no check at all.
# in the build, or the nightly refresh
# for each product: does the published price match the shop page?
# any mismatch -> fail, do not publish, and name the products7. Check every outbound link before publishing it
What this does: requests every link you are about to publish, not a sample, and leaves out anything that does not answer properly.
Why every one: a catalogue whose buy links fail is worse than one product being missing, and you are already in the loop.
And one trap to avoid: never build a web address out of an identifier unless you have confirmed that identifier belongs to the public facing namespace. Internal and public identifiers often look similar and are not the same, so an address you assembled yourself can look perfectly valid and lead nowhere.
# request it, require a proper answer, exclude anything elseThis makes your page agree with the shop at the moment you build it. Prices change on the platform without telling you. The real guarantee comes from how often you re-run the check, not from the check itself.
Related reading: the scrape that found four of thirty is the same distrust applied to how much of a catalogue you are actually seeing.
This page in the original Kyber Cypher voice: The API price is not the price