Scrape Product Prices with Python from JSON-LD
To scrape product prices with Python, fetch the page with `requests`, parse every `<script type="application/ld+json">` tag with `json.loads`, find the schema.org `Product` nodes, and read `offers`: `price`, `priceCurrency` and `availability`. Handle the real-world shapes: `@graph` arrays, lists of offers, `AggregateOffer` (`lowPrice`/`highPrice`), prices inside `priceSpecification`, and `ProductGroup` variants.
Below is a script that handles all of these, its output on live stores on 2026-10-20, the stores where the same approach returned nothing that day, and how E-commerce Product Scraper Pro handles the cases plain JSON-LD can't.
## Why JSON-LD
Many stores embed schema.org product data in their HTML for search engines, usually as JSON-LD. It's machine-readable data the store publishes on purpose, so it changes less often than the page's CSS classes. The fields you need are defined by schema.org:
* Offer: `price`, `priceCurrency`, `availability` (a URL such as `https://schema.org/InStock`), often `sku` and `url`.
* AggregateOffer: used "when a single product is associated with multiple offers", with `lowPrice`, `highPrice` and `offerCount`.
* ProductGroup: "a group of Products that vary only in certain well-described ways, such as by size, color", with the variants in `hasVariant`.
## The shapes you'll actually meet
Every row below is a live page I parsed on 2026-10-20:
Store (platform) | Where the price was
---|---
IKEA US | A plain `Offer` with `"price": "79"` as a string, `priceCurrency` alongside
Marks & Spencer | No top-level `price`: it's in `offers.priceSpecification` (a `UnitPriceSpecification` with `price`, `minPrice`, `maxPrice`, `priceCurrency`)
Nalgene (WooCommerce) | Inside `@graph`; a `ProductGroup` whose `hasVariant` items each have an offer; `priceSpecification` serialised as an object with a `"0"` key holding the currency
Death Wish Coffee (Shopify) | One `Product` with a list of `Offer`s, one per variant, each with its own `sku` and price
Skullcandy (Shopify) | A `ProductGroup` with `hasVariant`, plus a separate `Product` for the default variant
A parser that only reads `data["offers"]["price"]` gets two of these five. The script walks the whole document instead.
## The script
"""Extract product prices from schema.org JSON-LD. pip install requests beautifulsoup4"""
import json
import sys
import requests
from bs4 import BeautifulSoup
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/141.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
}
def walk(node):
"""Yield every dict in a JSON-LD document: top-level lists, @graph, nested values."""
if isinstance(node, list):
for item in node:
yield from walk(item)
elif isinstance(node, dict):
yield node
for value in node.values():
if isinstance(value, (dict, list)):
yield from walk(value)
def has_type(node, name):
t = node.get("@type")
return any(str(x).rsplit("/", 1)[-1] == name for x in (t if isinstance(t, list) else [t]) if x)
def number(value):
try:
return float(str(value).replace(",", "")) if value not in (None, "") else None
except ValueError:
return None
def price_spec(spec):
"""priceSpecification can be a dict, a list, or (WooCommerce) a dict with a "0" key."""
specs = spec if isinstance(spec, list) else [spec]
for s in specs:
if not isinstance(s, dict):
continue
nested = s.get("0")
price = s.get("price", nested.get("price") if isinstance(nested, dict) else None)
currency = s.get("priceCurrency") or (nested.get("priceCurrency") if isinstance(nested, dict) else None)
if price is not None:
return number(price), currency
return None, None
def offers(product):
"""One dict per offer: handles Offer, a list of Offers and AggregateOffer."""
raw = product.get("offers") or []
out = []
for o in raw if isinstance(raw, list) else [raw]:
if not isinstance(o, dict):
continue
if has_type(o, "AggregateOffer") and o.get("offers"):
out += offers({"offers": o["offers"]}) # real offers nested inside
continue
price, currency = number(o.get("price")), o.get("priceCurrency")
if has_type(o, "AggregateOffer"):
price = number(o.get("lowPrice"))
if price is None and "priceSpecification" in o:
price, spec_currency = price_spec(o["priceSpecification"])
currency = currency or spec_currency
out.append({
"price": price,
"high_price": number(o.get("highPrice")), # AggregateOffer only
"currency": currency,
"availability": str(o.get("availability") or "").rsplit("/", 1)[-1] or None,
"sku": o.get("sku") or product.get("sku"),
"url": o.get("url"),
})
return out
def extract(html):
"""All (product name, offer) pairs in the page's JSON-LD, de-duplicated."""
soup = BeautifulSoup(html, "html.parser")
rows, seen = [], set()
for tag in soup.find_all("script", type="application/ld+json"):
try:
doc = json.loads(tag.string or "")
except json.JSONDecodeError:
continue # malformed JSON-LD is common; skip it
for node in walk(doc): # also reaches ProductGroup.hasVariant
if not has_type(node, "Product"):
continue
for o in offers(node):
key = (o["sku"], o["price"], o["url"])
if o["price"] is not None and key not in seen:
seen.add(key)
rows.append({"name": node.get("name"), **o})
return rows
def fetch(url):
try:
r = requests.get(url, headers=HEADERS, timeout=20)
except requests.RequestException as e:
return None, f"request failed: {type(e).__name__}"
return r, f"HTTP {r.status_code}"
if __name__ == "__main__":
for url in sys.argv[1:]:
r, status = fetch(url)
rows = extract(r.text) if r is not None and r.ok else []
print(f"\n{url}\n {status}, {len(rows)} offer(s) in JSON-LD")
for row in rows[:3]:
print(f" {row['name'][:45]!r} | {row['sku']} | {row['price']} {row['currency']} | {row['availability']}")
Design notes:
* **`walk()` visits every dict**, so it finds `Product` nodes at the top level, in a list, inside `@graph`, and inside `ProductGroup.hasVariant` without special cases. The `seen` set removes the duplicates that this produces when a page describes the same variant twice (Skullcandy does).
* **`has_type()` accepts lists and full URLs**, because `@type` can be `["Product", "Thing"]` or `"http://schema.org/Product"`.
* **`number()` handles strings**, because `"79"` and `79` are both common.
* **Bad JSON is skipped, not fatal.** Some stores publish invalid JSON-LD.
* **A browser-like`User-Agent`** is sent because several stores answer the default `python-requests` agent differently. It doesn't get you past real bot protection (see below).
Run it with Python 3.12 (`pip install requests beautifulsoup4`):
python product_prices.py https://www.ikea.com/us/en/p/billy-bookcase-white-20522046/ https://www.marksandspencer.com/fair-isle-crew-neck-relaxed-jumper-with-wool/p/clp61248839 https://nalgene.com/product/32oz-wide-mouth-surrealist-print-bottles/ https://www.deathwishcoffee.com/products/power-surge-coffee https://www.skullcandy.com/products/crusher-540-active
Output on 2026-10-20:
https://www.ikea.com/us/en/p/billy-bookcase-white-20522046/
HTTP 200, 1 offer(s) in JSON-LD
'BILLY Bookcase - white 31 1/2x11x79 1/2 "' | 205.220.46 | 79.0 USD | InStock
https://www.marksandspencer.com/fair-isle-crew-neck-relaxed-jumper-with-wool/p/clp61248839
HTTP 200, 1 offer(s) in JSON-LD
'Fair Isle Crew Neck Relaxed Jumper' | P61248839 | 34.0 GBP | InStock
https://nalgene.com/product/32oz-wide-mouth-surrealist-print-bottles/
HTTP 200, 3 offer(s) in JSON-LD
'32oz Wide Mouth Surrealist Bottles - landscap' | 682024-0052 | 18.99 USD | None
'32oz Wide Mouth Surrealist Bottles - hummingb' | 682024-0051 | 18.99 USD | None
'32oz Wide Mouth Surrealist Bottles - astro-mo' | 682024-0050 | 18.99 USD | None
https://www.deathwishcoffee.com/products/power-surge-coffee
HTTP 200, 8 offer(s) in JSON-LD
'Power Surge Coffee' | LGPWSR16 | 19.99 USD | InStock
'Power Surge Coffee' | LGPWSR32K | 39.98 USD | InStock
'Power Surge Coffee' | LGPWSR48K | 59.97 USD | InStock
https://www.skullcandy.com/products/crusher-540-active
HTTP 200, 6 offer(s) in JSON-LD
'Crusher® 540 Active - Skullcandy X Baker Skat' | S6EVW-U054 | 229.99 USD | InStock
'Crusher® 540 Active - Smoke' | S6EVW-T013 | 209.99 USD | InStock
'Crusher® 540 Active - Forest' | S6EVW-T016 | 209.99 USD | InStock
Five stores, five shapes, one parser. Nalgene's variant offers carry no `availability`, so it prints `None`. That is "unknown", not "out of stock".
### Testing the AggregateOffer branch
None of the pages I tried on 2026-10-20 used `AggregateOffer`, so the branch is tested on a small inline document shaped like schema.org's definition, together with a broken tag:
from product_prices import extract
html = """
<script type="application/ld+json">
{"@context": "https://schema.org", "@type": "Product", "name": "Demo kettle", "sku": "K-1",
"offers": {"@type": "AggregateOffer", "lowPrice": "39.99", "highPrice": "54.00",
"offerCount": 4, "priceCurrency": "USD"}}
</script>
<script type="application/ld+json">{"@type": "Product", "name": "Broken", </script>
"""
for row in extract(html):
print(row)
{'name': 'Demo kettle', 'price': 39.99, 'high_price': 54.0, 'currency': 'USD', 'availability': None, 'sku': 'K-1', 'url': None}
An `AggregateOffer` is a price range, not a price. Store `lowPrice` and `highPrice` separately and don't compare a range with a single price in a price-change alert.
## Where it breaks
The same script, same day, same connection (one home connection outside the US; results depend on where and when you run it):
https://www.target.com/p/lego-marvel-spider-man-vs-anti-venom-heist-30725/-/A-94828796
HTTP 200, 0 offer(s) in JSON-LD
https://www.amazon.com/dp/B088NRLMPV
HTTP 200, 0 offer(s) in JSON-LD
https://www.sephora.com/product/peptide-glazing-fluid-P517531
HTTP 403, 0 offer(s) in JSON-LD
https://www.zalando.de/puma-club-ii-era-unisex-sneaker-low-whiteblackvapor-gray-pu115o0sb-a11.html
HTTP 403, 0 offer(s) in JSON-LD
Each failure is a different problem:
1. **The data is in the app's state, not JSON-LD.** Target's and Walmart's product pages returned HTTP 200 with the product title, but no Product JSON-LD in what `requests` received. Both pages contained a `__NEXT_DATA__` script (Target also `__TGT_DATA__`): the page is built from a JavaScript state object. You'd parse that object per site, or render the page.
2. **The store doesn't use JSON-LD for the price.** Amazon's page returned 200 with the price in its HTML (`a-price` elements), not in JSON-LD. That needs site-specific parsing.
3. **Bot protection.** Sephora and Zalando answered 403; John Lewis and Waitrose didn't answer at all within 20 seconds (`ReadTimeout`) in earlier tries. A different User-Agent doesn't fix this; it needs other network routes or a real browser, and respect for the site's terms.
4. **Variants.** Some pages describe only the selected variant. Nalgene and Death Wish publish every variant; other stores don't, and then sizes and colours need the store's own product data.
5. **Pagination and discovery.** JSON-LD describes one product page. To cover a category you need its product links first, through page parameters, "load more" APIs or the store's search, and a rule for when you've seen everything.
## Scaling up: the same pages through the actor
The actor starts with the same structured data and adds what the script lacks: per-site adapters (Shopify and WooCommerce product APIs, Walmart's page data, Amazon's HTML), a route ladder that moves from datacenter to a crawler User-Agent, residential proxies or a browser only when a page is blocked, variants as `variants[]`, pagination up to 200 pages per listing where the site supports it, and one status row per input explaining any failure. The README's supported-sites matrix says what works where; Sephora product pages are best-effort through a browser, and eBay, Best Buy, Home Depot, Costco and Etsy are unsupported.
"""Same pages through the Apify actor. pip install apify-client"""
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("constructive_calm/ecommerce-product-scraper-pro").call(
run_input={
"detailsUrls": [
{"url": "https://www.marksandspencer.com/fair-isle-crew-neck-relaxed-jumper-with-wool/p/clp61248839"},
{"url": "https://nalgene.com/product/32oz-wide-mouth-surrealist-print-bottles/"},
{"url": "https://www.target.com/p/lego-marvel-spider-man-vs-anti-venom-heist-30725/-/A-94828796"},
{"url": "https://www.amazon.com/dp/B088NRLMPV"},
],
"maxProductResults": 4,
},
max_total_charge_usd=0.10, # hard cap on what this run may cost
)
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
print(f"{row['site']:<16} {row['price']:>7} {row['currency']} inStock={row['inStock']}"
f" variants={row['variantCount']:<2} tier={row['tierUsed']:<11} from={'+'.join(row['dataSources'])}")
Output on 2026-10-20 (the client also streams the run's log while it waits):
marksandspencer 34 GBP inStock=True variants=0 tier=datacenter from=jsonld+meta
target 4.99 USD inStock=True variants=0 tier=crawler-ua from=jsonld+meta
amazon-us 9.99 USD inStock=True variants=12 tier=datacenter from=amazon-html
nalgene.com 18.99 USD inStock=True variants=3 tier=datacenter from=jsonld+woo-api
`dataSources` shows where each row came from. Target delivered through the crawler User-Agent route, where its page includes JSON-LD. Amazon came from its HTML. Nalgene combined JSON-LD with the WooCommerce Store API, and its row has a stock flag (`inStock=True`) where the page's JSON-LD had none.
Cost: the actor is pay per event, and these four products are one start plus four product rows on datacenter routes (no traffic fee): 4 × $0.0048 + $0.005 = **$0.0242** on Apify's FREE tier, 4 × $0.0008 + $0.005 = **$0.0082** on GOLD, from the README's pricing table. Residential and browser sites add per-MB and per-render fees. Apify's free plan gives $5 of usage a month with no credit card required; there's no in-actor free trial.
## During Black Friday and the holiday season
The script is enough to start a baseline for a handful of stores that serve JSON-LD: run it once a day from cron or Task Scheduler and append the rows to a CSV, beginning now, about five weeks before Black Friday (Friday 27 November 2026). In the sale window, prices change several times a day, so increase the frequency for the few products you care about and keep the daily readings for medians. Compare the sale price with your own 30-day median, not with the store's "was" price: how to spot fake Black Friday discounts shows the method, and the holiday sales calendar lists when each event's baseline should start.
Related: scraping product variants, Shopify products.json pagination, competitor price monitoring, and the e-commerce price tracker query builder. Respect each site's terms and robots rules, and keep request rates low.
_Disclosure: I built the actor and this site. The JSON-LD script above works on its own, without either._
Sources: schema.org Offer, schema.org AggregateOffer, schema.org ProductGroup, E-commerce Product Scraper Pro README (supported sites, pricing), and live requests to the listed product pages on 2026-10-20.