Competitor Price Monitoring with Google Shopping Results
Michael Carter · · 26 min read

Key takeaways
- In 30 Google Shopping checks of 8 products on November 17 and 18, 2026, only 142 of 1,022 offers (14%) were the product itself, sold new by a store; the rest were marketplace listings, other variants, accessories or other products.
- Google Shopping results show no GTIN, so match offers on the model number at the start of a word, or on the brand plus required words, and exclude other variants, bundles and accessories.
- Between checks minutes or hours apart, half or more of Google's offer list changed, but only 1 of 57 store prices for the same listing changed, by 67 cents.
- All 11 moves of the lowest matched price came from a store entering or leaving the list, so compare same-title listings only and confirm a new low on a second check before acting.
- Checking 500 products once a day takes 15,000 searches a month, $7.50 at BestSERP's $0.50 per 1,000 searches; checking them every hour costs $180.
Competitor price monitoring is checking, on a schedule, what other stores charge for the products you sell, so you can react when they change. Google Shopping results make it practical: one request returns offers from many stores for one product. Match those offers to your product, store each price with a timestamp, and alert when a competitor undercuts you.
This tutorial is for developers and small e-commerce teams who want their own price tracker instead of a subscription tool. You will build a Python program that checks eight real products through BestSERP, a Google SERP API, matches offers by model number and title, keeps price history in SQLite and raises alerts. Every output below comes from runs on November 17 and 18, 2026.
What competitor price monitoring with Google Shopping gives you
One search shows who sells a product and at what price. The /shopping endpoint returns that list as JSON: in our checks, 14 to 40 offers per request, each with a title, the store (source), the price as text and as a number (priceValue), and often an old price, a delivery note and a rating. Fill rates for each field are in our developer guide to Google Shopping, Places and Maps data.
The price monitoring use case page outlines a monitor; this post builds one on real data, which taught us three things:
- Most offers are not the product you sell. Of the 1,022 offers our checks returned, 142 (14%) survived matching.
- The list of stores changes more than the prices. Only half of the offers were still listed a few minutes later.
- Store prices barely moved. In 57 comparisons of the same listing at the same store, one price changed, by 67 cents. Each of the 11 moves of the lowest price came from a store appearing or disappearing.
Each run checks every product, keeps the offers that really are that product, stores one price per store in SQLite and compares it with the previous check. The November 18 run printed:

What you need
- Python 3.9 or newer (we used 3.12.10) and
requests(2.34.2). SQLite ships with Python (3.49.1 here). - A BestSERP API key in
BESTSERP_API_KEY. A new account gets 3,000 free searches; the quickstart shows the first request. - Optional: a Slack incoming webhook in
SLACK_WEBHOOK_URL. - A product list with model numbers. Ours has eight products that each fail to match in their own way:
| Product | Search query | Matched on |
|---|---|---|
| Breville Barista Express | Breville Barista Express BES870XL | Model BES870, or "barista express" |
| Samsung 990 PRO 2TB | Samsung 990 PRO 2TB MZ-V9P2T0B | Model MZ-V9P2T0B, or "990 pro" + "2tb" |
| KitchenAid Artisan mixer | KitchenAid Artisan KSM150PS stand mixer | Model KSM150PS only |
| Shark Navigator Lift-Away | Shark Navigator Lift-Away NV352 | Model NV352 only |
| Fujifilm Instax Mini 12 | Fujifilm Instax Mini 12 instant camera | "instax mini 12" |
| Garmin Forerunner 265 | Garmin Forerunner 265 | "forerunner 265", not "265s" |
| GoPro HERO13 Black | GoPro HERO13 Black | "hero13" + "black" |
| Logitech MX Master 3S | Logitech MX Master 3S | "mx master 3s", not "for mac" |
We replaced two products after the first run: all 39 offers for LEGO set 10316 were minifigures, display cases or look-alikes, and one of 33 offers for the Ninja AF101 air fryer carried its model number.
How to match the same product across stores
Match on the model number when a title carries one, on the brand plus required words when it does not, and drop titles that name another variant, a bundle or an accessory. Similar titles are not enough. One request for the Breville returned 39 offers, and only 7 were stores selling that machine new:

Why there is no GTIN to match on
A GTIN (Global Trade Item Number) is a product's barcode number: a UPC (GTIN-12) in North America, an EAN (GTIN-13) in Europe, a JAN in Japan or an ISBN for books, as Google Merchant Center's help on the GTIN attribute lists them (checked 2026-11-18). The page warns that "accurate matching to products can't be assured when GTIN is missing."
The Shopping results page does not print GTINs, so /shopping has no GTIN field (and no link to the store's product page). Match on titles, and keep your own GTINs as the internal key next to each model number.
The matching rules, in order
- Skip marketplace sellers such as eBay, Mercari, Poshmark and Alibaba, where most listings are used items or one-off sales. Keep them if those sellers are your real competitors.
- Drop used, refurbished, open-box, parts, rental and "compatible" listings.
- Drop each product's exclusions: other variants ("impress", "265s", "heatsink"), bundles, kits, accessories, and titles containing "for" plus the brand ("tamper for Breville").
- Accept a model number only at the start of a word, ignoring spaces and hyphens: KSM150PS matches "KSM150PSER" (a color code) but not "5KSM150PS", another model number. Without a model number, require the brand and every required word.
- Keep prices between half and double the median store price, one vote per store, so one store listing seven colors cannot drag the median down. This rule dropped a $22.99 GoPro HERO13 at Rent-A-Center.
- Keep the lowest price per store.
The rules live in products.json, not in the code. We changed them twice after reading real results; because every response is stored, one command matched the earlier checks again. Where all 1,022 offers went:

Step 1: Fetch Shopping results
Each product in products.json holds the query, the country and the matching rules; the optional my_price is your own price (two products have an example one), for an alert when a competitor is cheaper.
[
{"id": "breville-bes870", "name": "Breville Barista Express BES870XL", "query": "Breville Barista Express BES870XL",
"gl": "us", "brand": "breville", "models": ["BES870"], "words": ["barista express"],
"exclude": ["impress", "touch"], "my_price": 699.95},
{"id": "kitchenaid-ksm150ps", "name": "KitchenAid Artisan KSM150PS", "query": "KitchenAid Artisan KSM150PS stand mixer",
"gl": "us", "brand": "kitchenaid", "models": ["KSM150PS"], "words": [],
"exclude": ["bundle", "attachment", "bowl only", "cover"], "my_price": null}
]
fetch_shopping sends one /shopping request per product with noCache: true: BestSERP answers identical requests within an hour from cache at the normal price, so a fresh page costs nothing extra. It polls when a search answers 202, waits on 429 (never charged) and retries 502 and 503, which are refunded:
def fetch_shopping(q, gl="us", hl="en"):
"""One /shopping request; retries the cases that are never charged (429) or refunded (502, 503)."""
headers = {"X-API-KEY": os.environ["BESTSERP_API_KEY"]}
body = {"q": q, "gl": gl, "hl": hl, "noCache": True} # a price check wants a fresh page
for attempt in range(4):
resp = requests.post(f"{API}/shopping", headers=headers, json=body, timeout=130)
if resp.status_code == 202: # still running: poll the request ID
request_id = resp.json()["requestId"]
while resp.status_code == 202:
time.sleep(3)
resp = requests.get(f"{API}/requests/{request_id}", headers=headers, timeout=130)
if resp.status_code == 200:
return resp.json()
if resp.status_code in (429, 502, 503):
time.sleep(int(resp.headers.get("Retry-After", 10)))
continue
resp.raise_for_status() # 400, 401, 402: fix the request instead of retrying
raise RuntimeError(f"/shopping {q!r}: still failing after {attempt + 1} attempts")
durationMs ranged from 2.7 to 18 seconds per request (median 3.9). The /shopping reference lists every parameter, including pages for more offers (one credit per page).
Step 2: Match offers and store price history
The matching code turns each title into lowercase words. squash removes spaces and punctuation so "HERO 13" equals "HERO13", and has_model accepts a model number only at the start of a word:
def normalize(text):
"""Lowercase ASCII words separated by single spaces, with a space at each end."""
text = unicodedata.normalize("NFKD", text).encode("ascii", "ignore").decode().lower()
return " " + re.sub(r"[^a-z0-9]+", " ", text).strip() + " "
def squash(text):
"""Letters and digits only: "MZ-V9P2T0B/AM" and "mz v9p2t0b am" both become "mzv9p2t0bam"."""
return normalize(text).replace(" ", "")
def has_model(words, model):
"""Model number at the start of a word, spaces optional: KSM150PS matches KSM150PSER, not 5KSM150PS."""
return re.search(r"(?<![a-z0-9])" + " ?".join(squash(model)), words) is not None
def match(title, product):
"""Return 'model' or 'words' when the offer is this product, None when it is not."""
words = normalize(title)
if any(normalize(p) in words for p in GLOBAL_EXCLUDE + product["exclude"]):
return None # another variant, a bundle, an accessory or not new
if f" for {product['brand']} " in words:
return None # "Tamper for Breville Barista Express" is an accessory
if any(has_model(words, m) for m in product["models"]):
return "model"
flat = squash(title)
if product["words"] and product["brand"] in flat and all(
any(squash(alt) in flat for alt in w.split("|")) for w in product["words"]):
return "words"
return None
Two tables hold the history: checks, one row per request including the response itself (raw_json), and prices, one row per store and check. Keeping the response makes rules cheap to fix: python price_monitor.py --rematch rebuilds prices without a single new search. Each check also stores ipCountry, the country the search ran from (Step 3 explains why).
SCHEMA = """
CREATE TABLE IF NOT EXISTS checks (
id INTEGER PRIMARY KEY,
product_id TEXT NOT NULL,
checked_at TEXT NOT NULL, -- UTC, ISO 8601
gl TEXT NOT NULL,
ip_country TEXT, -- ipCountry: where the search ran from (missing when unknown)
request_id TEXT,
offers_returned INTEGER,
raw_json TEXT -- the response, so offers can be matched again later
);
CREATE TABLE IF NOT EXISTS prices (
check_id INTEGER NOT NULL REFERENCES checks(id),
product_id TEXT NOT NULL,
merchant TEXT NOT NULL,
price REAL NOT NULL,
currency TEXT,
title TEXT,
original_price TEXT, -- display text, for example "Usually $850"
position INTEGER,
match TEXT -- 'model' or 'words'
);
CREATE INDEX IF NOT EXISTS prices_by_product ON prices(product_id, merchant, check_id);
"""
pick_offers applies the store filter, the matching rules and the price band, then keeps each store's lowest price. The currency is the text before the number in price; store it, because priceValue has none.
def pick_offers(data, product):
"""Matched offers with a price, minus outliers, keeping the lowest price per merchant."""
matched = []
for item in data.get("shopping", []):
if (normalize(item["source"]).split() or [""])[0] in IGNORE_MERCHANTS:
continue # first word of "eBay - seller", "mercari.com"...
kind = match(item["title"], product)
if kind and item.get("priceValue"):
matched.append({**item, "match": kind})
lowest = {}
for o in matched:
lowest[o["source"]] = min(o["priceValue"], lowest.get(o["source"], o["priceValue"]))
if len(lowest) >= 3: # one vote per store; under half or over double the median is rarely a real offer
median = statistics.median(lowest.values())
matched = [o for o in matched if median / 2 <= o["priceValue"] <= median * 2]
best = {}
for o in matched:
if o["source"] not in best or o["priceValue"] < best[o["source"]]["priceValue"]:
best[o["source"]] = o
return list(best.values())
def save_prices(db, check_id, product, data):
rows = [(check_id, product["id"], o["source"], o["priceValue"], re.match(r"\D*", o["price"]).group().strip(),
o["title"], o.get("originalPrice"), o["position"], o["match"]) for o in pick_offers(data, product)]
db.executemany("INSERT INTO prices VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", rows)
def store(db, product, data):
for item in data.get("shopping", []): # merchant icons and thumbnails are not needed
item.pop("sourceIcon", None)
item.pop("imageUrl", None)
cur = db.execute(
"INSERT INTO checks (product_id, checked_at, gl, ip_country, request_id, offers_returned, raw_json)"
" VALUES (?, ?, ?, ?, ?, ?, ?)",
(product["id"], datetime.now(timezone.utc).isoformat(timespec="seconds"), product["gl"],
data.get("ipCountry"), data.get("requestId"), len(data.get("shopping", [])), json.dumps(data)))
save_prices(db, cur.lastrowid, product, data)
db.commit()
return cur.lastrowid
Step 3: Alert on price drops
After each check the program compares every store with its previous price and raises three kinds of alerts:
- PRICE DROP: the same listing at the same store fell by at least 3% (
DROP_ALERT_PCT). - NEW LOW: the product's lowest price is below the previous lowest, with the reason: a price cut, another listing at the same store, or a store that was not in the last check.
- UNDERCUT: the lowest price is below your
my_price.
A store's price counts as changed only when its title is the same as last time. On November 17 a Walmart marketplace seller went from $659 to $669, but those were two listings: the standard machine and the "BES870BSXL" Black Sesame color, a price rise that never happened.
Prices also depend on the country. A search runs from an IP in the gl country when one is free and otherwise from another country, and the response's ipCountry says which; the page may then differ from what local shoppers see. compare keeps such a check but prints a flag instead of alerts, and later checks never use it as their baseline.
def prices_at(db, check_id):
"""{merchant: (price, title)} for one check."""
rows = db.execute("SELECT merchant, price, title FROM prices WHERE check_id = ?", (check_id,))
return {merchant: (price, title) for merchant, price, title in rows}
def compare(db, product, check_id):
"""A summary line for this check, and the alerts it raises."""
name, gl, now = product["name"], product["gl"], prices_at(db, check_id)
returned, ip = db.execute("SELECT offers_returned, ip_country FROM checks WHERE id = ?", (check_id,)).fetchone()
flag = f" | FLAG: searched from a {ip} IP, not {gl}: no alerts" if ip and ip != gl else ""
if not now:
return f"{product['id']}: {returned} offers, none matched: check the query and rules{flag}", []
cur = db.execute("SELECT currency FROM prices WHERE check_id = ? LIMIT 1", (check_id,)).fetchone()[0]
low = min(now, key=lambda m: now[m][0])
low_price = now[low][0]
line = f"{product['id']}: {returned} offers, {len(now)} stores matched, lowest {cur}{low_price:,.2f} at {low}"
if flag: # the page may show another country's stores and prices: keep the check, compare nothing
return line + flag, []
alerts = []
prev = db.execute("SELECT id FROM checks WHERE product_id = ? AND id < ?"
" AND (ip_country IS NULL OR ip_country = gl) ORDER BY id DESC LIMIT 1",
(product["id"], check_id)).fetchone()
if prev and (before := prices_at(db, prev[0])):
# Compare a store's price only when it is the same listing (same title) as last time.
same = [m for m in now if m in before and now[m][1] == before[m][1]]
down = [m for m in same if now[m][0] < before[m][0]]
up = [m for m in same if now[m][0] > before[m][0]]
other = [m for m in now if m in before and m not in same]
line += (f" | vs last check: {len(down)} down, {len(up)} up, {len(same) - len(down) - len(up)} same,"
f" {len(other)} other listing, {len(set(now) - set(before))} new, {len(set(before) - set(now))} gone")
for m in down:
old, new = before[m][0], now[m][0]
if new <= old * (1 - DROP_ALERT_PCT / 100):
alerts.append(f"PRICE DROP {name} at {m}: {cur}{old:,.2f} -> {cur}{new:,.2f} ({new / old - 1:+.1%})")
old_low = min(price for price, _ in before.values())
if low_price < old_low:
why = "price cut" if low in same else "other listing" if low in other else "store not in the last check"
alerts.append(f"NEW LOW {name}: {cur}{low_price:,.2f} at {low}, {why} (lowest was {cur}{old_low:,.2f})")
if product.get("my_price") and low_price < product["my_price"]:
alerts.append(f"UNDERCUT {name}: {low} sells at {cur}{low_price:,.2f}, your price is {cur}{product['my_price']:,.2f}")
return line, alerts
def notify(alerts):
for text in alerts:
print("ALERT", text)
webhook = os.environ.get("SLACK_WEBHOOK_URL") # a Slack incoming webhook; never commit this URL
if webhook and alerts:
requests.post(webhook, json={"text": "\n".join(alerts)}, timeout=30).raise_for_status()
All eight checks on November 18 ran from US IPs. A test with gl set to another country ran from a US IP: the page still listed that country's stores, and the program printed the flag and no alerts.
notify prints the alerts and, when SLACK_WEBHOOK_URL is set, posts them as one message with a text field, the minimum payload in Slack's guide to incoming webhooks. The guide warns that the webhook URL contains a secret: keep it out of your repository. Our runs only printed.
What four runs over two days showed
The lowest price moved often, and never because a store changed its price. Between consecutive checks of the same product:
| Time between checks | Offers listed both times | Same store, same listing | Price changed | Lowest price moved |
|---|---|---|---|---|
| 3 to 6 minutes | 106 of 209 (51%) | 23 of 28 stores | 0 of 23 | 3 of 5 products |
| 2 hours | 129 of 257 (50%) | 23 of 47 | 1 of 23 | 2 of 7 |
| 10 hours | 97 of 250 (39%) | 11 of 34 | 0 of 11 | 6 of 7 |
The one change was 67 cents at a store listing the Breville around $1,117, likely a currency conversion. Across all offers seen twice, matched or not, 17 prices changed, mostly by cents at foreign stores. Every move of the lowest price came from a store entering or leaving the list:

Method. We ran the program at 15:44, 15:48 and 17:50 UTC on November 17, 2026 and at 03:56 UTC on November 18 (eight products, gl: "us", hl: "en", noCache: true). The first run used earlier rules and two products we later replaced; its stored responses were matched again with the final rules and the replaced products left out: 30 checks, analyzed by code/analyze.py. Eight products over 12 hours show how Google's list behaves, not how often prices change over weeks.
How often to check competitor prices
For most competitor price monitoring, check once a day, unless you can act faster than that. A check pays off only when someone, or your repricing script, reacts to it.
Our checks minutes and hours apart mostly recorded Google's list changing, not prices, so checking more often mainly adds noise and cost. Categories differ, though: Alberto Cavallo's NBER working paper on online competition (2018) found that online competition has raised how often large U.S. retailers change prices. Measure your own catalog with change_rate.py:
"""How often did store prices change between checks? Use it to tune how often you check.
python change_rate.py (reads PRICE_DB, default prices.db)
"""
import os
import sqlite3
SQL = """
WITH seq AS (
SELECT id, product_id, LAG(id) OVER (PARTITION BY product_id ORDER BY id) AS prev_id FROM checks
)
SELECT now.product_id, COUNT(*) AS compared, SUM(now.price <> before.price) AS changed
FROM prices now
JOIN seq ON seq.id = now.check_id
JOIN prices before ON before.check_id = seq.prev_id -- the product's previous check
AND before.merchant = now.merchant AND before.title = now.title -- same store, same listing
GROUP BY now.product_id ORDER BY now.product_id
"""
db = sqlite3.connect(os.environ.get("PRICE_DB", "prices.db"))
total = changed_total = 0
for product_id, compared, changed in db.execute(SQL):
print(f"{product_id:<24} {changed} of {compared} store prices changed since the previous check")
total, changed_total = total + compared, changed_total + changed
print(f"{'all products':<24} {changed_total} of {total}")
On our data it printed all products 1 of 57. Then pick a rhythm:
- Daily for most catalogs, at a fixed hour.
- Several times a day only when you reprice automatically and
change_rate.pyshows prices moving within a day. - Confirm before acting: treat a new low as real, or a store as gone, only when two checks in a row agree.
What competitor price monitoring costs for 50, 500 and 5,000 products
Each product check is one /shopping request, and BestSERP charges $0.50 per 1,000 searches after 3,000 free ones (pricing, checked 2026-11-18). For a 30-day month:
| Products | Once a day | Every 6 hours | Every hour |
|---|---|---|---|
| 50 | 1,500 searches, $0.75 | 6,000 searches, $3 | 36,000 searches, $18 |
| 500 | 15,000 searches, $7.50 | 60,000 searches, $30 | 360,000 searches, $180 |
| 5,000 | 150,000 searches, $75 | 600,000 searches, $300 | 3,600,000 searches, $1,800 |
- More offers per product:
pages: 2doubles each check's cost. Add pages only when stores you care about are missing. - More countries: each country is a separate check, so 500 products in 3 countries cost as much as 1,500 in one.
- Failures are free:
502and503answers are refunded. - Speed: at the default 60 searches per minute, 5,000 products take at least 84 minutes, so hourly checks need a higher limit (BestSERP raises it on request). Our program runs one search at a time; for bigger lists, see running thousands of searches reliably.
Limits of Google Shopping data for price monitoring
- Country level only. Targeting is by
glandhl, never by city or store, so regional and in-store prices are out of reach. - Prices as displayed.
priceValueis the item price, delivery is separate text, and neither says which currency. Withgl: "us", foreign stores such as ubuy.com.lb appeared with dollar prices. - Google's offer list, not every merchant. Amazon was not among our 1,094 offers, and big retailers came and went between requests: a missing store has not necessarily stopped selling.
- No GTIN, stock status or product page link, and desktop results only.
Common problems and fixes
| Problem | Cause | Fix |
|---|---|---|
| "none matched" | Titles lack your model number, or the product is no longer sold new | Add required words, or drop the product |
| A very low price from an unknown store | A wrong listing, a rental price or a dubious store | Add the store to IGNORE_MERCHANTS, or keep an allowlist of real competitors |
| A price rise that is not real | Another listing (color or edition) at the same store | Compare same-title listings only, as compare does |
| A competitor disappears | Google's list changed between requests | Never alert on missing stores |
| A line ends with "FLAG: searched from a ... IP" | No IP in the gl country was free |
None needed: the check is kept, without alerts |
Full code
The complete program, as run on November 17 and 18, 2026 with Python 3.12.10 and requests 2.34.2. Save it next to products.json, set BESTSERP_API_KEY and schedule python price_monitor.py with cron or Windows Task Scheduler. --history PRODUCT_ID prints one product's history.
"""Competitor price monitoring with Google Shopping results (BestSERP /shopping).
python price_monitor.py # check every product in products.json, store prices, print alerts
python price_monitor.py --history ID # print the stored price history of one product
python price_monitor.py --rematch # re-run matching on the stored responses after editing the rules
Needs BESTSERP_API_KEY in the environment. Optional: SLACK_WEBHOOK_URL (alerts are also posted to Slack),
PRICE_DB (path of the SQLite file, default prices.db).
Tested 2026-11-17 and 2026-11-18 with Python 3.12.10, requests 2.34.2 and SQLite 3.49.1.
"""
import argparse
import json
import os
import re
import sqlite3
import statistics
import sys
import time
import unicodedata
from datetime import datetime, timezone
from pathlib import Path
import requests
API = "https://api.bestserp.com"
HERE = Path(__file__).resolve().parent
DROP_ALERT_PCT = 3.0 # alert when a store's price falls by at least this much since the last check
GLOBAL_EXCLUDE = ["refurbished", "renewed", "used", "pre owned", "open box", "parts", "repair", "defective",
"rental", "replacement", "compatible"]
# Marketplaces where most listings come from individual sellers (often used or for parts). Remove a name
# here if those sellers are competitors you want to follow.
IGNORE_MERCHANTS = ["ebay", "mercari", "poshmark", "depop", "whatnot", "bonanza", "alibaba", "etsy"]
SCHEMA = """
CREATE TABLE IF NOT EXISTS checks (
id INTEGER PRIMARY KEY,
product_id TEXT NOT NULL,
checked_at TEXT NOT NULL, -- UTC, ISO 8601
gl TEXT NOT NULL,
ip_country TEXT, -- ipCountry: where the search ran from (missing when unknown)
request_id TEXT,
offers_returned INTEGER,
raw_json TEXT -- the response, so offers can be matched again later
);
CREATE TABLE IF NOT EXISTS prices (
check_id INTEGER NOT NULL REFERENCES checks(id),
product_id TEXT NOT NULL,
merchant TEXT NOT NULL,
price REAL NOT NULL,
currency TEXT,
title TEXT,
original_price TEXT, -- display text, for example "Usually $850"
position INTEGER,
match TEXT -- 'model' or 'words'
);
CREATE INDEX IF NOT EXISTS prices_by_product ON prices(product_id, merchant, check_id);
"""
def fetch_shopping(q, gl="us", hl="en"):
"""One /shopping request; retries the cases that are never charged (429) or refunded (502, 503)."""
headers = {"X-API-KEY": os.environ["BESTSERP_API_KEY"]}
body = {"q": q, "gl": gl, "hl": hl, "noCache": True} # a price check wants a fresh page
for attempt in range(4):
resp = requests.post(f"{API}/shopping", headers=headers, json=body, timeout=130)
if resp.status_code == 202: # still running: poll the request ID
request_id = resp.json()["requestId"]
while resp.status_code == 202:
time.sleep(3)
resp = requests.get(f"{API}/requests/{request_id}", headers=headers, timeout=130)
if resp.status_code == 200:
return resp.json()
if resp.status_code in (429, 502, 503):
time.sleep(int(resp.headers.get("Retry-After", 10)))
continue
resp.raise_for_status() # 400, 401, 402: fix the request instead of retrying
raise RuntimeError(f"/shopping {q!r}: still failing after {attempt + 1} attempts")
def normalize(text):
"""Lowercase ASCII words separated by single spaces, with a space at each end."""
text = unicodedata.normalize("NFKD", text).encode("ascii", "ignore").decode().lower()
return " " + re.sub(r"[^a-z0-9]+", " ", text).strip() + " "
def squash(text):
"""Letters and digits only: "MZ-V9P2T0B/AM" and "mz v9p2t0b am" both become "mzv9p2t0bam"."""
return normalize(text).replace(" ", "")
def has_model(words, model):
"""Model number at the start of a word, spaces optional: KSM150PS matches KSM150PSER, not 5KSM150PS."""
return re.search(r"(?<![a-z0-9])" + " ?".join(squash(model)), words) is not None
def match(title, product):
"""Return 'model' or 'words' when the offer is this product, None when it is not."""
words = normalize(title)
if any(normalize(p) in words for p in GLOBAL_EXCLUDE + product["exclude"]):
return None # another variant, a bundle, an accessory or not new
if f" for {product['brand']} " in words:
return None # "Tamper for Breville Barista Express" is an accessory
if any(has_model(words, m) for m in product["models"]):
return "model"
flat = squash(title)
if product["words"] and product["brand"] in flat and all(
any(squash(alt) in flat for alt in w.split("|")) for w in product["words"]):
return "words"
return None
def pick_offers(data, product):
"""Matched offers with a price, minus outliers, keeping the lowest price per merchant."""
matched = []
for item in data.get("shopping", []):
if (normalize(item["source"]).split() or [""])[0] in IGNORE_MERCHANTS:
continue # first word of "eBay - seller", "mercari.com"...
kind = match(item["title"], product)
if kind and item.get("priceValue"):
matched.append({**item, "match": kind})
lowest = {}
for o in matched:
lowest[o["source"]] = min(o["priceValue"], lowest.get(o["source"], o["priceValue"]))
if len(lowest) >= 3: # one vote per store; under half or over double the median is rarely a real offer
median = statistics.median(lowest.values())
matched = [o for o in matched if median / 2 <= o["priceValue"] <= median * 2]
best = {}
for o in matched:
if o["source"] not in best or o["priceValue"] < best[o["source"]]["priceValue"]:
best[o["source"]] = o
return list(best.values())
def save_prices(db, check_id, product, data):
rows = [(check_id, product["id"], o["source"], o["priceValue"], re.match(r"\D*", o["price"]).group().strip(),
o["title"], o.get("originalPrice"), o["position"], o["match"]) for o in pick_offers(data, product)]
db.executemany("INSERT INTO prices VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", rows)
def store(db, product, data):
for item in data.get("shopping", []): # merchant icons and thumbnails are not needed
item.pop("sourceIcon", None)
item.pop("imageUrl", None)
cur = db.execute(
"INSERT INTO checks (product_id, checked_at, gl, ip_country, request_id, offers_returned, raw_json)"
" VALUES (?, ?, ?, ?, ?, ?, ?)",
(product["id"], datetime.now(timezone.utc).isoformat(timespec="seconds"), product["gl"],
data.get("ipCountry"), data.get("requestId"), len(data.get("shopping", [])), json.dumps(data)))
save_prices(db, cur.lastrowid, product, data)
db.commit()
return cur.lastrowid
def prices_at(db, check_id):
"""{merchant: (price, title)} for one check."""
rows = db.execute("SELECT merchant, price, title FROM prices WHERE check_id = ?", (check_id,))
return {merchant: (price, title) for merchant, price, title in rows}
def compare(db, product, check_id):
"""A summary line for this check, and the alerts it raises."""
name, gl, now = product["name"], product["gl"], prices_at(db, check_id)
returned, ip = db.execute("SELECT offers_returned, ip_country FROM checks WHERE id = ?", (check_id,)).fetchone()
flag = f" | FLAG: searched from a {ip} IP, not {gl}: no alerts" if ip and ip != gl else ""
if not now:
return f"{product['id']}: {returned} offers, none matched: check the query and rules{flag}", []
cur = db.execute("SELECT currency FROM prices WHERE check_id = ? LIMIT 1", (check_id,)).fetchone()[0]
low = min(now, key=lambda m: now[m][0])
low_price = now[low][0]
line = f"{product['id']}: {returned} offers, {len(now)} stores matched, lowest {cur}{low_price:,.2f} at {low}"
if flag: # the page may show another country's stores and prices: keep the check, compare nothing
return line + flag, []
alerts = []
prev = db.execute("SELECT id FROM checks WHERE product_id = ? AND id < ?"
" AND (ip_country IS NULL OR ip_country = gl) ORDER BY id DESC LIMIT 1",
(product["id"], check_id)).fetchone()
if prev and (before := prices_at(db, prev[0])):
# Compare a store's price only when it is the same listing (same title) as last time.
same = [m for m in now if m in before and now[m][1] == before[m][1]]
down = [m for m in same if now[m][0] < before[m][0]]
up = [m for m in same if now[m][0] > before[m][0]]
other = [m for m in now if m in before and m not in same]
line += (f" | vs last check: {len(down)} down, {len(up)} up, {len(same) - len(down) - len(up)} same,"
f" {len(other)} other listing, {len(set(now) - set(before))} new, {len(set(before) - set(now))} gone")
for m in down:
old, new = before[m][0], now[m][0]
if new <= old * (1 - DROP_ALERT_PCT / 100):
alerts.append(f"PRICE DROP {name} at {m}: {cur}{old:,.2f} -> {cur}{new:,.2f} ({new / old - 1:+.1%})")
old_low = min(price for price, _ in before.values())
if low_price < old_low:
why = "price cut" if low in same else "other listing" if low in other else "store not in the last check"
alerts.append(f"NEW LOW {name}: {cur}{low_price:,.2f} at {low}, {why} (lowest was {cur}{old_low:,.2f})")
if product.get("my_price") and low_price < product["my_price"]:
alerts.append(f"UNDERCUT {name}: {low} sells at {cur}{low_price:,.2f}, your price is {cur}{product['my_price']:,.2f}")
return line, alerts
def notify(alerts):
for text in alerts:
print("ALERT", text)
webhook = os.environ.get("SLACK_WEBHOOK_URL") # a Slack incoming webhook; never commit this URL
if webhook and alerts:
requests.post(webhook, json={"text": "\n".join(alerts)}, timeout=30).raise_for_status()
def history(db, product_id):
checks = db.execute("SELECT id, checked_at FROM checks WHERE product_id = ? ORDER BY id", (product_id,)).fetchall()
table = {}
for check_id, _ in checks:
for merchant, (price, _) in prices_at(db, check_id).items():
table.setdefault(merchant, {})[check_id] = price
print(f"{'merchant':<32}" + "".join(f"{at[5:16].replace('T', ' '):>14}" for _, at in checks))
for merchant in sorted(table, key=lambda m: min(table[m].values())):
cells = (f"{table[merchant][c]:>14,.2f}" if c in table[merchant] else f"{'-':>14}" for c, _ in checks)
print(f"{merchant[:31]:<32}" + "".join(cells))
def main():
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # merchant names are not always ASCII
ap = argparse.ArgumentParser()
ap.add_argument("--history", metavar="PRODUCT_ID")
ap.add_argument("--rematch", action="store_true")
args = ap.parse_args()
products = {p["id"]: p for p in json.loads((HERE / "products.json").read_text(encoding="utf-8"))}
db = sqlite3.connect(os.environ.get("PRICE_DB", "prices.db"))
db.executescript(SCHEMA)
if args.history:
return history(db, args.history)
if args.rematch:
db.execute("DELETE FROM prices")
for check_id, product_id, raw in db.execute("SELECT id, product_id, raw_json FROM checks").fetchall():
if product_id in products: # products removed from the list keep their old checks only
save_prices(db, check_id, products[product_id], json.loads(raw))
db.commit()
return print("prices matched again from the stored responses")
alerts = []
for product in products.values():
try:
data = fetch_shopping(product["query"], product["gl"])
except (requests.RequestException, RuntimeError) as e: # failed searches are refunded: retry next run
print(f"{product['id']}: skipped, {e}")
continue
line, product_alerts = compare(db, product, store(db, product, data))
print(line)
alerts += product_alerts
notify(alerts)
if __name__ == "__main__":
main()
Next steps
Your competitor price monitoring job now matches offers carefully, keeps every response and alerts only on real changes. Next:
- Shopping, Places and Maps data in depth: the local search data guide and Google Places vs Google Maps results.
- The same pattern for rankings: detecting ranking drops automatically and a Python rank tracker with SQLite.
- The Google Shopping API page, and a free account with 3,000 searches to try it on your catalog.