SERP Competitor Analysis: Find Which Domains Own Your Keywords
Michael Carter · · 20 min read

Key takeaways
- SERP competitor analysis counts which domains hold the top 10 across a keyword set; for 28 US project management keywords on November 17 and 18, 2026, no site held more than 3.9% of the 463 page-one positions.
- Quora, Reddit and Facebook ranked on the same 5 keywords on both days, and content platforms such as these held 18% to 21% of all page-one positions on each day.
- One snapshot is not enough: 11 of 28 pages kept fewer than 20% of their sites overnight, and 5 of 10 rechecks a few minutes later replaced most of the page, all run from US IPs.
- Count positions and keywords separately: one site held 15 positions on November 17, but 7 of them were on a single page and it kept 3 positions the next day.
- The Jaccard index, shared keywords divided by the keywords either site ranks for, shows which sites compete for the same pages and keeps sites that rank for everything from looking like everyone's rival.
SERP competitor analysis finds the domains that compete with you in Google across a set of keywords: collect the top 10 for each keyword, count how many positions each domain holds, then measure how much their keyword sets overlap. The domains that keep appearing are your search competitors, whether or not you sell against them.
This tutorial is for SEO leads, content strategists and developers of SEO tools. It works through a real example, 28 project management keywords searched in the US on November 17 and 18, 2026, with a tested Python script, a domain-by-keyword heatmap, a check of what changed overnight, and a route for people who do not code.
Search competitors vs business competitors
A business competitor sells what you sell. A search competitor is any site that ranks for the keywords you want, whatever it sells: a forum, a video platform, a review site, a publisher, or a vendor you never meet in a sales call. The two lists overlap only partly, and SEO decisions need the second one.
| Business competitors | Search competitors | |
|---|---|---|
| Defined by | Product, price, customers | Rankings for your keywords |
| Where you find them | Sales calls, pricing pages, review sites | The results pages for your keyword set |
| For a project management app | Other project management vendors | Forums, Q&A sites, video, publishers, some vendors |
| What you compete for | The buyer's budget | Clicks on the same results page |
SERP analysis reads one results page to learn what a single keyword wants. Competitor analysis repeats that read across a keyword set and keeps what recurs: one domain on one page is noise, the same domain on a third of your pages is a competitor.
How SERP competitor analysis works
SERP competitor analysis turns a keyword set into a ranked list of domains and a map of who competes with whom. It takes four steps:
- Collect the top 10 organic results for every keyword in one market (country, language, device).
- Count the positions each domain holds, its share of all positions, and how many keywords it ranks for.
- Compare the leading domains pairwise with an overlap matrix, and draw the domain-by-keyword grid as a heatmap.
- Repeat the run on another day and keep what held.
The overlap measure is the Jaccard index, "a statistic used for gauging the similarity and diversity of sample sets": the size of the intersection of two sets divided by the size of their union, J(A, B) = |A ∩ B| / |A ∪ B|. It runs from 0 (no keyword in common) to 1 (exactly the same keywords). If one site ranks for 9 of your keywords, another for 3, and the 3 are among the 9, the index is 3 / 9 = 0.33.
Jaccard is better than a raw count of shared keywords because it corrects for size: a site that ranks for everything shares keywords with everyone, but its index with each of them stays low.
Step 1: Collect the top 10 for your keywords
Start with keywords from one market, in one country and language. For the example we wrote 28 US keywords a project management software company could target, in four groups:
| Group | Keywords | Examples |
|---|---|---|
| Software | 13 | project management software, gantt chart software, project tracking software |
| Alternatives | 3 | asana alternatives, trello alternatives, monday.com alternatives |
| Learn | 9 | what is project management, agile vs waterfall, raci chart |
| Templates | 3 | project timeline template, project management templates, project status report |
You need Python 3 with requests and a BestSERP key in the BESTSERP_API_KEY environment variable. BestSERP, the Google SERP API we run, gives 3,000 free searches, then charges $0.50 per 1,000; the quickstart and our guide to Google search results in Python cover the basics. We tested the script on November 18, 2026 with Python 3.12.10 and requests 2.34.2. Paste the five code blocks in Steps 1 to 3 into one file, serp_competitors.py.
"""SERP competitor analysis: which domains hold the top 10 across a keyword set, and how much their keyword sets overlap.
export BESTSERP_API_KEY=... # never hard-code the key
python serp_competitors.py keywords.txt --gl us --save raw/ --fresh --you wrike.com
Tested 2026-11-18 with Python 3.12.10 and requests 2.34.2. Each new keyword costs 1 search (1 credit).
Keywords already saved in --save are read from disk, so re-running the analysis costs nothing.
"""
import argparse
import csv
import html
import json
import os
import re
import time
from collections import defaultdict
from datetime import datetime, timezone
from pathlib import Path
import requests
API = "https://api.bestserp.com"
HEADERS = {"X-API-KEY": os.environ.get("BESTSERP_API_KEY", "")}
def fetch_serp(q, gl="us", hl="en", no_cache=False):
"""Top 10 for one keyword. Polls 202, waits on 429, retries 502/503 (refunded)."""
body = {"q": q, "gl": gl, "hl": hl, "num": 10}
if no_cache:
body["noCache"] = True # skip the 1-hour cache: a fresh page
for attempt in range(5):
r = requests.post(f"{API}/search", headers=HEADERS, json=body, timeout=130)
if r.status_code == 202: # still running: poll until it is done
rid = r.json()["requestId"]
while r.status_code == 202:
time.sleep(3)
r = requests.get(f"{API}/requests/{rid}", headers=HEADERS, timeout=130)
if r.status_code == 200:
return {**r.json(), "fetchedAt": datetime.now(timezone.utc).isoformat(timespec="seconds")}
if r.status_code in (429, 502, 503):
time.sleep(int(r.headers.get("Retry-After", 10 * (attempt + 1))))
continue
r.raise_for_status() # 400, 401, 402: fix the request, do not retry
raise RuntimeError(f"{q!r}: still failing after 5 attempts (HTTP {r.status_code})")
A 202 is polled, a 429 (60 searches a minute by default) waits for Retry-After, and a 502 or 503 is retried; failed searches are refunded. Each saved page gets a fetchedAt time in UTC, so you always know when a snapshot was taken.
def file_name(q):
return re.sub(r"[^a-z0-9]+", "-", q.lower()).strip("-") + ".json"
def collect(keywords, gl, hl, folder, no_cache=False):
"""One saved JSON file per keyword; only keywords not saved yet are fetched."""
Path(folder).mkdir(parents=True, exist_ok=True)
serps = {}
for q in keywords:
path = Path(folder, file_name(q))
if not path.exists():
serp = fetch_serp(q, gl, hl, no_cache)
path.write_text(json.dumps(serp, indent=1), encoding="utf-8")
print(f"fetched {q!r} from an IP in {serp.get('ipCountry', '?')}")
serps[q] = json.loads(path.read_text(encoding="utf-8"))
ran_from = serps[q].get("ipCountry", gl).lower() # the country the search ran from; missing when unknown
if ran_from != gl.lower():
print(f" warning: {q!r} ran from {ran_from}, not {gl}: fetch it again or leave it out")
return serps
Every later step reads the saved JSON, so you can change the counting rules and re-run for free. One keyword is one search: 28 keywords cost 28 credits, and 500 would cost about $0.25 after the free searches. --fresh sends noCache: true, which skips the one-hour cache.
The ipCountry check matters because a search runs from an IP in the gl country when one is free, and from another country otherwise; the response says which. All 28 of our November 18 searches ran from US IPs. Fetch any that did not again, or leave them out: they describe another market.
Step 2: Count domains and positions
Counting needs two decisions: what counts as one site, and whether several results on one page count once or several times. The second part makes both explicit.
PLATFORMS = { # results without a `domain` name their platform in `source` ("Reddit · r/projectmanagement")
"reddit": "reddit.com", "youtube": "youtube.com", "quora": "quora.com", "medium": "medium.com",
"linkedin": "linkedin.com", "facebook": "facebook.com", "x": "x.com", "instagram": "instagram.com",
"tiktok": "tiktok.com", "pinterest": "pinterest.com", "stack overflow": "stackoverflow.com",
}
TWO_PART_SUFFIXES = (".co.uk", ".org.uk", ".com.au", ".co.in", ".com.br", ".co.jp") # extend for your markets
def site_of(item):
"""The site that owns a result: `domain` (subdomains folded in), else the platform named in `source`."""
host = (item.get("domain") or "").lower().removeprefix("www.")
if not host:
name = item.get("source", "").split("·")[0].strip().lower()
return PLATFORMS.get(name, name or "?")
labels = host.split(".")
keep = 3 if host.endswith(TWO_PART_SUFFIXES) else 2
return ".".join(labels[-keep:]) # support.atlassian.com -> atlassian.com
def rankings(serps):
"""{keyword: {site: [positions it holds]}}: every organic result counts, position 1 is the top."""
out = {}
for q, serp in serps.items():
held = defaultdict(list)
for item in serp.get("organic", []):
held[site_of(item)].append(int(item["position"]))
out[q] = dict(held)
return out
def domain_table(ranks):
"""One row per site: positions held, share of all positions, keywords covered, average and best position."""
total = sum(len(p) for held in ranks.values() for p in held.values())
per_site = defaultdict(list) # site -> one list of positions per keyword it ranks for
for held in ranks.values():
for site, positions in held.items():
per_site[site].append(positions)
rows = []
for site, lists in per_site.items():
tops = [min(p) for p in lists] # the site's best position on each keyword
n = sum(map(len, lists))
rows.append({"site": site, "positions": n, "share": n / total, "keywords": len(lists),
"avg_position": sum(tops) / len(tops), "best": min(tops)})
return sorted(rows, key=lambda r: (-r["positions"], -r["keywords"], r["avg_position"]))
site_of folds subdomains into their site (support.atlassian.com becomes atlassian.com) and keeps two-part country suffixes such as .co.uk. Some results arrive without a domain and name their platform in source ("Reddit · r/projectmanagement"): 18 of 231 on November 18, all from Medium, YouTube, Quora, Reddit, Facebook, LinkedIn and Pinterest. For all 18, the platform matched the host of the final link, which we fetched by polling /requests/{requestId} (link resolution).
The table keeps two measures apart. Positions counts every result, so a site with three results on one page holds three. Keywords counts each site once per keyword. Share is positions divided by all organic positions in the run. Here is the November 18 run (trimmed):
fetched 'project management software' from an IP in us
fetched 'best project management software' from an IP in us
...
28 keywords, 231 organic positions, 163 sites
site positions share keywords avg pos best
quora.com 10 4.3% 9 3.4 2
medium.com 9 3.9% 7 5.6 3
reddit.com 7 3.0% 7 2.4 1
facebook.com 6 2.6% 5 5.0 3
youtube.com 5 2.2% 4 5.0 2
launchnotes.com 4 1.7% 4 4.5 1
ideascale.com 4 1.7% 4 4.8 3
larksuite.com 4 1.7% 3 3.3 1
ocmsolution.com 3 1.3% 3 1.0 1
adleaks.com 3 1.3% 3 6.0 6
Why keep both: on November 17, ones.com held the most positions (15), but 7 were on one page ("resource management software"), and it kept 3 positions the next day.
Step 3: Overlap matrix and heatmap
The overlap matrix compares the sites with the most keywords pairwise; heatmap_html draws it as an HTML table.
def keyword_sets(ranks):
sets = defaultdict(set)
for q, held in ranks.items():
for site in held:
sets[site].add(q)
return sets
def jaccard(a, b):
"""|A ∩ B| / |A ∪ B|: 0 = no shared keywords, 1 = exactly the same keywords."""
return len(a & b) / len(a | b) if a | b else 0.0
def overlap_matrix(sets, sites):
return [[jaccard(sets[a], sets[b]) for b in sites] for a in sites]
def competitors_of(you, sets, top=8):
"""Sites that share the most keywords with `you`, with the Jaccard index."""
mine = sets.get(you, set())
rows = [(s, len(mine & k), jaccard(mine, k)) for s, k in sets.items() if s != you and mine & k]
return sorted(rows, key=lambda r: (-r[1], -r[2]))[:top]
def heatmap_html(sites, matrix, path):
"""A dependency-free heatmap: an HTML table, darker cells = more keyword overlap."""
head = "".join(f"<th>{html.escape(s)}</th>" for s in sites)
rows = "".join(
f"<tr><th>{html.escape(a)}</th>" + "".join(
f'<td style="background:rgba(25,103,210,{v:.2f});color:{"#fff" if v > 0.5 else "#000"}">{v:.2f}</td>'
for v in row) + "</tr>" for a, row in zip(sites, matrix))
Path(path).write_text(f"<table style='font:13px sans-serif;border-collapse:collapse'>"
f"<tr><th></th>{head}</tr>{rows}</table>", encoding="utf-8")
The last part ties the steps together and writes domains.csv, overlap.csv and heatmap.html:
def main():
ap = argparse.ArgumentParser()
ap.add_argument("keywords", help="text file, one keyword per line")
ap.add_argument("--gl", default="us")
ap.add_argument("--hl", default="en")
ap.add_argument("--save", default="raw", help="folder for the raw JSON of each search")
ap.add_argument("--fresh", action="store_true", help="send noCache: true (a new page, not the 1-hour cache)")
ap.add_argument("--top", type=int, default=12, help="sites in the overlap matrix (most keywords first)")
ap.add_argument("--you", help="your domain, to list the sites that compete with it")
ap.add_argument("--out", default=".", help="folder for domains.csv, overlap.csv and heatmap.html")
args = ap.parse_args()
keywords = [k.strip() for k in Path(args.keywords).read_text(encoding="utf-8").splitlines() if k.strip()]
ranks = rankings(collect(keywords, args.gl, args.hl, args.save, args.fresh))
table = domain_table(ranks)
total = sum(r["positions"] for r in table)
print(f"\n{len(ranks)} keywords, {total} organic positions, {len(table)} sites\n")
print(f"{'site':<26}{'positions':>10}{'share':>7}{'keywords':>9}{'avg pos':>9}{'best':>6}")
for r in table[:15]:
print(f"{r['site']:<26}{r['positions']:>10}{r['share']:>7.1%}{r['keywords']:>9}"
f"{r['avg_position']:>9.1f}{r['best']:>6}")
sets = keyword_sets(ranks)
by_reach = sorted(table, key=lambda r: (-r["keywords"], r["avg_position"]))
sites = [r["site"] for r in by_reach[:args.top]]
matrix = overlap_matrix(sets, sites)
Path(args.out).mkdir(parents=True, exist_ok=True)
with open(Path(args.out, "domains.csv"), "w", newline="", encoding="utf-8") as f:
w = csv.DictWriter(f, fieldnames=list(table[0]))
w.writeheader()
w.writerows({**r, "share": round(r["share"], 4), "avg_position": round(r["avg_position"], 2)} for r in table)
with open(Path(args.out, "overlap.csv"), "w", newline="", encoding="utf-8") as f:
w = csv.writer(f)
w.writerow(["site"] + sites)
w.writerows([[a] + [f"{v:.2f}" for v in row] for a, row in zip(sites, matrix)])
heatmap_html(sites, matrix, Path(args.out, "heatmap.html"))
print(f"\nWrote domains.csv, overlap.csv and heatmap.html ({len(sites)} x {len(sites)} matrix) to {args.out}")
if args.you:
print(f"\nSites sharing the most keywords with {args.you} ({len(sets.get(args.you, ()))} keywords):")
for site, shared, j in competitors_of(args.you, sets):
print(f" {site:<26} {shared:>2} shared Jaccard {j:.2f}")
if __name__ == "__main__":
main()
Run it on your keyword file (add --you yourdomain.com to list the sites that share the most keywords with you):
export BESTSERP_API_KEY="YOUR_API_KEY"
python serp_competitors.py keywords.txt --gl us --save raw-2026-11-18/ --fresh --out out-2026-11-18/
Two pairs from the November 18 matrix show how to read it. ocmsolution.com and adleaks.com rank for exactly the same three keywords ("project management software", "best project management software" and "project tracking software"), so their index is 1.00: both are "best tools" list pages competing for the same commercial results. Reddit and Medium share 3 of the 11 keywords either one ranks for, an index of 0.27, one of the higher values in this set.
The grid below, drawn from the same saved JSON with a separate plotting script, shows the data behind the matrix for both days. Each cell holds a site's best position on November 17 (left half) and November 18 (right half); the right column shows how much of each page stayed the same overnight.

Read it by columns, then rows. Columns that fill in across groups (Quora, Reddit, Facebook) are broad competitors; a column with a few cells in one group is a topic specialist. Nearly blank rows, such as "work breakdown structure", are keywords where none of the ten ranks: there, you compete page by page.
Step 4: Run it again on another day
A competitor list from one snapshot can be wrong, so compare two runs before you act. This script reads two saved runs and reports what held:
"""Compare two runs of the same keyword set (for example two days): how much of the competitor picture held.
python compare_runs.py keywords.txt day1/ day2/
Reads the saved JSON only (no API calls, no credits). Tested 2026-11-18 with Python 3.12.10.
"""
import json
import sys
from pathlib import Path
from statistics import median
from serp_competitors import domain_table, file_name, jaccard, keyword_sets, rankings
def load(folder, keywords):
return {q: json.loads(Path(folder, file_name(q)).read_text(encoding="utf-8")) for q in keywords}
def main():
keywords = [k.strip() for k in Path(sys.argv[1]).read_text(encoding="utf-8").splitlines() if k.strip()]
a, b = rankings(load(sys.argv[2], keywords)), rankings(load(sys.argv[3], keywords))
same = {q: jaccard(set(a[q]), set(b[q])) for q in keywords}
print(f"Sites per keyword, run 1 vs run 2 (Jaccard): median {median(same.values()):.2f}; "
f"{sum(v >= 0.5 for v in same.values())} of {len(same)} keywords at 0.5 or more, "
f"{sum(v < 0.2 for v in same.values())} below 0.2")
ta, tb = domain_table(a), domain_table(b)
top_a, top_b = [r["site"] for r in ta[:10]], [r["site"] for r in tb[:10]]
print(f"Top 10 sites by positions: {len(set(top_a) & set(top_b))} in both runs")
sa, sb = keyword_sets(a), keyword_sets(b)
rows_b = {r["site"]: r for r in tb}
print(f"\n{'site':<26}{'positions':>11}{'share':>15}{'keywords':>10}{'in both':>9}")
for site in dict.fromkeys(top_a + top_b): # top 10 of either run, run 1 order first
ra = next((r for r in ta if r["site"] == site), {"positions": 0, "share": 0})
rb = rows_b.get(site, {"positions": 0, "share": 0})
print(f"{site:<26}{ra['positions']:>5} -> {rb['positions']:<3}{ra['share']:>7.1%} -> {rb['share']:<5.1%}"
f"{len(sa[site]):>4} -> {len(sb[site]):<3}{len(sa[site] & sb[site]):>5}")
both = sorted(((s, len(sa[s] & sb[s])) for s in sa), key=lambda r: -r[1])
print("\nSites on the same keyword in both runs:", ", ".join(f"{s} {n}" for s, n in both if n >= 2))
if __name__ == "__main__":
main()
Our two runs, about 12 hours apart (output trimmed):
Sites per keyword, run 1 vs run 2 (Jaccard): median 0.63; 15 of 28 keywords at 0.5 or more, 11 below 0.2
Top 10 sites by positions: 7 in both runs
site positions share keywords in both
ones.com 15 -> 3 6.5% -> 1.3% 8 -> 3 3
youtube.com 11 -> 5 4.7% -> 2.2% 9 -> 4 4
reddit.com 9 -> 7 3.9% -> 3.0% 9 -> 7 5
quora.com 8 -> 10 3.4% -> 4.3% 7 -> 9 5
facebook.com 6 -> 6 2.6% -> 2.6% 6 -> 5 5
medium.com 5 -> 9 2.2% -> 3.9% 5 -> 7 3
Sites on the same keyword in both runs: quora.com 5, reddit.com 5, facebook.com 5, youtube.com 4, ones.com 3, medium.com 3, launchnotes.com 3, pinterest.com 3, ...
The pages split into two kinds. On 15 of 28 keywords at least half the sites held; on 11, fewer than 20% did, and seven pages had no site in common at all. Page by page, change was mostly all or nothing.
Same-day rechecks showed the pattern within minutes. We fetched 10 keywords again a few minutes after the November 18 run, all from US IPs according to ipCountry: 5 pages kept at least half their sites (four were identical), and 5 replaced at least five sites. On November 17, 2 of 5 rechecks did the same. The country was not the cause, and the responses do not show what was. The practical rule holds either way: count a site as a competitor when it ranks on the same keywords in two runs, and treat everything else as provisional. For a running measure of how much your pages move, see how to build a SERP volatility index.
Reading the results
Three findings hold up across both runs, and each one changes a plan.
No site owns this keyword set. Across both days, 209 sites shared 463 page-one positions; the largest holders (Quora and ones.com) had 3.9% each. On November 18, 134 of 163 sites ranked for one keyword only. You compete keyword by keyword here, not against one rival.
Platforms are the steadiest rivals. Quora, Reddit and Facebook ranked on the same 5 keywords on both days, YouTube on 4. Platforms held 18% to 21% of all positions and 26% to 28% on the "learn" keywords, and 64% of their November 17 placements held the next day, against 53% for other sites. Competing with answers and threads calls for first-hand content more than another vendor comparison.
The big vendors rarely rank for their own category. asana.com, monday.com, trello.com and notion.com held no position in either run; clickup.com and smartsheet.com ranked for one keyword each on November 18. The software keywords went mostly to "best tools" list articles and review pages, like the ocmsolution.com and adleaks.com pair, plus a few smaller vendors' product pages.
One keyword shows the pattern up close. "what is a kanban board" returned the same 8 sites in all four fetches over two days, and 3 of them also rank across the set:

What to do with the list:
- Study the recurring sites' ranking pages: which formats (glossary entries, answer pages, lists) win across your set.
- Weight by position when you report: SEO share of voice weights each position by its expected clicks.
- Check your own pages with a
site:query; our tested list of Google search operators shows which operators work. - Compare with a larger sample, such as our study of Google's top 10 across 300 searches.
Doing it without code
For 10 to 20 keywords, a SERP competitor analysis works by hand with the same method.
- Find the domains. Search each keyword in a private browser window and note the domain of each top 10 result in a sheet with three columns: keyword, position, domain.
- Count them. In Google Sheets, select the data and choose Insert > Pivot table (Google's pivot table help), put domain under Rows and a count of keyword under Values, and sort.
- Watch your rivals. Our free SERP Checker takes your domain, up to 5 keywords and up to 3 competitor domains, checks Google's top 20 from a desktop browser in the country you choose and exports CSV; 3 free checks a day (checked November 18, 2026).
To pull results into a sheet without a server, see Google search results in Google Sheets; to build competitor reports into your product, see SERP data for SEO tools.
Method
- Keywords: the 28 keywords above, written for this example; no search volumes used.
- Market: BestSERP
/searchwithgl=us,hl=en,num=10(page one, desktop). - Run 1: November 17, 2026, 16:15 to 16:17 UTC, default cache settings (none of the 28 was served from cache). These responses predate the
ipCountryfield, so the country they ran from is not recorded. - Run 2: November 18, 2026, 03:56 to 03:59 UTC,
noCache: true;ipCountrywasusfor all 28. - Rechecks: 5 keywords at 16:24 UTC on November 17, 10 at 04:00 to 04:02 UTC on November 18 (all from US IPs),
noCache: true. - Counting: organic results only (video carousels on 5 November 18 pages not counted), 7 to 10 per page; sites grouped by
domainwith subdomains folded in, or by the platform insourcewhendomainwas missing. - Cost: 71 searches, 71 credits.
Common problems and fixes
| Problem | Cause | Fix |
|---|---|---|
| A site you never heard of tops the list | It holds several results on one or two pages | Read the keywords column, not only positions |
| The list changes between runs | Pages are replaced, sometimes within minutes | Compare two runs; trust sites that rank on the same keywords in both |
| Foreign sites in a US list | .co.uk or .com.au pages can rank in the US, or the search ran elsewhere |
Check ipCountry per search; keep the market fixed |
| Reddit or YouTube missing from the counts | Results without domain were dropped |
Map the platform in source, as site_of does |
429 rate_limited |
More than 60 searches a minute | Wait for Retry-After; the script does |
502 or 503 |
A search failed or the system was busy | Retry after a pause; failed searches are refunded |
Next steps
SERP competitor analysis costs one search per keyword per run, and every re-count after that is free. Collect, count, compare, and run it a second time before you decide.
- Before you write for any keyword on the list: the SERP analysis framework for reading one results page.
- To track the competitors you found over time: our rank tracking guide.
- To check where a few named rivals rank today, in any country: the free SERP Checker.