How to Get Google Search Results in Python (2026 Guide)
Michael Carter · · 21 min read · Last updated

Key takeaways
- Plain requests and BeautifulSoup stopped working on google.com when Google began requiring JavaScript in January 2025; a SERP API returns the same results page as JSON instead.
- Ask for depth with num, not page: num 20 cost 2 credits for positions 1 to 19, while page 2 alone cost 2 credits for positions 10 to 19, because every page up to the one you ask for is charged.
- Google's page 1 held 9 results in our test, so page 2 started at position 10, and in our full run the same page appeared at both position 6 and position 14, so de-duplicate before counting.
- After polling up to 2 minutes for final links, all 54 results in our November 18, 2026 run had a final URL and a domain; on November 17, before BestSERP's link fix, 23 of 56 had no domain and all 56 links stayed Google redirects.
- Retry 429, 502 and 503 after Retry-After, poll 202 answers for free, and never retry 400: failed searches are refunded and refused ones are never charged.
To get Google search results in Python, send each query to a SERP API with the requests library, read the organic results from the JSON, and save them with pandas. This guide builds one tested script that turns a keyword list into a CSV: up to 50 results per keyword, any country and language, with retries.
It is written for developers who need more than one call. If you only want a single request, the Python integration page has a minimal search function and the full status code table, and this tutorial does not repeat them. It adds what a real job needs: depth without overpaying, countries, retries, 202 polling and a clean CSV. New to the idea? Start with what a SERP API is. Every line below ran on November 17, 2026, and the full script again on November 18 after BestSERP's link fix, with Python 3.12.10, requests 2.34.2 and pandas 3.1.1; the outputs are real.
What you will build
You will build serp_to_csv.py, a script of about 110 lines that reads keywords from a CSV file, runs one search per row through BestSERP (bestserp.com), a Google SERP API, and writes every organic result to results.csv. Here is our run on three keywords (November 18, 2026):

What the script handles:
- A keyword list with a country (
gl) and language (hl) per row, defaulting to the US and English. - Depth: 10 to 50 results per keyword in a single request, the cheapest way to page.
- Failures: it polls
202answers, retries429,502and503afterRetry-After, and logs any keyword that still fails instead of stopping. - Final links: it waits up to 2 minutes for the final URLs that replace Google's redirects, and keeps every row groupable if a
domainis still missing. - Output: a pandas DataFrame saved as UTF-8 CSV that Excel opens correctly.
Why requests and BeautifulSoup no longer work on google.com
The classic tutorial, requests.get("https://www.google.com/search?q=...") parsed with BeautifulSoup, stopped working when Google began requiring JavaScript for Search in January 2025. TechCrunch reported on January 17, 2025 that Google made the change to protect its services "from bots and evolving forms of abuse and spam". A client that does not run JavaScript, such as requests, no longer gets a results page to parse.
What remains is a real browser with proxies, which is a project of its own; we measured how often Google blocks one in SERP API vs web scraping. Libraries that wrap google.com, such as googlesearch-python, hit the same wall, usually as HTTP 429 (why googlesearch-python fails and what to use instead). This guide uses an API, and the script never contacts google.com.
Prerequisites
You need three things:
Python 3.12 (what we tested with) and two packages:
pip install requests pandasAn API key. Sign up to get 3,000 free searches, valid for 30 days, then create a key in the dashboard (quickstart).
The key in an environment variable, never in the code:
export BESTSERP_API_KEY=YOUR_API_KEY # macOS, Linux $env:BESTSERP_API_KEY = "YOUR_API_KEY" # Windows PowerShell
Following every step below costs 10 credits: 1 for Step 1, 2 for the page test in Step 2, 6 for the full run and 1 for the 202 test in Step 4.
Step 1: Your first Google search results in Python
Send one search and look at the fields before you design the CSV. The script below uses a requests.Session, which reuses the connection and sends the key header on every call (see Session objects in the requests docs):
import os
import requests
session = requests.Session() # reuses the connection and sends the key with every request
session.headers["X-API-KEY"] = os.environ["BESTSERP_API_KEY"]
r = session.post("https://api.bestserp.com/search",
json={"q": "python web scraping", "gl": "us", "hl": "en"}, timeout=130)
r.raise_for_status()
data = r.json()
for item in data["organic"]:
kind = "redirect" if item.get("link", "").startswith("https://www.google.com/") else "final"
print(f"{item['position']:>2} {item.get('domain', '-'):<22} {item.get('source', '')[:28]:<28} {kind}")
print({k: data[k] for k in ("requestId", "credits", "cached", "durationMs", "linksPending", "unresolvedLinks")})
print("balance:", r.headers["X-Credits-Remaining"], "| searches left this minute:", r.headers["X-RateLimit-Remaining"])
Our output:
1 - YouTube · edureka! redirect
2 dev.to DEV Community redirect
3 - YouTube · Web Scraping Ninja redirect
4 - YouTube · Intellipaat redirect
5 keirolabs.cloud Keirolabs redirect
6 - LinkedIn · Suraj Kumar Soni redirect
7 - YouTube · Marketing Analytic redirect
8 proxy-seller.com Proxy-Seller redirect
9 - YouTube · buildwithpython redirect
{'requestId': '8e979e82-dbcb-40ce-8b12-39cb5415363d', 'credits': 1, 'cached': False, 'durationMs': 5531, 'linksPending': True, 'unresolvedLinks': 9}
balance: 2771 | searches left this minute: 58
Four details shape the rest of the script:
- Google showed 9 results, not 10.
numis a maximum, and a page often holds 8 or 9. - 6 of 9 results had no
domain. YouTube and LinkedIn results show a channel or author instead of a URL, sodomainis left out of the item until its final link arrives. Thesourcefield still says "YouTube · edureka!", which is why the code usesitem.get("domain")and neveritem["domain"]. - All 9 links were Google redirects (
google.com/goto?...). BestSERP replaces them with final URLs after answering (linksPending: True), but when we fetched the finished result 30 seconds later, all 9 were still redirects. That bug was fixed on November 18, 2026: in 61 searches through the public API that day, pollingGET /requests/{requestId}turned all 401 pending redirects into final URLs, median 34 seconds and at most 54 seconds after the request. The first answer's links are usually still redirects (they work, but are not the page's address), so the script below polls untillinksPendingis false, for up to 2 minutes (how link resolution works); in our November 18 run all 54 links arrived as final URLs. - The headers carry your budget.
X-Credits-RemainingandX-RateLimit-Remainingcome with every search, so a batch job can watch its balance and the 60-searches-a-minute limit without extra calls.
Step 2: Pagination and more results (num, page, cost)
To get more than one page, ask for more results with num (10, 20, 30, 40 or 50) in one request; do not loop over page. Both cost 1 credit per 10 results, but page: N returns only page N while charging for pages 1 to N.
We checked that with the same query and page: 2:
import os
import requests
session = requests.Session()
session.headers["X-API-KEY"] = os.environ["BESTSERP_API_KEY"]
r = session.post("https://api.bestserp.com/search",
json={"q": "python web scraping", "gl": "us", "hl": "en", "page": 2}, timeout=130)
r.raise_for_status()
data = r.json()
print("positions:", [item["position"] for item in data["organic"]])
print("credits:", data["credits"], "| searchParameters:", data["searchParameters"])
positions: [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
credits: 2 | searchParameters: {'q': 'python web scraping', 'gl': 'us', 'hl': 'en', 'type': 'search', 'num': 10, 'page': 2, 'engine': 'google'}
Page 2 started at position 10, because page 1 held 9 results, and it cost 2 credits. The full run below asked for num: 20 and got positions 1 to 19 in one response, also for 2 credits (with a different order than in Step 1: Google results move between requests). Looping over page 1 and page 2 would have cost 3 credits for the same 19 results.
| What you want | Request | Credits | What came back in our tests |
|---|---|---|---|
| Top 10 | num: 10 |
1 | Positions 1-9 |
| Top 20 | num: 20 |
2 | Positions 1-19 |
| Top 20, page by page | page: 1, then page: 2 |
3 | Positions 1-19, in two responses |
| Only results 11-20 | page: 2 |
2 | Positions 10-19 |
| Top 50, the maximum | num: 50 |
5 | Not tested |
Two limits apply: page × num / 10 can be at most 5, so 50 results is as deep as one request goes, and num: 15 is refused (400 invalid_request: num must be a multiple of 10). The script turns a depth you choose into one valid num:
num = min(50, max(10, -(-depth // 10) * 10)) # 10..50 in one request: never loop over pages
-(-depth // 10) rounds up, so a depth of 25 becomes num: 30. One more thing we saw: in the full run below, the same solix.com page appeared at positions 6 and 14 for "pandas read csv", so de-duplicate before you count results per site.
Step 3: Country and language (gl, hl)
Each search runs for one country (gl, a two-letter code) and one interface language (hl). Put them in the keyword file, one row per combination:
keyword,gl,hl
python web scraping,us,en
python web scraping,de,de
pandas read csv,,
The script reads the file as text and fills empty cells with the defaults, so the third row runs for the US in English:
jobs = pd.read_csv(keywords_csv, dtype=str).fillna({"gl": "us", "hl": "en"})
The country changes more than the order. For "python web scraping" on November 18, the US top 19 started with dev.to and held 11 YouTube videos; the German top 18 started with linkedin.com, held German sites such as neuefische.de and databasecamp.de, and only 7 YouTube videos. Only three sites appeared in both lists: YouTube, LinkedIn and Medium.
Targeting stops at the country. A location parameter is refused (400 unsupported_parameter: location is not supported yet; use gl (country) and hl (language)), so put the city in the query ("dentist austin") when it matters. The trade-offs between the two parameters are in how gl and hl change Google results.
Step 4: Retries, 202 polling and refunds
A batch job meets four answers besides 200, and each needs its own handling:
| Status | What it means | What the script does | Charged? |
|---|---|---|---|
202 |
Not finished within the request's timeout | Polls GET /requests/{requestId} every 2 s |
Reserved until it finishes |
429 |
More than 60 searches in a minute | Waits Retry-After seconds, retries |
No |
502 |
Google blocked the search or it timed out | Waits Retry-After (or 2 s, then 4 s), retries |
Refunded |
503 |
No capacity right now | Waits Retry-After, retries |
Refunded |
400, 401, 402 |
Your request, key or balance | Raises at once: retrying cannot help | No |
Polling is free and does not count toward the rate limit (async requests). Requests itself has no retry and no timeout by default; its docs warn that "Without a timeout, your code may hang for minutes or more", so every call sets timeout=130, above the API's 120-second maximum. The full list of codes is on the errors page. Here is the core of the script:
API = "https://api.bestserp.com"
RETRY = {429, 502, 503} # rate limited (not charged), search failed or busy (both refunded)
session = requests.Session()
session.headers["X-API-KEY"] = os.environ["BESTSERP_API_KEY"]
def log(message):
print(message, file=sys.stderr)
class SearchError(Exception):
def __init__(self, r):
try:
error = r.json().get("error", {})
except ValueError: # an HTML error page from a gateway, not the API's JSON
error = {"code": "gateway", "message": r.reason}
super().__init__(f"{r.status_code} {error.get('code')}: {error.get('message')}")
self.status = r.status_code
def search(q, gl="us", hl="en", num=10, tries=3, **params):
"""POST /search. Polls 202, retries 429/502/503, raises SearchError on anything else."""
body = {"q": q, "gl": gl, "hl": hl, "num": num, **params}
for attempt in range(1, tries + 1):
r = session.post(f"{API}/search", json=body, timeout=130)
if r.status_code == 202: # not finished within the timeout: poll (free, not rate limited)
request_id = r.json()["requestId"]
log(f" {q!r} ({gl}): 202, polling /requests/{request_id}")
while r.status_code == 202:
time.sleep(2)
r = session.get(f"{API}/requests/{request_id}", timeout=130)
if r.status_code == 200:
return r.json()
error = SearchError(r)
if r.status_code in RETRY and attempt < tries:
wait = float(r.headers.get("Retry-After", 2 ** attempt))
log(f" {q!r} ({gl}): {error}; retry {attempt} in {wait:.0f} s")
time.sleep(wait)
continue
raise error
Two choices are deliberate. tries=3 means two retries, matching the errors page's advice to retry 502 "once or twice". And SearchError survives a 502 that arrives as an HTML page instead of JSON.
Our first run met exactly that. The German search answered 502 four times in a row (three with Retry-After: 60, the last an HTML gateway page); the earlier version of the script logged it as failed, kept the other two keywords, and the credit was refunded. A second run six minutes later succeeded.
To see a real 202, we set the API's own timeout parameter to 1 second, shorter than any search takes:
from serp_to_csv import search
# timeout here is the API's parameter (1-120 s): answer 202 if the search is not done after 1 second
data = search("python requests retry", timeout=1)
print(len(data["organic"]), "results | credits:", data["credits"], "| durationMs:", data["durationMs"])
'python requests retry' (us): 202, polling /requests/588ad355-aeaa-4fc4-a653-c615d44333e5
9 results | credits: 1 | durationMs: 5357
A real 429 takes more than 60 searches in a minute, and a 502 cannot be ordered, so we tested those branches against a local stub server that answers scripted status codes (code/test_retries.py; no credits used). The log for "429, then 503, then 202 twice, then 200":
'stub' (us): 429 rate_limited: Too many requests; retry 1 in 1 s
'stub' (us): 503 busy: No capacity right now; retry 2 in 1 s
'stub' (us): 202, polling /requests/req-1
The same test confirmed that a 400 raises at once and that three 502 answers in a row raise after waits of 2 and 4 seconds. For jobs on a schedule, one more rule: an identical request within an hour may come from cache at the normal price, which noCache: true avoids.
Step 5: Save to pandas and CSV
Before anything is saved, the script waits for the final links. The first answer usually says linksPending: true, so this function polls the request, which is free and not rate limited, until the final URLs have replaced Google's redirects:
def wait_for_links(data, max_wait=120):
"""Poll until the final URLs replace Google's redirects: usually within 30 s, up to 2 minutes."""
deadline = time.monotonic() + max_wait
while data.get("linksPending") and time.monotonic() < deadline:
time.sleep(4) # polling is free and not rate limited
r = session.get(f"{API}/requests/{data['requestId']}", timeout=130)
if r.status_code == 200:
data = r.json()
if data.get("linksPending"):
log(f" {data['searchParameters']['q']!r}: links still pending after {max_wait} s, saved as they are")
return data
When a final URL arrives, BestSERP also fills domain from it. Each organic result then becomes one row, with the search's parameters and time attached so you can compare runs later:
def to_rows(data):
"""One row per organic result. domain is missing while a link is still a redirect: keep source too."""
p = data["searchParameters"]
searched_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
return [{"keyword": p["q"], "gl": p["gl"], "hl": p["hl"], "position": item["position"],
"domain": item.get("domain"), "source": item.get("source"), "title": item.get("title"),
"link": item.get("link"), "snippet": item.get("snippet"),
"searched_at": searched_at, "request_id": data["requestId"]}
for item in data.get("organic", [])]
The main loop runs the keywords in order, skips a keyword that still fails after retries, and builds the DataFrame:
def main(keywords_csv, out_csv, depth=20):
num = min(50, max(10, -(-depth // 10) * 10)) # 10..50 in one request: never loop over pages
jobs = pd.read_csv(keywords_csv, dtype=str).fillna({"gl": "us", "hl": "en"})
rows, credits, failed = [], 0, []
for job in jobs.itertuples(index=False):
started = time.monotonic()
try:
data = wait_for_links(search(job.keyword, job.gl, job.hl, num=num))
except (SearchError, requests.RequestException) as e:
log(f"FAILED {job.keyword!r} ({job.gl}): {e}")
failed.append(f"{job.keyword} ({job.gl})")
continue
credits += data["credits"]
rows += to_rows(data)
print(f"{job.keyword:<20} {job.gl}/{job.hl} {len(data['organic']):>2} results {data['credits']} credits"
f" {data['durationMs']:>5} ms total {time.monotonic() - started:>3.0f} s"
f" unresolved links: {data['unresolvedLinks']}")
if not rows:
sys.exit("no results saved")
df = pd.DataFrame(rows)
# The column to group by: domain, or the site name when a link stayed a redirect ("YouTube · channel").
df.insert(5, "site", df["domain"].fillna(df["source"].str.split(" · ").str[0]))
df.to_csv(out_csv, index=False, encoding="utf-8-sig") # the BOM lets Excel detect UTF-8
print(f"saved {len(df)} rows to {out_csv} | credits used: {credits} | failed: {failed or 'none'}")
return df
Two lines need a word. The site column fills a missing domain with the platform name taken from source, so "YouTube · edureka!" would become "YouTube". After the wait, none of our 54 results lacked a domain, so site equals domain on every row; the fallback only matters when a link is still a redirect after 2 minutes, which the script logs. And encoding="utf-8-sig" writes a byte order mark: pandas' `to_csv` writes plain UTF-8 by default, and Microsoft's Excel help says a UTF-8 CSV opens normally "if it was saved with BOM", which keeps German titles with "für" readable. unresolvedLinks was 0 for all three searches.
Here is the start of the file from our November 18 run:

Once the data is in pandas, questions about it take one line each. This analyze.py reads the CSV back:
import sys
import pandas as pd
df = pd.read_csv(sys.argv[1] if len(sys.argv) > 1 else "results.csv")
# How many results each search returned, and how deep it went
print(df.groupby(["keyword", "gl"])["position"].agg(results="count", deepest="max"), "\n")
# The sites with the most results across all searches
print(df.groupby("site").size().sort_values(ascending=False).head(5), "\n")
# Links still left as Google redirects, and results without a domain: both 0 when every link resolved
print("redirect links:", df["link"].str.startswith("https://www.google.com/").sum(), "of", len(df),
"| without domain:", df["domain"].isna().sum(), "\n")
# Same keyword in two countries: which sites rank in both, and where
both = df[df["keyword"] == "python web scraping"].pivot_table(
index="site", columns="gl", values="position", aggfunc="min")
print(both.dropna().sort_values("us"))
results deepest
keyword gl
pandas read csv us 17 17
python web scraping de 18 18
us 19 19
site
youtube.com 21
medium.com 4
linkedin.com 3
reddit.com 2
facebook.com 2
dtype: int64
redirect links: 0 of 54 | without domain: 0
gl de us
site
youtube.com 5.0 2.0
linkedin.com 1.0 8.0
medium.com 2.0 11.0
After the wait, all 54 links were final URLs and every result had a domain, so YouTube's 21 videos group under youtube.com. On November 17, before the link fix, the same script without the wait saved 56 rows with 23 missing domains and 56 redirect links, and site was what kept those rows groupable. One caveat remains: subdomains count as separate sites (blog.devgenius.io is not devgenius.io), so map the ones you care about before counting.
Full code
This is the complete serp_to_csv.py that produced the output above. Save it next to keywords.csv and run python serp_to_csv.py keywords.csv results.csv 20; the last argument is the depth.
"""Run a list of Google searches through the BestSERP API and save the organic results to CSV.
python serp_to_csv.py keywords.csv results.csv [depth]
keywords.csv has a header row: keyword,gl,hl (gl and hl may be left empty: us and en).
Tested 2026-11-17 with Python 3.12.10, requests 2.34.2 and pandas 3.1.1;
re-run 2026-11-18 with the wait for final links.
"""
import os
import sys
import time
from datetime import datetime, timezone
import pandas as pd
import requests
API = "https://api.bestserp.com"
RETRY = {429, 502, 503} # rate limited (not charged), search failed or busy (both refunded)
session = requests.Session()
session.headers["X-API-KEY"] = os.environ["BESTSERP_API_KEY"]
def log(message):
print(message, file=sys.stderr)
class SearchError(Exception):
def __init__(self, r):
try:
error = r.json().get("error", {})
except ValueError: # an HTML error page from a gateway, not the API's JSON
error = {"code": "gateway", "message": r.reason}
super().__init__(f"{r.status_code} {error.get('code')}: {error.get('message')}")
self.status = r.status_code
def search(q, gl="us", hl="en", num=10, tries=3, **params):
"""POST /search. Polls 202, retries 429/502/503, raises SearchError on anything else."""
body = {"q": q, "gl": gl, "hl": hl, "num": num, **params}
for attempt in range(1, tries + 1):
r = session.post(f"{API}/search", json=body, timeout=130)
if r.status_code == 202: # not finished within the timeout: poll (free, not rate limited)
request_id = r.json()["requestId"]
log(f" {q!r} ({gl}): 202, polling /requests/{request_id}")
while r.status_code == 202:
time.sleep(2)
r = session.get(f"{API}/requests/{request_id}", timeout=130)
if r.status_code == 200:
return r.json()
error = SearchError(r)
if r.status_code in RETRY and attempt < tries:
wait = float(r.headers.get("Retry-After", 2 ** attempt))
log(f" {q!r} ({gl}): {error}; retry {attempt} in {wait:.0f} s")
time.sleep(wait)
continue
raise error
def wait_for_links(data, max_wait=120):
"""Poll until the final URLs replace Google's redirects: usually within 30 s, up to 2 minutes."""
deadline = time.monotonic() + max_wait
while data.get("linksPending") and time.monotonic() < deadline:
time.sleep(4) # polling is free and not rate limited
r = session.get(f"{API}/requests/{data['requestId']}", timeout=130)
if r.status_code == 200:
data = r.json()
if data.get("linksPending"):
log(f" {data['searchParameters']['q']!r}: links still pending after {max_wait} s, saved as they are")
return data
def to_rows(data):
"""One row per organic result. domain is missing while a link is still a redirect: keep source too."""
p = data["searchParameters"]
searched_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
return [{"keyword": p["q"], "gl": p["gl"], "hl": p["hl"], "position": item["position"],
"domain": item.get("domain"), "source": item.get("source"), "title": item.get("title"),
"link": item.get("link"), "snippet": item.get("snippet"),
"searched_at": searched_at, "request_id": data["requestId"]}
for item in data.get("organic", [])]
def main(keywords_csv, out_csv, depth=20):
num = min(50, max(10, -(-depth // 10) * 10)) # 10..50 in one request: never loop over pages
jobs = pd.read_csv(keywords_csv, dtype=str).fillna({"gl": "us", "hl": "en"})
rows, credits, failed = [], 0, []
for job in jobs.itertuples(index=False):
started = time.monotonic()
try:
data = wait_for_links(search(job.keyword, job.gl, job.hl, num=num))
except (SearchError, requests.RequestException) as e:
log(f"FAILED {job.keyword!r} ({job.gl}): {e}")
failed.append(f"{job.keyword} ({job.gl})")
continue
credits += data["credits"]
rows += to_rows(data)
print(f"{job.keyword:<20} {job.gl}/{job.hl} {len(data['organic']):>2} results {data['credits']} credits"
f" {data['durationMs']:>5} ms total {time.monotonic() - started:>3.0f} s"
f" unresolved links: {data['unresolvedLinks']}")
if not rows:
sys.exit("no results saved")
df = pd.DataFrame(rows)
# The column to group by: domain, or the site name when a link stayed a redirect ("YouTube · channel").
df.insert(5, "site", df["domain"].fillna(df["source"].str.split(" · ").str[0]))
df.to_csv(out_csv, index=False, encoding="utf-8-sig") # the BOM lets Excel detect UTF-8
print(f"saved {len(df)} rows to {out_csv} | credits used: {credits} | failed: {failed or 'none'}")
return df
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2], int(sys.argv[3]) if len(sys.argv) > 3 else 20)
The script runs one search at a time: 38 to 45 seconds per keyword at a depth of 20 in our November 18 run, of which the search itself took about 11 seconds (durationMs) and the rest was the wait for final links; 128 seconds for all three, far below the limit of 60 searches a minute. That is fine for a few hundred keywords; for thousands, run several at once and pace them by the rate-limit headers, as in running bulk SERP API requests reliably.
Common problems and fixes
The errors below are the ones we hit or provoked while writing this guide; the 400 messages are copied from real responses, which cost nothing.
| Problem | Cause | Fix |
|---|---|---|
KeyError: 'BESTSERP_API_KEY' |
The variable is not set in this terminal | Export it in the same shell that runs Python |
400 invalid_request: num must be a multiple of 10 |
num: 15 or another odd value |
Use 10, 20, 30, 40 or 50 and slice |
400 invalid_request: page x num/10 must be at most 5 |
num: 20 with page: 3 asks for 60 results |
Stop at 50 results |
400 unknown_parameter: start |
A parameter from another API | Use num, or page for one page |
400 unsupported_parameter: location |
City targeting | Use gl and hl; put the city in q |
KeyError: 'domain' |
YouTube, LinkedIn and Medium results have no domain until their final links arrive | item.get("domain"), then the site column |
Links start with https://www.google.com/goto |
Read before link resolution finished (linksPending: true), or still pending after the 2-minute wait (the script logs it) |
Keep wait_for_links() in the loop; run a logged keyword again later; group on domain or site |
| The same page twice for one keyword | It appears on page 1 and page 2 | df.drop_duplicates(["keyword", "gl", "site", "title"]) |
| Accented letters look wrong in Excel | CSV saved without a BOM | encoding="utf-8-sig" |
502 search_failed after every retry |
Google kept blocking that search | Run the failed keywords again later; nothing was charged |
Next steps
You now have Google search results in Python as a DataFrame and a CSV, with the failure cases handled. Where to take it next:
- Track positions every day and keep the history: build a rank tracker in Python, which adds SQLite and a schedule.
- Run thousands of keywords: bulk SERP API requests.
- Compare markets properly: the gl and hl parameters explained.
- Turn a keyword's top 10 into a writing plan: create an SEO content brief with Python.
- Check parameters and fields as you extend the script: the web search reference and the Google Search API page.