SERP API vs Web Scraping: Which Is Better for Google Results?
Michael Carter · · 14 min read

Key takeaways
- In our November 2026 tests on 100 datacenter proxies, no Google search returned results while navigator.webdriver was exposed, and 28% did once it was hidden.
- IP quality set the ceiling: a residential connection got about 70% of searches through, while 62 of 100 datacenter proxies never returned a single result in five tries.
- Google's Images tab was not easier to scrape than web search: on the same proxies, 1 search in 100 returned results.
- In our cost model, scraping Google yourself costs $345 a month for 10,000 searches and $3,873 for 1 million, 5 to 9 times the cost of a SERP API.
- A SERP API does not end failures, it moves them to the provider: BestSERP refunds failed searches, but clients still retry 502 and 503 responses.
SERP API vs web scraping: for production work, a SERP API wins on cost and reliability. Scraping Google yourself makes sense for learning, tiny volumes or data no API returns. In our November 2026 tests, most searches from datacenter proxies hit a CAPTCHA, and our cost model puts building at 5 to 9 times the cost of buying.
This comparison is for developers and SEO teams deciding whether to build their own Google scraper or pay for an API. It draws on what we learned building the scraper behind BestSERP, a Google SERP API: paired blocking tests on 100 proxies, a total-cost table with every price sourced and every assumption labeled, and a verdict for each use case.
SERP API vs web scraping at a glance
A SERP API moves the hard part of getting Google results (access, parsing and upkeep) to a provider that charges per search. Scraping keeps all of it, and all of its cost, with you. If the term is new, start with what a SERP API is and how it works; a SERP is simply the search engine results page.
| SERP API | Scraping Google yourself | |
|---|---|---|
| What you get | Parsed results as JSON | Raw HTML that your own parser turns into data |
| What you run | Nothing: one HTTPS request per search | Headless browsers, proxies, a job queue and a parser |
| Blocks and CAPTCHAs | The provider's problem; BestSERP refunds failed searches | Yours: 72% of our datacenter-proxy searches were blocked even with the automation flag hidden |
| Cost | Per search: $0.50 per 1,000 at BestSERP | Proxy traffic, servers and engineer time: about $987 a month for 100,000 searches in our model |
| Another country | A parameter (gl) |
IP addresses that work in that country |
| When Google changes its pages | The provider updates its parser | You update yours |
| Control | Only the fields the provider returns | Anything on the page |
| Best for | Production and scheduled jobs | Learning, tiny volumes, special needs |
What scraping Google yourself involves in 2026
Scraping Google in 2026 means running a real browser, not sending a simple HTTP request. Google began requiring JavaScript for Search in mid-January 2025; a Google spokesperson told TechCrunch on January 17, 2025 that "Enabling JavaScript allows us to better protect our services and users from bots and evolving forms of abuse and spam." Many scrapers built on plain HTTP requests and an HTML parser stopped returning results.
A homemade scraper today has five moving parts:
- Browser workers. Each search loads Google in a real Chrome or Chromium browser, driven by a browser automation library such as Playwright. In our tests, a page load that returned results took a median of 3.9 seconds, and every browser holds memory.
- IP addresses. Google blocks by IP address, so steady volume needs a pool of proxies and a way to test them and drop the ones that stop working.
- Interstitial pages. Google can answer with a cookie consent page or a CAPTCHA instead of results; your code must recognize both.
- A parser. CSS selectors turn the page into titles, links and positions. When Google changes its markup, they silently return nothing.
- A queue, retries and monitoring. Something has to schedule searches, retry the blocked ones on another IP and tell you when the success rate drops.
If a Python library stopped working for you, why googlesearch-python fails with 429 errors covers that case.
Why Google blocks scrapers: what we measured
Google blocks traffic that looks automated. In our tests, three things decided almost every outcome: whether the browser announced that it was automated, the reputation of the IP address, and which Google tab was requested.
Method in brief: on November 8, 2026, while building BestSERP's scraper, we ran paired A/B tests on 100 datacenter proxies from one commercial provider. Every proxy ran every browser setup once, in its own random order, with 20 everyday queries (gl=us, hl=en); a search counted as blocked when Google redirected it to its CAPTCHA page.

The automation flag: 0% with it, 28% without
navigator.webdriver is a browser property that tells websites the browser is controlled by automation software. With it exposed, 0 of the 99 searches Google answered returned results. Hiding it, with the same browser and the same proxies, raised the rate to 28 of 100. In a second run, five different browser setups that all hid the flag landed between 23% and 29%: once the flag was gone, further browser changes bought nothing we could measure.
The IP address sets the ceiling
On a residential connection, our scraper got about 70% of searches through, against 28% on the datacenter proxies. Among the proxies, quality was close to all or nothing, and it belonged to the IP, not to chance.

Across five searches each, 62 of the 100 proxies never got a single result, while 15 got results every time and delivered 75 of the 135 successful searches. A scraper therefore pays for many IPs to keep a few, and has to keep testing them.
Other Google tabs are not easier
The Images tab returned results for 1 of 100 searches on the same proxies, worse than web search. We then took the 19 proxies that had passed web search and tried other tabs with Vietnam as the target country (gl=vn): Maps returned results 17 times, but News and Shopping only once each, and Videos and Places twice each. Every vertical is its own blocking problem.
One network setting broke every request
Through these proxies, all 27 attempts over HTTP/2 timed out after 20 seconds, while the same proxies answered over HTTP/1.1. Nothing in the error said why; we found it only because the test changed one setting at a time.
What a block looks like
All 831 blocked requests in our two main runs ended on google.com/sorry/, the CAPTCHA page that says "Our systems have detected unusual traffic from your computer network." We are not publishing a recipe for avoiding it. The point of these numbers is cost: at a 28% success rate, each successful search takes about 3.6 attempts, and every attempt burns proxy traffic and browser time.
How much does it cost to scrape Google? Build vs buy
In our model, scraping Google yourself costs about $345 a month for 10,000 searches and $3,873 for 1 million, against $70 and $565 for a SERP API including an hour of your engineer's time. Proxy traffic alone costs more per search than the API does.
Prices come from public pages checked on November 17, 2026; everything else is a labeled assumption.
| Input | Value | Basis |
|---|---|---|
| Attempts that return results | 70% | Our residential connection reached about 70%; datacenter proxies reached 28% |
| Data per attempt | 0.5 MB | Assumption. One results page we saved was 0.64 MB of HTML before compression, and pages also load scripts and images |
| Residential proxy traffic | $5, $4 and $3 per GB as volume grows | Public price lists of Bright Data and Oxylabs ranged from $2.50 to $8 per GB, by volume and promotion |
| Servers | $48 a month each: one up to 100,000 searches, two at 1 million | DigitalOcean Basic droplet, 4 vCPUs and 8 GB |
| Engineer time | $65.38 an hour | US BLS median wage for software developers, $135,980 a year in May 2025, before benefits and overhead |
| Maintenance | 4, 10 and 25 hours a month | Assumption: parser fixes, watching block rates, replacing IPs |
| One-time build | 80 hours ($5,230) | Assumption; left out of the monthly totals |
| SERP API | $0.50 per 1,000, plus 1 engineer hour a month | BestSERP pricing; failed searches are refunded |
And the result, per month:
| Cost line | 10,000 searches | 100,000 searches | 1,000,000 searches |
|---|---|---|---|
| Attempts needed | 14,286 | 142,857 | 1,428,571 |
| Proxy traffic | $36 (7.1 GB) | $286 (71 GB) | $2,143 (714 GB) |
| Servers | $48 | $48 | $96 |
| Engineer maintenance | $262 | $654 | $1,634 |
| Build total | $345 | $987 | $3,873 |
| Build per 1,000 searches | $34.52 | $9.87 | $3.87 |
| SERP API | $5 | $50 | $500 |
| Your engineer (1 hour) | $65 | $65 | $65 |
| Buy total | $70 | $115 | $565 |
| Buy per 1,000 searches | $7.04 | $1.15 | $0.57 |

Two assumptions move the result most: data per page and success rate. Halve the data to 0.25 MB and building 1 million searches drops to about $2,800 a month, still about five times the buy total. Proxy traffic and servers alone come to $2.24 per 1,000 searches at that volume, more than four times the API price before anyone spends an hour on maintenance. On cost alone, the SERP API vs web scraping question is settled at every volume we modeled.
What we left out makes building look cheaper than it is. Datacenter proxies cost less per IP, but 62 of our 100 never worked, and we could not measure how many searches a good IP sustains per day. Paid CAPTCHA-solving services would add a fee per solve; we modeled blocks as retries instead. Providers price differently from each other too: our SERP API pricing comparison normalizes them at several volumes.
Reliability and maintenance: what a SERP API does not fix
A SERP API does not make Google's blocking disappear; it moves the failure handling, and the cost of it, to the provider. You still see failures, but you stop paying for them.
BestSERP is not magic either. Across 52 searches on November 17, 2026, its median processing time was 4.2 seconds (3.4 to 11.1 seconds). Some searches fail with 502 search_failed when Google blocks or times out, and the system answers 503 when it is busy; both are refunded, and a retry usually succeeds. The status page shows the daily success rate. A client still needs failure handling; this is the whole "buy" side, tested on November 17, 2026 with Python 3.12.10 and requests 2.34.2:
import os
import sys
import time
import requests
API = "https://api.bestserp.com"
HEADERS = {"X-API-KEY": os.environ["BESTSERP_API_KEY"]}
def search(q, gl="us", hl="en", attempts=3):
for attempt in range(1, attempts + 1):
resp = requests.post(f"{API}/search", headers=HEADERS,
json={"q": q, "gl": gl, "hl": hl}, timeout=130)
if resp.status_code == 202: # still running: poll the request ID
request_id = resp.json()["requestId"]
while resp.status_code == 202:
time.sleep(3)
resp = requests.get(f"{API}/requests/{request_id}", headers=HEADERS, timeout=130)
if resp.status_code in (502, 503): # blocked, timed out or busy: refunded, retry
print(f"attempt {attempt}: {resp.status_code}, retrying", file=sys.stderr)
time.sleep(int(resp.headers.get("Retry-After", 5)))
continue
if resp.status_code == 429: # over the per-minute limit: wait, then retry
time.sleep(int(resp.headers.get("Retry-After", 60)))
continue
resp.raise_for_status() # 400, 401, 402: fix the request, never retry
return resp.json(), attempt
raise RuntimeError(f"no result after {attempts} attempts")
data, attempt = search("best running shoes")
for r in data["organic"][:5]:
print(r["position"], r.get("domain", "-"), r["title"][:60])
print(f"organic: {len(data['organic'])} | attempt: {attempt} | credits: {data['credits']}"
f" | cached: {data['cached']} | durationMs: {data['durationMs']}")
Our run, with a query from the blocking tests, shows the point in miniature: the first attempt failed with a 502 (refunded), and the retry returned results.
attempt 1: 502, retrying
1 runrepeat.com 7 Best Running Shoes in 2026
2 runnersworld.com The 15 Best Running Shoes of 2026
3 - What are the best running shoes that you'd actually buy ...
4 sixminutemile.com Best Running Shoes of the Year: 6 Top-Shelf Models
5 nike.com Men's Best Sellers Running Shoes
organic: 8 | attempt: 2 | credits: 1 | cached: False | durationMs: 9474
What stays with you after you buy:
- Retries and limits. Retry
502and503, wait out429(60 searches a minute by default), never retry400to402; the error reference lists every code. Treat fields such asdomainas optional. - The provider's coverage. BestSERP returns desktop results by country and language, without ads, People Also Ask, AI Overviews or city-level targeting.
- Provider risk. Products change and close; Google's own Custom Search JSON API is closed to new customers, as our Custom Search JSON API migration guide explains.
The build side carries the maintenance itself. When our scraper finds a page without results, its error message reads "no matches for this query, or Google changed its layout": the code cannot tell a real empty result from a broken parser, so a person has to look. In the week after the tests above, we changed how our own scraper reaches Google twice. Established tools face the same churn: when Google stopped honoring the parameter that showed 100 results per page in September 2025, rank trackers had to change how they collect positions, as our keyword rank tracking guide describes.
When building your own Google scraper makes sense
Build when the goal is learning, when the volume is tiny, or when you need something no API gives you.
- Learning. Driving a browser through a Google search teaches you how results pages render and change. Expect blocks.
- A few hundred searches a month. From your own connection, at a slow pace, a scraper can work well enough; ours got about 70% through on a residential line. Money barely matters at this size: 300 searches a month cost $0.15 at BestSERP's price, so you are trading your time, not your budget.
- Data the APIs do not return. BestSERP does not return ads, featured snippets, People Also Ask, knowledge panels, AI Overviews or mobile results. Coverage differs between providers, so ask for a sample response and see how to choose a SERP API. If no provider has what you need, such as a full-page screenshot, building is the honest answer.
- You already run a crawling fleet. A team that already operates browsers and good IP addresses at scale pays far less for one more target.
When a SERP API is the better choice
Buy when the results feed something that runs every day and people depend on it.
- Scheduled production jobs. Rank tracking, price checks and monitoring need the same searches to succeed every day; a failure leaves a gap in the data.
- Many countries. With an API, another market is a
glvalue. With a scraper, it is another set of IPs that work there, with their own block rates. - A small team. At 100,000 searches a month, our model assumes 10 engineer hours a month of upkeep; that time usually has a better use.
- AI agents. An agent needs an answer in seconds, not a browser fleet; our comparison of a search API vs a headless browser for agents goes deeper.
- Predictable cost. Failed searches are refunded, so the bill follows your volume, not Google's block rate.
To try it with code, the Google Search API page and the quickstart take a few minutes; our Python guide to Google search results builds a complete script with pagination and CSV export.
How we compared
We compared SERP API vs web scraping on cost and reliability, using our own measurements and public prices.
- Blocking tests (November 8, 2026). 100 datacenter proxies from one commercial provider; Playwright driving Chromium and Google Chrome; 20 everyday queries; gl=us, hl=en. Run 1 measured the automation flag and the Images tab (400 requests); run 2 compared five setups that hide the flag, plus the Images tab (600 requests). An earlier run testing HTTP/2 was stopped after 27 requests on 12 proxies, all timed out. The tab test used the 19 proxies that passed web search, gl=vn, 133 requests. The residential figure comes from a separate check on our own connection.
- Not tested. Residential proxy pools (the 70% is our own line), CAPTCHA-solving services and signed-in browser profiles.
- Cost model. Prices from the pages linked above, checked on November 17, 2026; assumptions labeled in the inputs table.
- BestSERP figures. Pricing and limits from our pricing page and docs; latency from 52 searches on November 17, 2026; the code sample was run the same day.
- Scope. This post compares cost and reliability. It does not cover the legal side of scraping, which depends on your jurisdiction and your use. This is general information, not legal advice.
Next steps
So, SERP API vs web scraping? Scrape to learn or for a handful of searches; buy once results feed something people rely on. From here:
- New to the topic: a practical guide to SERP APIs.
- Comparing providers: the best SERP API options in 2026.
- Ready to test: 3,000 free searches (valid for 30 days), then $0.50 per 1,000.