How to Scrape Customer Reviews (and How Many Each Platform Gives Up)

Scraping customer reviews is a pagination problem, and the platform decides how far the pagination goes. I picked one well-reviewed entity on each of six platforms and pulled reviews until the site stopped, showed a login page, or returned an empty page. Yelp gave up 6,134 of the 6,137 reviews it lists for a San Francisco restaurant. Google Maps handed over every one of a bakery’s 230 reviews and was still returning new pages at review 248 of 54,696 for Katz’s Delicatessen. TripAdvisor stopped at about 13,550 of the 18,674 it lists for the same deli. Trustpilot showed 20 reviews a page for ten pages and then its login form. Amazon showed 8 of 198,597 ratings on a product page and redirected the full list to sign-in. Glassdoor showed 3 of 211,114 and then asked for an account.

Those six numbers are the plan for any review project, before a line of code. The rest of this article shows the method behind them, the code for the two platforms that paginate cleanly, the pattern for detecting a wall, and where each platform’s data lives, with links to the guides that go deeper on Google Maps and Amazon.

What the reviews are used for

Reputation monitoring is the usual reason, tracking your own ratings across Google Maps, Yelp, and Trustpilot week by week. Competitor analysis is the second, reading what customers of the other brand complain about. Sentiment models need a training set of a few thousand labelled reviews, product teams read one-star reviews for defect reports, and fraud teams look for bursts of five-star reviews from accounts with one review each. Every one of those uses needs more than the first page, so the number that matters is how many reviews a platform lets you reach.

How many reviews you can actually get

The method is the same for all six platforms. Take one entity with a large, publicly stated review count. Fetch its reviews the way the site itself pages them, through the platform’s own pagination parameter or, for Google Maps and Yelp, through the HasData endpoints that page the same lists. Count what comes back, deduplicate by review ID, and keep going until one of three things happens: the site stops returning reviews, it returns a login page, or the page comes back empty. Amazon, Trustpilot, TripAdvisor, and Glassdoor answer a plain requests call with a connection check or a shell page, so those pages were rendered in a browser through a scraping API with residential proxies, one request per page, from one machine on one day.

Yelp is the only platform in the set that lets you reach the whole list by offset. The start parameter accepted every value up to the total, the call at offset 6,125 returned the last 9 reviews for Gary Danko and the call at 6,187 returned nothing, so the three missing reviews are ones Yelp itself filters. Tartine Bakery, with 9,194 listed, ended the same way at 9,162. Yelp keeps a second list, the “not recommended” reviews the site hides below the fold, and the API pages that list separately.

Google Maps pages by opaque token. The first page carries 8 reviews and every later page 10, each response holds the token for the next, and for the bakery the tokens ran out after 24 pages with exactly 230 unique reviews, the number the listing shows. For Katz’s I stopped at 25 pages and 248 reviews with the token still valid, so the ceiling on a 54,696-review place is your patience and your budget, not the platform. Smør Bakery, listed at 265, gave 253 unique reviews in 27 pages before the tokens ran out, so Google’s own count and the paginated list disagree by a few percent, probably reviews without text or reviews Google filters after counting.

TripAdvisor addresses pages by offset in the URL (-or10-, -or20-) with 15 review cards per page. Offsets 10, 100, 1,000, and 12,000 returned 15 cards, a binary search that retried every empty page put the end at 13,540, and single probes on other runs got cards at 13,620 and nothing at 13,680, so about 13,550 of the 18,674 listed reviews are reachable, give or take a hundred. The runs disagree because a blocked render also has zero cards, so the search below asks twice before it believes an empty page. The page’s language filter defaults to English, and the gap is most likely the other-language reviews the default view hides. Datacenter proxies got an empty shell titled “tripadvisor.com” on every request, residential ones got the page.

Trustpilot renders 20 reviews per page and puts the whole list into the page’s __NEXT_DATA__ JSON, which makes parsing a one-liner. The default view is the English subset, 2,849 of Spotify’s 5,783 reviews across 143 pages. Pages 1 to 10 returned their 20 reviews each, and page 11 and every page after it returned Trustpilot’s login form, so an anonymous client gets 200 reviews of 5,783. A plain requests call never got that far, since every attempt without a browser ended on a “Verifying Connection” page.

Amazon’s product page shows eight reviews under “Top reviews from the United States” and a link to see more. The link leads to /product-reviews/ASIN/, and that page redirected to the sign-in form on every attempt. The product data endpoint returns the totals, the star breakdown, and Amazon’s own AI summary of the reviews, but no review texts, because there are none on the page to return. Without a signed-in session, eight is the number.

Glassdoor’s company page rendered three reviews and a sign-in prompt, and pages 2 and 5 of the same list loaded Glassdoor’s login page instead. Both of those platforms made a decision about anonymous readers, and no client configuration changes it.

How to scrape customer reviews

The cleanest platform to start with is Yelp, because the list is addressable by offset and tells you when it ends. This script pulls the newest reviews for one business through the Yelp Reviews API, which pages the same list the website shows, and writes them to CSV:

import csv
import os

import requests

# Yelp shows 10 reviews per page on the site. The Yelp Reviews API pages the same list
# with start/num (num up to 49) and tells you when the list ends.
API = "https://api.hasdata.com/scrape/yelp/reviews"
HEADERS = {"x-api-key": os.environ["HASDATA_API_KEY"]}
PLACE_ID = "WavvLdfdP6g8aZTtbBQHTw"  # Gary Danko, San Francisco (from the Yelp Search API)

start, rows = 0, []
while True:
    params = {"placeId": PLACE_ID, "start": start, "num": 49, "sortBy": "dateDesc"}
    data = requests.get(API, headers=HEADERS, params=params, timeout=120).json()
    reviews = data.get("reviews", [])
    for r in reviews:
        rows.append({"id": r["id"], "date": r["date"], "rating": r["rating"], "user": r["user"]["name"], "text": r["comment"]["text"]})
    if not data["pagination"]["hasNextPage"] or not reviews:
        break
    start = data["pagination"]["nextPageStart"]
    if start >= 200:  # remove the cap for a full pull; 6,137 reviews is 126 calls
        break

with open("gary_danko_reviews.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)
print(len(rows), "reviews saved, newest:", rows[0]["date"], "oldest in this batch:", rows[-1]["date"])

The run saved 245 reviews in five calls, the newest from a week before the run and the oldest in that batch from almost three years back, which is how far 245 reviews reach on a restaurant that collects about ninety a year. The loop ends on hasNextPage, and the demo cap at 200 exists so the example finishes in seconds. A full pull is 126 calls, one per 49 reviews.

Google Maps has no offset. Each response carries a token for the next page and nothing tells you how many pages exist, so the loop runs until the token disappears and deduplicates by review ID along the way:

import json
import os

import requests

# Google Maps pages by opaque token, not by number: each response carries the token for the
# next page and nothing else tells you how many pages there are. Dedupe by reviewId, because a
# token fetched twice returns the same page.
API = "https://api.hasdata.com/scrape/google-maps/reviews"
HEADERS = {"x-api-key": os.environ["HASDATA_API_KEY"]}
DATA_ID = "0x89c2598f7ff4aa09:0x313547e757cb8cea"  # Katz's Delicatessen (from the Google Maps Search API)

seen, token, pages = {}, None, 0
with open("katz_reviews.jsonl", "w", encoding="utf-8") as out:
    while pages < 5:  # 5 pages for the demo; the token kept going past page 25 in my test
        params = {"dataId": DATA_ID, "sortBy": "newestFirst"}
        if token:
            params["nextPageToken"] = token
        data = requests.get(API, headers=HEADERS, params=params, timeout=120).json()
        for r in data.get("reviews", []):
            if r["reviewId"] not in seen:
                seen[r["reviewId"]] = r
                out.write(json.dumps({"id": r["reviewId"], "date": r["isoDate"], "rating": r["rating"], "text": r.get("snippet")}) + "\n")
        pages += 1
        token = data.get("pagination", {}).get("nextPageToken")
        if not token:
            break
print(pages, "pages,", len(seen), "unique reviews, more pages:", bool(token))

Five pages produced 48 unique reviews (8 on the first page, 10 on each of the next four) with the token still live. The dataId comes from the Google Maps Search API, and the four sort orders (mostRelevant, newestFirst, ratingHigh, ratingLow) change which reviews come first, which matters when you cap the pull. The full walkthrough of the place lookup, the fields, and the export is in the guide to scraping Google Maps reviews.

The third pattern is the one for platforms that stop you. Trustpilot’s list is paginated by page number, the reviews sit in the page’s __NEXT_DATA__ JSON, and after a certain page the site serves its login form instead of the list. A loop that treats the wall as a stop condition rather than an error ends cleanly with everything the platform allowed:

import json
import os
import re

import requests

# Trustpilot answers a plain request with a connection check, so the page goes through a
# rendering API. The reviews sit in the page's __NEXT_DATA__ JSON, 20 per page, and past
# a certain page the site returns its login page instead. Stop on the wall, not on an error.
API = "https://api.hasdata.com/scrape/web"
HEADERS = {"x-api-key": os.environ["HASDATA_API_KEY"], "Content-Type": "application/json"}
NEXT_DATA = re.compile(r'<script id="__NEXT_DATA__" type="application/json">(.*?)</script>', re.S)

reviews, page = [], 1
while page <= 3:  # raise the limit for a full pull; the wall decides where it ends
    body = {"url": f"https://www.trustpilot.com/review/www.spotify.com?page={page}",
            "jsRendering": True, "proxyType": "residential", "wait": 8000, "outputFormat": ["html"]}
    html = requests.post(API, headers=HEADERS, json=body, timeout=240).text
    match = NEXT_DATA.search(html)
    if not match or "Trustpilot Login" in html[:2000]:
        print(f"page {page}: login wall or no data, stopping")
        break
    props = json.loads(match.group(1))["props"]["pageProps"]
    batch = props.get("reviews", [])
    reviews.extend({"id": r["id"], "rating": r["rating"], "title": r["title"], "date": r["dates"]["publishedDate"], "text": r["text"]} for r in batch)
    total_pages = props["filters"]["pagination"]["totalPages"]
    print(f"page {page}/{total_pages}: {len(batch)} reviews ({props['filters']['pagination']['totalCount']} in the current language filter)")
    if page >= total_pages:
        break
    page += 1
print(len(reviews), "reviews collected")

Three pages, 60 reviews, and the pagination block told the script there were 143 pages of 2,849 English reviews, of which the anonymous session gets the first ten. The residential proxy with the eight-second wait was the combination that got past Trustpilot’s connection check, and the same request through datacenter proxies without the wait came back as the check page.

When a platform pages by offset and you want to know where the list ends without walking it, search for the end instead. TripAdvisor’s 18,674 reviews at 15 per page are 1,245 renders to walk, and a binary search over the offset finds the last page in about a dozen:

import os
import re

import requests

# Walking 18,674 reviews at 15 per page is 1,245 renders just to learn where the list ends.
# A binary search over the offset finds the end in about a dozen.
API = "https://api.hasdata.com/scrape/web"
HEADERS = {"x-api-key": os.environ["HASDATA_API_KEY"], "Content-Type": "application/json"}
PAGE = "https://www.tripadvisor.com/Restaurant_Review-g60763-d425787-Reviews-or{offset}-Katz_s_Deli-New_York_City_New_York.html"


def cards_at(offset):
    body = {"url": PAGE.format(offset=offset), "jsRendering": True, "proxyType": "residential", "outputFormat": ["html"]}
    html = requests.post(API, headers=HEADERS, json=body, timeout=240).text
    return len(re.findall(r'data-automation="reviewCard"', html))


def last_offset_with_reviews(low, high, step=10):
    while high - low > step:
        mid = (low + high) // 2 // step * step
        if cards_at(mid):
            low = mid
        else:
            high = mid
    return low


last = last_offset_with_reviews(0, 18674)
print(f"reviews stop between offset {last} and {last + 10}, about {last + 15:,} reachable of 18,674 listed")

The run printed reviews stop between offset 13530 and 13540, about 13,545 reachable of 18,674 listed after 24 renders, two per probe. Without the retry, one blocked render in the middle of the search moves the answer by hundreds of reviews, which is what happened on the first attempt.

Pagination and caps, platform by platform

Three pagination shapes cover the six platforms, and each one fails differently when you hit the end.

Two of those cells decide most projects. The language filter on TripAdvisor and Trustpilot means the number you see on the listing is not the number the default view will ever give you, and switching the filter is the difference between 2,849 and 5,783 reviews for Spotify. The sort order matters as soon as you cap a pull, because 200 reviews sorted by relevance and 200 sorted by date are different samples, and a sentiment model trained on the first learns the platform’s ranking rather than the customers’ opinion. Sort by date for monitoring and by rating for defect hunting, and record the sort with the data.

Deduplication is the other constant. Google Maps tokens fetched twice return the same page, Yelp’s offsets shift when a new review lands mid-pull, and Trustpilot’s page 1 changes under you as reviews arrive. Every platform gives each review an ID, and a set of IDs keeps the same review from being counted twice.

Where each platform’s reviews live

Where the reviews sit, how they paginate, and what a page costs differs per platform, and the table is the short form of the sections above.

Google Maps has the most reviews of any platform and the most complete access, and the dedicated guide covers the place lookup, the fields, and the export formats. Amazon’s review texts are behind a sign-in for the full list, so the reachable data is the product page, eight top reviews, the rating breakdown, and Amazon’s own review summary, which the product guide extracts together with prices and variants. Yelp is the platform where a full history is realistic, 6,134 reviews in 126 calls, and the Yelp guide shows the same pagination from Node.js. TripAdvisor and Trustpilot need a rendering client with residential proxies, which is what the Web Scraping API provides, and both stop where the results table says they stop. The credits add up differently per platform: the 230-review bakery on Google Maps is 24 pages and 120 credits, Gary Danko’s 6,134 Yelp reviews are 126 calls and 1,260 credits, Trustpilot’s 200 anonymous reviews are 10 renders and 150 credits, and TripAdvisor’s 13,550 reviews are 904 renders and about 13,600 credits, which is the number to put in front of whoever asked for the data. Glassdoor’s three anonymous reviews are a sample, and the pages the Glassdoor Listing API covers are job listings, so employer reviews stay behind the login.

Is it legal to scrape reviews?

Reviews on public pages can be collected as long as the site’s terms and the local law are respected, and the answer changes with the jurisdiction, the use, and the personal data inside the review text. Names and profile links are personal data under GDPR and the CCPA, so store what the analysis needs and drop the rest. The legal guide to web scraping goes through the cases and the rules.

Conclusion

The first question of a review project is how many reviews the platform will give you, and the answer is a number you can measure in an afternoon. Yelp and Google Maps give the whole list. TripAdvisor and Trustpilot give the default-language subset, and Trustpilot only its first ten pages. Amazon and Glassdoor give a handful and keep the rest for signed-in users. Page by offset where you can, by token where you must, stop on the wall instead of crashing into it, sort deliberately, deduplicate by ID, and read the platform guide before writing the platform’s scraper.