How to Scrape Google Shopping with Python, Even If You're a Beginner

Google Shopping puts the same product from several merchants on one screen with prices next to each other. The product data is available from any store. The comparison is what’s only here.

Two things about the page have changed since most tutorials on it were written, and both change the code you need. The search parameter is different, and the results are no longer in the HTML that a plain request gets back.

What the Google Shopping URL Does Now

The old shopping URL was google.com/search?q=books&tbm=shop. Request it today with a browser User-Agent and Google answers 200, then redirects to google.com/search?q=books&udm=28. The tbm=shop parameter still works as an entry point, but it isn’t where you land, and udm=28 is the address to write into new code.

Send the same request without a User-Agent header and the shopping parameter is dropped altogether. Both tbm=shop and udm=28 come back as google.com/search?q=books, an ordinary web search with no products in it at all. A script that forgets its headers doesn’t get blocked here, it quietly gets the wrong page.

That’s the page a browser draws. What a script receives is a different thing.

What a Plain Request Actually Returns

Fetching that URL with requests gives you 200 and about 92 KB, which looks like success until you parse it. That response holds two div elements, five script tags, and no class attributes at all. Stripped of its scripts it comes to 410 characters of visible text, and the text is a redirect notice telling the browser to follow a link if JavaScript hasn’t already done it.

Nothing in it matches sh-dgr__content, the class most older tutorials reach for, because there are no classes to match. There are no prices either. The products arrive after the JavaScript runs, so any approach built on parsing the first response finds an empty list and reports zero results on a search that worked.

That leaves two routes that do return data. Render the page in a real browser, or ask an API that renders it for you.

The class names in that panel are generated per layout, which is why copying one into a script buys you weeks rather than months.

Getting the Data Without Writing Code First

Before writing a scraper, it’s worth seeing the shape of the data you’ll be parsing. Sign up at HasData, open the Google SERP API section, and the request builder puts a query together for you.

Set the search type to Google Shopping. The same builder covers Images, News, ordinary SERP, Videos and Local results, and every parameter it exposes is described in the documentation. Fill in a keyword and run it.

What comes back is JSON you can read before committing to a parser.

The builder is a way to look at the response, not a way to run a job. The rest of this guide puts the same request in Python.

What One Page of Results Carries

A single request for books against google.com in the US returned 56 products. Not every product carries every field, which matters when you build a table out of them:

Price and merchant are always there, so a price monitor has what it needs on every row. Ratings are missing from about one product in ten, and originalPrice, which is what tells you something is discounted, shows up on 8 of the 56. Code that assumes a discount field on every row breaks on the first page it meets.

extractedPrice is worth knowing about. price is the string Google displays, $10.18, and extractedPrice is the same value as a number, so you can sort and compare without parsing currency out of text.

Scraping Google Shopping with Python

Install the two libraries. requests sends the query and pandas writes the spreadsheet:

pip install requests pandas openpyxl

The whole script fits in one file. It reads a keyword, calls the Google Shopping API, and saves what comes back:

import requests
import pandas as pd

API_URL = "https://api.hasdata.com/scrape/google/shopping"
HEADERS = {"x-api-key": "YOUR-API-KEY"}


def fetch_shopping(keyword, gl="us", hl="en"):
    params = {"q": keyword, "domain": "google.com", "gl": gl, "hl": hl}
    response = requests.get(API_URL, params=params, headers=HEADERS, timeout=60)
    response.raise_for_status()
    return response.json().get("shoppingResults", [])


def save(rows, filename="shopping_result.xlsx"):
    if not rows:
        print("no results")
        return
    df = pd.DataFrame(rows)
    df.to_excel(filename, index=False)
    print(f"{len(rows)} rows written to {filename}")


if __name__ == "__main__":
    save(fetch_shopping("books"))

raise_for_status matters more than it looks. Without it a failed call returns a dictionary with no shoppingResults key, pandas writes an empty file, and the script reports success.

Every column in that file comes from the JSON keys, so a new field on Google’s side turns into a new column rather than an error. Swapping to_excel for to_csv is the only change needed if you’d rather have CSV.

Reading One Product Across Several Stores

The results page gives one price per product, from whichever merchant Google puts first. The comparison lives a level deeper, and every result carries productId and immersiveProductPageToken to get you there.

Feeding that token to the Immersive Product API returns the merchant list along with the rating, the description, the specifications and the variants:

def fetch_offers(token, attempts=3):
    for attempt in range(attempts):
        response = requests.get(
            "https://api.hasdata.com/scrape/google/immersive-product",
            params={"pageToken": token, "moreStores": "true"},
            headers=HEADERS,
            timeout=60,
        )
        if response.status_code == 200:
            return response.json()["productResults"]
    response.raise_for_status()


results = fetch_shopping("books")
product = fetch_offers(results[0]["immersiveProductPageToken"])
for store in product["stores"]:
    print(store["name"], store["price"], store["shipping"], store["total"])

The retry loop is there because this endpoint renders the heaviest page in the article and answers 400 on some runs with the same parameters that succeed a moment later. Two of three calls failed that way while this section was being checked, so a single attempt with raise_for_status crashes on a request that was going to work.

For one paperback the three merchants came back like this:

Two of the three list the identical price and differ by a dollar once shipping is added. Sorting on price can’t separate them at all, and Google’s own “Best price” tag appears on the row with the lowest total rather than the lowest price. If you’re building a price monitor, total is the field to store.

The response carries a storesNextPageToken when Google has more merchants than it returned, which is how you page through a long offer list rather than settling for the first few.

Running a Browser Instead

Selenium drives a real Chrome, so the JavaScript runs and the products land in the DOM where a selector can reach them. That keeps the whole job on your own machine.

Modern Selenium finds and downloads the matching driver on its own, so the manual chromedriver.exe path that older guides show isn’t needed any more:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

URL = "https://www.google.com/search?q=books&udm=28"

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

try:
    driver.get(URL)
    WebDriverWait(driver, 15).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, "a h3"))
    )
    for card in driver.find_elements(By.CSS_SELECTOR, "a h3"):
        print(card.text)
finally:
    driver.quit()

The wait is the part worth copying. driver.get returns as soon as the document loads, which on this page is before any product exists, so a script that reads the DOM immediately gets the same empty list that requests gave it. WebDriverWait holds until something real appears.

Selectors are the weak point of this route. The class names on the shopping page are generated, they change without notice, and anchoring on a structural relationship like a heading inside a link survives longer than anchoring on a class. Our CSS selectors cheat sheet covers the patterns that hold up best. Even so, expect to revisit them, and log the row count per run so a redesign shows up as zeros rather than as silence.

Sending the Right Headers

If you write your own fetch layer, set a User-Agent that matches a browser you actually have. The advice to copy your own is right, and the string has to be current. A User-Agent from an iPhone running iOS 13 tells Google to serve a mobile layout with different markup, which is a second reason a desktop-shaped parser finds nothing.

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 15_8_0) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/153.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

Accept-Language does more here than it does on most sites, since it decides the currency and the merchant set you see. Ask for en-US from a European exit address and you can get US merchants with European prices, which is a mismatch that only shows up when someone questions the numbers weeks later.

Asking Through MCP

The same endpoint also answers an AI client over MCP, so a one-off question does not need a script at all. The configuration is one block, https://mcp.hasdata.com/api/mcp with the x-api-key header, and the client picks up the Shopping tool along with the rest. After that:

The model received the same parsed JSON the Python above works with, and nothing between the question and the table touched HTML. One limit worth knowing. Ask an agent to monitor a thousand products and a good one will not call this tool a thousand times. It will write the Python from the sections above and run it, because code is cheaper for it than tokens, and a script’s output does not vary. At volume, even the AI writes the script.

Which Shopping API

If the API route is the one you want, the split that matters is whether Google Shopping is a first-class endpoint or a general scraper pointed at a Shopping URL. A dedicated endpoint returns parsed products. A general scraper returns the page’s HTML and leaves the parsing to you, which on this page means the JavaScript problem from the start of the article all over again.

The dedicated endpoint this guide uses returned 66 parsed products for the benchmark query, with the filters and refinements blocks included, in 3.6 seconds. Whichever provider you evaluate, run one query and count what comes back parsed. That single check separates the two kinds faster than any feature page.

Where Each Route Fits

The three approaches differ in what you maintain rather than in what they can reach.

The first column is there because it’s what most tutorials teach, and it returns nothing on this page. Between the other two, Selenium keeps everything on your machine and costs you the selectors, while the API costs credits and gives you fields with names that don’t move.

Whichever you pick, keep the raw response. When a number looks wrong three weeks later, the saved JSON answers the question and a re-run doesn’t, because the prices will have changed underneath you.