How to Scrape and Download Google Images with Python

Google Images builds its result grid in the browser after the page loads. A plain requests call returns 89 KB of markup without a single img element in it, so a Beautiful Soup parser has nothing to collect.

Two routes still hand over image URLs. This guide covers the HasData Google Images API from the dashboard and from Python, then Google’s own Custom Search JSON API, and it ends with what the raw HTML holds now. Saving the files needs some care, because a third of the URLs carry a query string where the extension should be.

Each route returns a different amount for a different price.

The rest of this article works through all three, starting with the one that fills a folder fastest.

Benefits of Image Scraping

Training sets are the biggest use case, where a few thousand labelled images of a category beat a folder filled by hand. Content teams pull them to see which visuals rank for a product term before commissioning new photography. Marketplaces run them against their own listings to find sellers reusing a competitor’s photos. The volumes stay smaller than Google Maps or search results scraping, and so does the range of things people do with them.

Tools for Web Scraping Google Images

An API that returns image URLs as JSON is the short path, and the rest of this article spends most of its time there. Writing a parser against the page itself is the other option, and it has become the harder one for a reason worth understanding before starting.

Creating a Custom Web Scraper

Writing your own scraper works in almost any programming language, Python with Requests, Beautiful Soup and urllib being the usual starting point.

Images differ from text results in one way that matters. A page of search results keeps its titles and links in the HTML, while the image grid arrives later, so the same selectors-and-Requests recipe that works on a SERP returns an empty list here.

Using Online Services

Ready-made services need no programming, though they hand back only the fields they decided to expose. Browser plug-ins sit in the same category, either rigid or requiring code to customize.

Get Image Data in the API Playground

The Google Images API product page carries a playground that runs a live query without an account, which is the fastest way to see the response shape and decide whether these fields are the ones you need. Signing in adds the API Playground tab, where you pick the API, fill in the parameters, and then either run the request there, copy the equivalent code in any supported language, or copy a ready prompt for wiring the endpoint into your own AI agent. An account also comes with 1,000 free credits.

These are the parameters the endpoint accepts.

Running the request returns JSON with an imagesResults array of about a hundred entries. Each one carries position, title, source, link, thumbnail, original, originalWidth and originalHeight, where original is the full-size file and thumbnail the preview Google serves in the grid. One request costs 5 credits, so the sign-up grant covers 200 of them.

Copying the JSON out of the playground works for a one-off. For anything repeated, the same call fits in a few lines of Python.

Google Images Scraper in Python with the API

The examples use Python 3 and one dependency. Our Python scraping guide covers setting up the environment.

Step 1. Install the Dependencies

Requests handles both the API call and the image downloads:

pip install requests

os, re and urllib come with Python, so the API route needs nothing else. The Beautiful Soup example at the end of the article adds beautifulsoup4.

Step 2. Set the Endpoint and Parameters

Put the key, the endpoint and the search into variables, since the keyword is used again later as the folder name:

import requests

keyword = "Coffee"
api_url = "https://api.hasdata.com/scrape/google/images"
headers = {"x-api-key": "YOUR-API-KEY"}

params = {
    "q": keyword,
    "domain": "google.com",
}

The table above covers the parameters this endpoint takes, and the documentation carries the accepted values for each. There is a second route to related data: the Google SERP API returns the Images tab of a regular web search, while this endpoint searches Google Images directly. The two surfaces rank differently, so the results overlap without matching, and which one fits depends on whether you want image search or the image strip of a web search.

Step 3. Send the Request and Read the Response

Wrapping the call in try keeps a network error from killing the run, and checking the status code before parsing avoids a JSONDecodeError on an error page:

try:
    response = requests.get(api_url, params=params, headers=headers)
    if response.status_code == 200:
        data = response.json()
        for image in data["imagesResults"]:
            print(image["title"], image["original"])
    else:
        print("API request failed with status", response.status_code)
except Exception as error:
    print("Failed to make the API request:", error)

A 401 means the key is missing or wrong, and a 429 means the account’s concurrency limit was hit, which on the free plan is one request at a time.

Step 4. Save the Images with the Right Extension

The original field holds the full-size URL, and that is where file naming breaks. Taking everything after the last dot works for photo.jpg and produces jpeg?width=640&crop=smart&auto=webp&s=be for a Reddit preview link. Across 100 URLs from a single response, 68 ended in a clean extension and 32 carried a query string behind it, so roughly a third of a folder ends up with names no image viewer opens.

Reading the extension from the URL path rather than the whole URL fixes those 32. A second check earns its place too. Of the 258 links a three-page run got through, 41 answered with text/html instead of an image, and writing that HTML into a .jpg file leaves a corrupt image behind:

import os
import re
from pathlib import PurePosixPath
from urllib.parse import urlparse

IMAGE_TYPES = {
    "image/jpeg": "jpg", "image/png": "png", "image/webp": "webp",
    "image/gif": "gif", "image/avif": "avif", "image/bmp": "bmp",
    "image/svg+xml": "svg",
}

def extension_for(url, content_type):
    from_path = PurePosixPath(urlparse(url).path).suffix.lstrip(".").lower()
    if from_path in ("jpg", "jpeg", "png", "webp", "gif", "avif", "bmp", "svg"):
        return "jpg" if from_path == "jpeg" else from_path
    return IMAGE_TYPES.get(content_type.split(";")[0].strip().lower(), "jpg")

urlparse drops the query string and the fragment, PurePosixPath.suffix takes what is left after the last dot in the path alone, and the Content-Type header covers URLs that carry no extension at all.

Step 5. Download the Files

The download loop creates a folder named after the keyword, then saves each image under its title. It continues from the data returned in step 3:

browser_headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"
}

folder_name = re.sub(r"[^\w\-]+", "_", keyword)
os.makedirs(folder_name, exist_ok=True)

for image in data["imagesResults"]:
    image_url = image["original"]
    try:
        image_response = requests.get(image_url, headers=browser_headers, timeout=30)
        content_type = image_response.headers.get("Content-Type", "")
        if image_response.status_code != 200 or not content_type.startswith("image/"):
            print("Skipped", image_url[:60], content_type or image_response.status_code)
            continue
        title = re.sub(r"[^\w\-]+", "_", image["title"])[:60]
        file_name = f"{image['position']}_{title}.{extension_for(image_url, content_type)}"
        with open(os.path.join(folder_name, file_name), "wb") as file:
            file.write(image_response.content)
            print("Saved", file_name)
    except Exception as error:
        print("Failed to download", image_url[:60], error)

Sending a browser User-Agent on the image request changes what comes back more often than it looks. Requesting 25 image links twice, once with a browser user agent and once with the default one Requests sends, 8 returned the image only in the first case and text/html or text/plain in the second. None went the other way.

The position prefix on the file name earns its place for a duller reason. Titles repeat across pages, and without the prefix 7 names collided in a three-page run and overwrote files already on disk.

Step 6. Collect More Than One Page

ijn moves through the result pages, 0 being the first. Three consecutive values returned 297 unique image links, with two repeats between the first page and the second, which is why the loop keeps a set of URLs already seen:

seen = set()
for ijn in range(3):
    data = requests.get(api_url, params={**params, "ijn": ijn}, headers=headers).json()
    for image in data.get("imagesResults", []):
        seen.add(image["original"])
print(len(seen), "unique image URLs")

Every page costs another 5 credits, so a thousand images works out to about ten requests.

The Complete Script

The pieces above joined up, paging three deep and saving as it goes:

import os
import re
import requests
from pathlib import PurePosixPath
from urllib.parse import urlparse

keyword = "Coffee"
api_url = "https://api.hasdata.com/scrape/google/images"
headers = {"x-api-key": "YOUR-API-KEY"}
params = {"q": keyword, "domain": "google.com"}
browser_headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"
}

IMAGE_TYPES = {
    "image/jpeg": "jpg", "image/png": "png", "image/webp": "webp",
    "image/gif": "gif", "image/avif": "avif", "image/bmp": "bmp",
    "image/svg+xml": "svg",
}

def extension_for(url, content_type):
    from_path = PurePosixPath(urlparse(url).path).suffix.lstrip(".").lower()
    if from_path in ("jpg", "jpeg", "png", "webp", "gif", "avif", "bmp", "svg"):
        return "jpg" if from_path == "jpeg" else from_path
    return IMAGE_TYPES.get(content_type.split(";")[0].strip().lower(), "jpg")

folder_name = re.sub(r"[^\w\-]+", "_", keyword)
os.makedirs(folder_name, exist_ok=True)
seen = set()

try:
    for ijn in range(3):
        response = requests.get(api_url, params={**params, "ijn": ijn}, headers=headers)
        if response.status_code != 200:
            print("API request failed with status", response.status_code)
            break

        for image in response.json().get("imagesResults", []):
            image_url = image["original"]
            if image_url in seen:
                continue
            seen.add(image_url)
            try:
                image_response = requests.get(image_url, headers=browser_headers, timeout=30)
                content_type = image_response.headers.get("Content-Type", "")
                if image_response.status_code != 200 or not content_type.startswith("image/"):
                    print("Skipped", image_url[:60], content_type or image_response.status_code)
                    continue
                title = re.sub(r"[^\w\-]+", "_", image["title"])[:60]
                file_name = f"{ijn}_{image['position']}_{title}.{extension_for(image_url, content_type)}"
                with open(os.path.join(folder_name, file_name), "wb") as file:
                    file.write(image_response.content)
                    print("Saved", file_name)
            except Exception as error:
                print("Failed to download", image_url[:60], error)
except Exception as error:
    print("Failed to make the API request:", error)

The three API calls finish in seconds and the downloads dominate everything after that. A full run over three pages was still working through the links after ten minutes, so a large keyword list is worth pointing at a thread pool.

Google’s Own Custom Search JSON API

Google sells image results through the Custom Search JSON API. Two things are needed before the first call, an API key and a Programmable Search Engine, whose cx id identifies the engine in every request. A new engine searches only the sites listed in it, so covering the whole web means switching that option on first.

Setting searchType=image turns the response from web results into image results, and the filters cover size, colour, type and usage rights.

The limits decide whether it fits the job. Google gives 100 free queries a day and charges for anything past that. num accepts values from 1 to 10, and the API returns only the first 100 results for a query, so a full hundred images costs ten calls and the free tier tops out around a thousand images a day. The endpoint in the previous section returns its hundred in a single request, which is the difference worth weighing when the target runs to tens of thousands of files.

Scraping Google Images with Requests and BeautifulSoup

The direct route skips the API and parses google.com. It returns no images:

import requests
from bs4 import BeautifulSoup

url = "https://www.google.com/search"
params = {"q": "coffee", "tbm": "isch"}
headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"
}

response = requests.get(url, params=params, headers=headers)
soup = BeautifulSoup(response.content, "html.parser")

print(response.status_code, len(response.content) // 1024, "KB", response.url)
print(len(soup.find_all("img")), "img elements")

The response comes back 200 with 89 KB of markup, and the final URL is rewritten from tbm=isch to udm=2, the parameter Google moved image search to. find_all('img') returns an empty list. Of those 89 KB, 88 sit inside five script tags, and searching the raw markup for encrypted-tbn0.gstatic.com thumbnails or any .jpg, .png or .webp address turns up nothing at all. The grid is assembled in the browser from data fetched after the page loads, which leaves a parser nothing to work with, whether it looks for img tags or for URLs in the script payload.

Driving the page with a browser used to be the way around that. Chromium loading the same URL, headless and headed alike, ended up on a document whose title was the request URL and whose body held no image elements either, before or after scrolling, so that route needs work of its own before it returns a single thumbnail. The API route above stays the shorter path to a folder of files.

Which Route to Use

Volume decides this one. Anything past a few hundred images belongs on the Google Images endpoint, where one call covers a hundred and ijn keeps going. A hundred a day for a side project fits inside the free tier of Google’s Custom Search JSON API without a credit balance to watch.

Whichever route supplies the URLs, the download step decides whether the files open afterwards. Reading the extension from the URL path, sending a browser user agent, and checking the Content-Type before writing bytes are what separate a folder of images from a folder of unopenable files.