Web Scraping for SEO: Technical Guide

SEO scraping is the automated extraction of SERP data, on-page elements, and competitor content for analysis. It replaces manual analysis with programmatic scale.

This guide shows you how to build SEO scrapers in Python that run unattended. You will extract organic rankings, AI Overviews, competitor keywords, content structure, and technical SEO issues.

Core Tech Stack for SEO Scraping

Every script in this guide is built from the same parts, and the table names the tool used here next to its common alternative.

The scripts use httpx and parsel rather than the more common requests and BeautifulSoup pair. httpx has the same call shape as requests, adds HTTP/2 through its http2 extra and an async client, and its httpx.Client reuses connections when an audit fetches hundreds of pages from one host (the scripts below call httpx.get for clarity, and moving them onto a Client is a one-line change). parsel runs the same XPath and CSS selectors Scrapy uses, so a selector written here moves into a spider unchanged. requests and BeautifulSoup do the same two jobs with a larger stock of examples, and every script in this guide translates to them line by line. Selenium and Playwright drive a real browser and belong where a page builds its content with JavaScript. Here that rendering happens on request inside the HasData Web Scraping API, so no browser runs on your machine.

Common Use Cases

1. Rank tracking across locations and devices.
2. SERP feature analysis, from PAA questions to AI Overviews and Featured Snippets.
3. Competitor audits of headings, word counts, and schema markup.
4. Technical SEO monitoring of your own site for redirect chains, missing meta tags, and noindex flags.

Prerequisites and Setup

Install the required Python libraries for HTTP requests and HTML parsing.

pip install httpx parsel pandas requests

What each library does:

• httpx: HTTP client for every page fetch in this guide.
• parsel: XPath and CSS selector engine (used by Scrapy).
• pandas: Data structuring and CSV export.
• requests: Standard HTTP client for API calls.

HasData API Setup

For scraping Google SERP or sites with JavaScript rendering, use the HasData API. It handles headless browsers, rendering, and proxy rotation automatically.

1. Sign up at HasData.com and copy your API key from the dashboard. The same key works for both the Web Scraping API (for any website) and the Google SERP API (specifically for search results).
2. Test the connection with a simple request.

import requests
API_KEY = "HASDATA_API_KEY"
url = "https://api.hasdata.com/scrape/web"
payload = {
    "url": "https://books.toscrape.com/",
    "proxyType": "datacenter"
}
headers = {"x-api-key": API_KEY, "Content-Type": "application/json"}
response = requests.post(url, json=payload, headers=headers)
print(response.status_code, len(response.json()["content"]))

Once you have a successful response, you are ready to start building the scrapers.

Scraping Your Own Website for Technical SEO Audit

Before analyzing competitors, audit your own pages. A custom scraper checks the entire site structure, detects technical errors, and verifies metadata in minutes, at a page count no manual check reaches.

Standard SEO crawlers like Screaming Frog do this well, but the checks you can add are limited to what their GUI exposes. A Python scraper gives you direct access to raw HTML. You control what you extract and how you structure the output.

Bulk Metadata Extractor

This script extracts core on-page SEO elements from a list of URLs, which a sitemap parse or a crawl of your own domain supplies. It identifies missing canonicals, duplicate titles, and accidental noindex tags.

import httpx
from parsel import Selector
import pandas as pd
import time
import os

OUTPUT_DIR = "output/1_metadata"

URLS = [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://hasdata.com/blog/python-for-seo"
]


def scrape_metadata(url: str) -> dict:
    """Scrapes SEO metadata from a single URL. Returns a dict of metadata fields."""
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
    }
    try:
        response = httpx.get(url, headers=headers, follow_redirects=True, timeout=10.0)

        if response.status_code != 200:
            return {"url": url, "status": response.status_code, "error": f"HTTP {response.status_code}"}

        selector = Selector(text=response.text)

        title = selector.xpath("//title/text()").get() or ""
        meta_desc = selector.xpath("//meta[@name='description']/@content").get() or ""

        return {
            "url": url,
            "final_url": str(response.url),
            "status": response.status_code,
            "title": title,
            "title_length": len(title),
            "meta_desc": meta_desc,
            "meta_desc_length": len(meta_desc),
            "canonical": selector.xpath("//link[@rel='canonical']/@href").get(),
            "h1": selector.xpath("//h1/text()").get(),
            "h1_count": len(selector.xpath("//h1").getall()),
            "word_count": len(" ".join(selector.xpath("//body//text()").getall()).split()),
            "robots_meta": selector.xpath("//meta[@name='robots']/@content").get(),
            "og_title": selector.xpath("//meta[@property='og:title']/@content").get(),
            "og_description": selector.xpath("//meta[@property='og:description']/@content").get(),
            "og_image": selector.xpath("//meta[@property='og:image']/@content").get(),
        }

    except Exception as e:
        return {"url": url, "status": 0, "error": str(e)}


def run_metadata_audit(urls: list = None, output_dir: str = OUTPUT_DIR) -> list:
    """
    Audits a list of URLs for SEO metadata.
    Saves results to {output_dir}/seo_audit.csv
    Returns list of result dicts.
    """
    if urls is None:
        urls = URLS

    os.makedirs(output_dir, exist_ok=True)
    results = []

    print(f"Auditing {len(urls)} pages...")
    for url in urls:
        print(f"  Scraping: {url}")
        data = scrape_metadata(url)
        results.append(data)
        time.sleep(1)

    df = pd.DataFrame(results)
    out_path = os.path.join(output_dir, "seo_audit.csv")
    df.to_csv(out_path, index=False)
    print(f"Saved {len(df)} rows → {out_path}\n")
    return results


if __name__ == "__main__":
    run_metadata_audit()

What This Detects:

Sort the CSV by any of these columns and the pages that need attention come first.

Redirect Chain Detector

Long redirect chains (A to B to C to D) waste crawl budget and increase page load latency. Standard crawlers report the final status, but they often miss the intermediate hops. This script traces the full path of every URL.

import httpx
import pandas as pd
import os

OUTPUT_DIR = "output/2_redirects"

URLS_TO_CHECK = [
    "http://httpbin.org/redirect/3",
    "http://httpbin.org/redirect/1",
    "http://httpbin.org/status/200",
]


def check_redirect_chain(url: str) -> list:
    """
    Follows all redirects for a single URL.
    Returns a list of dicts, one per hop (including the final destination).
    """
    history = []
    try:
        response = httpx.get(url, follow_redirects=True, timeout=10.0)

        for resp in response.history:
            history.append({"url": str(resp.url), "status": resp.status_code})

        history.append({"url": str(response.url), "status": response.status_code})

    except Exception as e:
        history.append({"url": url, "status": 0, "error": str(e)})

    return history


def run_redirect_audit(urls: list = None, output_dir: str = OUTPUT_DIR) -> list:
    """
    Checks redirect chains for a list of URLs.
    Saves all hops to {output_dir}/redirect_chains.csv
    Returns flat list of hop dicts.
    """
    if urls is None:
        urls = URLS_TO_CHECK

    os.makedirs(output_dir, exist_ok=True)
    all_chains = []

    for url in urls:
        print(f"\nChecking: {url}")
        chain = check_redirect_chain(url)

        for i, step in enumerate(chain):
            step["original_url"] = url
            step["hop_number"] = i + 1
            all_chains.append(step)
            print(f"  [{step['status']}] → {step['url']}")

        if len(chain) > 2:
            print(f"  ⚠ Chain length: {len(chain)} (recommended ≤ 2)")

    df = pd.DataFrame(all_chains)
    out_path = os.path.join(output_dir, "redirect_chains.csv")
    df.to_csv(out_path, index=False)
    print(f"\nSaved {len(all_chains)} hops → {out_path}\n")
    return all_chains


if __name__ == "__main__":
    run_redirect_audit()

What This Detects:

Every hop is a row in the CSV, so a loop or a domain change is visible on its own line rather than hidden behind the final status code.

What a Plain Fetch Misses

The scripts above read the HTML the server sends, without running JavaScript, and the obvious question is how often that HTML is not what Google indexes. I took the top 10 results for 30 queries, half commercial and half informational, 248 pages in all, and fetched each one twice: once with httpx exactly as the metadata extractor does, once through the Web Scraping API with jsRendering on and a two-second wait. 145 pages answered both requests with a real document. 83 refused or never answered the plain request (41 with a 403, the rest timeouts or connection errors), the API declined 33 URLs, 31 of them among those 83, 4 pages answered with a body under 500 bytes, and 14 Reddit threads came back as an empty shell on one side, so the comparison below is on the 145.

The same measurement as shares of the 145 pages puts the three tiers side by side.

The fields an audit reads hold up better than the folklore suggests. The title matched on 133 of the 145 pages and differed on 12, the canonical and the meta description each changed on 22 pages (15%), and on 6 and 8 of those the tag existed only after rendering, which is the case that makes a plain audit report a missing tag on a page that has one. Meta robots moved on 4 pages and hreflang on 4. JSON-LD changed on 22 pages: 8 had structured data only after rendering, 9 only in the plain HTML, and 5 gained types. Taken together, 38 of the 145 pages (26%) differed on at least one of the six fields the metadata extractor reads, 18 of 82 commercial pages and 20 of 63 informational ones.

The bigger gap is in what the plain fetch never saw. 83 of the 248 pages, a third of the top 10, did not answer a plain httpx request at all: 41 sent a 403 (IKEA, Costco, Capterra, GoDaddy and PCMag among them) and 40 timed out or reset the connection. The counts that depend on the whole page moved most, the H2 count on 49 pages (34%) and the internal link count on 97 (67%), because menus, footers, related posts and comment widgets arrive with JavaScript. So on titles, canonicals and robots directives the extractor is right on 85% to 97% of the pages it can reach, and the two places to switch to the rendered fetch through the Web Scraping API are the pages that refuse the plain request and the audits that count headings or links.

What the Site Audit Gives You

You can audit thousands of pages for technical issues without Screaming Frog’s GUI. You can version control your audits in CSV format and automate them via cron or GitHub Actions.

Technical health decides whether a page can rank. What it ranks for depends on content, so the next step is to look at the headings, vocabulary, and schema markup of the pages that already rank.

Scraping Competitor Websites for Content Analysis

While technical health is the foundation, content drives rankings. To outrank a competitor, you need to know what they publish and how they structure it. Three scripts pull that out of top-ranking pages, starting with the heading structure, then the recurring phrases, then the schema markup.

Content Structure Analysis (H2/H3 & Word Count)

Search engines reward pages that cover a topic fully. By scraping the heading hierarchy (H1-H6) of top-ranking pages, you can create a “master outline” that covers all subtopics your competitors address.

This function extracts the skeleton of any article, allowing you to visualize its logical flow and depth.

import httpx
from parsel import Selector
import pandas as pd
import time
import os
from urllib.parse import urlparse

OUTPUT_DIR = "output/3_structure"

COMPETITOR_URLS = [
    "https://books.toscrape.com/",
    "https://hasdata.com/blog/python-for-seo"
]


def extract_content_structure(url: str) -> dict:
    """
    Extracts content structure from a single URL.
    Returns a dict with summary stats and heading lists.
    """
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
    }
    try:
        response = httpx.get(url, headers=headers, follow_redirects=True, timeout=10.0)

        if response.status_code != 200:
            return {"url": url, "error": f"HTTP {response.status_code}"}

        selector = Selector(text=response.text)

        return {
            "url": url,
            "h1": selector.xpath("//h1//text()").get(),
            "h2_list": [h.strip() for h in selector.xpath("//h2//text()").getall() if h.strip()],
            "h3_list": [h.strip() for h in selector.xpath("//h3//text()").getall() if h.strip()],
            "h2_count": len(selector.xpath("//h2").getall()),
            "h3_count": len(selector.xpath("//h3").getall()),
            "word_count": len(" ".join(selector.xpath("//body//text()").getall()).split()),
            "paragraph_count": len(selector.xpath("//p").getall()),
            "image_count": len(selector.xpath("//img").getall()),
            "internal_links": len([h for h in selector.xpath("//a/@href").getall() if h.startswith("/") or urlparse(h).netloc == urlparse(url).netloc]),
            "external_links": len([h for h in selector.xpath("//a/@href").getall() if h.startswith("http") and urlparse(h).netloc != urlparse(url).netloc]),
        }

    except Exception as e:
        return {"url": url, "error": str(e)}


def run_content_structure_audit(urls: list = None, output_dir: str = OUTPUT_DIR) -> list:
    """
    Analyzes content structure for a list of URLs.
    For each URL saves two files in {output_dir}/{domain}/:
      - summary.csv   — one row with counts (word_count, h2_count, etc.)
      - headings.csv  — one row per heading (level | text)
    Also saves a combined structure_summary.csv across all URLs.
    Returns list of result dicts.
    """
    if urls is None:
        urls = COMPETITOR_URLS

    os.makedirs(output_dir, exist_ok=True)
    results = []

    for url in urls:
        print(f"  Analyzing: {url}")
        data = extract_content_structure(url)
        results.append(data)

        # Per-site subfolder
        domain = urlparse(url).netloc.replace(".", "_")
        site_dir = os.path.join(output_dir, domain)
        os.makedirs(site_dir, exist_ok=True)

        # Summary row (no list columns)
        summary_row = {k: v for k, v in data.items() if k not in ("h2_list", "h3_list")}
        pd.DataFrame([summary_row]).to_csv(os.path.join(site_dir, "summary.csv"), index=False)

        # Headings flat (one heading per row)
        headings_rows = []
        for h2 in data.get("h2_list", []):
            headings_rows.append({"level": "H2", "text": h2})
        for h3 in data.get("h3_list", []):
            headings_rows.append({"level": "H3", "text": h3})
        pd.DataFrame(headings_rows).to_csv(os.path.join(site_dir, "headings.csv"), index=False)

        print(f"Saved → {site_dir}/summary.csv + headings.csv")
        time.sleep(1)

    # Combined summary across all URLs
    all_summary = [{k: v for k, v in r.items() if k not in ("h2_list", "h3_list")} for r in results]
    combined_path = os.path.join(output_dir, "structure_summary.csv")
    pd.DataFrame(all_summary).to_csv(combined_path, index=False)
    print(f"Combined summary → {combined_path}\n")

    return results


if __name__ == "__main__":
    run_content_structure_audit()

What This Reveals:

For JavaScript-rendered competitor sites, use HasData’s Web Scraping API with jsRendering: true to get the full DOM.

Keyword Extraction (N-Gram Analysis)

Competitors use specific phrases repeatedly, and the phrases that recur across several ranking pages show the vocabulary the top results share on a topic. Extracting them is a cheap way to see which terms a page competing for the query is expected to cover.

This script extracts the 20 most frequent 2-word and 3-word phrases from competitor content. You compare this list against your own content to find missing terminology. Text inside pre, code, script, and style is skipped, otherwise a tutorial page’s code samples take over the list (the first run on the page below returned else print and window datalayer as top phrases).

import httpx
from parsel import Selector
from collections import Counter
import re
import pandas as pd
import os
from urllib.parse import urlparse

OUTPUT_DIR = "output/4_keywords"

DEFAULT_URL = "https://hasdata.com/blog/python-for-seo"

STOPWORDS = {
    "the", "is", "at", "which", "on", "a", "an", "as", "are", "was", "were",
    "be", "been", "in", "of", "to", "for", "with", "and", "or", "but", "not",
    "this", "that", "by", "from", "it", "can", "will", "you", "your", "has",
    "have", "had", "all", "its", "our", "we", "do", "so", "if", "about",
    "what", "how", "more", "than", "when", "their", "also", "into", "other",
}


def extract_ngrams(text: str, n: int = 2, top_n: int = 20) -> list:
    """Extracts top N n-grams from text after filtering stopwords."""
    words = re.findall(r"\b[a-z]+\b", text.lower())
    words = [w for w in words if w not in STOPWORDS and len(w) > 2]
    ngrams = zip(*[words[i:] for i in range(n)])
    ngram_list = [" ".join(gram) for gram in ngrams]
    return Counter(ngram_list).most_common(top_n)


def analyze_competitor_keywords(url: str) -> dict:
    """
    Fetches a page and extracts top bigrams and trigrams from body text.
    Returns dict with url, bigrams, trigrams.
    """
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"}
    try:
        response = httpx.get(url, headers=headers, timeout=15.0)
        selector = Selector(text=response.text)

        body_text = " ".join(selector.xpath("(//article | //main)//text()[not(ancestor::pre) and not(ancestor::code) and not(ancestor::script) and not(ancestor::style)]").getall())
        if not body_text.strip():
            body_text = " ".join(selector.xpath("//body//p//text()").getall())

        return {
            "url": url,
            "bigrams": extract_ngrams(body_text, n=2, top_n=20),
            "trigrams": extract_ngrams(body_text, n=3, top_n=15),
        }

    except Exception as e:
        return {"url": url, "error": str(e)}


def run_keyword_extraction(url: str = None, output_dir: str = OUTPUT_DIR) -> dict:
    """
    Runs keyword extraction for a URL.
    Saves bigrams and trigrams to {output_dir}/keywords_{domain}.csv
    Returns result dict.
    """
    if url is None:
        url = DEFAULT_URL

    os.makedirs(output_dir, exist_ok=True)
    result = analyze_competitor_keywords(url)

    if "error" in result:
        print(f"Error: {result['error']}")
        return result

    domain = urlparse(url).netloc.replace(".", "_")
    rows = []
    for phrase, count in result.get("bigrams", []):
        rows.append({"type": "bigram", "phrase": phrase, "count": count})
    for phrase, count in result.get("trigrams", []):
        rows.append({"type": "trigram", "phrase": phrase, "count": count})

    df = pd.DataFrame(rows)
    out_path = os.path.join(output_dir, f"keywords_{domain}.csv")
    df.to_csv(out_path, index=False)

    print(f"Top 2-Word Phrases:")
    for phrase, count in result["bigrams"]:
        print(f"  {phrase}: {count}")
    print(f"\nTop 3-Word Phrases:")
    for phrase, count in result["trigrams"]:
        print(f"  {phrase}: {count}")
    print(f"\nSaved → {out_path}\n")

    return result


if __name__ == "__main__":
    run_keyword_extraction()

Example Output:

Top 2-Word Phrases:
  output example: 12
  seo tools: 7
  google trends: 6
  example script: 6
  code output: 6
  serp api: 5
  code script: 5
  web scraping: 4
  serp intent: 4
  trends api: 3

Top 3-Word Phrases:
  output example script: 6
  code output example: 6
  google trends api: 3
  web scraping api: 2
  use cases seo: 2

The lists are cut to their top entries. “output example” leads because that page labels every code block with it, the kind of boilerplate you strike out by hand before comparing lists.

How to Use This:

1. Run this script on your top 3 competitors.
2. Export their keyword lists to CSV.
3. Compare against your own content using Ctrl+F in your draft.
4. If competitors mention ‘serp features’ frequently and you do not, you may be underweighting a core topic.

The list is a coverage check. Google expects certain terminology in content that covers a topic fully, and a phrase every competitor uses and you never do marks a subtopic you skipped.

This is frequency-based analysis, not TF-IDF. The content gap analyzer in our Python SEO guide runs the same n-grams across several competitors at once and lists the phrases your page is missing.

Schema Markup Extraction

Structured data (schema.org markup) tells Google what a page is, and types such as Product, Article, and BreadcrumbList still produce rich results. FAQPage and HowTo lost their rich results for most sites in 2023, yet competitors keep shipping the markup, so an audit reads the types they use and checks them against what Google still rewards for your page type.

This script extracts all JSON-LD structured data from a competitor page.

import httpx
from parsel import Selector
import json
import os
from urllib.parse import urlparse

OUTPUT_DIR = "output/5_markup"

DEFAULT_URL = "https://www.nytimes.com/wirecutter/reviews/best-standing-desk/"


def extract_schema(url: str) -> dict:
    """
    Extracts all JSON-LD schema blocks from a page.
    Returns dict with url, schemas list, and schema_types list.
    """
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"}
    try:
        response = httpx.get(url, headers=headers, timeout=15.0)
        selector = Selector(text=response.text)

        raw_schemas = selector.xpath("//script[@type='application/ld+json']/text()").getall()

        parsed_schemas = []
        for raw in raw_schemas:
            try:
                parsed_schemas.append(json.loads(raw))
            except json.JSONDecodeError:
                continue

        return {
            "url": url,
            "schema_count": len(parsed_schemas),
            "schema_types": [s.get("@type") for s in parsed_schemas if "@type" in s],
            "schemas": parsed_schemas,
        }

    except Exception as e:
        return {"url": url, "error": str(e)}


def run_markup_extraction(url: str = None, output_dir: str = OUTPUT_DIR) -> dict:
    """
    Extracts schema markup from a URL and saves to JSON.
    Saves to {output_dir}/schema_{domain}.json
    Returns result dict.
    """
    if url is None:
        url = DEFAULT_URL

    os.makedirs(output_dir, exist_ok=True)
    result = extract_schema(url)

    if "error" in result:
        print(f"Error: {result['error']}")
        return result

    print(f"Schema Types Found: {result['schema_types']}")
    for schema in result["schemas"]:
        schema_type = schema.get("@type", "Unknown")
        print(f"\n--- {schema_type} ---")
        print(json.dumps(schema, indent=2)[:400])

    domain = urlparse(url).netloc.replace(".", "_")
    out_path = os.path.join(output_dir, f"schema_{domain}.json")
    with open(out_path, "w", encoding="utf-8") as f:
        json.dump(result, f, indent=2, ensure_ascii=False)
    print(f"\nSaved → {out_path}\n")

    return result


if __name__ == "__main__":
    run_markup_extraction()

Example Output:

Schema Types Found: ['Article', 'ItemList', 'BreadcrumbList']

--- Article ---
{
  "@type": "Article",
  "@context": "http://schema.org",
  "inLanguage": "en-US",
  "alternativeHeadline": "The 3 Best Standing Desks in 2026",
  "author": [
    {
      "@type": "Person",
      "name": "Kaitlyn Wells",

--- ItemList ---
{
  "@type": "ItemList",
  "@context": "http://schema.org",
  "numberOfItems": 2,
  "itemListElement": [
    {
      "@type": "ListItem",
      "position": 1,
      "item": {
        "@type": "Product",
        "name": "Uplift V3 Standing Desk",

--- BreadcrumbList ---
{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    {
      "@type": "ListItem",
      "position": 1,
      "name": "Office",
      "item": "https://www.nytimes.com/wirecutter/office/"

Each block is cut at 400 characters by the script, which is enough to read the type and the first entity without scrolling through a whole product list.

What This Tells You:

A competitor that shows up in People Also Ask got there because Google picked a passage from the page, not because of FAQPage markup. Compare the types across the top 10 and copy the ones that still produce a rich result for your page type.

What the Competitor Audit Gives You

You can audit competitor content structure, extract their keyword patterns, and reverse-engineer their schema strategy. This data tells you exactly what topics to cover, which phrases to emphasize, and which structured data to implement.

But the SERP is where content competes. The next section shows how to scrape Google Search results directly to see what ranks, what AI Overviews say, and what questions users ask.

Scraping Google Search Results for Strategic Intelligence

Analyzing your own pages is useful, but rankings are decided on the SERP, and you cannot optimize for rankings you do not measure. The SERP has three layers worth scraping: what ranks (organic), what Google synthesizes (AI Overviews), and what users want to know next (PAA and Related Searches).

We will use the HasData Google SERP API. It returns structured JSON, eliminating the need to parse volatile HTML or manage headless browsers for JavaScript rendering.

Extracting Organic Rankings

This is your baseline rank tracker, the self-hosted counterpart of the rank tracking APIs that sell the same positions as a feed. For every keyword, you need position, URL, title, and snippet to detect title rewrites and identify competitors.

import requests
from urllib.parse import urlparse
import pandas as pd
import os

OUTPUT_DIR = "output/6_rankings"
API_KEY = "HASDATA_API_KEY"  # https://app.hasdata.com/

KEYWORDS = [
    "python web scraping tutorial",
    "seo audit tools",
]


def scrape_organic_results(keyword: str, api_key: str = None) -> list:
    """
    Fetches organic search results for a keyword via HasData SERP API.
    Returns list of result dicts (position, title, url, snippet, domain).
    """
    if api_key is None:
        api_key = API_KEY

    url = "https://api.hasdata.com/scrape/google/serp"
    headers = {"x-api-key": api_key}
    params = {
        "q": keyword,
        "gl": "us",
        "hl": "en",
        "location": "New York,New York,United States",
    }

    try:
        response = requests.get(url, headers=headers, params=params, timeout=30)

        if response.status_code != 200:
            print(f"  Error {response.status_code}: {response.text[:200]}")
            return []

        data = response.json()
        organic = data.get("organicResults", [])

        return [
            {
                "keyword": keyword,
                "position": item.get("position"),
                "title": item.get("title"),
                "url": item.get("link"),
                "snippet": item.get("snippet"),
                "domain": urlparse(item.get("link", "")).netloc,
            }
            for item in organic
        ]

    except Exception as e:
        print(f"  Exception: {e}")
        return []


def run_organic_rankings(
    keywords: list = None, api_key: str = None, output_dir: str = OUTPUT_DIR
) -> list:
    """
    Scrapes organic rankings for a list of keywords.
    Saves one CSV per keyword to {output_dir}/organic_{keyword}.csv
    Also saves a combined file: organic_all.csv
    Returns flat list of all result dicts.
    """
    if keywords is None:
        keywords = KEYWORDS

    os.makedirs(output_dir, exist_ok=True)
    all_data = []

    for kw in keywords:
        print(f"  Scraping: {kw}")
        results = scrape_organic_results(kw, api_key=api_key)
        all_data.extend(results)

        if results:
            safe_kw = kw.replace(" ", "_")
            out_path = os.path.join(output_dir, f"organic_{safe_kw}.csv")
            pd.DataFrame(results).to_csv(out_path, index=False)
            print(f"  {len(results)} results → {out_path}")

    if all_data:
        combined_path = os.path.join(output_dir, "organic_all.csv")
        pd.DataFrame(all_data).to_csv(combined_path, index=False)
        print(f"\nCombined: {len(all_data)} rows → {combined_path}\n")

    return all_data


if __name__ == "__main__":
    run_organic_rankings()

What this tells you:

• Your current position for target keywords.
• Title tag patterns Google rewards (length, structure, keyword placement).
• Snippet length distribution (short vs. detailed).

Extracting AI Overviews

AI Overviews appear on a share of Google searches that depends on query intent, measured in our AI Overview study. If Google synthesizes an answer above your organic result, you must analyze its content and citations.

Monitor whether your domain is cited in AI Overviews for your target keywords, one keyword here and hundreds of keywords on a schedule with the monitor from the Python SEO guide. Track how Google frames the topic.

The SERP response carries the overview inline when Google renders it with the results. Otherwise it carries a short-lived pageToken, and the Google AI Overview API returns the same textBlocks and references for that token. The script handles both cases and retries on a 429, which the API returns when the plan’s concurrency is exceeded.

import json
import os
import time

import requests

OUTPUT_DIR = "output/7_ai_overviews"
API_KEY = "HASDATA_API_KEY"  # https://app.hasdata.com/
DEFAULT_KEYWORD = "what is seo scraping"
SERP_URL = "https://api.hasdata.com/scrape/google/serp"
AIO_URL = "https://api.hasdata.com/scrape/google/ai-overview"


def _get(url, params, api_key):
    """One GET with a retry on 429, which the API returns when the plan's concurrency is exceeded."""
    for attempt in range(4):
        response = requests.get(url, headers={"x-api-key": api_key}, params=params, timeout=120)
        if response.status_code != 429:
            response.raise_for_status()
            return response.json()
        time.sleep(5 * (attempt + 1))
    response.raise_for_status()


def get_ai_overview(keyword: str, api_key: str = None) -> dict | None:
    """
    Returns the AI Overview for a keyword as a dict with "textBlocks" and "references", or None.
    Google loads the overview asynchronously, so the SERP response often carries only a short-lived
    pageToken; the AI Overview endpoint turns that token into the full block.
    """
    if api_key is None:
        api_key = API_KEY
    data = _get(SERP_URL, {"q": keyword, "gl": "us", "hl": "en"}, api_key)
    overview = data.get("aiOverview")
    if not overview:
        return None
    if "textBlocks" in overview:
        return overview
    if "pageToken" in overview:
        follow = _get(AIO_URL, {"pageToken": overview["pageToken"]}, api_key)
        return follow.get("aiOverview") or follow
    return None


def overview_text(overview: dict) -> str:
    """Flattens textBlocks (paragraphs, lists with items) into plain text."""
    parts = []
    for block in overview.get("textBlocks", []):
        if block.get("snippet"):
            parts.append(block["snippet"])
        for item in block.get("list", []):
            title = item.get("title", "")
            parts.append(f"- {title} {item.get('snippet', '')}".strip())
    return "\n".join(parts)


def run_ai_overviews(
    keyword: str = None, api_key: str = None, output_dir: str = OUTPUT_DIR
) -> dict | None:
    """
    Fetches and displays the AI Overview for a keyword.
    Saves result to {output_dir}/ai_overview_{keyword}.json
    Returns the overview dict or None.
    """
    if keyword is None:
        keyword = DEFAULT_KEYWORD

    os.makedirs(output_dir, exist_ok=True)
    overview = get_ai_overview(keyword, api_key=api_key)

    if overview:
        print(f"\n--- AI Overview for '{keyword}' ---")
        print(overview_text(overview)[:600])

        references = overview.get("references", [])
        if references:
            print(f"\nCited sources ({len(references)}):")
            for ref in references:
                print(f"  - {ref.get('title')}: {ref.get('link')}")

        safe_kw = keyword.replace(" ", "_")
        out_path = os.path.join(output_dir, f"ai_overview_{safe_kw}.json")
        with open(out_path, "w", encoding="utf-8") as f:
            json.dump({"keyword": keyword, "ai_overview": overview}, f, indent=2, ensure_ascii=False)
        print(f"\nSaved → {out_path}\n")
    else:
        print(f"No AI Overview found for '{keyword}'")

    return overview


if __name__ == "__main__":
    run_ai_overviews()

Use Case:

• Audit which competitor domains are cited most frequently across your keyword set.
• Reverse-engineer their content patterns (depth, citation density, schema usage).
• Track when AI Overviews appear or disappear for your keywords (indicates query intent shifts).

Extracting People Also Ask (PAA)

PAA questions are intent signals. They tell you what subtopics users expect in a complete answer, and following the questions that open under each answer grows the list into a PAA topic graph.

import requests
import pandas as pd
import os

OUTPUT_DIR = "output/8_paa"
API_KEY = "HASDATA_API_KEY"  # https://app.hasdata.com/
DEFAULT_KEYWORD = "how to do seo for a website"


def scrape_paa(keyword: str, api_key: str = None) -> list:
    """
    Fetches People Also Ask questions for a keyword.
    Returns list of dicts: keyword, question, snippet, source.
    """
    if api_key is None:
        api_key = API_KEY

    url = "https://api.hasdata.com/scrape/google/serp"
    headers = {"x-api-key": api_key}
    params = {"q": keyword, "gl": "us", "hl": "en"}

    try:
        response = requests.get(url, headers=headers, params=params, timeout=30)

        if response.status_code != 200:
            print(f"  Error {response.status_code}: {response.text[:200]}")
            return []

        data = response.json()
        questions = []
        for item in data.get("relatedQuestions", []):
            questions.append(
                {
                    "keyword": keyword,
                    "question": item.get("question"),
                    "snippet": item.get("snippet"),
                    "source": item.get("link"),
                }
            )
        return questions

    except Exception as e:
        print(f"  Exception: {e}")
        return []


def run_people_also_ask(
    keyword: str = None, api_key: str = None, output_dir: str = OUTPUT_DIR
) -> list:
    """
    Fetches PAA questions for a keyword, prints and saves them.
    Saves to {output_dir}/paa_{keyword}.csv
    Returns list of question dicts.
    """
    if keyword is None:
        keyword = DEFAULT_KEYWORD

    os.makedirs(output_dir, exist_ok=True)
    paa_data = scrape_paa(keyword, api_key=api_key)

    if not paa_data:
        print(f"No PAA questions found for '{keyword}'")
        return []

    print(f"\n--- People Also Ask: '{keyword}' ---")
    for q in paa_data:
        print(f"Q: {q['question']}")
        snippet = (q.get("snippet") or "")[:120]
        print(f"A: {snippet}...\n")

    safe_kw = keyword.replace(" ", "_")
    out_path = os.path.join(output_dir, f"paa_{safe_kw}.csv")
    pd.DataFrame(paa_data).to_csv(out_path, index=False)
    print(f"Saved {len(paa_data)} questions → {out_path}\n")

    return paa_data


if __name__ == "__main__":
    run_people_also_ask()

Use Case:

• Map PAA questions to H2/H3 sections in your content.
• Identify semantic gaps (questions competitors answer but you do not).
• Monitor PAA volatility (new questions appearing indicates shifting user interest).

Extracting Related Searches

Related Searches appear at the bottom of the SERP. They represent semantic variations and adjacent intents. These are keyword expansion opportunities.

Use Case: Discover long-tail modifiers. Identify cluster topics for pillar pages.

import requests
import pandas as pd
import os

OUTPUT_DIR = "output/9_related"
API_KEY = "HASDATA_API_KEY"  # https://app.hasdata.com/
DEFAULT_KEYWORD = "seo scraping tools"


def get_related_searches(keyword: str, api_key: str = None) -> list:
    """
    Fetches related search queries for a keyword from Google SERP.
    Returns a list of query strings.
    """
    if api_key is None:
        api_key = API_KEY

    url = "https://api.hasdata.com/scrape/google/serp"
    params = {"q": keyword, "location": "United States", "gl": "us", "hl": "en"}
    headers = {"x-api-key": api_key}

    try:
        response = requests.get(url, params=params, headers=headers, timeout=30)

        if response.status_code != 200:
            print(f"  Error {response.status_code}: {response.text[:200]}")
            return []

        data = response.json()
        return [item.get("query") for item in data.get("relatedSearches", []) if item.get("query")]

    except Exception as e:
        print(f"  Exception: {e}")
        return []


def run_related_searches(
    keyword: str = None, api_key: str = None, output_dir: str = OUTPUT_DIR
) -> list:
    """
    Fetches related searches for a keyword, prints and saves them.
    Saves to {output_dir}/related_{keyword}.csv
    Returns list of query strings.
    """
    if keyword is None:
        keyword = DEFAULT_KEYWORD

    os.makedirs(output_dir, exist_ok=True)
    queries = get_related_searches(keyword, api_key=api_key)

    if not queries:
        print(f"No related searches found for '{keyword}'")
        return []

    print(f"\n--- Related Searches for '{keyword}' ---")
    for q in queries:
        print(f"  - {q}")

    safe_kw = keyword.replace(" ", "_")
    out_path = os.path.join(output_dir, f"related_{safe_kw}.csv")
    pd.DataFrame({"keyword": keyword, "related_query": queries}).to_csv(out_path, index=False)
    print(f"\nSaved {len(queries)} queries → {out_path}\n")

    return queries


if __name__ == "__main__":
    run_related_searches()

Use Case:

• Identify content clusters (group related terms with high overlap).
• Discover informational vs. transactional modifiers.
• Build internal linking architecture around these semantic connections.

Complete Workflow Example

The nine scripts above become nine files, and the workflow below imports all of them. Save each script under the file name its import line uses:

Put the workflow file next to them and run it. Modes 1 and 3 need the API key, mode 2 runs without it.

import json
import os
from datetime import datetime
from urllib.parse import urlparse

from _1_metadata_extractor         import scrape_metadata, run_metadata_audit
from _2_redirect_chain_detector    import check_redirect_chain, run_redirect_audit
from _3_content_structure_analyzer import extract_content_structure, run_content_structure_audit
from _4_keyword_extraction         import analyze_competitor_keywords, run_keyword_extraction
from _5_markup_extraction          import extract_schema, run_markup_extraction
from _6_organic_rankings           import scrape_organic_results, run_organic_rankings
from _7_ai_overviews               import get_ai_overview, run_ai_overviews
from _8_people_also_ask            import scrape_paa, run_people_also_ask
from _9_related_searches           import get_related_searches, run_related_searches

OUTPUT_DIR = "output"
API_KEY = "HASDATA_API_KEY"  # https://app.hasdata.com/


#  MODE 1 — SERP AUDIT  (keyword → scripts 6, 7, 8, 9)

def serp_audit(keyword: str, api_key: str = None) -> dict:
    """
    Runs a full SERP audit for a keyword.
    """
    if api_key is None:
        api_key = API_KEY

    _header(f"SERP AUDIT: '{keyword}'")

    print("[6/9] Organic Rankings")
    organic = run_organic_rankings(keywords=[keyword], api_key=api_key)

    print("[7/9] AI Overview")
    ai_overview = run_ai_overviews(keyword=keyword, api_key=api_key)

    print("[8/9] People Also Ask")
    paa = run_people_also_ask(keyword=keyword, api_key=api_key)

    print("[9/9] Related Searches")
    related = run_related_searches(keyword=keyword, api_key=api_key)

    summary = {
        "mode": "serp_audit",
        "keyword": keyword,
        "timestamp": datetime.now().isoformat(),
        "organic_results": len(organic),
        "ai_overview_triggered": bool(ai_overview),
        "paa_questions": len(paa),
        "related_searches": len(related),
        "top_organic_urls": [r.get("url") for r in organic[:5]],
        "paa_questions_list": [q.get("question") for q in paa],
        "related_queries": related,
    }

    _save_summary(summary, label=keyword)
    _footer_serp(summary)
    return summary


#  MODE 2 — SITE AUDIT  (url → scripts 1, 2, 3, 4, 5)

def site_audit(url: str) -> dict:
    """
    Runs a full on-page audit for a single URL.
    """
    domain = urlparse(url).netloc
    _header(f"SITE AUDIT: {url}")

    print("[1/5] Metadata")
    metadata_list = run_metadata_audit(urls=[url])
    metadata = metadata_list[0] if metadata_list else {}

    print("[2/5] Redirect Chain")
    chain = check_redirect_chain(url)
    redirect_hops = len(chain)
    has_redirect_issue = redirect_hops > 2

    print("[3/5] Content Structure")
    structure_list = run_content_structure_audit(urls=[url])
    structure = structure_list[0] if structure_list else {}

    print("[4/5] Keyword Extraction")
    keywords = run_keyword_extraction(url=url)

    print("[5/5] Schema Markup")
    schema = run_markup_extraction(url=url)

    summary = {
        "mode": "site_audit",
        "url": url,
        "domain": domain,
        "timestamp": datetime.now().isoformat(),
        # Metadata
        "title": metadata.get("title"),
        "title_length": metadata.get("title_length"),
        "meta_desc_length": metadata.get("meta_desc_length"),
        "h1": metadata.get("h1"),
        "word_count": structure.get("word_count"),
        # Redirects
        "redirect_hops": redirect_hops,
        "redirect_warning": has_redirect_issue,
        # Structure
        "h2_count": structure.get("h2_count"),
        "h3_count": structure.get("h3_count"),
        "internal_links": structure.get("internal_links"),
        "external_links": structure.get("external_links"),
        # Schema
        "schema_types": schema.get("schema_types", []),
        # Keywords
        "top_bigrams": [phrase for phrase, _ in (keywords.get("bigrams") or [])[:5]],
        "top_trigrams": [phrase for phrase, _ in (keywords.get("trigrams") or [])[:5]],
    }

    _save_summary(summary, label=domain)
    _footer_site(summary)
    return summary


#  MODE 3 — FULL AUDIT  (keyword + url → all 9 scripts)

def full_audit(keyword: str, url: str, api_key: str = None) -> dict:
    """
    Runs both serp_audit and site_audit.
    Returns a combined summary dict with both results.
    """
    if api_key is None:
        api_key = API_KEY

    serp_summary = serp_audit(keyword=keyword, api_key=api_key)
    site_summary = site_audit(url=url)

    combined = {
        "mode": "full_audit",
        "keyword": keyword,
        "url": url,
        "timestamp": datetime.now().isoformat(),
        "serp": serp_summary,
        "site": site_summary,
    }

    _save_summary(combined, label=f"full_{keyword.replace(' ', '_')}")
    return combined


#  Helpers

def _header(title: str) -> None:
    print(f"\n{'═' * 60}")
    print(f"  {title}")
    print(f"{'═' * 60}\n")


def _save_summary(summary: dict, label: str) -> None:
    os.makedirs(OUTPUT_DIR, exist_ok=True)
    safe = label.replace(" ", "_").replace("/", "_").replace(".", "_")
    path = os.path.join(OUTPUT_DIR, f"summary_{safe}.json")
    with open(path, "w", encoding="utf-8") as f:
        json.dump(summary, f, indent=2, ensure_ascii=False)
    print(f"\nSummary saved → {path}")


def _footer_serp(s: dict) -> None:
    print(f"\n{'═' * 60}")
    print(f"  SERP AUDIT DONE — '{s['keyword']}'")
    print(f"  Organic     : {s['organic_results']} results")
    print(f"  AI Overview : {'Yes' if s['ai_overview_triggered'] else 'No'}")
    print(f"  PAA         : {s['paa_questions']} questions")
    print(f"  Related     : {s['related_searches']} queries")
    print(f"{'═' * 60}\n")


def _footer_site(s: dict) -> None:
    print(f"\n{'═' * 60}")
    print(f"  SITE AUDIT DONE — {s['url']}")
    print(f"  Title       : {s['title']} ({s['title_length']} chars)")
    print(f"  Words       : {s['word_count']}")
    print(f"  Headings    : {s['h2_count']} H2 / {s['h3_count']} H3")
    print(f"  Redirects   : {s['redirect_hops']} hop(s)" + (" ⚠" if s['redirect_warning'] else ""))
    print(f"  Schema      : {s['schema_types'] or 'none'}")
    print(f"  Top bigrams : {', '.join(s['top_bigrams'])}")
    print(f"{'═' * 60}\n")


#  Entry point — edit mode and values below

if __name__ == "__main__":
    DEMO_KEYWORD = "python web scraping"
    DEMO_URL = "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"

    # Pick one:

    # Mode 1 — SERP only (requires API key)
    # serp_audit(keyword=DEMO_KEYWORD, api_key=API_KEY)

    # Mode 2 — Site only (no API key needed)
    # site_audit(url=DEMO_URL)

    # Mode 3 — Full audit (requires API key)
    full_audit(keyword=DEMO_KEYWORD, url=DEMO_URL, api_key=API_KEY)

On the demo keyword and URL the full audit ran in 29 seconds: four SERP API calls, the AI Overview follow-up among them, and five page fetches, one request at a time. No manual copying or screenshots, just structured data ready for analysis.

The seven Python scripts for SEO automation take the same SERP data further, into intent clustering, trend monitoring, and content gap analysis.

When to Use Scraping vs Traditional Tools

Use scraping when you need:

• Real-time data (not cached databases)
• Custom location or device targeting
• Bulk analysis across thousands of keywords
• Integration with your internal dashboards

Use traditional tools (Ahrefs, Semrush) when you need:

• Historical trend data
• Backlink analysis
• Competitor domain overviews

The scripts in this guide replace a dashboard subscription with data you collect yourself, on your schedule and for the exact keywords and pages you care about. Start with the site audit, which needs no API key, and add the SERP scripts once a keyword set is worth tracking every week.