How to Parse JSON in Python, From APIs to JSON Hidden in HTML

A Python scraper meets JSON in an API response, where the whole body is JSON and response.json() does the work, and inside the HTML itself, where sites embed structured data for search engines and frameworks embed the page state for their own hydration. I checked 300 sites across five categories for this article, and of the 189 that answer a plain request, 83% carry machine-readable JSON somewhere in their markup. On news sites the share is 98%. Parsing that JSON is a shorter and sturdier path to the data than writing CSS selectors against the layout.

This guide covers the whole path, from the json module basics through API responses with the error handling they actually need and streaming for responses that do not fit in memory, to the kinds of JSON hiding in HTML and the measurement of how often that JSON is there.

Where Scrapers Meet JSON

Each source of JSON arrives differently and asks for a different first move.

The embedded kinds get their own measured section, because how often they appear decides whether they are worth checking first. The examples ahead need pip install requests beautifulsoup4 lxml, and the streaming section adds ijson.

Parsing Basics

The standard library’s json module has four functions, and their names pair up. loads and dumps work with strings, load and dump work with file objects.

import json

json_string = '{"name": "John Doe", "age": 30, "skills": ["python", "sql"]}'

person = json.loads(json_string)        # string to Python dict
print(person["name"], person["skills"][0])

with open("person.json", "w", encoding="utf-8") as f:
    json.dump(person, f, indent=2)      # dict to file, pretty-printed

with open("person.json", encoding="utf-8") as f:
    same_person = json.load(f)          # file back to dict

back_to_text = json.dumps(same_person)  # dict back to a string
print(back_to_text)

Parsed JSON is an ordinary dictionary. Reading, adding, changing and deleting keys is plain dict work, with no JSON-specific API involved:

person["city"] = "New York"     # the key is new, so this adds it
person["city"] = "Boston"       # the key exists, so the same line replaces the value
del person["city"]              # and this removes it again
print(person.get("city"))       # None, rather than the KeyError that person["city"] would raise

Adding and updating are the same assignment, and which one happens depends only on whether the key was already there. That is worth knowing for scraped JSON, where a field you expect may simply be absent, which is what .get() is for.

The indent=2 argument is the pretty-printer. Without it dumps writes one long line, and with it the output becomes readable for debugging.

Types map between the two worlds almost one to one:

The gaps run in the other direction. Python sets, bytes, datetime objects and custom classes have no JSON form, so json.dumps raises TypeError on them until you convert them yourself or pass a default= function.

Parsing API Responses

An API response is JSON end to end, and response.json() parses it in one call. It still deserves a real example rather than a toy, so this request pulls a repository record from the GitHub API, a response with nested objects and lists:

import requests

response = requests.get("https://api.github.com/repos/psf/requests", timeout=30)
response.raise_for_status()
repo = response.json()

print(repo["name"])                    # requests
print(repo["owner"]["login"])          # psf
print(repo["license"]["spdx_id"])      # Apache-2.0
print(repo["topics"][:3])              # ['client', 'cookies', 'forhumans']
print(repo["stargazers_count"])        # 54274

response.json() is JSON parsing applied to the body, with the encoding handled for you (the official json module documentation covers the underlying machinery). Nested values chain through keys and indexes the same way they would on any dict of dicts. The timeout and raise_for_status guards matter here for the same reasons they do in any Python scraper: Requests waits forever by default, and an error page fed to a parser produces confusing failures two lines later.

When the Response Is Not JSON at All

Every scraper that parses external JSON eventually receives something else. An HTML error page, a CAPTCHA interstitial and a truncated body all reach json.loads sooner or later, and each one raises json.JSONDecodeError, an exception carrying enough context to tell you what actually arrived:

import requests

response = requests.get("https://api.github.com/repos/psf/requests", timeout=30)

try:
    data = response.json()
except requests.exceptions.JSONDecodeError as error:
    print(f"not JSON: {error.msg} at position {error.pos}")
    print(f"body starts with: {error.doc[:80]!r}")
    raise SystemExit(1)

The exception class matters here. response.json() raises requests.exceptions.JSONDecodeError, which is not guaranteed to be a subclass of the standard library’s json.JSONDecodeError (with simplejson installed it is not), so catching the Requests class is the version that works everywhere. For plain json.loads calls, catch json.JSONDecodeError itself. Either class carries the same fields: an HTML page fails at position 0 with Expecting value, and error.doc[:80] shows the <html> that came instead of the data, while a body cut off mid-transfer fails near the end. That difference is worth logging, since the first means the site refused you and the second means the connection did.

Checking response.headers["Content-Type"] before parsing keeps this branch quiet on sites known to serve HTML error pages with status 200, and keeping raise_for_status() ahead of .json() lets a plain 404 or 500 surface as an HTTP error rather than a parse error.

When Keys Repeat

The JSON specification leaves duplicate keys undefined, and real feeds ship them anyway. Python’s parser does not raise on them. It keeps the last value and silently drops the rest:

import json

print(json.loads('{"price": 10, "price": 20}'))

The output is {'price': 20}, and the 10 is gone without a trace. When the dropped values matter, object_pairs_hook receives every pair before the dict swallows them:

def collect(pairs):
    out = {}
    for key, value in pairs:
        if key in out:
            prev = out[key]
            out[key] = prev + [value] if isinstance(prev, list) else [prev, value]
        else:
            out[key] = value
    return out

print(json.loads('{"price": 10, "price": 20}', object_pairs_hook=collect))

This prints {'price': [10, 20]}, with both values kept. Older advice suggests strict=False for duplicate keys, and that parameter has nothing to do with them. It only permits raw control characters inside strings, so a body with a literal newline inside a value parses instead of raising.

Streaming Large Responses with ijson

json.load builds the whole structure in memory before your code sees the first record. On a 100 MB response that costs real RAM. I measured both approaches on a 100 MB array of 1,136,080 flat records:

Both runs read every record and computed the same sum. The streaming version holds one record at a time, so peak memory stays flat no matter how large the response grows, at the price of running slower:

import ijson

total = 0
with open("big.json", "rb") as f:
    for item in ijson.items(f, "item"):
        total += float(item["price"])   # one record in memory at a time

The second argument is the path to iterate, and "item" means each element of a top-level array. ijson accepts any file-like object, so a streamed download plugs in directly through ijson.items(response.raw, "item") after requests.get(url, stream=True). Reach for it when responses stop fitting comfortably, and stay with json.load below that line, since it is faster and simpler. Faster drop-in parsers like orjson exist too, but a scraper spends its time waiting on the network, so parse speed rarely changes anything.

JSON Hidden Inside HTML

Sites embed JSON in their pages for their own reasons, and a scraper can extract it instead of parsing the layout. The data arrives typed and named, the way the site’s own developers see it, and it survives redesigns that break every CSS selector.

application/ld+json

Structured data for search engines sits in <script type="application/ld+json"> blocks: articles with headlines and dates, products with names and prices, job postings with salaries. The site maintains this block for machines to read, which makes it the most stable data source on the page. This example takes the freshest article from a news front page and reads its headline, date and author without touching the article’s layout at all:

import json
import re
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

UA = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                    "(KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

response = requests.get("https://www.theguardian.com/international", headers=UA, timeout=30)
response.raise_for_status()
soup = BeautifulSoup(response.content, "lxml")

# take the first dated article link and fetch the article itself
link = soup.find("a", href=re.compile(r"/20\d{2}/"))
article = requests.get(urljoin(response.url, link["href"]), headers=UA, timeout=30)
soup = BeautifulSoup(article.content, "lxml")

for script in soup.find_all("script", type="application/ld+json"):
    try:
        data = json.loads(script.string)
    except json.JSONDecodeError:
        continue                      # one malformed block should not kill the loop
    items = data if isinstance(data, list) else data.get("@graph", [data])
    for item in items:
        if "NewsArticle" in (item.get("@type") or ""):
            print(item["headline"])
            print(item["datePublished"], "|", item["author"][0]["name"])
            break

On the run behind this article it printed the headline of that morning’s lead story with its publication timestamp and byline. The wrapper logic exists because sites package the blocks differently. Some pages carry several ld+json scripts, some wrap everything in a single @graph array, and some put a bare object in each block, so the loop normalizes all three shapes before filtering by @type, skips a malformed block instead of dying on it, and matches NewsArticle inside multi-type arrays too.

NEXT_DATA and Framework State

Sites built on Next.js ship their entire page state as one JSON document in a script tag with a known id. Whatever the page renders, the data behind it is in there:

import json

import requests
from bs4 import BeautifulSoup

UA = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                    "(KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

response = requests.get("https://www.bbc.com/news", headers=UA, timeout=30)
response.raise_for_status()
soup = BeautifulSoup(response.content, "lxml")

tag = soup.find("script", id="__NEXT_DATA__")
if tag is None:
    raise SystemExit("no __NEXT_DATA__ on this page")
state = json.loads(tag.string)

page = state["props"]["pageProps"]
print(len(json.dumps(page)), "bytes of page state")

On bbc.com this run found roughly 100 KB of page state, every headline and link on the page included, in one parse and zero selectors. The inner structure differs per site, so the practical workflow is to dump state to a file once, read it in an editor to find the path to your data, and hardcode that path. In the 300-site scan, 35 sites carried __NEXT_DATA__, with the highest shares among real estate and SaaS sites.

Other frameworks do the same under different names. window.__INITIAL_STATE__, __PRELOADED_STATE__, __APOLLO_STATE__, __NUXT__ and __remixContext are all page state serialized into a script, and they differ from __NEXT_DATA__ only in how the object is wrapped.

Finding the Path to Your Data

A framework state blob is big. The BBC one above is around 100 KB, and reading it by eye to locate one field gets old fast. Fifteen lines of recursion answer the question directly, printing every path at which a key appears:

def find_paths(node, wanted, path="state"):
    if isinstance(node, dict):
        for key, value in node.items():
            here = f"{path}[{key!r}]"
            if key == wanted:
                yield here
            yield from find_paths(value, wanted, here)
    elif isinstance(node, list):
        for i, value in enumerate(node[:20]):
            yield from find_paths(value, wanted, f"{path}[{i}]")


for path in list(find_paths(state, "title"))[:3]:
    print(path)

Against the bbc.com state from the previous example, this printed state['props']['pageProps']['navigation']['mainNavigation'][0]['title'] and two siblings. Run it once with the field name you saw on the page, copy the path it prints into the scraper, and delete the helper. The list slice caps how deep it walks into long arrays, since the twentieth item of a list rarely has keys the first one lacks.

Inline var data and Other State Assignments

State assignments and plain var data = {...} scripts sit in anonymous script tags, where the JSON to extract lives inside JavaScript source rather than standing alone. raw_decode handles that. It parses one complete value from the start of a string and ignores whatever follows, so you point it at the character after = and it stops by itself at the matching brace:

import json

import requests

UA = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                    "(KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"}

html = requests.get("https://www.newyorker.com/", headers=UA, timeout=30).text

marker = html.find("window.__PRELOADED_STATE__")
if marker == -1:
    raise SystemExit("no state assignment on this page")

start = html.index("=", marker) + 1
state, _ = json.JSONDecoder().raw_decode(html[start:].lstrip())
print(len(json.dumps(state)), "bytes,", "top keys:", list(state)[:4])

On newyorker.com this cut 1.18 MB of __PRELOADED_STATE__ out of the homepage source and parsed it in one call. The one thing raw_decode cannot survive is state that is not valid JSON, JavaScript-only values being the usual reason. cbc.ca’s __INITIAL_STATE__ fails 35 KB in on a literal undefined sitting where JSON expects a value. When that happens, the page usually also carries one of the cleaner kinds above.

How Often the Data Is Already There

Checking for embedded JSON before writing selectors pays five times out of six, and that number comes from measuring it. The scan took 300 sites, 60 each across shops, news, job boards, real estate and SaaS, fetched each homepage plus one content page linked from it, and looked for every kind of embedded JSON above.

The first finding is about reachability rather than JSON. 111 of the 300 sites do not answer a plain Python request, 87 of them refusing with HTTP 403 and the rest timing out or failing the connection, and shops and real estate portals block hardest, with 33 and 37 of 60 unreachable. Embedded JSON is only readable after the fetch succeeds, so for those sites the fetching problem comes first, whether through your own proxy setup or a Web Scraping API that returns the rendered HTML.

Among the 189 sites that do answer, the embedded JSON is close to ubiquitous:

The same shares as a picture, category by category.

News is the standout. 93% of reachable news sites carry JSON-LD and 24 of them declare full NewsArticle objects, so headlines, timestamps and authors are available without parsing an article layout even once. The most common types overall are Organization (106 sites), WebSite (76) and ImageObject (68), which identify the site rather than its content, so the useful signal is whether a content page carries a content type like NewsArticle, Product or JobPosting. That is why the scan fetched a content page and not just the homepage, and it reached one on 118 of the 189 sites.

The practical rule the numbers support is to view the source before writing selectors. Searching a page for ld+json, __NEXT_DATA__ and __INITIAL_STATE__ takes thirty seconds, and most reachable sites reward it with something to parse.

Converting JSON to CSV

Flat JSON converts to CSV with the standard library alone. csv comes with Python, and nothing needs installing:

import csv
import json

records = json.loads('''[
  {"name": "requests", "stars": 54274, "language": "Python"},
  {"name": "httpx", "stars": 14567, "language": "Python"}
]''')

with open("repos.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "stars", "language"])
    writer.writeheader()
    writer.writerows(records)

Nested JSON needs flattening first, and pandas.json_normalize does it in one call, turning nested keys into dotted column names:

import pandas as pd

repo = {"name": "requests", "stars": 54274,
        "license": {"spdx_id": "Apache-2.0", "name": "Apache License 2.0"}}

df = pd.json_normalize(repo)
print(list(df.columns))   # ['name', 'stars', 'license.spdx_id', 'license.name']
df.to_csv("repos.csv", index=False)

The same DataFrame writes to Excel through df.to_excel("repos.xlsx", index=False), which needs openpyxl installed. pd.DataFrame on a dict of plain scalars raises ValueError asking for an index, and json_normalize handles that shape without complaint, so for single API records it is the safer default.

Conclusion

The json module covers the mechanics in four functions, and the scraping-specific skill is knowing where the JSON is. Check the API first, then the page source for ld+json and framework state, and only then reach for selectors. On the sites measured here, that order usually resolves the data question before any layout parsing starts, and the JSON version of the data carries names, types and structure the layout never will.

The failure paths deserve the same respect as the happy path. Every guard in this article is a line or two, and each one converts a confusing crash into a log line that says what the server actually sent.