How to Use cURL in Python

A curl command is a request written out in full, and the fastest way to run it in Python is to translate it into a requests call. The command usually comes from an API reference or from Chrome’s DevTools (right-click a request, Copy as cURL), and every flag maps onto a requests argument, -H onto headers, -d onto data, -b onto cookies, -x onto proxies. To see how much survives the translation, I copied 47 page requests from Chrome 151 and replayed each one four ways: with curl itself, with the Python that the open-source converter produced, with curl_cffi impersonating Chrome, and with a bare requests.get(). The converted code kept all 521 headers and got exactly what curl got, 37 pages and 10 blocks. The impersonating client changed nothing on these pages, and the bare call lost three more sites.

The rest of this article is the flag table behind that translation, the flags that vanish on the way, the measurement in full, and the three other routes: curl_cffi for sites that check the TLS handshake, subprocess for running the command as it is, and PycURL for the projects that want libcurl’s own options.

Where the curl command comes from

The command comes from one of two places, and the source shapes how long it is. API documentation gives you a short command with two or three headers and a body, the kind that translates in a minute. Chrome’s DevTools give you the long kind. Open the Network tab, find the request that carried the data (the Fetch/XHR filter narrows the list), right-click it, and choose Copy, then Copy as cURL (bash), or Copy as cURL (cmd) on Windows if you want to paste it into a Windows shell. Every header the browser sent is in the command, cookies included as a -b flag, and --compressed at the end tells curl to ask for gzip and unpack it. For the quotes API on the infinite-scroll practice page, Chrome 151 produces this:

curl 'https://quotes.toscrape.com/api/quotes?page=1' \
  -H 'accept: */*' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'priority: u=1, i' \
  -H 'referer: https://quotes.toscrape.com/scroll' \
  -H 'sec-ch-ua: "Not=A?Brand";v="99", "Google Chrome";v="151", "Chromium";v="151"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36' \
  -H 'x-requested-with: XMLHttpRequest' \
  --compressed

Paste that into a terminal and it returns the same JSON the page received. The point of the copy is that it is a complete, working request with the exact headers that the server accepted a second ago, and the same button opens our list of DevTools tricks for scraping. The sec-ch-ua and sec-fetch-* headers are what Chrome sends about itself. A server can check them, most do not, and the section on curl_cffi below shows which ones a plain Python client cannot fake.

Translating curl flags to Python

Every flag in a copied command has a counterpart in requests, and the mapping is mechanical enough that converters exist for it. The open-source curlconverter library does it in the terminal or in the browser, and HasData’s own curl to Python converter runs the same kind of translation on the site. The table is what those tools apply, and knowing it means you can read their output and fix it when the command uses a flag they skip.

-L, --compressed, --http2, and --retry vanish in translation without a trace. The first two are harmless, because requests follows redirects and decompresses on its own, with one twist in the other direction, a command without -L will follow redirects in Python that curl would have stopped at. The last two change behaviour, since the converted script speaks HTTP/1.1 and never retries. And a command without --max-time produces a call without timeout=, which waits forever on a stalled server where curl at least keeps running in a terminal you can see. Add timeout= to every converted call.

requests

The command above, translated by hand, is a dozen lines. The query string moves into params=, the headers into a dict, and --compressed disappears because requests already asks for compression:

import requests

headers = {
    "accept": "*/*",
    "accept-language": "en-US,en;q=0.9",
    "referer": "https://quotes.toscrape.com/scroll",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
    "x-requested-with": "XMLHttpRequest",
}
response = requests.get("https://quotes.toscrape.com/api/quotes", params={"page": 2}, headers=headers, timeout=30)
response.raise_for_status()
data = response.json()
print(response.status_code, len(data["quotes"]), "quotes, has_next =", data["has_next"])
print(data["quotes"][0]["author"]["name"], "-", data["quotes"][0]["text"][:50])

The run printed 200 10 quotes, has_next = True. I dropped the sec-ch-ua and sec-fetch-* headers because this server ignores them, and that is the first judgment call a converter cannot make for you. Keep all the headers when a request fails without them, strip the browser-identity ones when it does not, since sec-ch-ua claiming Chrome 151 while the TLS handshake says Python is the inconsistency anti-bot vendors look for, even if none of the 47 pages measured below acted on it.

httpx

httpx takes the same arguments with two differences that matter for curl commands. It speaks HTTP/2 when installed with the h2 extra (pip install 'httpx[http2]'), which is what --http2 asks for, and it does not follow redirects unless told to, the same default as curl without -L:

import httpx

headers = {"accept": "application/json", "user-agent": "Mozilla/5.0"}

with httpx.Client(http2=True, headers=headers, follow_redirects=True, timeout=30) as client:
    response = client.get("https://quotes.toscrape.com/api/quotes", params={"page": 1})
    print(response.status_code, response.http_version, len(response.json()["quotes"]), "quotes")
    plain = client.get("https://quotes.toscrape.com/random")
    print(plain.status_code, plain.http_version, len(plain.text), "bytes of HTML")

Both calls came back over HTTP/2. For a copied command that is just a GET with headers, httpx and requests are interchangeable, and the choice follows whatever the rest of the project uses.

What gets lost in translation

The converter’s promise is that the Python it produces makes the same request as the command you pasted. I tested that on 47 pages, the shops, news front pages, documentation sites, developer platforms, market data, and job boards from the dynamic-content survey, chosen before running anything and with robots.txt respected. Chrome 151 opened each page once through Playwright, and the headers of the document request became a Copy as cURL command, the same headers DevTools would give you (twelve on 40 of the pages, fewer where the site served Chrome a challenge), minus cookies, since a first visit has none. Each command was then replayed four ways from the same machine: curl 8 on Windows with -L --max-time 30, the requests code that curlconverter 4.12 produced with a 30-second timeout added, curl_cffi 0.16 with the same headers and impersonate="chrome", and a bare requests.get(url) with no headers at all. A 200 without a challenge title counted as getting the page. A 401, 403, 429, or 503, or a title such as “Just a moment” or “Access Denied”, counted as blocked.

The same four columns as a picture:

The converter kept every one of the 521 headers across the 47 commands and produced the same verdict as curl on all 47. On the 37 pages both fetched, the body sizes matched byte for byte, which rules out quiet differences such as a mobile layout served to one client and not the other. The three clients that carried the copied headers behaved identically, and the fourth, the bare call, lost three sites.

The Verge and crates.io answer python-requests/2.34.2 with a 403 and answer the copied headers with the page. RemoteOK dropped the bare connection. Ten sites blocked all four clients, curl included: Cloudflare’s “Just a moment” page on Stack Overflow, Product Hunt, and npm, Cloudflare’s “Attention Required” on IKEA, a challenge page on Newegg and on CoinGecko, “Access Denied” on H&M, and the block pages of Indeed, Glassdoor, and Zillow. Those pages want a JavaScript challenge solved or a different IP, and no amount of header copying gets a plain HTTP client through them.

The sample also shows what did not matter here. The TLS fingerprint changed nothing. curl_cffi with Chrome’s handshake fetched exactly the pages curl and requests fetched and was blocked on exactly the same ten, so on these 47 landing pages the copied headers did all the work and the handshake did none. Time was not lost in translation either. The converted scripts took 1.22 s per request at the median against 0.88 s for curl, and the difference is the Python interpreter starting for each script in the test, since curl_cffi and the bare call, run inside one process, came in at 0.80 s and 0.75 s. Where the handshake does decide is documented with JA3 and JA4 measurements in the guide to scraping without getting blocked, and one request per site from a home connection did not trigger those checks.

When requests is not enough, curl_cffi

A converted command carries every header the browser sent and none of the handshake. Chrome negotiates TLS with a particular set of cipher suites, extensions, and ALPN values, and opens HTTP/2 with particular SETTINGS frames. Python’s ssl module produces a different set, and anti-bot vendors fingerprint both. Measured on the same machine, requests presents the JA4 fingerprint t13d1712h1_ab0a1bf427ad_882d495ac381 (17 cipher suites, 12 extensions, HTTP/1.1) while Chrome presents t13d1516h2_8daaf6152771_806a8c22fdea, and no header can change that. This is what curl_cffi is for. It is a Python binding to a fork of curl that reproduces browser handshakes, and it keeps the requests call shape:

from curl_cffi import requests

headers = {
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "accept-language": "en-US,en;q=0.9",
}
response = requests.get("https://tls.browserleaks.com/json", headers=headers, impersonate="chrome", timeout=30)
report = response.json()
print(response.status_code, "| JA4:", report.get("ja4"), "| HTTP/2 fingerprint:", report.get("akamai_hash"))
print("user-agent sent:", report.get("user_agent"))

The fingerprint service reported a JA4 of t13d1516h2_8daaf6152771_806a8c22fdea, Chrome’s value, and the HTTP/2 fingerprint 52d84b11737d980aef856699f885ca86, from a Python script. impersonate="chrome" in curl_cffi 0.16 also fills in Chrome 150’s own user-agent, sec-ch-ua, and accept headers unless you override them, so the copied command’s headers and the handshake finally agree. Version-pinned targets exist for sites that compare the two closely, 44 of them in 0.16.2, from "chrome142" and "safari18_0" to "firefox147" and the Android and iOS variants. In the 47-page test above this made no difference, because none of those sites checked the handshake on a single request, so treat curl_cffi as the second step, taken when a header-complete request still comes back with a challenge. The guide to scraping without getting blocked measures how far impersonation gets against Cloudflare, DataDome, and Akamai, and where it stops. Past that point, the remaining option is a managed browser behind rotating proxies, which is what HasData’s Web Scraping API does for one credit per plain request through datacenter proxies or ten with JavaScript rendering (five and fifteen through residential proxies), billed only on success, with 1,000 free credits at sign-up.

Running curl directly with subprocess

The shortest route from a copied command to Python skips the translation. subprocess runs the command and hands back its output:

import json
import subprocess

# Pass a list, not a shell string: no quoting surprises and no shell injection
# when part of the command comes from user input.
command = [
    "curl", "--silent", "--show-error", "--fail", "--max-time", "30",
    "https://quotes.toscrape.com/api/quotes?page=1",
    "-H", "accept: application/json",
    "--compressed",
]
result = subprocess.run(command, capture_output=True, text=True, check=True)
data = json.loads(result.stdout)
print(len(data["quotes"]), "quotes via subprocess, exit code", result.returncode)

--fail makes curl exit non-zero on a 4xx or 5xx, and check=True turns that into a CalledProcessError, so a blocked request raises instead of handing you an error page as data. The approach is right for a one-off command, for curl features requests lacks (--resolve, --http3, FTP, --unix-socket), and for scripts that already live in a shell environment. It is wrong as a general HTTP client. Every response is text or bytes you parse yourself, there is no session object carrying cookies between calls, text=True decodes stdout with the console’s encoding rather than the page’s, and each request pays for a process start. Ten thousand pages through subprocess is ten thousand process launches.

PycURL for low-level control

PycURL is the Python binding to libcurl itself, the library inside the curl binary. It exposes every CURLOPT_* option as setopt() calls, which makes it verbose and precise. The current release, 7.47.0, ships wheels for CPython 3.10 through 3.14 on Windows, macOS, and Linux, each bundling its own libcurl (8.20.0 on Windows, with OpenSSL and Schannel), so pip install pycurl finishes in seconds on the three platforms most people use:

import json
from io import BytesIO
from urllib.parse import urlencode

import pycurl

# GET: libcurl writes the body into a buffer you own.
buffer = BytesIO()
curl = pycurl.Curl()
curl.setopt(pycurl.URL, "https://quotes.toscrape.com/api/quotes?page=1")
curl.setopt(pycurl.HTTPHEADER, ["accept: application/json", "user-agent: Mozilla/5.0"])
curl.setopt(pycurl.ACCEPT_ENCODING, "")        # the --compressed equivalent
curl.setopt(pycurl.FOLLOWLOCATION, True)       # -L
curl.setopt(pycurl.TIMEOUT, 30)                # --max-time 30
curl.setopt(pycurl.WRITEDATA, buffer)
curl.perform()
status = curl.getinfo(pycurl.RESPONSE_CODE)
print("GET", status, len(json.loads(buffer.getvalue())["quotes"]), "quotes")

# POST: reuse the handle, swap the URL and attach a form body (-d).
buffer = BytesIO()
curl.setopt(pycurl.URL, "https://httpbin.org/post")
curl.setopt(pycurl.POSTFIELDS, urlencode({"field": "value"}))
curl.setopt(pycurl.WRITEDATA, buffer)
curl.perform()
print("POST", curl.getinfo(pycurl.RESPONSE_CODE), json.loads(buffer.getvalue())["form"])
curl.close()

The run printed GET 200 10 quotes and POST 200 {'field': 'value'}. The mapping from a curl flag to a setopt constant is closer than the mapping to requests, because both are libcurl underneath: -H is HTTPHEADER, -L is FOLLOWLOCATION, --compressed is ACCEPT_ENCODING set to an empty string, -x is PROXY, -u is USERPWD. What PycURL buys over requests is that completeness and libcurl’s transfer machinery, CurlMulti for hundreds of concurrent handles in one thread, and the protocols curl supports beyond HTTP. What it costs is the code above, three times the length of the requests version for the same two calls, and bytes-in-buffers instead of a Response object.

Installing PycURL

PycURL’s reputation for painful installs comes from the years when PyPI carried only a source distribution. Building it meant a C compiler, libcurl headers, and an SSL backend that matched the one libcurl was compiled against, and a mismatch produced the libcurl link-time ssl backend (openssl) is different from compile-time ssl backend error that fills the old forum threads. Wheels arrived with 7.45.3, and every release since covers the mainstream platforms, so the reputation now applies to the edges: PyPy, Alpine and other musl-based images, unusual architectures, or a pinned old version. On those, the PycURL install docs still describe the source build.

One more install problem has nothing to do with PycURL. pip install curl fails with No matching distribution found for curl because no package of that name exists on PyPI. The binding is pycurl, the browser-impersonating fork is curl_cffi, and the command-line tool comes with the operating system.

Conclusion

A copied curl command is already the answer to how a request should look, and Python only needs it translated. Run it through a converter or the table above, add timeout=, keep the headers, and the requests call behaves like the command did on 47 out of 47 pages in this test. Reach for curl_cffi when a site checks the handshake rather than the headers, which none of these 47 did on a single request, and it costs one import to find out. subprocess is for one-off commands and the few curl features Python clients lack, and PycURL is for projects that want libcurl’s own options and protocols rather than an HTTP client. pip install curl installs nothing, because that package does not exist.