Building a LinkedIn Scraper in Python

LinkedIn cuts anonymous visitors off almost immediately. In our test, plain requests with full browser headers fetched exactly one public profile cleanly, and from the second request on LinkedIn answered with its 999 status code, 26 times out of the remaining 29. A headless browser did no better. Any working LinkedIn scraper is built around that fact, so this guide starts with the wall itself, measures how far each access route gets, and then walks through the route that reliably returns public profile data, plus working Python scrapers for job listings, LinkedIn Learning, and the Top Content hub.

How Far Each Route Gets Before the Authwall

The authwall is LinkedIn’s login screen served in place of the page an anonymous visitor asked for. Depending on the client it arrives as a redirect to linkedin.com/authwall or as a response with status 999, LinkedIn’s own anti-abuse code. To measure when it appears, we collected 265 public profile URLs and fetched them in a single stream with delays, through four different routes:

The first request often passes, which is exactly what makes the wall confusing to debug. A script works on the test run, then returns login pages in production. The 999 responses in the requests row and the /authwall redirects in the Selenium row started on the second profile, and a 2.5-second delay between requests did not prevent them.

Detection is cheap, so build it in before parsing. With requests, check response.status_code == 999 and whether authwall, login, or signup appears in the final response.url after redirects. In a browser, check driver.current_url the same way. A card-shaped page that parses into empty fields usually means the check was skipped and a login page went into the parser.

The wall also has a legal backstory. In hiQ Labs v. LinkedIn, the best-known court case about web scraping, the Ninth Circuit twice held that collecting public LinkedIn pages does not violate the CFAA, and the case still ended with hiQ bound by an injunction after LinkedIn’s breach-of-contract claims survived. What that means in practice for scraping public pages is covered in our breakdown of web scraping law.

What Each Method Can and Cannot Get

The routes to scrape LinkedIn trade coverage against account risk, and the wall results above decide most cells of this comparison:

Automating a login with account credentials goes against LinkedIn’s User Agreement and can get the account restricted or banned, so the cookie and browser-profile rows are for accounts you own and accept the risk for. The two routes that need no account at all are the proxy route, which the wall limits hard, and the SERP route, which the next section covers.

Scraping Public Profiles from Google Search Results

Google indexes public LinkedIn profiles, so profile data can come from search results instead of LinkedIn’s servers. The wall never appears because no request touches LinkedIn. Our linkedin-serp-scraper repository packages this route with LLM-based structuring and CSV/JSON export, and the core of it fits in one script.

A site:linkedin.com/in query with a role keyword returns profile URLs, and the result titles already carry the name and headline. This script collects them through the Google SERP API, paginating until it has 300 results:

import requests
import json

api_key = "PUT-YOUR-API-KEY"
profiles, start = [], 0

while len(profiles) < 300:
    response = requests.get(
        "https://api.hasdata.com/scrape/google/serp",
        params={
            "q": 'site:linkedin.com/in "python developer"',
            "location": "Austin,Texas,United States",
            "deviceType": "desktop",
            "start": start,
        },
        headers={"x-api-key": api_key},
        timeout=90,
    )
    response.raise_for_status()
    organic = response.json().get("organicResults") or []
    if not organic:
        break
    for item in organic:
        link = (item.get("link") or "").split("?")[0]
        if "linkedin.com/in/" in link and not any(p["url"] == link for p in profiles):
            title = (item.get("title") or "").rsplit(" | ", 1)[0]
            profiles.append({
                "url": link,
                "name": title.split(" - ")[0].strip(),
                "headline": title.split(" - ", 1)[1].strip() if " - " in title else "",
                "snippet": item.get("snippet") or "",
            })
    start += len(organic)

with open("profiles.json", "w", encoding="utf-8") as f:
    json.dump(profiles, f, ensure_ascii=False, indent=2)
print(f"collected {len(profiles)} profiles")

In our run, 34 SERP calls produced 265 unique profiles. Each collected record carries the fields below, measured across those 265 profiles:

That is enough for lead lists and candidate sourcing, which is what most LinkedIn scraping is for, and our lead generation guide shows where such lists go next. The repository version also passes the snippets through an LLM to split locations, roles, and companies into separate columns before the CSV export.

For fields the snippet does not carry, work experience or education, the profile page itself is still needed. Fetching the collected URLs through the Web Scraping API with residential proxies and JS rendering returned the full page for 4 profiles out of 10 in our test, with the JSON-LD Person block and the experience section intact on every profile we field-checked, while the other 6 came back as authwall or partial pages. Treat per-URL enrichment as a lossy second pass over the URLs the SERP route found, and keep the SERP fields as the base record.

Other Ways to Extract Data from LinkedIn

The measured routes above cover collection at scale. The remaining options each fit a narrower job, and their code is below.

The Official LinkedIn API

LinkedIn’s own API is built for apps that act on behalf of a signed-in member, so it fits integrations more than data collection. Job search endpoints are not open to individual developers, daily request limits are low, and an individual account only gets test-company access to many endpoints. Registration happens on the developer page, which issues a Client ID and Client Secret for the OAuth flow. The flow below needs pip install requests-oauthlib, and the authorization URL must carry the scopes your app was approved for, since LinkedIn grants them per product:

from requests_oauthlib import OAuth2Session

cl_id = "PUT-YOUR-CLIENT-ID"
cl_secret = "PUT-YOUR-CLIENT-SECRET"

linkedin = OAuth2Session(cl_id, redirect_uri="http://localhost:8080/callback")
authorization_url, state = linkedin.authorization_url(
    "https://www.linkedin.com/oauth/v2/authorization"
)
print("Go to:", authorization_url)

response = input("Put full URL: ")
linkedin.fetch_token(
    "https://www.linkedin.com/oauth/v2/accessToken",
    client_secret=cl_secret,
    authorization_response=response,
)

The link goes into a browser, LinkedIn redirects back with a code, and the pasted URL turns into a token. From there the LinkedIn API documentation lists the endpoints the token can reach.

Requests with Proxies or Session Cookies

The requests library reaches public pages directly until the wall intervenes, one clean profile per IP before the 999 responses start. Routing each request through a different proxy resets that counter per IP, which is what proxy rotation is for, and the mechanics of wiring a pool into requests are in our proxy guide.

The cookie variant reuses a logged-in session instead. The li_at session cookie is in DevTools, on the Network tab of any request to LinkedIn after login:

The cookie travels in the headers with a current User-Agent string:

import requests

url = "https://www.linkedin.com/jobs/search?position=1&pageNum=0"
headers = {
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "accept-language": "en-US,en;q=0.9",
    "referer": "https://www.linkedin.com/",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
    "cookie": "YOUR-COOKIES",
}

response = requests.get(url, headers=headers)
print(response.status_code, len(response.text))

Requests now arrive as your account, which lifts the wall and moves the risk onto the account itself.

Selenium or Playwright with a Browser Profile

A browser profile that is already logged in keeps the session between runs, so the script never touches the login form. Selenium points at the user-data root and names the profile inside it (Chrome must be fully closed first, or the profile is locked):

from selenium import webdriver
from selenium.webdriver.common.by import By

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(r"--user-data-dir=C:\Users\Admin\AppData\Local\Google\Chrome\User Data")
chrome_options.add_argument("--profile-directory=Profile 1")

driver = webdriver.Chrome(options=chrome_options)
driver.get("https://www.linkedin.com/jobs/search?position=1&pageNum=0")

Playwright drives the same flow with launch_persistent_context, and everything below about parsing applies to it unchanged. For a first-time login inside the script, the form fields are username, password, and a submit button:

Filling them and submitting completes the login once, and the profile keeps the session afterward:

driver.get("https://linkedin.com/uas/login")
driver.find_element(By.ID, "username").send_keys("PUT-YOUR-LOGIN")
driver.find_element(By.ID, "password").send_keys("PUT-YOUR-PASSWORD")
driver.find_element(By.XPATH, "//button[@type='submit']").click()

Without a logged-in session, a headless browser is just a slower way to meet the wall. Our measurement above shows it, 2 profiles out of 12, with /authwall redirects starting from the second URL. Stealth plugins exist for both Selenium and Playwright, and their results vary with LinkedIn’s detection updates, so a session you own remains the stable version of this route.

Web Scraping API

For pages the wall guards loosely, the job search, Learning, and Top Content pages among them, a scraping API removes the proxy and rendering work. The request names the target URL and the proxy type, and the response carries the rendered HTML:

import requests
import json

payload = json.dumps({
    "url": "https://www.linkedin.com/jobs/search?position=1&pageNum=0",
    "proxyCountry": "US",
    "proxyType": "datacenter",
    "jsRendering": True,
})
headers = {"Content-Type": "application/json", "x-api-key": "PUT-YOUR-API-KEY"}

response = requests.post("https://api.hasdata.com/scrape/web", headers=headers, data=payload, timeout=180)
response.raise_for_status()
job_content = response.json()["content"]

The three scrapers below all start from this snippet and differ only in the URL and the parsing, so their code shows the changed parts. Each needs beautifulsoup4 and requests installed:

pip install beautifulsoup4 requests

Parsing stays on your side of the wire, so the same BeautifulSoup code works no matter which fetching route produced the HTML.

Scraping LinkedIn Job Listings

The job search page serves 60 vacancy cards per page to an anonymous visitor, and its URL parameters do the filtering. We re-ran all of them, f_SB2 included, and the page returned its full 60 cards with every selector below intact. A ready-made version of this scraper is in Google Colaboratory.

The URL takes these parameters, all optional and freely combinable:

The list view carries the title, company, location, link, and posted date per card. Details beyond that are on each vacancy’s own page, which is a separate request per job:

import requests
import json
import csv
from bs4 import BeautifulSoup

api_key = "PUT-YOUR-API-KEY"
ln_url = (
    "https://www.linkedin.com/jobs/search?f_SB2=1&f_E=1&f_TPR="
    "&location=United%20States&keywords=Data%20Scientist&f_JT=F&position=1&pageNum=0"
)

payload = json.dumps({
    "url": ln_url,
    "proxyCountry": "US",
    "proxyType": "datacenter",
    "jsRendering": True,
})
headers = {"Content-Type": "application/json", "x-api-key": api_key}
response = requests.post("https://api.hasdata.com/scrape/web", headers=headers, data=payload, timeout=180)
response.raise_for_status()

soup = BeautifulSoup(response.json()["content"], "html.parser")
job_data = []
job_list = soup.find("ul", class_="jobs-search__results-list")

for job in (job_list.find_all("li") if job_list else []):
    def text_of(tag, cls):
        el = job.find(tag, class_=cls)
        return el.get_text(strip=True) if el else "-"

    date_el = job.find("time", class_="job-search-card__listdate")
    job_data.append({
        "job_title": text_of("h3", "base-search-card__title"),
        "company": text_of("h4", "base-search-card__subtitle"),
        "location": text_of("span", "job-search-card__location"),
        "job_link": job.find("a")["href"] if job.find("a") else "-",
        "posted_date": date_el["datetime"] if date_el else "-",
    })

if job_data:
    with open("job_data.csv", "w", newline="", encoding="utf-8") as csvfile:
        writer = csv.DictWriter(csvfile, fieldnames=job_data[0].keys())
        writer.writeheader()
        writer.writerows(job_data)
print(f"saved {len(job_data)} jobs")

Our verification run collected all 60 cards, 55 of them with a machine-readable posted date, the rest freshly promoted listings without one:

Paginating through pageNum extends the same script to deeper result pages, one request per page.

Scraping LinkedIn Learning

The Learning catalog answers anonymous requests the same way, and in our check it served 50 course cards with every selector we re-verified still in place. The ready-made script is in Google Colaboratory.

The catalog filters through its own parameter set:

Only the URL and the parsing block change against the jobs scraper (the fetch itself is the snippet from the Web Scraping API section):

ln_url = (
    "https://www.linkedin.com/learning/search?sortBy=RELEVANCE"
    "&difficultyLevel=BEGINNER&entityType=COURSE&durationV2=&softwareNames="
)

learn_data = []
learn_list = soup.find("ul", class_="results-list")

for learn in (learn_list.find_all("li") if learn_list else []):
    def text_of(tag, cls):
        el = learn.find(tag, class_=cls)
        return el.get_text(strip=True) if el else "-"

    learn_data.append({
        "title": text_of("h3", "base-search-card__title"),
        "author": text_of("h4", "base-search-card__subtitle"),
        "type": text_of("p", "base-search-card__identifier"),
        "link": learn.find("a")["href"] if learn.find("a") else "-",
    })

The card fields are the course title, the author, the material type, and the link:

Course descriptions and instructor profiles are one request deeper, on the course pages themselves.

Scraping the Top Content Hub

The page at linkedin.com/pulse/topics/home/, which used to host the collaborative-articles hub, now redirects to the Top Content hub, and its old content-hub-* markup is gone. Any scraper still parsing those classes returns empty lists. The current page serves two card types, and both parse without logging in:

Editor’s Picks cards carry a category, a topic title, a like count, and a link to the topic’s page. Topic Categories cards carry the category name and its post count. This parsing block handles both, fed by the fetch snippet from the Web Scraping API section with the hub’s URL swapped in:

ln_url = "https://www.linkedin.com/top-content/"

soup = BeautifulSoup(response.json()["content"], "html.parser")

picks = []
for card in soup.find_all("div", class_="editor-topic"):
    texts = list(card.stripped_strings)
    link = card.find("a")
    picks.append({
        "category": texts[0] if texts else "-",
        "title": texts[2] if len(texts) > 2 else "-",
        "likes": texts[3] if len(texts) > 3 else "-",
        "link": link["href"] if link else "-",
    })

categories = []
for card in soup.find_all("div", class_="topic-category"):
    texts = list(card.stripped_strings)
    link = card.find("a")
    categories.append({
        "name": texts[1] if len(texts) > 1 else "-",
        "posts": texts[2] if len(texts) > 2 else "-",
        "link": link["href"] if link else "-",
    })

print(f"{len(picks)} picks, {len(categories)} categories")

Our run returned 8 Editor’s Picks, led by “Career Advancement Tips” at 916K likes, and 40 categories ranging from Business Strategy at 92K posts down to Event Planning at 9K. Each topic link leads to a page of curated posts, so a crawler that follows them collects the actual content one hop deeper.

Conclusion

The measurements settle the method choice. For profile data at scale, collect from Google’s index, where the authwall never enters the picture, and the linkedin-serp-scraper repository holds the full version of that pipeline. For the pages LinkedIn serves anonymously, the job search, Learning, and Top Content scrapers above run through the Web Scraping API without proxy management. Session-based methods still make sense for data only your own account can see, with the account risk that entails, and direct requests without rotation or a session ends at the wall almost immediately.