Web Scraping with C# in 2026

C# runs scraping scripts efficiently on Windows, Linux, and macOS. The async/await model and built-in thread pool handle high-concurrency workloads without requiring external frameworks. The NuGet ecosystem covers everything from HTML parsing to full browser automation, and the same code compiles to a self-contained binary that ships without a runtime dependency.

HttpClient is built into .NET and handles all HTTP requests without additional packages. HtmlAgilityPack parses the downloaded HTML and lets you query elements by XPath or CSS selector. AngleSharp is the modern alternative with a CSS-first API and LINQ integration. For pages that inject content via JavaScript, Playwright for .NET controls Chrome, Firefox, or WebKit and manages the browser binary automatically.

Setting up the environment

C# scrapers run on .NET, available at the official .NET website. You can write code in Visual Studio or Visual Studio Code. Visual Studio ships with a full project management UI and a NuGet GUI. Visual Studio Code with the C# Dev Kit extension covers the same ground at a smaller footprint.

Verify the installation:

dotnet --version

All examples target .NET 8, the current LTS release. Create a new console project:

dotnet new console -n Scraper
cd Scraper

Add NuGet packages with dotnet add package PackageName and run the project with dotnet run. The examples below use top-level statements, which are available since .NET 6.

C# libraries for web scraping

ScrapySharp appears in older tutorials but has had no meaningful development since the early 2010s. AngleSharp covers the same CSS-selector-based use case on an actively maintained codebase.

HttpClient vs headless browser: benchmark

All six approaches were tested against 50 pages of books.toscrape.com (1,000 books, 5 fields per book) on .NET 8.0.424, Windows 11.

HtmlAgilityPack and AngleSharp finish within 2% of each other because at 50 pages, network round-trip time dominates. The parallel run with SemaphoreSlim(5) delivers a 4× speedup over sequential without changing the parser.

Headless browser approaches take 2.4–3× longer than sequential HttpClient on a site that returns static HTML. The overhead is the browser process: rendering a full page, executing CSS, and maintaining a DevTools Protocol connection for every navigation. On a JavaScript-rendered page where HttpClient returns an empty container, this overhead is the price of correctness. On static HTML it adds nothing.

Fetching pages with HttpClient

HttpClient is part of the .NET standard library. No NuGet packages needed. The recommended pattern is one shared instance per application. Creating a new instance per request exhausts the socket pool on high-volume scrapers:

using System.Net.Http;

var client = new HttpClient();
client.DefaultRequestHeaders.Add(
    "User-Agent",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36"
);

var html = await client.GetStringAsync("https://books.toscrape.com/");
Console.WriteLine(html[..500]);

GetStringAsync returns the full HTML as a string. Pass it to HtmlAgilityPack or AngleSharp for parsing. Servers that check headers return 403 when the User-Agent is missing or looks like a bot, so set it on every client you instantiate.

For pages that return 429 or 503 intermittently, a retry wrapper handles the common cases:

async Task<string> FetchWithRetry(HttpClient client, string url, int maxRetries = 3)
{
    for (var attempt = 0; attempt < maxRetries; attempt++)
    {
        var response = await client.GetAsync(url);
        if (response.IsSuccessStatusCode)
            return await response.Content.ReadAsStringAsync();

        if ((int)response.StatusCode is 429 or 503 or 504)
            await Task.Delay(TimeSpan.FromSeconds(Math.Pow(2, attempt)));
        else
            response.EnsureSuccessStatusCode();
    }
    throw new Exception($"Failed after {maxRetries} retries");
}

The delay doubles on each attempt: 1 s, 2 s, 4 s. Most transient rate-limit windows clear inside that time.

To route requests through a proxy, configure HttpClientHandler:

var proxy = new WebProxy("http://PROXY_IP:PORT")
{
    Credentials = new NetworkCredential("username", "password")
};
var handler = new HttpClientHandler { Proxy = proxy };
var client = new HttpClient(handler);

For large lists of URLs, SemaphoreSlim limits concurrent requests without blocking threads:

var semaphore = new SemaphoreSlim(5);

var tasks = urls.Select(async url =>
{
    await semaphore.WaitAsync();
    try { return await FetchWithRetry(client, url); }
    finally { semaphore.Release(); }
});

var pages = await Task.WhenAll(tasks);

Five concurrent requests is a reasonable starting point. Raise it if the server handles it without returning 429, lower it if you see errors.

For login-protected pages that track state via cookies, attach a CookieContainer to the handler. After the login POST, subsequent requests from the same client carry the session cookie automatically:

var cookies = new CookieContainer();
var handler = new HttpClientHandler { CookieContainer = cookies };
var client = new HttpClient(handler);

var loginData = new FormUrlEncodedContent(new[]
{
    new KeyValuePair<string, string>("username", "user"),
    new KeyValuePair<string, string>("password", "password")
});

await client.PostAsync("https://quotes.toscrape.com/login", loginData);

var html = await client.GetStringAsync("https://quotes.toscrape.com/");

This approach works when the login flow is a standard HTML form POST. Pages that validate JavaScript execution during login (bot detection middleware) need Playwright for .NET instead.

Parsing HTML with HtmlAgilityPack

Install the package:

dotnet add package HtmlAgilityPack

HtmlAgilityPack builds a DOM tree from the HTML string and exposes it through XPath selectors. The example below fetches all book titles and prices from books.toscrape.com:

using HtmlAgilityPack;

var web = new HtmlWeb();
var doc = web.Load("https://books.toscrape.com/");

foreach (var book in doc.DocumentNode.SelectNodes("//article[@class='product_pod']"))
{
    var title = book.SelectSingleNode(".//h3/a")?.GetAttributeValue("title", "");
    var price = book.SelectSingleNode(".//p[@class='price_color']")?.InnerText.Trim();
    Console.WriteLine($"{title}: {price}");
}

SelectNodes takes an XPath expression and returns an HtmlNodeCollection. .InnerText gives the text content of a node. .GetAttributeValue("attr", fallback) reads an attribute value and returns the fallback string when the attribute is absent.

XPath covers filtering patterns that CSS handles in a browser. A few common ones:

Note that [@class='product'] matches only elements whose class attribute is exactly product. For elements with multiple classes (e.g., class="product featured"), use contains(): //div[contains(@class,'product')].

Scraping multiple pages

The loop below collects all 1,000 books across 50 pages of books.toscrape.com:

using System.Net.Http;
using HtmlAgilityPack;

var client = new HttpClient();
var baseUrl = "https://books.toscrape.com/catalogue/";
var nextPath = "page-1.html";
var books = new List<(string Title, string Price)>();

while (nextPath != null)
{
    var html = await client.GetStringAsync(baseUrl + nextPath);
    var doc = new HtmlDocument();
    doc.LoadHtml(html);

    foreach (var book in doc.DocumentNode.SelectNodes("//article[@class='product_pod']")
        ?? Enumerable.Empty<HtmlNode>())
    {
        var title = book.SelectSingleNode(".//h3/a")?.GetAttributeValue("title", "") ?? "";
        var price = book.SelectSingleNode(".//p[@class='price_color']")?.InnerText.Trim() ?? "";
        books.Add((title, price));
    }

    var nextLink = doc.DocumentNode.SelectSingleNode("//li[@class='next']/a");
    nextPath = nextLink?.GetAttributeValue("href", null);
    if (nextPath != null) await Task.Delay(500);
}

Console.WriteLine($"Scraped {books.Count} books");

?? Enumerable.Empty<HtmlNode>() guards against the null that SelectNodes returns on the last page when no nodes match. Task.Delay(500) adds a half-second pause between requests, which is enough to stay below most server rate limits.

Modern HTML parsing with AngleSharp

Install the package:

dotnet add package AngleSharp

AngleSharp implements the full W3C HTML5 parsing spec and exposes a CSS-first API. The same books example with AngleSharp:

using AngleSharp;

var config = Configuration.Default.WithDefaultLoader();
var context = BrowsingContext.New(config);
var document = await context.OpenAsync("https://books.toscrape.com/");

foreach (var book in document.QuerySelectorAll("article.product_pod"))
{
    var title = book.QuerySelector("h3 a")?.GetAttribute("title");
    var price = book.QuerySelector("p.price_color")?.TextContent.Trim();
    Console.WriteLine($"{title}: {price}");
}

QuerySelectorAll accepts any CSS selector. .TextContent is the equivalent of .innerText in a browser.

AngleSharp also exposes a LINQ interface over the DOM for those who prefer method chains over selector strings:

var titles = document
    .GetElementsByTagName("article")
    .Where(el => el.ClassName == "product_pod")
    .Select(el => el.QuerySelector("h3 a")?.GetAttribute("title"))
    .Where(t => t != null)
    .ToList();

HtmlAgilityPack is the safer choice when the target HTML is heavily malformed. It applies the same lenient error correction that browsers use. On clean or standards-compliant HTML, both libraries produce the same results and the decision comes down to whether the team prefers XPath or CSS selectors.

JavaScript-rendered pages with Playwright for .NET

Install the package, build the project to generate the install script, then download the Chromium binary:

dotnet add package Microsoft.Playwright
dotnet build
pwsh bin/Debug/net8.0/playwright.ps1 install chromium

Playwright for .NET manages the browser binary, so there is no separate ChromeDriver download that needs to match the installed Chrome version.

The example below scrapes quotes from the JavaScript-rendered version of quotes.toscrape.com, where the quote list is injected after page load:

using Microsoft.Playwright;

var playwright = await Playwright.CreateAsync();
await using var browser = await playwright.Chromium.LaunchAsync(new BrowserTypeLaunchOptions
{
    Headless = true
});
var page = await browser.NewPageAsync();

await page.GotoAsync("https://quotes.toscrape.com/js/");
await page.WaitForSelectorAsync("div.quote");

var quotes = await page.EvaluateAsync<string[]>(
    "[...document.querySelectorAll('.quote .text')].map(el => el.innerText)"
);

foreach (var quote in quotes)
    Console.WriteLine(quote);

WaitForSelectorAsync blocks until the selector appears in the DOM. EvaluateAsync runs a JavaScript expression in the page context and returns the result as a typed .NET value.

For login-protected pages, Playwright handles the full session flow:

await page.GotoAsync("https://quotes.toscrape.com/login");
await page.FillAsync("input[name='username']", "user");
await page.FillAsync("input[name='password']", "password");
await page.ClickAsync("input[type='submit']");
await page.WaitForURLAsync("**/");

Subsequent page.GotoAsync calls after the login carry the session cookie automatically. Playwright also supports Firefox and WebKit from the same API: swap playwright.Chromium for playwright.Firefox or playwright.Webkit without changing any other code.

When a scraper returns empty results, taking a screenshot at the point of failure is the fastest way to diagnose whether the page actually loaded:

await page.ScreenshotAsync(new PageScreenshotOptions { Path = "debug.png", FullPage = true });

Add that line before the EvaluateAsync call when debugging. Common culprits are a cookie consent dialog covering the content, or a redirect to a login page that WaitForSelectorAsync never sees.

PuppeteerSharp

PuppeteerSharp is the C# port of Puppeteer. It drives Chromium and downloads the browser binary on first run:

dotnet add package PuppeteerSharp

using PuppeteerSharp;

await new BrowserFetcher().DownloadAsync();

await using var browser = await Puppeteer.LaunchAsync(new LaunchOptions { Headless = true });
await using var page = await browser.NewPageAsync();

await page.GoToAsync("https://quotes.toscrape.com/js/");
await page.WaitForSelectorAsync("div.quote");

var quotes = await page.EvaluateExpressionAsync<string[]>(
    "[...document.querySelectorAll('.quote .text')].map(el => el.innerText)"
);

foreach (var quote in quotes)
    Console.WriteLine(quote);

The API and behavior are nearly identical to Playwright. Pick Playwright for .NET for new projects because it supports multiple browser engines and has a more modern async/await API. PuppeteerSharp is a better fit when porting existing Puppeteer scripts from Node.js and you want to keep the API delta small.

Selenium

Selenium drives real browsers and predates both Playwright and PuppeteerSharp. Its NuGet package is Selenium.WebDriver, and it requires a matching browser driver binary (ChromeDriver for Chrome, geckodriver for Firefox) that you manage separately. The driver version must correspond to the installed browser version, which breaks when the browser auto-updates.

dotnet add package Selenium.WebDriver
dotnet add package Selenium.WebDriver.ChromeDriver

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Support.UI;

var options = new ChromeOptions();
options.AddArgument("--headless");

using var driver = new ChromeDriver(options);
driver.Navigate().GoToUrl("https://quotes.toscrape.com/js/");

var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
wait.Until(d => d.FindElements(By.CssSelector("div.quote")).Count > 0);

foreach (var quote in driver.FindElements(By.CssSelector(".quote .text")))
    Console.WriteLine(quote.Text);

driver.Quit();

WebDriverWait polls until the condition returns true or the timeout expires. FindElements with a CSS selector returns all matching elements. Use FindElement (singular) when you expect exactly one.

Selenium remains the right choice when cross-browser coverage (Chrome, Firefox, Edge, Safari) or existing Selenium test infrastructure is the deciding factor. For scraping alone, Playwright for .NET covers the same functionality without the manual driver management.

Saving scraped data

The .NET standard library covers JSON without extra packages. For JSON, System.Text.Json serializes the list directly:

using System.Text.Json;

var json = JsonSerializer.Serialize(books, new JsonSerializerOptions { WriteIndented = true });
await File.WriteAllTextAsync("books.json", json);

For CSV, CsvHelper is the standard NuGet package for anything beyond a trivial two-column file:

dotnet add package CsvHelper

using CsvHelper;
using System.Globalization;

await using var writer = new StreamWriter("books.csv");
await using var csv = new CsvWriter(writer, CultureInfo.InvariantCulture);
await csv.WriteRecordsAsync(books);

CsvHelper infers column headers from property or field names and handles quoting and escaping automatically. For a plain record type like (string Title, string Price), it writes the header row on its own.

Using the HasData Web Scraping API

The HasData Web Scraping API handles page fetching, JavaScript rendering, and proxy routing server-side. You send a URL and receive rendered HTML or structured data without managing any browser process in your own environment.

After signing up at HasData to get an API key, the call uses HttpClient with the built-in System.Net.Http.Json helpers:

using System.Net.Http;
using System.Net.Http.Json;
using System.Text.Json;

var client = new HttpClient();
client.DefaultRequestHeaders.Add("x-api-key", "YOUR-API-KEY");

var payload = new
{
    url = "https://example.com/",
    extractRules = new
    {
        Title = "h1",
        Description = "p"
    }
};

var response = await client.PostAsJsonAsync("https://api.hasdata.com/scrape/web", payload);
var json = await response.Content.ReadFromJsonAsync<JsonElement>();

var extractedData = json.GetProperty("extractedData");
Console.WriteLine($"Title: {extractedData.GetProperty("Title")}");
Console.WriteLine($"Description: {extractedData.GetProperty("Description")}");

extractRules takes CSS selectors as values. Append @href to extract an attribute instead of text content: "Link": "a @href". The response key is extractedData.

For a product store with multiple fields, the payload changes to match the page structure:

var payload = new
{
    url = "https://demo.opencart.com/",
    extractRules = new
    {
        Title       = "h4",
        Link        = "h4 > a @href",
        Description = "p",
        OldPrice    = "span.price-old",
        NewPrice    = "span.price-new"
    }
};

The outputFormat field controls what the API returns. Set it to "html" to get the rendered HTML string, "markdown" to get the page as Markdown (useful for feeding into LLM pipelines), or omit it to get structured JSON from extractRules. The response key is always extractedData regardless of which format you use with extraction rules.

The API is the practical alternative to Playwright for .NET when the page requires JavaScript rendering but you want to avoid running a Chromium process inside your deployment environment.

FAQ

Which C# library should I start with for web scraping?

Start with HttpClient and HtmlAgilityPack. HttpClient is part of .NET itself and fetches the HTML. HtmlAgilityPack installs via a single NuGet package and parses it with XPath. If you prefer CSS selectors, replace HtmlAgilityPack with AngleSharp.

When does HttpClient fail and Playwright become necessary?

HttpClient fetches exactly what the server sends before any JavaScript runs. Open view-source: on the target URL. If the data you need appears there, HttpClient handles it. If the data is absent from the source but visible in the rendered browser, JavaScript is inserting it after load and Playwright for .NET is the right tool.

What happened to PhantomJS in C#?

PhantomJS development stopped in 2018. PhantomJSDriver was removed from Selenium 4, released in 2021. Code using new PhantomJSDriver() does not compile against current Selenium packages. Use Playwright for .NET or PuppeteerSharp for JavaScript-rendered pages.

When is Selenium worth using over Playwright?

Selenium is the right choice when you need cross-browser coverage across Chrome, Firefox, Edge, and Safari from a single test suite, or when the team already has Selenium-based infrastructure and switching costs outweigh the benefits. For scraping alone, Playwright covers the same scenarios with a simpler API and without the manual ChromeDriver binary that needs to match the installed browser version.

Can I run a C# scraper on Linux or in a Docker container?

Yes. .NET 8 runs on Linux natively. For headless browser scraping with Playwright for .NET, add the --with-deps flag to the install command: playwright.ps1 install --with-deps chromium. This installs the system dependencies (fonts, shared libraries) that Chromium needs on a minimal Linux image. Docker images based on mcr.microsoft.com/dotnet/runtime:8.0 work with Playwright after running the install step during the image build.