Best Ways to Find All URLs on Any Website

Find all URLs on a domain by using a site crawler, parsing the sitemap file, exploring robots.txt, applying search engine queries with operators, or writing a custom scraping script. Each method provides different levels of control and depth, depending on your technical skills and data access needs. 

How to Find All URLs on a Domain

There are five main ways to get all the links from a site:

1. Website Crawlers. Use a ready-made crawler that scans the whole site and lists all the links it finds.
2. Sitemaps & robots.txt. If the site has a sitemap.xml, you can pull links directly from there.
3. SEO Tools. Many SEO tools come with built-in features to collect site links.
4. Search Engine Queries. If you only need links that match a specific pattern, you can scrape them from search engine results.
5. Write Your Own Script. This case is ideal for developers or technical users who require custom scraping logic, precise control over link extraction, or integration with specific tools.

We’ll go through each method step by step. 

Method 1, a Website Crawler

The most reliable way to collect all URLs from a website is to use a crawler. It doesn’t rely on a sitemap and ignores most site protections since a specialized tool does the crawling.

For this example, we’ll use HasData’s web crawler, which is available after you sign up. You can find it in your dashboard under no-code scrapers.

To run it, fill in the main fields:

1. Limit. Maximum number of pages to crawl (0 = no limit).
2. URLs. Starting URLs.
3. maxDepth. How many link levels to follow from each starting URL.

Optionally, you can set RegEx patterns to include or skip certain paths. You can also choose the output format.

Once launched, the task will start crawling the site (or multiple sites), and you just need to wait for it to finish. You can track progress on the right side of the screen. After the crawl finishes, you can download the output file.

If you prefer using a script, Method 5 runs the same crawl through the API.

Method 2, Sitemaps and robots.txt

If the website has a sitemap that lists all its URLs, you can parse it. But keep in mind, not every site has a sitemap.

Using robots.txt

Unlike the sitemap, robots.txt is always in the root folder and named exactly that: 

https://vuejs.org/robots.txt

Robots.txt holds data for bots visiting a website, including where the sitemap is and what it’s called.

However, many developers don’t add the sitemap path there, which makes it much harder to find.

Locating Sitemaps

When robots.txt declares no sitemap, it is worth looking in the usual place by hand. In most cases, the sitemap is located at the domain’s root. But webmasters often split it by topic or compress it to save bandwidth. Based on the name and type, sitemaps usually fall into these categories:

• Root sitemap. By convention, it’s usually here:
  https://your-domain.com/sitemap.xml
• Index sitemap. On larger sites, you’ll often see sitemap_index.xml or sitemap-index.xml, which point to multiple smaller sitemaps.
• Specialized sitemaps. Big e-commerce or news sites may have:
  sitemap-products.xml, sitemap-news.xml, sitemap-images.xml, etc.
• Compressed sitemaps. It’s common to use .gz for compression:
  sitemap.xml.gz
• Custom names. Technically, any name or extension is allowed. But the path must be listed in robots.txt or submitted through Webmaster Tools.

In general, start by checking the default sitemap URL.

Parsing Sitemap XML

With the sitemap in hand, the next step is parsing it for the URLs. A sitemap usually looks like this:

You need to extract everything inside the <loc>...</loc> tags. You can use regular expressions or convert XML to CSV, whatever works best for you. I find it easiest to write a small Python script that scrapes the sitemap, extracts the links, and saves them to a TXT file.

The whole job is a request, one XML parse, and a file write. vuejs.org serves its sitemap to any client, so the script below runs as printed:

import requests
import xml.etree.ElementTree as ET

sitemap_url = "https://vuejs.org/sitemap.xml"
output_file = "sitemap_links.txt"

response = requests.get(sitemap_url)
response.raise_for_status()

# sitemap tags live in a namespace, so the query has to name it
root = ET.fromstring(response.content)
namespace = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
links = [loc.text for loc in root.findall('.//ns:loc', namespace)]

with open(output_file, 'w', encoding='utf-8') as f:
    for link in links:
        f.write(link + '\n')

The run against vuejs.org wrote 110 URLs:

A server that refuses a plain request for the sitemap will usually answer one that arrives through a residential exit, which is what a web scraping API supplies.

We’ll use HasData’s web scraping API as an example. You’ll need an API key, which you can get after signing up.

The API takes the target in a JSON payload and returns JSON, with the fetched page in the content field. One detail is worth a line: jsRendering stays off, so the sitemap arrives as raw XML instead of a browser’s rendering of it, and the parse stays the same four lines:

import requests
import json
import xml.etree.ElementTree as ET

api_key = "YOUR-API-KEY"
sitemap_url = "https://vuejs.org/sitemap.xml"

url = "https://api.hasdata.com/scrape/web"

payload = json.dumps({
  "url": sitemap_url,
  "proxyType": "datacenter",
  "proxyCountry": "US",
  "jsRendering": False
})
headers = {
  'Content-Type': 'application/json',
  'x-api-key': api_key
}

response = requests.post(url, headers=headers, data=payload)
response.raise_for_status()

root = ET.fromstring(response.json().get("content"))
namespace = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
links = [loc.text for loc in root.findall('.//ns:loc', namespace)]

with open("sitemap_links.txt", 'w', encoding='utf-8') as f:
    for link in links:
        f.write(link + '\n')

The result is the same, but this method works even on sites that block direct requests.

Method 3, SEO Tools

Many SEO tools can collect links from a website. But they come with limitations. For example, the free version of Screaming Frog only allows up to 500 URLs.

To collect links with Screaming Frog, download it from the official website, install it, and launch it. Enter your domain and start the crawl:

Wait for the crawl to finish. If needed, export the data to a file:

Remember that if the site has anti-scraping protection, the tool might not reach all pages. In that case, try adjusting the crawl speed, user agents, and other headers. These settings are accessible in the configuration panel. 

This method returns more than URLs. You also get the status code of each page, so it’s useful if you want to check for things like broken links on your site. 

Listing the Links in DevTools

For a single page, the browser already has the tool. Open the page, press F12, and paste this into the Console tab:

copy([...document.querySelectorAll('a[href]')]
  .map(a => a.href.split('#')[0])
  .filter((u, i, all) => u.startsWith(location.origin) && all.indexOf(u) === i))

copy() puts the result on the clipboard. What lands there is every unique same-origin URL the page links to, fragment anchors stripped. On the Vue.js home page that comes to 29 links. The snippet reads one page instead of crawling, so it shows the structure around the page you are on, and it costs nothing and installs nothing.

Free Online URL Extractors

A few hosted tools list a site’s URLs with no code at all. Their free tiers cover a look around a small site, so the useful question about each one is what it returns and where it stops.

• Firecrawl has a map endpoint that returns a site’s URLs in one call. It takes a limit, the sitemap can be included, skipped, or used alone, and a map call costs one credit. Small volumes run without an API key at reduced rate limits.
• Olostep folds URL mapping into a general scraping API, where Map sits next to scrape, crawl, search, and answer operations.
• Simplescraper is a point-and-click scraper that also works through lists. It crawls URLs in bulk in its cloud, or scrapes a listing page and then each result that page links to.
• SiteGPT collects a site’s pages for a different goal. You give it a site or sitemap link and it fetches every page it finds as chatbot training material, so the page list is a byproduct of training rather than the product.

Method 4, Search-Engine Queries

If the previous examples didn’t work for you, or if you only want to collect specific pages that match certain criteria, you can try using search engine results instead.

Search operators narrow the SERP to the links you want, and then you scrape the results. Here are the main operators you can use:

You can mix and match these operators to filter for the exact pages you need.

Now let’s write a script to extract those links from the SERP. We’ll use HasData’s SERP API for that. You’ll need an API key, which you can find in your dashboard after signing up on our site.

The script builds one request, reads organicResults out of the response, and saves the links:

import requests
import json


api_key = "YOUR-API-KEY"


query = "site:hasdata.com inurl:blog"
location = "Austin,Texas,United States"
device_type = "desktop"
num_results = 100


base_url = "https://api.hasdata.com/scrape/google/serp"


params = {
    "q": query,
    "location": location,
    "deviceType": device_type,
    "num": num_results
}


headers = {
    "Content-Type": "application/json",
    "x-api-key": api_key
}


response = requests.get(base_url, headers=headers, params=params)
response.raise_for_status()


data = response.json().get("organicResults", [])
urls = [entry["link"] for entry in data if "link" in entry]


output_file = "serp.json"
with open(output_file, "w", encoding="utf-8") as f:
    json.dump(urls, f, indent=2)

One call returns one page of results, about ten organic rows, so a site with more pages than that takes the same request repeated with a start offset of 10, 20, and so on until the results run out.

This will give you a JSON file with links to indexed pages that match your search criteria.

This method won’t give you every page on the site, only the ones that are indexed and match your query, but that’s the point. It’s a filtered, targeted approach.

Method 5, Scraping URLs with a Custom Script

The Crawler API does the crawling on our side. You hand it a start URL and a page limit, it follows the links it finds, and the finished job returns one JSON row per page it reached. The script below starts a job, waits for it to finish, and saves every URL:

import time
import requests
import json

API_KEY = "YOUR-API-KEY"

headers = {
    "x-api-key": API_KEY,
    "Content-Type": "application/json"
}

def start_crawl():
    payload = {
        "limit": 50,
        "urls": ["https://vuejs.org"],
        "outputFormat": ["json"]
    }
    response = requests.post(
        "https://api.hasdata.com/scrapers/crawler/jobs",
        json=payload,
        headers=headers
    )
    response.raise_for_status()
    job_id = response.json().get("id")
    print(f"Started job with ID: {job_id}")
    return job_id

def poll_job(job_id):
    while True:
        response = requests.get(
            f"https://api.hasdata.com/scrapers/jobs/{job_id}",
            headers=headers
        )
        data = response.json()
        status = data.get("status")
        print(f"Job status: {status}")
        if status in ["finished", "failed", "cancelled"]:
            return data
        time.sleep(10)

def download_and_extract_urls(json_url, job_id):
    output_path = f"results_{job_id}.json"
    response = requests.get(json_url)
    response.raise_for_status()
    raw_data = response.json()
    urls = [entry["url"] for entry in raw_data if "url" in entry]
    with open(output_path, "w", encoding="utf-8") as f:
        json.dump(urls, f, indent=2)
    print(f"Saved {len(urls)} URLs to {output_path}")

job_id = start_crawl()
result = poll_job(job_id)
download_and_extract_urls(result["data"]["json"], job_id)

start_crawl sends the job and returns its ID. limit caps the crawled pages, so a crawl of a site of unknown size costs a known number of credits at most. The crawl runs on HasData’s side, and nothing here needs a proxy or a browser.

poll_job asks for the status every ten seconds until the job reports finished, failed, or cancelled. A finished job carries download links for the results in data, keyed by format.

download_and_extract_urls reads the JSON result. Each row describes one crawled page:

{
  "url": "https://vuejs.org/about/faq.html",
  "statusCode": 200,
  "depth": 1,
  "title": "Frequently Asked Questions | Vue.js",
  "parentUrl": "https://vuejs.org"
}

depth and parentUrl say where the crawl found each page, and statusCode separates pages that exist from pages the site merely links to. A bare list of URLs carries neither.

The same endpoint also takes aiExtractRules for pulling fields out of each crawled page with an LLM. With those rules in the payload the rows change shape. Each one carries the model’s answer in aiResponse and drops the url field. A crawl that lists URLs therefore runs without rules, and extraction with an LLM runs as its own job.

Which Method Actually Finds the Pages

Which of the five to reach for depends on how the site is built, and the usual advice that a sitemap is the most accurate one holds for only some of them. Nine domains were checked with three of these methods, picked for how they are built rather than for what they sell.

The crawl below follows href attributes in the HTML the server returns, which is what your own crawler does. It renders no JavaScript, and it stops at 300 pages.

Only five of the nine publish a sitemap.xml. On three of the rest, a crawl found between 232 and 295 URLs while the sitemap route found zero, so a script that reads only the sitemap reports that the site has no pages. shop.tesla.com answered 403 to a plain request on the homepage, the sitemap and robots.txt alike, which is a different thing from having nothing to find.

The reverse case is just as sharp. On gohugo.io the sitemap lists 792 URLs and the crawl found 2, because the homepage carries seven href attributes in total and six of them are external sponsor banners. Its navigation is built client side, so a crawl that renders nothing has nothing to follow.

The Row Worth Copying Down

vuejs.org is where both methods are wrong on their own:

Two methods, 246 URLs between them, and eight in common. Whichever one you pick, you miss more than half. That is the argument for running two and merging, rather than for choosing the best.

What robots.txt Adds

Almost nothing, on this sample. Where a site declares a sitemap in robots.txt the declared file was the same one at /sitemap.xml every time, so the robots route returned an identical set. It is still worth reading, because a site can declare a sitemap that is not at the conventional path, and four of these nine declared no sitemap at all.

Crawl Noise

A crawl collects things a person would not call a page. On the two Shopify storefronts, 5% and 9% of what the crawl returned carried a query string or a paging segment. On the seven others it was zero. Faceted navigation is where a crawler drowns, and it is exactly where a sitemap is most likely to exist.

Conclusion

Five methods, from ready-made crawlers through sitemaps and SEO tools to search operators such as site:, inurl:, intitle: and intext:.

One of the simplest ways to get started is with our Website Crawler, which lets you quickly list all URLs on any site without technical hassle. For more control, our Web Scraping API returns the HTML and leaves the parsing to you.

Each method here comes with a script that runs. Which one fits depends on how the site is built, and the measurement above is the shortest way to tell.