Web Scraping with Go: Complete Guide 2026

Go goroutines start at around 2 KB each, so running 100 concurrent requests adds almost no overhead. The same workload in Python requires asyncio or a thread pool. Colly runs page visits concurrently, with parallelism and delay controlled by a single LimitRule. GoQuery handles HTML parsing. For JavaScript-rendered pages, chromedp drives Chrome directly.

Getting Started with Go

Before scraping data step-by-step and exploring various libraries, let’s prepare and set up our environment. First and foremost, let’s install Git so we can directly fetch Go libraries from GitHub. Download the required version from the official website and install it. If you are a beginner and haven’t used Git before, we recommend keeping the default settings unchanged.

Now it’s time actually to install Go. To do this, simply visit the official Go website, download the installation file, and follow the instructions provided during installation.

To be sure that you have successfully installed Go, you can use the command “go version”:

C:\Scripts>go version

go version go1.20.5 windows/amd64

You can use any text editor to write code, but it’s better to use specialized tools for convenience and syntax highlighting. We will use Visual Studio Code.

Inspecting the Target Website

Before scraping web pages, it’s important to analyze the target website. This is necessary to find exactly where the information we need is located. We should know which tags and classes contain the required elements. For example, we can search for data inside <div> tags with a specific class or use other specific selectors to precisely locate the desired information on the web page. As an example, we will use two websites - “example” and “store example”.

Example Page

To analyze the webpage “example.com,” we must study its structure and content. By examining the page’s HTML code, we can identify the tags and classes that contain the necessary information. To do this, go to the page and open the DevTools (press F12 or right-click on the screen and select “Inspect”).

As we can see, the page title is located within the h1 tag, and the rest of the text is stored within the p tags. We can now use CSS selectors or XPATH to extract the desired information.

Example Store

This website has much more data and a structure closer to reality. Each item has a div element with the class “col,” and inside this div, you can find the following information:

1. The image is inside a div tag with the class “image” within a nested “a” tag with the “href” attribute.
2. The product name is in an “h4” tag. The link to the product page is also inside it in a nested “a” tag with the “href” attribute.
3. The product description is in a “p” tag.
4. The price is in a div tag with the class “price.” It also has nested tags:
5. The original price is in a “span” tag with the class “price-old.”
6. The discounted price is in a “span” tag with the class “price-new.”
7. The tax information is in a “span” tag with the class “price-tax.”

Now that we know the structure of both websites we will be scraping, we can select the libraries.

Get Data with HasData API

The HasData Web Scraping API handles page fetching, JavaScript rendering, and proxy routing server-side. You send a URL and get back the rendered HTML or structured data.

We will use the net/http package to make requests. It is part of the Go standard library, so no installation is needed.

You will also need an API key, which you can get after signing up at HasData.

Using Example

Let’s get the data for example.com. Declare the imports:

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "strings"
)

Build the request with the URL and extraction rules:

func main() {
    apiURL := "https://api.hasdata.com/scrape/web"

    payload := strings.NewReader(`{
        "url": "https://example.com/",
        "extractRules": {
            "Title": "h1",
            "Description": "p"
        }
    }`)

    client := &http.Client{}
    req, err := http.NewRequest("POST", apiURL, payload)
    if err != nil {
        fmt.Println(err)
        return
    }

    req.Header.Add("x-api-key", "YOUR-API-KEY")
    req.Header.Add("Content-Type", "application/json")

    res, err := client.Do(req)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer res.Body.Close()

    body, err := io.ReadAll(res.Body)
    if err != nil {
        fmt.Println(err)
        return
    }

    var response map[string]interface{}
    if err := json.Unmarshal(body, &response); err != nil {
        fmt.Println(err)
        return
    }

    extractedData := response["extractedData"].(map[string]interface{})
    fmt.Println("Title:", extractedData["Title"])
    fmt.Println("Description:", extractedData["Description"])
}

For the product store, replace the payload with the store URL and its selectors:

    payload := strings.NewReader(`{
        "url": "https://demo.opencart.com/",
        "extractRules": {
            "Title":       "h4",
            "Link":        "h4 > a @href",
            "Description": "p",
            "Old":         "span.price-old",
            "New":         "span.price-new",
            "Tax":         "span.price-tax",
            "Image":       ".image > a @href"
        }
    }`)

And read the corresponding fields from extractedData:

    fmt.Println("Titles:", extractedData["Title"])
    fmt.Println("Links:", extractedData["Link"])
    fmt.Println("Old Prices:", extractedData["Old"])
    fmt.Println("New Prices:", extractedData["New"])

The extractRules map uses CSS selectors as values. Append @href to a selector to extract an attribute instead of text content, as in "h4 > a @href".

Easy Parsing with GoQuery

GoQuery is a popular Go library that provides a convenient way to parse HTML or XML documents and extract data using CSS selectors. It is based on jQuery, a widely used JavaScript library for manipulating and browsing HTML documents.

Install GoQuery Library

To use the GoQuery library, install it with:

go get github.com/PuerkitoBio/goquery

After that, import it in your Go file alongside the standard net/http package.

Using Example

Include the necessary libraries and declare the main function:

package main

import (
    "fmt"
    "log"
    "net/http"

    "github.com/PuerkitoBio/goquery"
)

func main() { }

Make a request and load the response into a GoQuery document:

    url := "https://example.com"

    resp, err := http.Get(url)
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()

    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil {
        log.Fatal(err)
    }

Extract the page title using a CSS selector:

    doc.Find("h1").Each(func(_ int, s *goquery.Selection) {
        fmt.Println(s.Text())
    })

Output:

Example Domain

For the product store, replace the URL and selectors:

    url := "https://demo.opencart.com/"

    doc.Find("div.col").Each(func(_ int, s *goquery.Selection) {
        image := s.Find(".image a").AttrOr("href", "")
        productName := s.Find("h4 a").Text()
        productLink := s.Find("h4 a").AttrOr("href", "")
        description := s.Find("p").Text()
        oldPrice := s.Find(".price-old").Text()
        newPrice := s.Find(".price-new").Text()
        tax := s.Find(".price-tax").Text()

        fmt.Println("Image:", image)
        fmt.Println("Product Name:", productName)
        fmt.Println("Product Link:", productLink)
        fmt.Println("Description:", description)
        fmt.Println("Old Price:", oldPrice)
        fmt.Println("New Price:", newPrice)
        fmt.Println("Tax:", tax)
        fmt.Println()
    })

GoQuery works only with static HTML. If the target page renders content via JavaScript, the response body will not contain it and doc.Find will return empty results.

Fast Parallel Crawling with Colly

Colly is an HTTP-based scraping framework. It fetches pages over HTTP, parses the HTML response, and triggers callbacks for matching elements. There is no browser, no JavaScript engine, and no headless Chrome. Colly does not execute JavaScript and does not render dynamic content.

What Colly does exceptionally well is concurrent crawling. Each page visit runs on a goroutine, and the built-in rate limiter controls how many run in parallel. On a static site with thousands of pages, this makes Colly significantly faster than any single-threaded scraper.

Install Colly

go get -u github.com/gocolly/colly/v2

This pulls in the v2 API, which is the current stable version.

Using Example

Create a collector and set a parallelism limit:

package main

import (
    "fmt"
    "log"
    "time"

    "github.com/gocolly/colly/v2"
)

func main() {
    c := colly.NewCollector(
        colly.Async(true),
    )

    c.Limit(&colly.LimitRule{
        DomainGlob:  "*",
        Parallelism: 8,
        Delay:       100 * time.Millisecond,
    })

    c.OnHTML("div.col", func(e *colly.HTMLElement) {
        image := e.ChildAttr("div.image a", "href")
        productName := e.ChildText("h4 a")
        productLink := e.ChildAttr("h4 a", "href")
        oldPrice := e.ChildText(".price-old")
        newPrice := e.ChildText(".price-new")

        fmt.Println("Product:", productName)
        fmt.Println("Link:", productLink)
        fmt.Println("Image:", image)
        fmt.Println("Old Price:", oldPrice)
        fmt.Println("New Price:", newPrice)
        fmt.Println()
    })

    err := c.Visit("https://demo.opencart.com/")
    if err != nil {
        log.Fatal(err)
    }

    c.Wait()
}

Async(true) makes Visit non-blocking. c.Wait() at the end holds the program until all goroutines finish. The LimitRule caps concurrent requests at 8 and adds a 100ms delay between them, which keeps the scraper well below thresholds that trigger rate limiting.

For JavaScript-rendered pages, use chromedp instead.

JavaScript-Rendered Pages with chromedp

chromedp drives Chrome over the DevTools Protocol. It is the standard Go answer for pages that render content via JavaScript after the initial load.

Install chromedp

go get github.com/chromedp/chromedp

Chrome or Chromium must be installed on the system.

Using Example

The JavaScript version of quotes.toscrape.com at /js/ injects its quote list after page load. A plain HTTP request returns an empty container.

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/chromedp/chromedp"
)

func main() {
    ctx, cancel := chromedp.NewContext(context.Background())
    defer cancel()

    var quotes []string
    err := chromedp.Run(ctx,
        chromedp.Navigate("https://quotes.toscrape.com/js/"),
        chromedp.WaitVisible(`div.quote`, chromedp.ByQuery),
        chromedp.Evaluate(
            `[...document.querySelectorAll('.quote .text')].map(el => el.innerText)`,
            &quotes,
        ),
    )
    if err != nil {
        log.Fatal(err)
    }

    for _, q := range quotes {
        fmt.Println(q)
    }
}

WaitVisible blocks until the selector appears in the DOM. Evaluate runs a JavaScript expression in the page context and unmarshals the result into the Go variable (a slice of strings in this case).

Data Storage and Processing

The Colly example from the previous section collects products into a [][]string slice. Writing that slice to CSV takes about ten lines of standard library code.

package main

import (
    "encoding/csv"
    "fmt"
    "log"
    "os"
    "time"

    "github.com/gocolly/colly/v2"
)

func main() {
    c := colly.NewCollector(colly.Async(true))
    c.Limit(&colly.LimitRule{DomainGlob: "*", Parallelism: 8, Delay: 100 * time.Millisecond})

    var data [][]string

    c.OnHTML("div.col", func(e *colly.HTMLElement) {
        image := e.ChildAttr("div.image a", "href")
        productName := e.ChildText("h4 a")
        productLink := e.ChildAttr("h4 a", "href")
        description := e.ChildText("p")
        oldPrice := e.ChildText(".price-old")
        newPrice := e.ChildText(".price-new")
        tax := e.ChildText(".price-tax")

        data = append(data, []string{image, productName, productLink, description, oldPrice, newPrice, tax})
    })

    if err := c.Visit("https://demo.opencart.com/"); err != nil {
        log.Fatal(err)
    }
    c.Wait()

    file, err := os.Create("data.csv")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    writer := csv.NewWriter(file)
    writer.Comma = ';'
    defer writer.Flush()

    writer.Write([]string{"Image", "Product Name", "Product Link", "Description", "Old Price", "New Price", "Tax"})
    writer.WriteAll(data)

    fmt.Printf("Saved %d rows\n", len(data))
}

After running the script, a data.csv file appears in the same directory. Real datasets often need a cleaning pass to remove empty rows, strip whitespace, or normalize prices to a single currency format before analysis.

Best Practices

Set rate limits. colly.LimitRule controls both concurrency and delay. Start with Parallelism: 8 and Delay: 100ms, then adjust based on server response times.

Handle errors at the collector level. c.OnError catches all request failures in one place instead of checking each visit individually.

Use context.WithTimeout with chromedp. Headless Chrome sessions can hang on unresponsive pages. A timeout context kills the session and returns an error instead of blocking the program.

Respect robots.txt. Colly checks it by default (colly.NewCollector() respects robots.txt unless disabled with colly.IgnoreRobotsTxt()).

Go’s concurrency model means scraping bottlenecks usually show up at the network or target server before the Go process itself becomes the constraint.