How AI Web Scraping Works and When It Beats a Parser

Businesses and researchers pull data off the web to price against competitors and to watch a market move. Web scraping, the process of extracting data from websites, has been a common practice for some time.

Recently, the use of AI in various industries has been gaining traction, and it has already improved and optimized certain activities. This article covers where AI helps in scraping and in the processing that follows.

Traditional Web Scraping Techniques

We have already devoted many articles to scraping, from descriptions of what it is to scraping guides in various programming languages. Our blog contains articles about scraping in Python, NodeJS, C#, and other programming languages and even how to scrape websites using Google Sheets. 

If you want to learn more about web scraping learn how to write scraping scripts for any website, be it Zillow, Amazon, Google Maps, or anything else, we suggest you check out the detailed tutorials on our blog.

How Can Artificial Intelligence Be Used in Web Scraping

 At first, programs and tools based on artificial intelligence were only used by large companies. This was because of the significant amount of web data that had to be processed - even though it came with a high price. But with advances in AI and machine learning technology, these AI tools have become more accessible for everyone to use.

To better understand the advantages of using AI, let’s look at which areas they can benefit and how they can be used there.

But these are only some of the areas and uses of AI. So let’s talk more about where and how AI can be beneficial.

AI Applications in Web Scraping for Various Areas

It’s hard to think of any area where AI can’t be used. Wherever there is data, the use of such models can be beneficial.

Although, there are some areas in which they are most commonly used:

1. E-commerce and Retail.
2. Healthcare and Medicine.
3. Finance and Banking.
4. Manufacturing and Supply Chain.
5. Marketing and Advertising.
6. Transportation and Logistics.
7. Education and E-Learning.
8. Data Mining.
9. Natural Language Processing (NLP).
10. Cybersecurity.

One of the advantages of AI is that it can be trained on any data using machine learning. That is, there are no areas where data would be inappropriate. Anyone can make a narrowly focused model when discussing scientific development or their business.

Ways to Integrate AI in Different Workflows

If we look closer at how Artificial Intelligence (AI) can be used, it becomes clear that AI is great for data processing and analysis. For example, in social media like LinkedIn, AI can help generate leads and analyze competitors. In finance, AI can detect recent changes and compile summary analytics of the market.

But when it comes to e-commerce businesses specifically, these benefits are even more significant. Using AI, you can monitor what your competitors are doing, learn from their successes, apply their best practices to your business operations, and more.

With development or maintenance tasks like finding reliable proxy servers for each website, collecting relevant links, or aiding in web scraping activities, there’s no question that AI has something valuable to offer here too.

AI-Powered Data Scraping in Action

Since high-quality AI models appeared relatively recently, most users want to learn how to use them or know about their features. So, let’s look at how exactly you can use AI models for scraping and data processing, what rules to follow, and what limitations you will face.

However, before we move on to examples of using AI for scraping and processing data, let’s look at how to build prompts so that the AI model understands you. To start, you need to remember that the quality of the result depends entirely on you and how you formulated the task.

It is also worth specifying that you need language AI models like ChatGPT (OpenAI) for scraping and data collection. Based on our own experience, observe the following rules to get a good result:

1. Let the model know that you will be specifying additional parameters. The AI model must be set up before you use it. You can say, “Now I will specify conditions that must be met for further data processing.
2. Next, set the parameters. The AI model doesn’t know how you want it to act. If you want it to act in a way that is not “default,” specify this in the prompt. For example, if you don’t want to use complex vocabulary in the response, you could specify something like: “Avoid using complex vocabulary” or “Use natural language.”
3. Specify the style or person from whom you would like to receive a response. For example, “act as a developer” or “use the official style.”
4. If you want to compose some small text that will use your writing style, give an example of your text to the AI and ask them to remember the style and use it in the future: “Here’s an example of my text. I want you to use the same style next”.

Now that you know how to build prompts for an AI model, let’s look at options for what and how they can be used in scraping and data extraction.

Data Scraping and Processing

And the first thing you can use the language AI model for is processing and parsing text. You can solve almost any word-processing task with an AI language model. For example, you can ask the AI to put the keywords you want in your text or to remove stop words. The biggest plus, however, is that the language model doesn’t care if you’re serving plain text or text that uses hypertext markup language or something else so that you can use it as a web scraping tool.

When models took four thousand tokens, this was manual work. You opened DevTools, copied the outerHTML of the container, pasted it into a chat window and asked for JSON back. That only ever worked as a demonstration, because the answer came back unvalidated and nothing about it repeated without a person in the loop.

The automated version starts one step earlier, at the fetch. Fetch the page with a scraping request, reduce the markup to the block that holds the records, send that with a schema describing the fields you want, and validate what comes back before it reaches your database. The tools built for this do all four for you. The step that defeats most hand-rolled versions is the first one, because fetching at any volume means proxies, rendering, and retries before a model ever sees the markup.

Data Analysis

AI handles text well, as the sentiment and translation cases above show.

You can give it a set of data, which will help you analyze and extract information from it. It runs the statistics and points at the patterns, and it drafts the chart code if you ask for one.

You can gain valuable insight into a specific product or service segment using AI. You can determine which products are the most popular, identify the top-performing sellers and detect which products and sellers have lower sales. By understanding this data in advance, you can make informed decisions about improving your business.

Also, AI can help you quickly analyze customer feedback, making it easier to identify what they like and dislike. When you use AI for web scraping, you can gain insights into your customers’ experiences that may have been overlooked. You’ll be able to find patterns in their responses and learn how best to improve your services or products for maximum satisfaction.

Text Classification

The language model can classify text into predefined categories or labels. This can be useful for sentiment analysis, topic classification, spam detection, or intent recognition.

Named Entity Recognition (NER)

AI can be used to identify and extract essential pieces of information from texts. This technology is helpful for businesses that want to analyze their competitors’ pages and researchers looking for critical data from different sources. Named entities such as people’s names, organizational names, locations, or dates can all be identified and extracted with the help of AI web scraping techniques.

Sentiment Analysis

AI can be used to determine the sentiment behind the text, such as customer reviews or social media posts. This is much more difficult to accomplish manually than simply analyzing ratings on a marketplace product page. Language models benefit this task, providing insights into whether an opinion expressed in words is positive, negative, or neutral.

Language Translation

Machine translation used to be frustrating, with words translated one for one, context ignored, and meaning sometimes skewed. Although automatic translators have improved recently, AI could offer a more reliable solution.

AI models learn from previously collected data in a specific language, remembering patterns and standard phrases to interpret the meaning beyond individual words. Therefore, translations don’t consider only literal definitions but also consider how they fit within their broader context for better accuracy.

Text Generation

AI is mighty when it comes to generating text based on data. For example, suppose you have a service or product to offer. In that case, web scraping can help you collect leads and information from their websites and then provide AI with that data to automate the generation of tailored letters for each potential lead. You can create highly personalized offers that will be of interest and benefit based on what they need - all without ever having to write it by hand.

Scraping Code Generation

Using ChatGPT for scraping can be limited, but using AI to create scrapers or crawlers can help you overcome those limitations. You may still need help with other problems, such as solving captchas, using proxies, and avoiding blocking or JavaScript rendering. Fortunately, different use cases, such as web scraping API, can make your tasks easier and provide reliable results.

For example, we can use this prompt:

Write a scraper on Python to get data and save it to a table from the website quotes.toscrape.com with this structure:

<body>…</body>

The result will be the following scraper, which, as our tests show, we can immediately save and use:

import requests
from bs4 import BeautifulSoup
import pandas as pd

# Send a GET request to the website
url = 'http://quotes.toscrape.com'
response = requests.get(url)

# Create a BeautifulSoup object
soup = BeautifulSoup(response.text, 'html.parser')

# Find all the quote elements
quote_elements = soup.find_all('div', class_='quote')

# Initialize lists to store the extracted data
quotes = []
authors = []
tags = []

# Extract data from each quote element
for quote in quote_elements:
    # Extract the quote text
    quote_text = quote.find('span', class_='text').text
    quotes.append(quote_text)

    # Extract the author name
    author_name = quote.find('small', class_='author').text
    authors.append(author_name)

    # Extract the tags
    tag_elements = quote.find_all('a', class_='tag')
    tag_list = [tag.text for tag in tag_elements]
    tags.append(tag_list)

# Create a DataFrame from the extracted data
data = {'Quote': quotes, 'Author': authors, 'Tags': tags}
df = pd.DataFrame(data)

# Save the DataFrame to a CSV file
df.to_csv('quotes.csv', index=False)

Let’s do the same prompt, only we use NodeJS instead of Python:

const axios = require('axios');
const cheerio = require('cheerio');
const fs = require('fs');
const ObjectsToCsv = require('objects-to-csv');

// Send a GET request to the website
const url = 'http://quotes.toscrape.com';
axios.get(url)
  .then(response => {
    // Load the HTML response into Cheerio
    const $ = cheerio.load(response.data);

    // Find all the quote elements
    const quoteElements = $('.quote');

    // Initialize arrays to store the extracted data
    const quotes = [];
    const authors = [];
    const tags = [];

    // Extract data from each quote element
    quoteElements.each((index, element) => {
      const quoteElement = $(element);

      // Extract the quote text
      const quoteText = quoteElement.find('.text').text().trim();
      quotes.push(quoteText);

      // Extract the author name
      const authorName = quoteElement.find('.author').text().trim();
      authors.push(authorName);

      // Extract the tags
      const tagElements = quoteElement.find('.tag');
      const tagList = tagElements.toArray().map(tag => $(tag).text().trim());
      tags.push(tagList);
    });

    // Create an array of objects with the extracted data
    const data = quotes.map((quote, index) => {
      return {
        Quote: quote,
        Author: authors[index],
        Tags: tags[index]
      };
    });

    // Convert the data array to CSV format
    const csv = new ObjectsToCsv(data);

    // Save the CSV file
    csv.toDisk('quotes.csv', { append: false })
      .then(() => {
        console.log('Data successfully saved to quotes.csv');
      })
      .catch(err => {
        console.error('Error saving data to quotes.csv:', err);
      });
  })
  .catch(error => {
    console.error('Error retrieving data:', error);
  });

And for the last example, we will use the R programming language:

library(rvest)
library(writexl)

# Specify the URL of the website
url <- "http://quotes.toscrape.com"

# Send a GET request to the website and read the HTML content
page <- read_html(url)

# Extract the quote elements using CSS selectors
quote_elements <- page %>% html_nodes(".quote")

# Initialize empty vectors to store the extracted data
quotes <- character()
authors <- character()
tags <- list()

# Extract data from each quote element
for (quote_element in quote_elements) {
  # Extract the quote text
  quote <- quote_element %>% html_node(".text") %>% html_text() %>% trimws()
  quotes <- c(quotes, quote)

  # Extract the author name
  author <- quote_element %>% html_node(".author") %>% html_text() %>% trimws()
  authors <- c(authors, author)

  # Extract the tags
  tag_nodes <- quote_element %>% html_nodes(".tag")
  tag_list <- tag_nodes %>% html_text() %>% trimws()
  tags <- c(tags, list(tag_list))
}

# Create a data frame with the extracted data
data <- data.frame(Quote = quotes, Author = authors, Tags = tags, stringsAsFactors = FALSE)

# Specify the output file path
output_file <- "quotes.xlsx"

# Save the data frame to an Excel file
write_xlsx(data, path = output_file)

# Print a success message
cat("Data successfully saved to", output_file, "\n")

As we can see, we can use any programming language, and all we have to do is to set up and prepare the environment correctly.

Text Summarization

AI is also a great tool to help you sift through long texts and summarize them quickly. For instance, if you gather a list of potential leads but are not sure which ones might be interested in your offer or are the best fit for your business, AI can help by condensing large amounts of content into smaller summaries that capture the core points and essential details.

Question Answering

You can not only use ready-made language models. With the help of machine learning, you can create your own tailored to your datasets. For instance, if you’re running a business and want to lighten the workload of technical support staff, artificial intelligence might be just what you need.

Feed the model real user questions along with the answers your support staff gave. Once enough of those pairs are in, it answers most product questions on its own, and the ones it can’t answer go to a person.

Data-Driven Decision Making

When faced with complex decisions, you can discuss your problem with AI, and it reads the data you have and suggests what to do next.

However, this does not mean you must rely entirely on the AI to make decisions. It can only help and suggest options, but you must decide.

Model or Parser, and What Each One Costs

Three quantities set the bill, and only one of them belongs to the page. Input tokens are what you send, which the table above shows is mostly a property of your reduction. Output tokens are the records that come back, priced higher per token than input by every current provider, though on a page of markup they are the smaller half. Attempts are the multiplier, because a retry after a malformed answer pays full price for the input all over again.

That arithmetic separates model-based extraction from a selector-based scraper. A parser that knows where the price lives costs nothing per extraction once written, and costs developer time whenever the layout changes. A model costs tokens on every run, and survives markup churn inside the block it reads. The symmetry is not perfect. The reduction step above needs a selector for that block, so a model-based pipeline breaks the same way a parser does when the container itself moves, and only tolerates the churn inside it.

For a large crawl, run the parser and put a model behind it for the pages where it fails. “Fails” needs a definition in code rather than in judgement, and the two that work are a required field coming back empty and a schema validation error.

One caveat about that validation, because it is easy to over-trust. Checking the response against a schema proves its shape and not its truth, and a hallucinated price passes a number field as easily as a real one. The cheap guard is asserting that each extracted string appears verbatim in the source you sent, which costs nothing and catches invention.

Challenges of Using AI

Web scraping can present some challenges, mainly if you use artificial intelligence (AI) to do the work. A model costs a call per page and can invent a value that a schema check waves through, so it doesn’t fit every scraping task.

Context Limits

Model-based extraction runs inside a context window, and raw HTML fills one faster than page weight suggests. Measured with the o200k_base tokenizer, real markup runs 2.9 to 5.3 characters per token, so a hundred thousand tokens is only about 300 to 380 KB of HTML. A Wikipedia article measured 235 KB and 71,483 tokens. An MDN reference page measured 248 KB and 69,892.

Those numbers are why the reduction step is not optional, and the order of the reductions matters more than most write-ups admit.

Stripping scripts and styles saved between 0.6% and 61%, depending entirely on how much of the page is inline JavaScript. Reducing to the text of the block that holds the records saved 87% to 96% every time. The first reduction is the one people reach for, the second is the one that pays, and a pipeline that only strips tags leaves an order of magnitude on the table.

Context windows have grown by roughly two orders of magnitude since this technique appeared, from four thousand tokens to hundreds of thousands. That widened the margin rather than removing the step. A raw multi-megabyte page still does not fit in most windows, and what makes it admissible is the reduction.

Work with External Resources

AI web scraping takes two steps where a link suggests one. You fetch the page yourself and hand the model the part of it that matters, because the model knows only what is in the prompt.

AI can be Wrong

AI can generate answers based on the training data, but unlike humans, AI cannot think critically. This means that even if its answer is wrong, AI may still be very confident in its response.

Conclusion and Takeaways

Our findings show that using AI can be great for handling data. Such models do an excellent job of analyzing and assisting in processing. However, AI can only be used as an assistant tool that also needs human control.

A model reads unstructured markup without a selector for every field, which is what makes it useful on pages that change under you.

The limit is cost per page rather than capacity, it can make wrong decisions when the source data is disputable, and it needs the page handed to it.