Immobilienscout24 is Germany's leading real estate portal, boasting a vast database of properties. Similar to popular US platforms like Redfin and Zillow, Immobilienscout24 offers a user-friendly interface for searching and comparing apartments, houses, commercial spaces, and land. Its extensive filtering options make it easy to find the perfect match.
Unlike its American counterparts, Immobilienscout24 caters specifically to the German real estate market, known for its stability. Access to this data is invaluable to analysts, investors, and real estate agencies, making Immobilienscout24.de an attractive target for data scraping.
Zillow Scraper is the tool for real estate agents, investors, and market researchers. Its easy-to-use interface requires no coding knowledge and allows users to…
Redfin Scraper is an innovative tool that provides real estate agents, investors, and data analysts with a convenient way of collecting and organizing property-related…
Introduction to Immobilienscout24.de Scraping
As we've mentioned, Immobilienscout24.de is an excellent resource for finding properties in Germany. Unfortunately, the site is primarily in German. While there are developer sections with documentation for the Immobilienscout24.de official API, we'll discuss that in more detail later. Before we dive into data extraction from Immobilienscout24.de, let's familiarize ourselves with key terms commonly used in listings. While the site's interface is quite user-friendly, understanding these terms will significantly enhance our ability to accurately and efficiently extract the information we need. Key terms include:
- Miete – Rent. This indicates the monthly cost of renting a property.
- Kaufpreis – Purchase price. This specifies the total cost of buying a property.
- Nebenkosten – Additional costs. These are extra fees added to the base rent or purchase price (e.g., utilities).
- Wohnfläche – Living area. This refers to the total square footage of all living spaces in the property.
- Zimmer – Rooms. The number of rooms is a crucial factor when searching for a property.
By understanding these terms, you can fine-tune your data extraction process and obtain the most relevant information from listings.
Preparing for Web Scraping
Before we build a data extraction tool, we must first identify the specific data we aim to collect. This includes pinpointing its exact location on the target website and understanding its underlying structure. This initial analysis will guide us in selecting the most suitable data extraction techniques.
To begin, we'll conduct a thorough examination of the website's structure. This will help us identify the page elements that house the desired data. Once we have a clear understanding of the data's organization, we can explore various web scraping methods and choose the most effective approach for our particular use case.
Analyzing Website Structure
Let's head over to the site and see what kind of data we can pull from the fields:
The most crucial components for us will be:
- Filters: These allow us to personalize our search and retrieve precisely the data we need.
- Listings: Each listing on the page provides all the necessary information, from the title and address to details about the agency, cost, amenities, and other parameters.
Let's start with the filters on the page. There are actually many more filtering options available than what's initially displayed at the top. If we navigate to the dedicated filters page, we'll find a much broader range of criteria. These filters allow you to fine-tune your search by property type, location, price range, number of rooms, and additional details like whether pets are allowed, the year of construction, or the type of heating system.
More advanced options include excluding certain property types, filtering by energy efficiency, or even specifying a minimum internet speed. This flexibility makes it easy to extract precisely the data you need for your analysis.
To personalize the query according to the necessary filters, it is necessary to change the query address. By default, the search string looks like this:
https://www.immobilienscout24.de/Suche/de/Let's prepare a basic script that will allow us to dynamically form the necessary link, and most of the basic filters will be placed as variables. As a programming language, we will choose Python, as it is one of the most convenient for developing scraping scripts.
To begin, we will create a variable in which we will store the base URL for searching for real estate:
base_url = "https://www.immobilienscout24.de/Suche/de/"Next, we'll introduce variables to represent different search filters. We also will include comments to explain their purpose:
state = "baden-wuerttemberg" # State in Germany
city = "stuttgart" # City
district_1 = "degerloch" # Main district
district_2 = "degerloch" # Sub-district (optional)
property_type = "wohnung" # Property type (e.g., 'wohnung' for apartment)
features = "mit-balkon" # Specific features (e.g., 'mit-balkon' for with balcony)
action = "mieten" # Action (e.g., 'mieten' for rent, 'kaufen' for buy)
price_type = "rentpermonth" # Pricing type (e.g., 'rentpermonth', 'totalprice')
origin = "result_list" # Origin of the search (usually 'result_list' when the search comes from the result list)At the end, we will collect these variables into a common link:
constructed_url = (
f"{base_url}{state}/{city}/{district_1}/{district_2}/"
f"{property_type}-{features}-{action}?"
f"pricetype={price_type}&enteredFrom={origin}"
)This approach will enable us to customize our queries more effectively in the future by simply modifying variables. It's a more convenient method that reduces manual work. However, if necessary, you can still manually input the final data collection URL. If the search results page doesn't provide sufficient information, you can always scrape the required data from the specific listing page. In general, the individual property page will contain the following information:
As you can see, there is significantly more information available on the listing page. Even without the data exclusive to registered users with "Plus" accounts, the volume of details is substantially higher.
In future examples, we will explore scraping both general search results pages and individual property listings.
Choosing a Web Scraping Method
Before we dive into specific examples of data scraping, let's outline the various methods available to extract the information you need. The optimal approach hinges on your technical skills, the specific task at hand, and the amount of time you're willing to invest in data collection. For scraping ImmobilienScout24.de, you can consider these options:
- Manual Data Extraction. While the least efficient method, it requires no technical skills. It's suitable for small-scale, one-time data gathering from a handful of listings.
- Utilizing ImmoScout24 API. The platform offers a rich set of APIs that allow interaction with its content. Refer to their official developer page for more details. However, using these APIs for data scraping can present certain challenges:
- Leveraging Third-Party APIs. Consider using third-party tools designed for real estate data extraction. This option is often simpler and less restrictive. Moreover, it can help circumvent common scraping obstacles.
- Building a Custom Scraper. This involves developing a scraper from scratch using a programming language of your choice. Be prepared to handle challenges like CAPTCHAs, which ImmoScout24 frequently employs to detect suspicious activity.
In this article, we'll explore how to use HasData's API to scrape real estate data from ImmoScout24 and delve into the creation of a custom Python scraper using libraries like Requests, BeautifulSoup4, and Selenium.
Scraping Property Listings Data
Let's start our guide by exploring methods to scrape immoscout24 data from real estate listings. To automate the process of generating the necessary URLs, we'll leverage the script we discussed earlier.
Each data extraction method has its pros and cons. It's important to note that using libraries like BeautifulSoup and Requests directly is unlikely to yield successful results, as you'll often encounter CAPTCHAs. Therefore, our primary focus will be on web scraping APIs and the Selenium library, which can simulate human behavior.
Using Requests & BeautifulSoup
As we discussed earlier, extracting data using Requests and BeautifulSoup from this particular website is highly unlikely to be successful due to various technical challenges. However, for illustrative purposes, let's explore how this could be attempted by expanding upon our previous script.
First, import the libraries:
import requests
from bs4 import BeautifulSoupAfter creating the link, set the User Agent in request headers and go on it to view the page source or any error messages:
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
}
response = requests.get(constructed_url, headers=headers)
if response.status_code == 200:
soup = BeautifulSoup(response.text, 'html.parser')
print(soup.prettify())
else:
print(f"Failed to retrieve the page. Status code: {response.status_code}")Instead of getting the whole webpage's code, we use CSS selectors to pinpoint and extract only the exact data we need:
soup = BeautifulSoup(response.text, 'html.parser')
properties = soup.select(".grid-item.result-list-entry__data-container")
for property in properties:
title = property.select_one(".result-list-entry__brand-title").text.strip()
price = property.select_one(".result-list-entry__primary-criterion dd").text.strip()
area = property.select("dl.result-list-entry__primary-criterion dd")[1].text.strip()
rooms = property.select("dl.result-list-entry__primary-criterion dd")[2].text.strip()
location = property.select_one(".result-list-entry__address").text.strip()
print(f"Title: {title}")
print(f"Price: {price}")
print(f"Area: {area}")
print(f"Rooms: {rooms}")
print(f"Location: {location}")
print("-" * 50)Unfortunately, attempting to execute this code will result in a 401 error. Inspecting the page's source code will reveal a request to solve a CAPTCHA or even information about blocking. As we initially cautioned, this approach was explored purely for theoretical purposes and to satisfy curiosity about its potential feasibility.
Using Web Scraping API
The second method we'll explore involves using HasData's web scraping API, which helps bypass anti-bot measures like blocking and CAPTCHAs. This API is quite user-friendly, making it suitable for programmers of all skill levels.
Overall, it's very similar to the previous example, except that the request to the website is made through the API. Additionally, you can view and run a ready-made script using Google Colaboratory.
To begin, let's import the necessary libraries:
import request
import json
from bs4 import BeautifulSoupNext, we'll set our HasData API key and the URL of the real estate listings page. You can use the code from the previous example to generate this URL, but we'll set it manually here to avoid repetition:
api_key = "YOUR-API-KEY"
scout_url = "https://www.immobilienscout24.de/Suche/de/baden-wuerttemberg/stuttgart/degerloch/degerloch/wohnung-mit-balkon-mieten?pricetype=rentpermonth&enteredFrom=result_list"You can obtain your HasData API key from your account after signing up on our website.
Set the headers and endpoint for the web scraping API:
api_url = "https://api.hasdata.com/scrape/web"
headers = {
'Content-Type': 'application/json',
'x-api-key': api_key
}Then, let's set the parameters for the request itself. Specifically, we'll use German residential proxies to reduce the risk of being blocked, and we'll enable JavaScript execution on the page:
payload = json.dumps({
"url": scout_url,
"proxyType": "residential",
"proxyCountry": "DE",
"blockResources": False,
"blockAds": False,
"blockUrls": [],
"jsScenario": [],
"screenshot": False,
"jsRendering": True,
"excludeHtml": False,
"extractEmails": False,
"wait": 10
})Now, all that's left is to execute the request and parse the page's HTML using BeautifulSoup:
response = requests.post(api_url, headers=headers, data=payload)
if response.status_code == 200:
response_data = response.json()
html_content = response_data.get("content")
if html_content:
soup = BeautifulSoup(html_content, 'html.parser')
properties = soup.select(".grid-item.result-list-entry__data-container")
data = []
for property in properties:
title = property.select_one(".result-list-entry__brand-title").get_text(strip=True)
price = property.select_one("dl.result-list-entry__primary-criterion dd").get_text(strip=True)
area = property.select("dl.result-list-entry__primary-criterion dd")[1].get_text(strip=True)
rooms = property.select("dl.result-list-entry__primary-criterion dd")[2].get_text(strip=True)
location = property.select_one(".result-list-entry__address").get_text(strip=True)
data.append([title, price, area, rooms, location])
print(f"Title: {title}")
print(f"Price: {price}")
print(f"Area: {area}")
print(f"Rooms: {rooms}")
print(f"Location: {location}")
print("-" * 50)
else:
print("Failed to retrieve HTML content.")
else:
print(f"API request failed: {response.status_code}, error message: {response.text}")As a result, we'll obtain the following data after running the script:
To enhance your analysis of market trends, consider incorporating web scraping into the script. This technique can retrieve additional data, such as real estate agents information or other relevant details. To implement this, simply define new variables with the appropriate CSS selectors for the desired elements.
Using Selenium for Dynamic Content
The final method we'll explore involves using a modified version of the Selenium library and headless browsers for data scraping. Before diving into the script itself, let's examine what happens when we navigate to a real estate search page using Selenium.
For this, we import the necessary libraries, navigate to the website, and introduce a brief delay:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
chrome_options = Options()
driver = webdriver.Chrome(options=chrome_options)
url = "https://www.immobilienscout24.de/Suche/de/baden-wuerttemberg/stuttgart/degerloch/degerloch/wohnung-mit-balkon-mieten?pricetype=rentpermonth&enteredFrom=result_list"
driver.get(url)
time.sleep(10)
driver.quit()During script execution, we encounter a 401 error on Scout24, along with a warning that our activity appears suspicious and robot-like. In case of errors, they provide a request code and ask us to contact support.
Unfortunately, as you can see, the Selenium webdriver is unequivocally identified as a bot and isn't even offered a CAPTCHA. However, there's a solution to this. SeleniumBase, a modified version of Selenium, can help us circumvent such issues.
Primarily used by testers, SeleniumBase is well-suited for conducting tests. Nevertheless, it offers features that allow us to bypass the blocking mechanisms on Scout24. SeleniumBase's UC Mode (Undetected-Chromedriver Mode) enables bots to mimic human behavior, evading anti-bot systems that attempt to block them or trigger CAPTCHAs. However, this might not be sufficient as the website often requires solving a CAPTCHA.
We have several ways:
- Utilize German residential proxies.
- Employ CAPTCHA-solving services.
- Manually solve the CAPTCHA and save cookies, session data, and local storage, to restore them later.
We'll opt for the last approach. First, we need to visit the page, solve the CAPTCHA, and save the data. For this, we'll use a separate script:
from seleniumbase import SB
import pickle
import time
with SB(uc=True) as sb:
sb.open("https://www.immobilienscout24.de/Suche/de/baden-wuerttemberg/stuttgart/degerloch/degerloch/wohnung-mit-balkon-mieten?pricetype=rentpermonth&enteredFrom=result_list")
time.sleep(20)
cookies = sb.get_cookies()
with open('cookies.pkl', 'wb') as file:
pickle.dump(cookies, file)
local_storage = sb.execute_script("return JSON.stringify(window.localStorage);")
with open('local_storage.json', 'w', encoding='utf-8') as file:
file.write(local_storage)
session_storage = sb.execute_script("return JSON.stringify(window.sessionStorage);")
with open('session_storage.json', 'w', encoding='utf-8') as file:
file.write(session_storage)We've introduced a 20-second delay, providing ample time to solve the CAPTCHA. Even if you're not German, you can easily solve it by selecting the image with the most occurrences.
Next, let's create a new script where we import the previously saved data and then parse the required page as in previous examples:
from selenium.webdriver.common.by import By
from seleniumbase import SB
import pickle
import csv
import json
with SB(uc=True) as sb:
sb.open("https://www.immobilienscout24.de")
with open('cookies.pkl', 'rb') as file:
cookies = pickle.load(file)
for cookie in cookies:
sb.add_cookie(cookie)
with open('local_storage.json', 'r', encoding='utf-8') as file:
local_storage = file.read()
sb.execute_script(f"var items = JSON.parse(arguments[0]); for (var key in items) {{ window.localStorage.setItem(key, items[key]); }}", local_storage)
with open('session_storage.json', 'r', encoding='utf-8') as file:
session_storage = file.read()
sb.execute_script(f"var items = JSON.parse(arguments[0]); for (var key in items) {{ window.sessionStorage.setItem(key, items[key]); }}", session_storage)
sb.open("https://www.immobilienscout24.de/Suche/de/baden-wuerttemberg/stuttgart/degerloch/degerloch/wohnung-mit-balkon-mieten?pricetype=rentpermonth&enteredFrom=result_list")
properties = sb.find_elements(By.CSS_SELECTOR, ".grid-item.result-list-entry__data-container")
for property in properties:
title = property.find_element(By.CSS_SELECTOR, ".result-list-entry__brand-title").text.strip()
price = property.find_element(By.CSS_SELECTOR, "dl.result-list-entry__primary-criterion dd").text.strip()
area = property.find_elements(By.CSS_SELECTOR, "dl.result-list-entry__primary-criterion dd")[1].text.strip()
rooms = property.find_elements(By.CSS_SELECTOR, "dl.result-list-entry__primary-criterion dd")[2].text.strip()
location = property.find_element(By.CSS_SELECTOR, ".result-list-entry__address").text.strip()
print(f"Title: {title}")
print(f"Price: {price}")
print(f"Area: {area}")
print(f"Rooms: {rooms}")
print(f"Location: {location}")
print("-" * 50)Since the obtained data aligns perfectly with the previous results from HasData's API, we'll skip a detailed analysis of the script. Notably, this approach effectively addresses most challenges when scraping ImmobillenScout24.de.
For maximum reliability, it's recommended to supplement the script with German proxies and a CAPTCHA-solving service. Despite its effectiveness, this method doesn't guarantee complete protection against blocks.
Automate the collection of information on real estate prices, property characteristics, and more with Zillow Scraper API to make your market analysis faster, more…
With this powerful tool, you can easily scrape accurate data from the Apartments.com website without any coding knowledge or experience. Get valuable insights into…
Scraping Property Data
Scraping real estate property shares many similarities with collecting data from real estate listings. The primary differences lie in the specific elements and links that need to be targeted.
Given that we’ve already covered general challenges in real estate scraping, let’s dive into practical examples using HasData's API and SeleniumBase. This will allow us to bypass the more basic approach involving Requests and BeautifulSoup.
Using Web Scraping API
Let's start by exploring a simpler method using HasData's web scraping API. As before, you'll need a personal API key, which can be obtained from your account after registering on our website.
You can also find, run, and experiment with the ready-made script in Colab Research.
Most of the script that interacts with HasData's API will remain unchanged, except for the page link:
from bs4 import BeautifulSoup
import requests
import json
import csv
api_key = "YOUR-API-KEY"
scout_url = "https://www.immobilienscout24.de/expose/153073358?referrer=RESULT_LIST_LISTING&searchId=cb9e25ee-fcc3-3365-be73-71e453b7abf5&searchUrl=%2Fde%2Fbaden-wuerttemberg%2Fstuttgart%2Fdegerloch%2Fdegerloch%2Fwohnung-mit-balkon-mieten%3Fpricetype%3Drentpermonth&searchType=district&fairPrice=FAIR_OFFER#/"
api_url = "https://api.hasdata.com/scrape/web"
headers = {
'Content-Type': 'application/json',
'x-api-key': api_key
}
payload = json.dumps({
"url": scout_url,
"proxyType": "residential",
"proxyCountry": "DE",
"blockResources": False,
"blockAds": False,
"blockUrls": [],
"jsScenario": [],
"screenshot": False,
"jsRendering": True,
"excludeHtml": False,
"extractEmails": False,
"wait": 10
})
response = requests.post(api_url, headers=headers, data=payload)
if response.status_code == 200:
response_data = response.json()
html_content = response_data.get("content")
if html_content:
soup = BeautifulSoup(html_content, 'html.parser')
# Here will be another code
else:
print("Failed to retrieve HTML content.")
else:
print(f"API request failed: {response.status_code}, error message: {response.text}")However, the selectors for the data we want to extract will need to be modified:
data = {}
title = soup.find('h1', id='expose-title')
data['title'] = title.get_text(strip=True) if title else None
address = soup.find('span', class_='zip-region-and-country')
data['address'] = address.get_text(strip=True) if address else None
cold_rent = soup.find('div', class_='is24qa-kaltmiete-main is24-value font-semibold')
data['cold_rent'] = cold_rent.get_text(strip=True) if cold_rent else None
warm_rent = soup.find('div', class_='is24qa-warmmiete-main is24-value font-semibold')
data['warm_rent'] = warm_rent.get_text(strip=True) if warm_rent else None
area = soup.find('div', class_='is24qa-flaeche-main is24-value font-semibold')
data['area'] = area.get_text(strip=True) if area else None
rooms = soup.find('div', class_='is24qa-zi-main is24-value font-semibold')
data['rooms'] = rooms.get_text(strip=True) if rooms else None
floor = soup.find('dd', class_='is24qa-etage')
data['floor'] = floor.get_text(strip=True) if floor else None
availability = soup.find('dd', class_='is24qa-bezugsfrei-ab')
data['availability'] = availability.get_text(strip=True) if availability else None
for key, value in data.items():
print(f"{key}: {value}")As a result, we'll obtain the following data:
Since there is a lot of data on the page, you can add absolutely any variables and selectors you need to the script, which will allow you to extract all the necessary information about a specific real estate property.
Using Selenium for Dynamic Content
If we use Selenium, or more specifically SeleniumBase, the process remains largely the same. We'll simply need to replace the search URL and update the selectors. The bulk of the script will stay unchanged:
from selenium.webdriver.common.by import By
from seleniumbase import SB
import pickle
import csv
with SB(uc=True) as sb:
sb.open("https://www.immobilienscout24.de")
with open('cookies.pkl', 'rb') as file:
cookies = pickle.load(file)
for cookie in cookies:
sb.add_cookie(cookie)
with open('local_storage.json', 'r', encoding='utf-8') as file:
local_storage = file.read()
sb.execute_script(f"var items = JSON.parse(arguments[0]); for (var key in items) {{ window.localStorage.setItem(key, items[key]); }}", local_storage)
with open('session_storage.json', 'r', encoding='utf-8') as file:
session_storage = file.read()
sb.execute_script(f"var items = JSON.parse(arguments[0]); for (var key in items) {{ window.sessionStorage.setItem(key, items[key]); }}", session_storage)
sb.open("https://www.immobilienscout24.de/expose/153073358?referrer=RESULT_LIST_LISTING&searchId=cb9e25ee-fcc3-3365-be73-71e453b7abf5&searchUrl=%2Fde%2Fbaden-wuerttemberg%2Fstuttgart%2Fdegerloch%2Fdegerloch%2Fwohnung-mit-balkon-mieten%3Fpricetype%3Drentpermonth&searchType=district&fairPrice=FAIR_OFFER#/") We'll only modify the section where we extract the required data:
data = {}
title = sb.get_text("h1#expose-title")
data['title'] = title.strip() if title else None
address = sb.get_text("span.zip-region-and-country")
data['address'] = address.strip() if address else None
cold_rent = sb.get_text("div.is24qa-kaltmiete-main.is24-value.font-semibold")
data['cold_rent'] = cold_rent.strip() if cold_rent else None
warm_rent = sb.get_text("div.is24qa-warmmiete-main.is24-value.font-semibold")
data['warm_rent'] = warm_rent.strip() if warm_rent else None
area = sb.get_text("div.is24qa-flaeche-main.is24-value.font-semibold")
data['area'] = area.strip() if area else None
rooms = sb.get_text("div.is24qa-zi-main.is24-value.font-semibold")
data['rooms'] = rooms.strip() if rooms else None
floor = sb.get_text("dd.is24qa-etage")
data['floor'] = floor.strip() if floor else None
availability = sb.get_text("dd.is24qa-bezugsfrei-ab")
data['availability'] = availability.strip() if availability else None
for key, value in data.items():
print(f"{key}: {value}")However, before running this script, it's essential to execute the previously created script that saves cookies, session data, and local storage. Additionally, we recommend creating a variable or file to store all necessary URLs for scraping. This allows you to iterate through these URLs within a single script.
Storing Scraped Data
In the previous examples, we used simple print statements to display data on the screen. While this is convenient for quick checks, it's not practical for long-term storage or sharing. Let's explore how to save our data to files.
Two common file formats for storing structured data are JSON and CSV. To work with these formats, we'll need to import the corresponding Python libraries: json and csv.
import csv
import jsonA CSV (Comma-Separated Values) file is a plain text file that stores tabular data in a human-readable format. Each line in a CSV file represents a row, and each value within a row is separated by a comma:
with open('property_details.csv', mode='w', newline='', encoding='utf-8') as file:
writer = csv.writer(file)
writer.writerow(data.keys())
writer.writerow(data.values())The with statement ensures that the file is properly closed even if an exception occurs. The csv.writer object provides methods for writing rows to the CSV file.
JSON (JavaScript Object Notation) is a lightweight data-interchange format that is easy for humans to read and write and for machines to parse and generate:
with open('property_details.json', mode='w', encoding='utf-8') as file:
json.dump(data, file, ensure_ascii=False, indent=4)The json.dump() function converts a Python object (in this case, data) into a JSON string and writes it to the specified file. The ensure_ascii=False argument allows non-ASCII characters to be encoded correctly, and indent=4 adds indentation for better readability.
Conclusion and Takeaways
This article delves into the challenges and solutions associated with web scraping ImmobilienScout24.de. We've explored a range of techniques and tools to effectively extract data from this platform, providing practical examples and code snippets.
Our findings indicate that specialized web scraping APIs offer the most robust and convenient approach. By leveraging these APIs, developers can streamline the data extraction process, eliminating the need to handle website blocks, CAPTCHAs, and other common obstacles.
For those seeking more granular control over the scraping process, we've also examined Selenium Base. However, it's important to note that using Selenium Base often requires additional measures, such as proxy servers and CAPTCHA-solving services, to ensure the script's stability and reliability.
