Web Scraping Best Practices: 13 Tips and Tricks for Scraping Any Website

There are a few reasons most scrapers break, such as the wrong data source, fragile selectors, no retry logic, or too many requests in a short time, and more.

At SerpApi, we extract from more than 100 search engines and any websites every day, mostly without browser automation. These are 13 web scraping best practices we learned while building our APIs:

• Part 1 (tips 1 - 9) help you find and extract the data.
• Part 2 (tips 10 - 13) keep your scraper running without getting blocked or breaking easily.

New to web scraping? Start with our beginner's guide to building a web scraper

🐍 All code examples in this blog is using Python.

Web scraping techniques to find and extract the data

1. Browser Dev Tools

Dev tools is possibly one of the very first thing that needs to be checked before writing actual code.

I wrote quite a lot of code before looking at DevTools, with the realization that instead of parsing data, I can make a simple HTTP request, either GET or POST to the website API/server and get a JSON response with all the necessary data that is already structured.

Under DevTools, you can find the Elements, Network, and Source/Application tabs.

The following sections will go from easiest to hardest methods, with tools that can be used to get the job done.

2. CSS Selectors or/and XPath

CSS Selectors

CSS selectors are patterns used to select and match the element(s) you want to style or extract from an HTML page.

Normally, for building websites CSS selectors is use to help style the website, while in web scraping those selectors are used to match certain HTML nodes (elements) from which we want data to be extracted.

We've covered types of selectors with actual code examples in the web scraping with CSS selectors blog post and which selector works best for a particular task, and how to test them in the browser.

There're a few often-used types of selectors:

Some website change their HTML regularly, so pick the selectors wisely. Here are some tips:

• IDs, data-* attributes (e.g. [data-testid="price"]), and aria-* attributes rarely change.
• Generated class names like .css-1x2y3z or .X5PpBb can change on every website build.
• Long position-based paths like div > div:nth-child(3) > span break as soon as a new element is added.

XPath

XPath is a query language that is used to navigate the DOM, and it can be more flexible than CSS selectors.

It's mostly helpful when an HTML element doesn't have a selector name, or has a very weird location in messy HTML. XPath uses expressions to create a query:

//div[@class='content']//text()[not(ancestor::div/@class[contains(., 'code')])][normalize-space()]

You might be interested in: XPath Cheat Sheet

SelectorGadget Chrome Extension

SelectorGadget extension allows you to quickly grab CSS selector(s) by clicking on the desired element in your browser. It returns CSS selector(s).

Example of selecting certain HTML elements and the returned CSS selector(s):

We've shown how SelectorGadget works in a separate web scraping with CSS selectors blog post.

3. Scraping from Meta Elements

Selectors are not the only place where valuable data could be extracted. Some of the data can be directly in the HTML <meta> elements that load no matter if the page is static or dynamic.

For example, on most blogs you can extract the title, description, and image from Open Graph <meta> elements:

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"
}

html = requests.get("https://serpapi.com/blog/how-to-build-a-web-scraper-a-beginners-guide/", headers=headers, timeout=30).text
soup = BeautifulSoup(html, "html.parser")

open_graph = {tag["property"]: tag.get("content") for tag in soup.select('meta[property^="og:"]')}

print(open_graph.get("og:title"), open_graph.get("og:description"), open_graph.get("og:image"), sep="\n")

📌 Select <meta> elements by attribute, not by index like soup.select("meta")[13]. The index changes as soon as the website adds one more <meta> element.

4. Scraping From Tables

Tables can be scraped with parsel or bs4, but pandas simplifies this task a lot with its read_html() method:

pip install pandas lxml

A basic example of extracting table data from Wikipedia:

import pandas as pd

url = "https://en.wikipedia.org/wiki/History_of_Python"

# Wikipedia asks scripts to send a descriptive User-Agent with contact info
headers = {"User-Agent": "PythonVersionsScraper/1.0 (you@example.com)"}

table = pd.read_html(url, match="Release date", storage_options=headers)[0]  # [0] = first matched table
print(table.head())

table.to_csv("python_versions.csv", index=False)

Outputs:

   Version Latest  micro version   Release date End of full support End of security fixes
0      0.9              0.9.9[2]  1991-02-20[2]    1993-07-29[a][2]      1993-07-29[a][2]
1      1.0              1.0.4[2]  1994-01-26[2]    1994-07-14[a][2]      1994-07-14[a][2]
2      1.1              1.1.1[2]  1994-10-11[2]    1994-11-10[a][2]      1994-11-10[a][2]
3      1.2                   NaN  1995-04-13[2]         Unsupported           Unsupported
4      1.3                   NaN  1995-10-13[2]         Unsupported           Unsupported

📌 storage_options sends your headers with the request (pandas 2.1+). Without it, Wikipedia blocks the default Python User-Agent. If you already have the HTML as a string, wrap it in io.StringIO() first. pandas 3 no longer accepts raw HTML strings.

You'll still need some data cleaning to make this data usable 🙂

Have a look at the gotchas that could happen with read_html():

5. Scraping with Regular Expressions

Scraping with regular expressions in Python is possible with the re module.

Why scrape data with regular expressions in the first place?

1. If the HTML structure is very messy.
2. If there are no CSS selectors and XPath doesn't work either.
3. If the data you want is directly in the text string.
4. Similar reasons to those mentioned above.

There're a few main methods that could be used:

import re

dummy_text = '''
Contrary to popular belief, Lorem Ipsum is not simply random text. 
It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. 
Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure
Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature,
discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of 
"de Finibus Bonorum et Malorum" (The Extremes of Good and Evil) by Cicero, written in 45 BC. 
This book is a treatise on the theory of ethics, very popular during the Renaissance.
The first line of Lorem Ipsum, "Lorem ipsum dolor sit amet..", 
comes from a line in section 1.10.32.
'''

dates = re.findall(r'\d{1}\.\d{2}\.\d{2}', dummy_text) # https://regex101.com/r/VKYiA9/1
years_bc = re.findall(r'\d+\s?\bBC\b', dummy_text)     # https://regex101.com/r/ApypoB/1

print(dates)
print(years_bc)

# ['1.10.32', '1.10.33', '1.10.32']
# ['45 BC', '45 BC']

Here's a visualization of what is being matched by the regular expressions above:

The above regular expressions read like this:

6. Python Web Scraping Tools

In this section, we'll go over the most popular Python web scraping tools that can extract data from static and dynamic websites.

Pick the tool based on the website:

• Parsing libraries for static pages,
• Browser automation for pages that need JavaScript or interaction
• A framework for large crawls.

Parsel

Parsel extracts data from HTML, JSON and XML documents with XPath and CSS selectors support, and could be combined with regular expressions. It uses the lxml parser under the hood.

The great thing I like about parsel (apart from XPath support) is that it returns None if certain data is not present, so there's no need for a lot of ugly try/except blocks.

pip install parsel

A practical example using parsel. The following script scrapes quotes, authors, and tags from quotes.toscrape.com, a website made for scraping practice, and follows the "Next" button for up to 3 pages:

import json

import requests
from parsel import Selector

BASE_URL = "https://quotes.toscrape.com"
max_pages = 3

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"
}

quotes = []
next_page = "/"

for _ in range(max_pages):
    html = requests.get(f"{BASE_URL}{next_page}", headers=headers, timeout=30).text
    selector = Selector(text=html)

    for quote in selector.css("div.quote"):
        quotes.append({
            "text": quote.css("span.text::text").get(),
            "author": quote.css("small.author::text").get(),
            "tags": quote.css("div.tags a.tag::text").getall(),
        })

    next_page = selector.css("li.next a::attr(href)").get()  # None on the last page

    if not next_page:
        break

print(json.dumps(quotes[:2], indent=2, ensure_ascii=False))

with open("quotes.json", "w", encoding="utf-8") as f:
    json.dump(quotes, f, indent=2, ensure_ascii=False)

print(f"Saved {len(quotes)} quotes to quotes.json")

BeautifulSoup

BeautifulSoup also extracts data from HTML/XML documents and can be combined with the lxml parser and regular expressions.

Unlike parsel, it doesn't support XPath, lacks a getall() method, and needs a lot of try/except to handle missing data. However, it can create new HTML nodes with methods like wrap(). That's handy when data isn't properly structured, e.g. a table without a <table> element that you can wrap and then parse with pandas.

pip install beautifulsoup4 lxml

The same quotes scraper using BeautifulSoup. Notice that select_one() returns None when nothing matches, so the "Next" link needs an if check before reading its href:

import json

import requests
from bs4 import BeautifulSoup

BASE_URL = "https://quotes.toscrape.com"
max_pages = 3

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"
}

quotes = []
next_page = "/"

for _ in range(max_pages):
    html = requests.get(f"{BASE_URL}{next_page}", headers=headers, timeout=30).text
    soup = BeautifulSoup(html, "lxml")

    for quote in soup.select("div.quote"):
        quotes.append({
            "text": quote.select_one("span.text").get_text(),
            "author": quote.select_one("small.author").get_text(),
            "tags": [tag.get_text() for tag in quote.select("div.tags a.tag")],
        })

    next_link = soup.select_one("li.next a")  # None on the last page

    if not next_link:
        break

    next_page = next_link["href"]

print(json.dumps(quotes[:2], indent=2, ensure_ascii=False))

with open("quotes.json", "w", encoding="utf-8") as f:
    json.dump(quotes, f, indent=2, ensure_ascii=False)

print(f"Saved {len(quotes)} quotes to quotes.json")

Playwright

Browser automation is handy when you need to perform some sort of interaction with the website, for example, scrolling, clicks and similar things.

Such things could be done without browser automation; this is how we tend to do it at SerpApi. It can be very complicated, but on the flip side, the reward is much faster data extraction.

playwright is a modern alternative to selenium. It can perform pretty much all interactions as the user would do i.e clicks, scrolls and many more. To keep it fast, wait for the element you need instead of a fixed time.sleep(), and block resources you don't need:

pip install playwright
playwright install chromium

Practical example of website interaction using playwright and parsel to extract the data.

import json
from playwright.sync_api import sync_playwright

BLOCKED = {"image", "media", "font"}

with sync_playwright() as playwright:
    browser = playwright.chromium.launch(headless=True)
    page = browser.new_page()

    # skip resources we don't need
    page.route(
        "**/*",
        lambda route: route.abort()
        if route.request.resource_type in BLOCKED
        else route.continue_(),
    )

    page.goto("https://quotes.toscrape.com/js/")
    page.wait_for_selector(".quote")

    quotes = [
        {
            "text": quote.locator(".text").inner_text(),
            "author": quote.locator(".author").inner_text(),
        }
        for quote in page.locator(".quote").all()
    ]

    browser.close()

print(json.dumps(quotes, indent=2, ensure_ascii=False))

Selenium

selenium is very similar to playwright but a little older. Since Selenium 4.6, Selenium Manager downloads the browser driver for you, so you no longer need webdriver-manager.

See our Selenium web scraping in Python guide, or scraping dynamic websites with Puppeteer if you use JavaScript.

Scrapy

scrapy is a high-level web scraping framework designed to scrape data at scale and can be used to create a whole ETL pipeline.

However, you have to keep in mind that it's bulky and could be quite confusing, and while it provides a lot of things for you, most of those things you may not need.

pip install scrapy

The same quotes scraper as a Scrapy spider. Scrapy handles the requests, delays, robots.txt, and saving the results for you:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    max_pages = 3

    custom_settings = {
        "ROBOTSTXT_OBEY": True,  # respect robots.txt
        "DOWNLOAD_DELAY": 1,     # wait 1 second between requests
    }

    def parse(self, response, page=1):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }

        next_page = response.css("li.next a::attr(href)").get()

        if next_page and page < self.max_pages:
            yield response.follow(next_page, callback=self.parse, cb_kwargs={"page": page + 1})

Save it as quotes_spider.py, then run it and save the results to a file:

scrapy runspider quotes_spider.py -o quotes.json

To learn more, start with the official Scrapy tutorial.

7. XHR Requests

XHR request allows to talk to the server by making a request and getting data back in response. It's one of the first things that you can check before writing actual code. Those requests can be used to get data dircetly from website's "source" without the need to use parsing libraries/frameworks.

To find a certain XHR request you need:

1. Open browser dev tools (F12)
2. Network
3. Fetch/XHR
4. Refresh page as data may come on page update.
5. Click through every request and see if there's any data you want.

If you find the request with the data you want, you can preview the data (example from the Poocoin.app):

How to extract data from an XHR request

When making XHR request, we need to pass URL parameters so that server can understand and "reply" to us.

Here's a simple illustration of that process:

To find those headers and URL query parameters, we need to go to a certain URL and look at Headers and Payload tabs and make sure we see what request method is used (GET, POST, etc).

We can do it like so:

1. Copy URL as CURL (Bash) and use it with online CURL runner or tools such as Insomnia.
2. Copy request URL under headers tab.

From Insomnia (URL copied from the XHR->Headers tab).

📌Keep in mind that some of the passed URL parameters need to be scraped and passed to the URL beforehand (before making request to the server/api). URL can have some sort of a unique token or something different and can't be worked without it.

If the response is successful and you want to make an exact request in the script, those parameters could be automatically generated with tools such as Insomnia (or other alternatives) where you can test different types of requests with different parameters and headers.

Simple example but the same approach will be on other websites with or without passing URL parameters and headers:

import requests

# https://requests.readthedocs.io/en/latest/user/quickstart/#json-response-content
html = requests.get('https://api.chucknorris.io/jokes/random').json()
print(html['value'])

# Once, due to an engine stall of his F-22 Raptor during a Dessert Storm sorte', Chuck Norris had to ejaculate over the Red Sea.

8. Page Source

This is the next thing that could be checked after Dev Tools -> XHR. It's about looking at page source and trying to find the data there, that is either hidden in the rendered HTML or can't be scraped with selectors because it's being rendered by JavaScript.

One of the ways to find if there's the data you want is in the inline JSON or not:

1. select and copy any type of data you want to extract (title, name, etc.)
2. open page source CTRL + U
3. find the data CTRL + F, if some of the occurrences will be inside <script> elements then congratulation, you found inline JSON or something similar 🙂

Here's a visual example of how inline JSON could look like (from the TripAdvisor):

When we found that there's a data we want locates in the inline JSON, there're a few ways to extract it:

• using regular expression to extract parts of the inline JSON.
• (if needed) once again using regular expression(s) to extract portion of the data: links, dates, emails, etc.
• using regular expression to parse inline JSON and covert it to usable JSON that could be used as a Python dict.

Extract inline JSON data with Regex

The following example is from our Google Arts & Culture - Artists blog post where we select() all <script> elements and then extract other data from matched element(s) using regular expressions:

from bs4 import BeautifulSoup
import requests, json, re, lxml

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36'
}

params = {
    'tab': 'pop'
}

html = requests.get(f'https://artsandculture.google.com/category/artist', params=params, headers=headers, timeout=30)
soup = BeautifulSoup(html.text, 'lxml') 

# 👇👇👇 data extraction
all_script_tags = soup.select('script')

# https://regex101.com/r/JQXUBm/1
portion_of_script_tags = re.search(r'("stella\.common\.cobject",".*)\[\["A",\["stella\.pr","PrefixedAssets', str(all_script_tags)).group(1)

# https://regex101.com/r/c9m9B0/1
authors = re.findall(r'"stella\.common\.cobject","(.*?)","\d+', str(portion_of_script_tags))

print(json.dumps(authors, indent=2, ensure_ascii=False))

Extract JSON-LD and Next.js data without Regex

If the JSON sits in its own <script> element, you don't need regex at all. Two common ones:

• <script type="application/ld+json">: structured data (schema.org) that websites add for SEO, often with names, prices, ratings, dates, and authors.
• <script id="__NEXT_DATA__">: full page data on Next.js websites that use the Pages Router. Newer Next.js websites built with the App Router don't have it. Their data sits in self.__next_f.push(...) calls inside <script> elements, which you'll need to extract with regex.

from bs4 import BeautifulSoup
import requests, json

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36"
}

html = requests.get("https://serpapi.com/blog/how-to-build-a-web-scraper-a-beginners-guide/", headers=headers, timeout=30)
soup = BeautifulSoup(html.text, "lxml")

# JSON-LD structured data
for script in soup.select('script[type="application/ld+json"]'):
    data = json.loads(script.string)
    print(data.get("@type"), data.get("headline"), data.get("datePublished"))

# Next.js page data (if the website uses Next.js)
next_data = soup.select_one("script#__NEXT_DATA__")

if next_data:
    page_data = json.loads(next_data.string)
    print(page_data["props"].keys())

9. Reverse Engineering & Debugging

The great examples of reverse engineering on our blog:

• scrape Walmart Search for a specific store
• reverse engineering Google Finance charts.

Make sure to check them both as here we're not going to duplicate the same information.

Sources tab

One of the approaches, when something complex needs to be extracted, could be done under the Source tab.

It could be done by degubbing website JS source code from certain files with debugger breakpoints (Dev tools -> sources -> debugger) by trying to trace what is going on in the code and how can we intercept/create by ourselves data and use it the parser.

We've also done a SerpApi Podcast about scraping dynamic websites which is essentially the topic of using Source tab.

In this video Ilya told about his approach to investigating and reproducing Google Maps data URL parameter which defines data about certain Google Maps place. This parameter hold data about place id, latitude and longitude.

Application tab

A similar thing could be done in the Dev tools -> Application tab where we see, for example, cookies data and either intercept it on reverse engineer it by understanding how this cookie was built.

Ilya, one of the engineers at SerpApi has written in more detail about reverse engineering Location cookies from Walmart and his approach for such task.

Keep your scraper running without getting blocked or breaking

Finding the data is half of the job. The other half is making sure the website keeps letting you in, and that you notice when something changes.

10. Send Headers and Reuse a Session

The default requests User-Agent (python-requests/x.y) is an easy signal for bot detection. Send the headers a real browser sends (copy them from Dev Tools → Network → Headers), and reuse one Session so connections and cookies carry over between requests:

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
})

response = session.get("https://news.ycombinator.com/", timeout=30)
print(response.status_code)

📌 Headers alone won't get you past advanced bot protection, but missing headers will get you blocked on websites that would otherwise let you through.

11. Add Timeouts, Retries, and Backoff

Servers return 429 Too Many Requests or 503 errors. Without a timeout, a request can hang forever. Without retries, one bad response stops the whole script:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=3,                                     # max retries per request
    backoff_factor=1,                            # waits 1s, 2s, 4s between retries
    status_forcelist=[429, 500, 502, 503, 504],  # retry on these status codes
    respect_retry_after_header=True,             # honor the server's Retry-After header
)

session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))

response = session.get("https://news.ycombinator.com/", timeout=(5, 30))  # (connect, read)
response.raise_for_status()

print(response.text)

12. Slow Down and Respect robots.txt

Sending requests as fast as possible is the quickest way to get your IP blocked. Check robots.txt, honor the crawl delay if there is one, and add a random pause between requests:

import random
import time
from urllib.robotparser import RobotFileParser

import requests

USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"
BASE_URL = "https://news.ycombinator.com/"
max_pages = 3

robots = RobotFileParser(f"{BASE_URL}robots.txt")
robots.read()
delay = robots.crawl_delay(USER_AGENT)

session = requests.Session()
session.headers.update({"User-Agent": USER_AGENT})

for page in range(1, max_pages + 1):
    url = f"{BASE_URL}?p={page}"

    if not robots.can_fetch(USER_AGENT, url):
        print(f"Skipping {url}: disallowed by robots.txt")
        continue

    response = session.get(url, timeout=30)
    print(page, response.status_code, len(response.text))

    time.sleep(delay or random.uniform(1, 3))

Also:

• Cache pages while developing. Save the HTML to a file or use requests-cache, so you don't hit the website on every test run.
• Check the website's terms of service and be careful with personal data. Privacy laws like GDPR apply even when the data is public.

13. Validate Your Data and Know When to Use an API

Websites change their HTML without any warning. The worst case isn't a crash. It's a scraper that keeps running and quietly saves empty data. A simple check after each run tells you when something changed:

import json

REQUIRED_FIELDS = ("title", "link")


def validate(items, max_missing_ratio=0.2):
    if not items:
        raise ValueError("Parsed 0 items. The page layout may have changed.")

    incomplete = [item for item in items if any(not item.get(field) for field in REQUIRED_FIELDS)]

    if len(incomplete) / len(items) > max_missing_ratio:
        raise ValueError(f"{len(incomplete)} of {len(items)} items are missing required fields.")

    return items


items = [
    {"title": "Show HN: A tiny web scraper", "link": "https://example.com/1"},
    {"title": "Ask HN: Favorite parsing library?", "link": "https://example.com/2"},
]

with open("results.json", "w", encoding="utf-8") as f:
    json.dump(validate(items), f, indent=2, ensure_ascii=False)

print(f"Saved {len(items)} items to results.json")

All of the above adds up. Proxies, CAPTCHAs, JavaScript rendering, and layout changes turn a small script into something you have to maintain all the time. Before building a scraper, check if the website has an official API.

For search engines, this is exactly what we do at SerpApi. Here's how to get Google search results as JSON with the SerpApi Python library:

pip install serpapi

import json
import os

import serpapi

client = serpapi.Client(api_key=os.environ["SERPAPI_API_KEY"])

results = client.search({
    "engine": "google",
    "q": "web scraping best practices",
    "gl": "us",
    "hl": "en",
})

for result in results.get("organic_results", []):
    print(result["position"], result["title"], result["link"])

with open("google_results.json", "w", encoding="utf-8") as f:
    json.dump(dict(results), f, indent=2, ensure_ascii=False)

The same client works for Google Maps, Amazon, YouTube, and other APIs. You can start with the free plan.

FAQ

Is web scraping legal?

Scraping publicly available data is generally lower risk than scraping data behind a login, but it depends on your country, the website's terms of service, and what you do with the data. Personal data and copyrighted content have extra rules. This isn't legal advice, so check with a lawyer for your specific use case.

What's the best language for web scraping?

Python is the most common choice because of libraries like requests, BeautifulSoup, parsel, and playwright. JavaScript (Node.js) and Ruby work well too. See our beginner's guide for examples in all three.

How do I scrape a website without getting blocked?

Send browser-like headers, add retries with backoff, and slow down your requests (tips 10–12). Only use browser automation when you need it. For heavily protected websites, an API that manages proxies and CAPTCHAs is usually more reliable.

Can AI do web scraping?

AI models are good at turning messy HTML into structured data, but they don't replace the request part. See using AI to parse HTML into structured data.