n8n Web Scraping: 7 Steps to Your First Workflow

On this page

Sponsored contribution.

Bots are now responsible for more traffic than humans. The 2026 Imperva Bad Bot Report shows that in 2025 bot traffic accounted for 53% of all web traffic. That share mixes good bots (search crawlers) with bad ones, and sites respond by filtering automated traffic harder. A careless scraper that ignores site rules can stop working within hours. You don't need to write a full app to run a polite, reliable scraper. With n8n web scraping, you just drag blocks onto a canvas and your own server does the work. This guide will walk you through setting up a self-hosted instance, pulling clean fields from a site on a schedule and getting alerts when things go wrong.

Before you start with n8n web scraping

• A VPS with 2 GB of RAM and Docker installed
• A domain or subdomain pointed at the server for HTTPS
• A Google account or a Postgres database for results
• A Telegram or Slack channel for alerts

n8n web scraping tutorial: 7 steps on your own server

Here is the full n8n web scraping flow, from an empty VPS to scheduled tasks with alerts.

Step 1. Deploy n8n in Docker on a VPS

Docker Compose is the easiest way to host your n8n web scraping stack. Save the config below as docker-compose.yml and run docker compose up -d.

services:
  n8n:
    image: docker.n8n.io/n8nio/n8n:2.42.4
    restart: unless-stopped
    ports:
      - "127.0.0.1:5678:5678"
    environment:
      - N8N_HOST=n8n.example.com
      - N8N_PROTOCOL=https
      - WEBHOOK_URL=https://n8n.example.com/
      - GENERIC_TIMEZONE=Europe/Berlin
      - EXECUTIONS_DATA_PRUNE=true
      - EXECUTIONS_DATA_MAX_AGE=168
    volumes:
      - n8n_data:/home/node/.n8n
volumes:
  n8n_data:

Some of these lines will save you trouble later:

• 127.0.0.1 keeps the editor off the open internet. To reach it from outside, put a reverse proxy with SSL in front of it.
• GENERIC_TIMEZONE sets the timezone your schedules run in. Without it, n8n defaults to New York time.
• Pruning keeps the database small. A job that runs every 15 minutes logs about 670 executions a week, and this config deletes anything older than 7 days.
• Pinning the image tag means a surprise update won't break your flows overnight.

If you're on CyberPanel, you can also launch the n8n container from the Docker Manager, and the panel will handle the domain and SSL for you.

Step 2. Pick a source and check robots.txt and site rules

Every n8n web scraping project should start with site.com/robots.txt. Look for Disallow lines that cover your path and for a Crawl-delay value. Then read the terms of service, because they can ban scraping even when robots.txt is silent. A missing file isn't permission: the ToS still apply.

Next, figure out how the page loads:

• Press Ctrl+U. If the text you need is in the raw source, simple HTML parsing is enough.
• If it isn't, open DevTools, go to Network and filter by Fetch/XHR. The page often calls a JSON API that you can query directly.
• Search the source for __NEXT_DATA__ or application/ld+json. Those tags are likely to contain the full product data.
• If none of this works, you'll need a headless browser. For your initial attempt, pick an easier target.

Step 3. Configure the HTTP Request node

Add a Manual Trigger for testing and then connect an HTTP Request node. This node decides whether the target answers at all, so take your time with the settings:

• Response: set the format to Text and save to a field called html.
• Headers: send a descriptive User-Agent that names your scraper and includes a contact email, plus an Accept-Language header.
• Timeout: allow 15–30 seconds (15,000–30,000 ms) so that one stuck URL doesn't bring the whole thing to a halt.
• Retries: in Settings turn on Retry On Fail with 3 attempts and 5,000ms between them.
• Batching: for a long list of URLs, send one item per batch with a 2,000 ms (2-second) interval.

Your connection matters as much as those settings. A scheduled workflow sends the same requests at regular intervals, so it needs a stable, fast route to the target. Many teams add ISP proxies to their workflow infrastructure for this: proxies give you static IP addresses registered to real internet providers and a steady connection, so every run starts from the same predictable address with consistent latency. Paste the proxy URL into the Proxy option of the HTTP Request node. A static IP also lets you ask the site owner to allowlist your web scraper.

Step 4. Extract fields with the HTML node and CSS selectors

This is where n8n web scraping turns raw HTML into clean fields. Add an HTML node after the HTTP Request node and set the operation to Extract HTML Content. Point it at the html property and map each field:

Key	CSS selector	Return value
title	h2.product-title	Text
price	span.price	Text
link	a.product-link	Attribute: href
image	img.thumb	Attribute: src

Selectors are usually the first thing to break, so set them up carefully:

• Turn Return Array on for lists and then use Split Out to get one item per product.
• Test the selectors in the browser console first. document.querySelectorAll('span.price').length should be the same as the number of prices you can see.
• Use data-* attributes where you can. The generated classes (e.g. .css-1x2y3z) are going to change on every deploy.
• Clean up prices with {{ $json.price.replace(/[^0-9.]/g, '') }} in Edit Fields.

The JSON output you're after looks like { "title": "Desk Lamp", "price": "19.99", "link": "/p/desk-lamp", "image": "/img/lamp.jpg" }. Flat key-value items like this map straight onto columns in the next step.

Step 5. Save results to Google Sheets or a database

Google Sheets is fine for a few thousand rows. You can use Append or Update Row and match on the link column so you don't end up with duplicate rows from repeat jobs.

If you want price history, use Postgres and Insert a new row on every run, with a scraped_at field set to {{ $now }}. For a table of current prices only, use Insert or Update on the link column instead.

Step 6. Set up a cron schedule

Replace the Manual Trigger with a Schedule Trigger and a custom expression. For example, 7 */6 * * * fires at minute 7 of every sixth hour. It uses the same five fields as a Linux crontab.

A few rules for picking run times:

• Stay off minute :00. That's when thousands of other jobs fire too.
• Add a Wait with a random delay (e.g. {{Math.floor(Math.random() * 120)}} seconds).
• Prices may need only a daily check, while news may need an hourly one.
• Don't let runs overlap: a job that takes 20 minutes shouldn't run every 15.

If you also want to start the flow on demand then put a webhook trigger alongside the schedule. Then publish the flow so it goes live.

Step 7. Add error handling and notifications

Handle errors at two levels:

• Inside the flow. In the HTTP Request settings set On Error to Continue (using error output) and have the failures go to a log sheet. That way, one dead URL doesn't stop the whole batch.
• Across flows. Create a small flow which starts with an Error Trigger and posts to either Telegram or Slack. Set it as the Error Workflow in your web scraping workflow's settings. Include {{ $json.execution.url }} so you can jump straight to the failed execution.

The failures to watch out for in n8n web scraping are the silent ones. A CAPTCHA page or a site redesign often still returns a 200 status code, just with blank fields. Add an IF check to ensure the title isn't blank and fire an alert if the item count drops to zero. Layouts change constantly, so review your selectors monthly.

Common n8n scraping problems and quick fixes

Symptom	Likely cause	Fix
Empty fields, status 200	JavaScript content or redesign	Find the JSON source; update selectors
403 or 429 errors	Rate limit or IP block	Slow down, add random delays, ask to be allowlisted
Jobs fire at the wrong hour	No timezone	Define GENERIC_TIMEZONE, restart
Duplicate rows	Plain append	Switch to an upsert

Wrapping up

Most of what makes an n8n web scraping setup reliable happens prior to the first run: the timezone and pruning settings in Docker, the robots.txt and terms check, sensible timeouts with a stable IP and selectors that can withstand a site redesign. After that, upsert your results, schedule jobs off the hour, and alert on both errors and empty results.

Expect to tune the flow over time. Check the error log between selector reviews, and this workflow can continue for months without surprises.