How to Scrape Websites With n8n (Tested Guide)

Updated 2026-10-09 · 1151 words

This page may contain affiliate links. If you buy through them we may earn a commission at no extra cost to you.

Direct answer: scraping a website in n8n means an HTTP Request node to fetch the page, an HTML Extract node using CSS selectors to pull the data you want out of the returned HTML, and, for anything paginated, a loop that advances to the next page until there isn't one. That covers static pages in well under half an hour. The complication is JavaScript-rendered sites, which n8n can't render on its own, so you route those through a headless browser API instead. Here's the workflow we built and tested against both page types.

Step 1: Fetch the page with an HTTP Request node

Add an HTTP Request node, set the method to GET, and point it at the target URL. Set a realistic User-Agent header in the node's options rather than leaving n8n's default; some sites block requests that obviously look like a script, and a browser-like User-Agent string is the simplest way to avoid that first, cheapest layer of blocking. Run the node once and inspect the raw HTML in the output to confirm you're getting the content you expect, not a CAPTCHA page or an error response disguised as a 200 status.

Step 2: Extract the data with CSS selectors

Add an HTML Extract node after the HTTP Request node, pointed at the HTML field from the previous step's output. Define one extraction rule per piece of data you want, a CSS selector plus a key name, and the node returns each match as a field on the output item. Open the target page in your browser's dev tools first and confirm the exact selector (a class name, a data attribute) actually matches what you think it does; sites restructure their markup often enough that a selector that worked last month can silently return nothing today, which is a quieter failure than an outright error and easy to miss if you're not checking output row counts.

Step 3: Handle pagination properly

Most scraping tutorials stop at a single page, which is rarely the real use case. For sites with numbered pages or a "next" link, wrap the fetch-and-extract steps in a loop using a Split in Batches node or a simple counter, incrementing a page-number query parameter until the extraction returns zero results or the "next" link stops appearing. Hardcoding a fixed page count works until the site adds or removes content, so prefer a loop that exits on an actual stopping condition over one that assumes a specific number of pages will always exist.

Step 4: Handle JavaScript-rendered pages

If the data you need only appears after client-side JavaScript runs, React, Vue, or any site that builds its content dynamically, the plain HTTP Request node won't see it; you'll get an HTML shell with none of the actual content. The fix is routing the request through a headless browser API instead: a service like Browserless or ScrapingBee runs a real browser, waits for the page to render, and returns the final HTML over a simple HTTP call you make from the same HTTP Request node, just pointed at the service's endpoint instead of the target site directly. From there, the HTML Extract step works exactly the same as it does for a static page. Self-hosting your own headless browser via Docker is possible if you're already comfortable with that infrastructure, but a managed API is the faster, lower-maintenance starting point for most teams.

Step 5: Respect the site and the law

Before scraping anything at real volume, check the site's robots.txt file and terms of service. A robots.txt disallow rule isn't legally binding on its own in most places, but ignoring it alongside a site's terms of service meaningfully increases your exposure if the site owner ever objects. Never scrape personal data without a legitimate basis, never scrape content behind a login for a purpose the site didn't intend, and never hit a server hard enough to degrade its performance for other visitors. If an official API exists for the data you need, use it; a scraper is a workaround for when no API exists, not a default first choice just because it's possible.

Step 6: Rate-limit yourself

Add a Wait node between requests, even a second or two, rather than firing requests as fast as n8n can send them. Scraping too fast is the single most common reason a working scraper gets blocked within minutes of real use; most sites rate-limit by IP, and a burst of rapid requests is the clearest possible signal that you're a script, not a browsing human. If you're scraping at genuine scale, a rotating proxy service helps avoid IP-based blocks, but treat needing one as a cue to double-check whether the scale you're operating at still fits within what the site's terms actually permit.

Step 7: Schedule and store the results

Once the extraction logic works reliably, wrap it in a Schedule Trigger so it runs on its own instead of manually, our guide to scheduling workflows in n8n covers the trigger setup and the timezone details that trip people up. Write the extracted records somewhere queryable, a Google Sheet, Airtable, or a database node, and dedupe against what you've already collected so a re-run doesn't pile up duplicate rows for content that hasn't actually changed.

Static pages vs. JavaScript-rendered pages

When NOT to scrape with n8n

Skip scraping entirely if the site publishes an official API; it's more stable, won't break on a redesign, and carries none of the legal ambiguity scraping does. Skip it too if the data lives behind a login wall and your use case isn't something the site's terms clearly permit, or if you need real-time data at high frequency, a scraper polling every few seconds is both expensive to run reliably and far more likely to get your IP blocked than a service designed to be queried that often. For high-volume or JS-heavy scraping at real scale, a dedicated scraping platform built for exactly that job will outperform a general automation tool stretched to cover it.

If you're pulling scraped data into a workflow that also needs to react to events rather than run on a timer, our guide to using webhooks in n8n covers that trigger type. For cleaning up messy scraped records before they reach a spreadsheet or database, our guide to automating data entry with AI covers that step, and our best n8n workflow templates roundup includes a few data-collection templates worth starting from instead of building the extraction logic from a blank canvas.

Save this to Pinterest