🛠️ How-to guide

<aside> 🕸️

WHAT IT DOES

A website crawl reads pages from a public site or help center and ingests them as content — then keeps them current on a schedule. This page covers the full setup; for monitoring and managing a crawl afterwards, see Crawls.

</aside>

Before you start

Step by step

  1. Open Sources → Add source → Website crawl (or start from the Crawls page with New crawl).
  2. Enter a Crawl name and the Starting URL.
  3. Set Max depth (how many links deep) and Max links per page.
  4. Choose a Refresh frequency — One-time, Hourly, Daily, Weekly, or Monthly.
  5. (Optional) Open Advanced for load delay, headless rendering, content type, elements to ignore, sitemap/robots options, and authentication.
  6. Adjust the Default tags if you like (a site tag is filled in for you).
  7. Select Start crawl. You're taken to the crawl's detail page to watch progress.

The website-crawl setup: enter a starting URL, set depth and refresh frequency, with more control under Advanced.

The website-crawl setup: enter a starting URL, set depth and refresh frequency, with more control under Advanced.

Every setting

Basic

Setting Default Details
Crawl name — Required. A label for this crawl.
Starting URL — Required. Must begin with http:// or https://.
Max depth 3 How many links deep to follow (1–10).
Max links per page 5 How many links to follow on each page (1–10).
Refresh frequency One-time One-time, Hourly, Daily, Weekly, or Monthly.
Default tags site: <name> Applied to every page the crawl ingests.

Advanced

Setting Default Details
Page load delay 5s Seconds to wait before reading a page (0–60), for slow-loading sites.
Headless browsing On Renders the page fully — needed for sites built with JavaScript. Turn off for a faster raw fetch.
Page content type Markdown Markdown keeps structure; Text is flatter.
In-page elements to ignore — Hide banners, headers, or footers so they don't clutter answers.
Discover URLs from sitemap.xml Off Seeds every URL in the site's sitemap (overrides the depth and links limits).
Respect robots.txt On Skips paths the site disallows for crawlers. Turn off only for sites you own.
Incremental re-crawl On On refreshes, skips pages that haven't changed — faster and cheaper.
Authentication — A login page URL and sign-in steps for content behind a login.