🛠️ How-to guide · Last updated: September 17, 2026

<aside> 🕸️

WHAT IT DOES

A website crawl reads pages from a public site or help center and ingests them as content — then keeps them current on a schedule. This page covers the full setup; for monitoring and managing a crawl afterwards, see Crawls & Syncs.

</aside>

Before you start

• Access: Editor, Content Manager, Publisher, or Admin (see Roles & Permissions).
• The starting URL of the site or section you want to crawl.
• For content behind a login: the login page URL and a short sign-in script (optional).

<aside> ⭐ TIP

Start narrow — depth 3 and a Daily refresh — and widen only if you're missing pages. It keeps the crawl fast and your content focused on what residents actually ask about.

</aside>

Step by step

1. Open Sources → Add source → Website crawl (or start from the Crawls & Syncs page with New crawl).
2. Enter a Crawl name and the Starting URL.
3. Set Max depth (how many link-levels deep), Max links per page (set 0 to follow all links), and Max pages (the total to visit; 0 = no limit).
4. Choose a Refresh frequency — One-time, Hourly, Daily, Weekly, Monthly, or a Custom cron.
5. (Optional) Open Advanced for load delay, headless rendering, content type, elements to ignore, sitemap/robots options, and authentication.
6. Adjust the Default tags if you like (a site tag is filled in for you).
7. Select Start crawl. You're taken to the crawl's detail page to watch progress.

The website-crawl setup — a starting URL, Max depth, Max links per page (0 = all links), and the new Max pages (0 = no limit), with more under Advanced.

Every setting

Basic

Advanced