🛠️ How-to guide · Last updated: October 1, 2026

<aside> 🕸️

WHAT IT DOES

Crawls & Syncs keep content flowing in automatically. A website crawl pulls pages straight from your public site or help center and refreshes them on a schedule — and this is also the home for your SharePoint, cloud storage, database, API Crawl, and API Push sources, so everything that updates on a timer lives in one place, with no manual re-uploading.

</aside>

The Crawls & Syncs page

The Crawls page lists your website crawls (and any SharePoint or cloud-storage syncs) with a live progress strip while they run — pages fetched, in flight, and failed. Filter by source (Website, SharePoint, Cloud, plus API Crawl and API Push once you have one) or status (Active, Idle, Failed), and switch between card and list views. Open an API Crawl or API Push source to see each run and every record it brought in.

The Crawls page — website crawls with live progress; here one is actively crawling, one is idle, and one has failed pages.

Starting a crawl

Select New crawl and choose Website. (The same menu also starts an API Crawl or API Push — see Add a source: API Crawl and Add a source: API Push.) You'll set a few basics, with more control available under Advanced.

Basic options

• Crawl name — a label for this crawl.
• Starting URL — where the crawl begins, e.g. https://riverton.gov.
• Max depth — how many link-levels deep to follow (up to 10, default 3).
• Max links per page — how many links to follow on each page; set 0 to follow all (default 10).
• Max pages — the total number of pages a run will visit; 0 means no limit.
• Refresh frequency — One-time, Hourly, Daily, Weekly, or Monthly.
• Default tags — labels applied to every page found; a site tag is suggested for you.

Advanced options

• Page load delay — seconds to wait before reading a page (0–60, default 5), for sites that load slowly.
• Headless browsing — on by default; needed for sites that build their pages with JavaScript.
• Page content type — Markdown (keeps structure) or Text (flatter).
• In-page elements to ignore — hide banners, headers, or footers so they don't clutter answers.