🛠️ How-to guide

<aside> 🕸️

WHAT IT DOES

Crawls pull content straight from your public website or help center and keep it fresh on a schedule — so your agents always answer from your latest pages, with no manual re-uploading.

</aside>

The Crawls page

The Crawls page lists your website crawls (and any SharePoint or cloud-storage syncs) with a live progress strip while they run — pages fetched, in flight, and failed. Filter by source (Website, SharePoint, Cloud) or status (Active, Idle, Failed), and switch between card and list views.

The Crawls page — website crawls with live progress; here one is actively crawling, one is idle, and one has failed pages.

Starting a crawl

Select New crawl and choose Website. You'll set a few basics, with more control available under Advanced.

Basic options

• Crawl name — a label for this crawl.
• Starting URL — where the crawl begins, e.g. https://riverton.gov.
• Max depth — how many links deep to follow (1–10, default 3).
• Max links per page — how many links to follow on each page (1–10, default 5).
• Refresh frequency — One-time, Hourly, Daily, Weekly, or Monthly.
• Default tags — labels applied to every page found; a site tag is suggested for you.

Advanced options

• Page load delay — seconds to wait before reading a page (0–60, default 5), for sites that load slowly.
• Headless browsing — on by default; needed for sites that build their pages with JavaScript.
• Page content type — Markdown (keeps structure) or Text (flatter).
• In-page elements to ignore — hide banners, headers, or footers so they don't clutter answers.
• Discover URLs from sitemap.xml — use the site's sitemap to find pages.