Airgentic Help
The General tab controls the overall behaviour of the web crawler: whether it runs, how it identifies itself, and how politely it treats target websites. → Open Crawler Settings (General tab)
Enable or disable this data source. When disabled, the web crawler will not run for this service. Use this to pause crawling without losing your other settings.
The number of milliseconds the crawler waits between page requests to the same domain. This helps prevent overloading servers and reduces the risk of being rate-limited or blocked.
The User-Agent string the crawler sends with each HTTP request. Many servers log or restrict access by user agent.
Mozilla/5.0 (compatible; MyCrawler/1.0)) if the site or your CDN expects a specific identity (for example, an “Airgentic” user agent that you have allowlisted).When enabled, all http:// URLs are automatically converted to https:// before crawling. This ensures the crawler uses secure connections when the site supports them, and avoids duplicate content between HTTP and HTTPS versions.
← Back to Crawler settings overview
Link rewrites send visitors to a different address than the one that was crawled. The crawler still reads the original page, but search results and the assistant's citations link to the rewritten address. Use them when the crawled address is not the one you want people to land on: a file served from a CDN that has a friendlier public page, or content that your website shows inside another page.
Each rule has two parts:
{0} is the whole crawled address, and {1}, {2} or {name} are the groups captured by the match. Add |urlencode, |base64 or |squiz_remote_content to encode a value, for example {0|urlencode}. Write {{ and }} for literal braces.For example, the match ^https://cdn\.example\.com/(.*)$ with the link https://www.example.com/{1} turns https://cdn.example.com/files/report.pdf into https://www.example.com/files/report.pdf.
Use Try an address to see which rule matches a crawled address and the link it produces before you save. Saving updates the index so that existing pages pick up the new links. If your official links come from a search field instead, this section is read-only and you change them in Search Fields.