How the crawler behaves
What the runner fetches, how politely, and what it refuses to reach. WebtraxBot honours robots.txt and Crawl-delay.
Identity
The runner sends User-Agent: Mozilla/5.0 (compatible; WebtraxBot/1.0; +https://webtrax.dev/bot). Site owners can control it in robots.txt with User-agent: WebtraxBot; webtrax.dev/bot explains how.
What it fetches
- The start URL. If it redirects within the same site (HTTP to HTTPS, or adding or dropping
www.), the final address becomes the site; a redirect to another site stops the check (exit 3). robots.txt, thensitemap.xml, sitemap indexes and anySitemap:lines, as extra starting points.- Pages on the same site, breadth first, up to
max_pagesandmax_depth. Each URL is fetched once: a link to a page the crawl fetches is answered by that page’s result, and redirects stop at a page the crawl already has. - Every other link target (external links, pages beyond the limits) once, with
HEAD, confirmed withGETfor 403, 404, 405 and 501, which some servers answer wrongly forHEAD. - External sites are checked, never crawled.
Politeness
robots.txtDisallow rules are honoured: those URLs are never requested.- At most 2 requests per second to each host by default, or the site’s
Crawl-delayif it is longer (capped at 30 seconds). - A
429or503withRetry-Afteris waited out and retried; the host’s rate is halved for the rest of the run. A link that stays rate limited is reported as rate limited, never as broken. - Server errors and timeouts get one retry before they count.
- HTML pages are read up to 5 MB (after decompression); other content types are not downloaded.
The browser stage
With --browser, after the crawl the runner loads a sample of the crawled pages in Chromium, once each, in a fresh browser context with service workers and downloads blocked. All of Chromium’s traffic, including images, scripts and WebSockets, goes through a local egress proxy that applies the same rules as the crawler, and Chromium is not allowed to resolve names itself. Performance numbers are measured through that proxy on one load, so compare runs rather than reading single numbers as exact.
What it refuses to reach
Every connection is checked against the address it is about to use, on every redirect hop, so a DNS answer that changes cannot sneak past. By default the runner refuses private, loopback and link-local addresses (allowed for local targets and with --allow-private), and it always refuses cloud metadata addresses such as 169.254.169.254, which hand out credentials.