Webtrax docs

How the crawler behaves

What the runner fetches, how politely, and what it refuses to reach. WebtraxBot honours robots.txt and Crawl-delay.

Identity

The runner sends User-Agent: Mozilla/5.0 (compatible; WebtraxBot/1.0; +https://webtrax.dev/bot). Site owners can control it in robots.txt with User-agent: WebtraxBot; webtrax.dev/bot explains how.

What it fetches

Politeness

The browser stage

With --browser, after the crawl the runner loads a sample of the crawled pages in Chromium, once each, in a fresh browser context with service workers and downloads blocked. All of Chromium’s traffic, including images, scripts and WebSockets, goes through a local egress proxy that applies the same rules as the crawler, and Chromium is not allowed to resolve names itself. Performance numbers are measured through that proxy on one load, so compare runs rather than reading single numbers as exact.

What it refuses to reach

Every connection is checked against the address it is about to use, on every redirect hop, so a DNS answer that changes cannot sneak past. By default the runner refuses private, loopback and link-local addresses (allowed for local targets and with --allow-private), and it always refuses cloud metadata addresses such as 169.254.169.254, which hand out credentials.