Bot Documentation

WebAuditFixBot

WebAuditFixBot is the automated crawler operated by WebAuditFix.com. It runs on-demand security, privacy, and web-standards audits, triggered by a visitor submitting a URL through WebAuditFix.com or the partner API.

User-Agent String
WebAuditFixBot/1.0 (+https://webauditfix.com/bot)
Every request from WebAuditFixBot carries this exact user-agent. The URL in the user-agent string links to this page. The bot does not spoof browser user-agents.

What this bot does

WebAuditFixBot performs a one-off audit of a public website at the request of a visitor to WebAuditFix.com. Each audit is a fixed set of at most seven HTTP requests to the target origin: the submitted URL, plus the well-known paths /robots.txt, /security.txt, /sitemap.xml, /.well-known/security.txt, /llms.txt, /privacy-policy, and /cookie-policy.

The bot does not crawl beyond those paths. It does not follow cross-origin links. It does not run JavaScript beyond what a default browser navigation requires. It does not submit forms, log in, or interact with authenticated areas.

Policy compliance

Category
Security — scans public websites for security, privacy, and standards issues
Verified
Consent model
Visitor-initiated — every scan is triggered by an interactive submission at WebAuditFix.com or via an issued partner API key. No continuous or speculative crawling.
Visitor
robots.txt
Fetched and parsed at the start of every scan. If the origin's robots.txt disallows WebAuditFixBot for the target URL, the scan halts without further requests and the requesting user sees a message pointing them to this page.
Enforced
AI training
Crawled content is never used to train machine-learning models. No content is sold or licensed to third parties for that purpose.
None
Crawl scope
Limited to the submitted URL's origin. No cross-domain following, no depth-first crawl, no sitemap-driven expansion.
Scoped
Rate
Sequential requests from a single worker per scan. Each scan is at most seven HTTP requests, typically completing in under 5 seconds. Rate-limited to 10 scans per hour per source IP.
Polite
Owner block
Site owners can block WebAuditFixBot at any time by adding Disallow: / under a User-agent: WebAuditFixBot block in their robots.txt.
Owner action

Crawl behaviour

  • Fetches and parses robots.txt before every scan. Honours Disallow and Allow directives for user-agent WebAuditFixBot and for the wildcard *.
  • If robots.txt disallows the target URL, no further requests are made — the scan is marked "failed: disallowed by robots.txt" and no result page is generated.
  • Does not follow cross-origin links, submit forms, trigger account actions, or interact with authenticated areas.
  • Identifies itself in every request via the user-agent above. Does not spoof browser user-agents.
  • All requests are signed with Web Bot Auth (RFC 9421 HTTP message signatures) — origin operators can verify signature at https://webauditfix.com/.well-known/http-message-signatures-directory.
  • Fetches originate from the same server range published in the WebAuditFix.com SPF record; a Cloudflare verified-bot listing (when granted) is the canonical source of authorised IPs.

Data collected and retention

Data type Purpose Retention Access
HTTP response headers, redirect chain, status codes Security-header + transport analysis Retained with the scan record indefinitely at /results/<scan_id>. Removed on request to [email protected]. Anyone who has the scan URL
HTML head + first ~500 KB of body Standards, accessibility, and structured-data checks 1 hour in Redis, then discarded. Aggregate check results only are stored long-term. Anyone who has the scan URL, during the retention window
robots.txt, security.txt, sitemap.xml, /.well-known files Standards + vulnerability-disclosure discoverability Retained as scan evidence indefinitely at /results/<scan_id>. Anyone who has the scan URL
Aggregated compliance scores (0–100 per category) Sector benchmarks + public "N scans completed" counter Retained indefinitely, anonymised Public (aggregate only, no per-site identifiers)

No crawled content is used for AI training or sold to third parties. Removal requests for individual scan records go to [email protected]. Full policy at /privacy-policy.

How to allow or block WebAuditFixBot

To explicitly allow WebAuditFixBot (recommended if your site is behind a Cloudflare bot-management plan):

User-agent: WebAuditFixBot
Allow: /

To block WebAuditFixBot entirely — for example, if a scan was triggered by a third party without your consent:

User-agent: WebAuditFixBot
Disallow: /

WebAuditFixBot honours the block on the very next scan attempt — the scan halts before any content is fetched. If you continue to see requests carrying the WebAuditFixBot user-agent after publishing a Disallow: / directive, this is a policy violation and we treat it as such — please report to the email below with a request log so we can investigate.

Contact and abuse reporting

✉
Abuse & enquiries
[email protected]
Report unexpected crawl activity, request removal of a specific scan record, or ask about WebAuditFixBot's behaviour. Responses within one business day.