Skip to content
OpenSEO
Get started

Crawler

OpenSEOBot

OpenSEOBot is the crawler behind the OpenSEO site audit. It visits a website only when someone runs a site audit of it, or has scheduled one, to check it for technical SEO issues such as broken links, redirects and missing titles. It does not build a search index, and it does not collect content for training AI models.

It identifies itself with this user agent:

OpenSEOBot/0.1 (+https://openseo.gg/bot)

How it crawls

  • Two requests at a time to start. While the site answers quickly it adds more, up to six at a time and never more than four a second. It halves that as soon as the site slows down or answers with errors.
  • It follows robots.txt: the rules for OpenSEOBot, or for * when there are none, including Crawl-delay up to 10 seconds.
  • When a site keeps answering 429 Too Many Requests, it backs off and then stops.
  • Each audit is capped at the number of pages its owner chose.

What it fetches

  • Pages it finds through links and XML sitemaps, starting from the address it was given.
  • robots.txt, sitemaps and llms.txt, and the http:// version of the home page, to check that it redirects to HTTPS.
  • The images, scripts and stylesheets pages use, to check that they load.
  • Links to other sites, up to 300 per audit, to check that they are not broken.

How to slow it down or block it

To keep it out of your whole site, add this to robots.txt:

User-agent: OpenSEOBot
Disallow: /

A Disallow for part of the site, or a Crawl-delay, works the same way. The change applies from the next audit.

A page your robots.txt disallows is never fetched. If your firewall or bot protection refuses OpenSEOBot instead, the audit reports the page as blocked. Its owner can choose to fetch blocked pages through a proxy service, which does not use this user agent.

Self-hosted copies

OpenSEO is open source, so anyone can run their own copy. Those copies send the same user agent from their own servers, and they follow the same rules unless the person running them changed the code.

Questions or problems

If OpenSEOBot is causing trouble for your site, open an issue on GitHub with your domain and the time of the visits.