Web crawler
What is a web crawler?
A web crawler, also called a spider or bot, is an automated program that browses the web by requesting pages and following the links on them. Search engines use crawlers to discover content for their index; SEO tools use them to audit sites; AI companies use them to collect content for models and live answers.
Well-known crawlers include Googlebot, Bingbot, Applebot, and AI crawlers such as GPTBot, OAI-SearchBot and ClaudeBot. Reputable crawlers identify themselves in their user agent and follow robots.txt. Bad bots often ignore robots.txt and pretend to be a browser or Googlebot.
Why web crawler matters for SEO
Crawlers are how your pages get into search engines and, increasingly, AI answers. Knowing which bots visit, and controlling them with robots.txt, lets you allow the useful ones and limit those that only cost you bandwidth.
Running a crawler over your own site is also the fastest way to find broken links, redirects and missing tags, because it sees the site the way a search engine does. A site audit is exactly that crawl, and the technical half of an SEO audit.
Crawlers have limits. Many don’t run JavaScript, most only follow links in plain <a href> elements, and all of them give up on pages that time out or return errors. If your content only appears after a click or a script, some crawlers will never see it.
How to control web crawlers
- Use robots.txt to allow or disallow crawlers by user agent.
- Use noindex for pages you want crawled but kept out of search results.
- Check server logs to see which crawlers visit and how often.
- Block abusive bots at the firewall or CDN, since they may ignore robots.txt.
Web crawler FAQ
Is a web crawler the same as a scraper?
They overlap. A crawler discovers pages by following links; a scraper extracts specific data from pages. Many tools do both.
Do all crawlers obey robots.txt?
No. Robots.txt is a voluntary standard. Search engine and most AI company crawlers follow it, but malicious bots often don’t.
Related SEO terms
Googlebot is Google’s web crawler. It fetches pages so Google Search can index them, using a smartphone crawler for most sites.
Crawling is how search engines discover pages: automated bots fetch URLs and follow the links on them to find more pages.
Robots.txt is a plain-text file at the root of a site that tells crawlers which URLs they may and may not request.
AI crawlers (GPTBot, ClaudeBot)
AI crawlers are bots that AI companies use to collect web content for model training or for AI search answers, such as GPTBot and ClaudeBot.
A user agent is the software making a web request, identified by a user-agent string such as Googlebot’s that servers and robots.txt can read.
Log file analysis is reviewing your server’s access logs to see exactly which URLs search engine bots requested, when and with what result.