Crawling

Crawling is how search engines find and download pages with automated programs called crawlers. How Googlebot discovers URLs and what limits crawling.

Crawling is the first stage of search. A search engine's automated programs, called crawlers, bots or spiders, fetch pages from the web so they can be analyzed. Google's main crawler is Googlebot, which comes in a smartphone and a desktop version.

Before a page can be crawled, Google has to discover its URL. There is no central list of all web pages, so Google finds new URLs mainly through links from pages it already knows, such as a category page that links to a new article, and through sitemaps that site owners submit.

Google decides algorithmically which sites to crawl, how often and how many pages to fetch from each. Its crawlers try not to overload a site, and they slow down when a server responds with errors.

You can control crawling with a robots.txt file, which tells crawlers which URLs they may access. Google notes that robots.txt is not a way to keep a page out of search results; that takes a noindex rule.

For most sites, how much Google crawls is not a problem. Google's crawl budget guide is written for very large sites, or large ones whose pages change daily.

Related terms

Sources

  1. In-depth guide to how Google Search works
  2. Googlebot
  3. Introduction to robots.txt
  4. Optimize your crawl budget

All terms