What is Googlebot?
"Googlebot" is the term for Google's search robot that crawls websites on the internet.
In reality, "Googlebot" is not just a single bot, as the name might suggest, but rather thousands, and perhaps even millions, of computers with the "Googlebot" software installed, crawling the internet.
Googlebot's tasks are to discover new pages on the internet, re-crawl pages with potentially updated content and collect information about these pages, so that Google's database is constantly kept up to date and their algorithms can always serve the best content to their users.
How does Googlebot work?
Googlebot works by the software accessing pages on the internet and "scanning" their content. The way Googlebot finds pages to scan is primarily by following links on the pages it accesses.
In addition to following links on pages being crawled, Googlebot also uses sitemaps to find new pages to crawl.
It is therefore a process that never stops, as Googlebot can always find new pages that need to be crawled.
See the illustration below for a better visual explanation:
How often does Googlebot visit my website?
If you are interested in knowing how often Googlebot crawls your website, this can be found in Google Search Console under "Legacy tools and reports" -> "Crawl stats":
You will then be presented with three graphs showing various information about how often Google crawls your website.
It can be a good idea to keep a regular eye on these graphs, as a large spike or a significant drop may indicate that a problem has occurred on your website that should be investigated further.
If you see a large spike in the graphs without having added a lot of new content yourself, this can typically indicate that Google has been given access to crawl sorting and filtering pages or other pages with thin content. This is something that should be addressed as quickly as possible if that is the case.
If you see a significant drop in the graphs without having removed content from your website yourself, this may indicate that Google has been denied access to certain pages through robots.txt, or that content has been deleted from your website.
Different Googlebots
There is not just one Googlebot — there are many different Googlebots with different "user-agents" associated with them. For example, there are Googlebots for both desktop and mobile, as well as Googlebots for images and video.
Here you can see an overview of the most common Googlebots:
Would you like to know more about how Googlebot affects your website's visibility?
Fill in the form below, or call us on 30 12 42 72 to get advice on how to optimise your content for Googlebot!
