What is crawling?
Crawling is when a search engine sends a search robot, e.g. Googlebot, to a given page to read its content. In other words, it is the search engines' reading of a page on your website.
This should not be confused with indexing, which is when search engines have crawled a page and subsequently index it, thereby displaying it in search results.
What causes search engines to crawl a page?
There are a few different ways in which search engine crawlers can find a page/URL and subsequently crawl it:
- There is a link to the relevant page in the website's XML sitemap
- There is an internal link to the relevant page on the website
- There is an external link to the page on a third-party website
- A large spike in traffic to the given page
To ensure your pages get indexed, it is therefore beneficial to have an XML sitemap set up for your website. This way, you help search engines find the pages on your website, which can ultimately lead to new categories, product pages, blog posts, etc. being indexed more quickly.
How can I see which of my pages Google is crawling/has crawled?
First and foremost, you can get a good overview of which of your pages Google has crawled/generally crawls by going into Google Search Console under "Coverage":
Here you can get an overview of the pages on your website that Google has crawled, and you can see whether they are indexed (valid), or whether they are excluded from indexing for one reason or another.
Another way to see which pages Google is crawling is by checking your website's server logs and looking for the Googlebot user agent. Here you can easily see when Googlebot has visited your website and which pages it has visited.
Want to ensure your pages are correctly crawled and indexed?
Fill in the form below, or call us on 30 12 42 72 to get help with your SEO strategy!
