What is robots.txt?
Robots.txt is a file containing instructions for various web crawlers, such as Googlebot. It is the first file search engines will attempt to locate on your website, as it can contain important instructions that the search engine is interested in following. In SEO, it is an important file to be familiar with and ensure is properly optimised.
What can you use the robots.txt file for?
You can use the robots.txt file to set up instructions for web crawlers such as Googlebot, regarding which pages on your domain you do not want to be crawled or visited by web crawlers.
Typically, the robots.txt file is used to exclude pages from search engines that should not be crawled. In this way, you can optimise your crawl budget and ensure that Google focuses its efforts on the pages that are important to your website, while also preventing unnecessary pages with duplicate content and thin content from being indexed.
How do I set up instructions in robots.txt?
First and foremost, you need to locate your robots.txt file. This will be placed in the root of your web hosting and named "robots.txt". You will therefore need to connect to your web hosting via FTP.
If you cannot find a "robots.txt" file in the root of your web hosting, you need to create your robots.txt file. You do this simply by placing a file named "robots.txt" in the root of your web hosting.
After that, it is time to set up the instructions in your robots.txt file.
Which instructions and blocks should be set up in a robots.txt file will always depend on the individual case.
The actual setup of instructions in the robots.txt file is somewhat technical. The example below will, for instance, exclude all pages on the domain from being crawled by all crawlers.
User-agent: *
Disallow: /
As can be seen, you first specify at the top which web crawlers the instructions are directed at — in the above case, the asterisk means the instructions apply to all web crawlers.
Below this, you then set up your "allow" and "disallow" rules.
If, for example, you want to exclude all files in the subdirectory "tag" but allow the file "statement.html" in the "tag" subdirectory, you would insert the following instruction:
User-agent: *
Disallow: /tag/
Allow:/tag/udtalelse.html
These disallow and allow rules can therefore be combined in many different ways.
It only takes a single typo or an incorrect disallow instruction in your robots.txt file to cause major damage to your SEO. You should therefore be very careful if you venture into making changes to it, and preferably refrain entirely if you are not 100% certain what you are doing.
Are you unsure which instructions you should set up in your robots.txt, or whether you should set up any instructions at all?
Contact us at 30 12 42 72, and we will be happy to investigate which instructions we believe you should set up or remove from your robots.txt file!
