{"id":5142,"date":"2021-05-24T11:09:04","date_gmt":"2021-05-24T09:09:04","guid":{"rendered":"https:\/\/help.marketingminer.com\/?post_type=kb&p=5142"},"modified":"2021-08-06T15:18:01","modified_gmt":"2021-08-06T13:18:01","slug":"robots-txt-what-is-it-and-how-does-it-work","status":"publish","type":"kb","link":"https:\/\/help.marketingminer.com\/en\/article\/robots-txt-what-is-it-and-how-does-it-work\/","title":{"rendered":"Robots.txt: what is it and how does it work"},"content":{"rendered":"\n

Robots.txt<\/strong> is a plain text file that tells web robots (most often search engine crawlers) which web pages or files can or cannot be crawled. It\u2019s good practice to also include your sitemap location in the robots.txt file. <\/p>\n\n\n\n

Here\u2019s the example of the MM robots.txt file:\u00a0https:\/\/www.marketingminer.com\/robots.txt<\/a>.<\/p>\n\n\n\n

\"Marketing<\/figure><\/div>\n\n\n\n

What is robots.txt used for? <\/h2>\n\n\n\n

By using robots.txt file, you prevent crawlers from accessing specific pages\/files. It keeps your resources under control and it prevents overloading the server with bots. In addition, it also helps to optimize a crawl budget<\/a> by allowing the crawling of important pages.<\/p>\n\n\n

\n
\n <\/i>\n <\/div>\n
\n TIP<\/strong>: You want to pay attention to crawl budget optimization especially when you have a big site with 100k+ landing pages.\u00a0 <\/div>\n <\/div>\n <\/div>\n\n\n

Many people use robots.txt to prevent search engine bots from indexing pages (e.g. admin pages, pages with sensitive information,…). Although, Google advises using a noindex<\/strong> directive for pages you don\u2019t want to index in the search engine. <\/p>\n\n\n\n

Remember, if you use the noindex tag on your site, it doesn\u2019t help you save your crawl budget as the crawler still needs to visit the page to be able to discover the noindex directive. <\/p>\n\n\n\n

How does robots.txt work? <\/h2>\n\n\n\n

The primary goal of search engine crawlers is to crawl web pages to discover content, index it, and then serve it to users in the search engines.  <\/p>\n\n\n\n

When the crawl attempts to visit a page, it automatically looks for your robots.txt file located in the root of the website <\/strong>(for example: https:\/\/www.marketingminer.com\/robots.txt<\/a>). If your robots.txt doesn\u2019t exist or it isn\u2019t located at the root of your website folder, the crawler will automatically check all your web pages<\/strong>.  <\/p>\n\n\n\n

Unfortunately the robots.txt directives are not supported by all crawlers (e.g. email address scrapers etc.). For this reason, it\u2019s important to remember that some search engines ignore them completely.<\/p>\n\n\n\n

Example robots.txt:<\/h2>\n\n\n\n
User-agent: user-agent-name\nDisallow: pages you don't want to be crawled\n<\/code><\/pre>\n\n\n\n
In the example below, you can see that Googlebot<\/strong> (user-agent name) is restricted from accessing all pages that contain \/blog\/<\/strong> string in the URL. <\/p>\n\n\n\n
User-agent: Googlebot \nDisallow: \/blog\/\n<\/code><\/pre>\n\n\n\n
Below, you can find the most common format of a robots.txt file. Most CMS programs already have a robots.txt file in place. It means that, all user agents (represented by an asterisk<\/strong> *) can crawl all pages (there is nothing blocked in the disallow section). <\/p>\n\n\n\n
User-agent: *\nDisallow:\n<\/code><\/pre>\n\n\n\n
Be careful with the command below, as this directive blocks all bots from crawling your site (even your homepage!). <\/p>\n\n\n\n
User-agent: *\nDisallow: \/\n<\/code><\/pre>\n\n\n\n
It often happens that web developers forget to remove the slash from the disallow section in robots.txt file after putting the site live. And then many website owners are wondering why their site is not being indexed yet. <\/p>\n\n\n\n
Robots.txt syntax<\/h2>\n\n\n\n
In the simple form, robots.txt file looks as follows:<\/p>\n\n\n\n