{"id":2403,"date":"2020-06-24T16:53:27","date_gmt":"2020-06-24T14:53:27","guid":{"rendered":"https:\/\/help.marketingminer.com\/?post_type=kb&#038;p=2403"},"modified":"2020-10-13T10:03:41","modified_gmt":"2020-10-13T08:03:41","slug":"what-is-a-crawl-budget","status":"publish","type":"kb","link":"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/","title":{"rendered":"What is a crawl budget"},"content":{"rendered":"\n<p><strong>A Crawl Budget is a volume of URLs that a crawler can crawl in a certain period of time.<\/strong> Its amount is most commonly stated in the quantity of URLs scanned by a specific robot (crawler) in one day. It depends on several factors and the most crucial ones are content quality, website load speed and internal linking structure.<\/p>\n\n\n\n<h2>How to identify a web crawl budget<\/h2>\n\n\n\n<p>User needs <strong>access log<\/strong> file to identify crawl budget from a specific crawlers (robots) of a search engine. Access log is a file on a server that records all the requests that are processed by a particular server. The output data then return as:<\/p>\n\n\n\n<ul><li>User-agent&nbsp;(It is used to identify requests from a crawler to server)<\/li><li>IP<\/li><li>URL of a request<\/li><li>Date and time of a request<\/li><li>&#8230;and many others<\/li><\/ul>\n\n\n\n<p>It enables SEO specialists or anyone else to closely analyze information about requests performed by search engine crawlers.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"886\" height=\"428\" src=\"https:\/\/help.marketingminer.com\/wp-content\/uploads\/2020\/06\/crawl-budget.png\" alt=\"Crawl budget definition\" class=\"wp-image-1530\" srcset=\"https:\/\/help.marketingminer.com\/wp-content\/uploads\/2020\/06\/crawl-budget.png 886w, https:\/\/help.marketingminer.com\/wp-content\/uploads\/2020\/06\/crawl-budget-300x145.png 300w, https:\/\/help.marketingminer.com\/wp-content\/uploads\/2020\/06\/crawl-budget-768x371.png 768w\" sizes=\"(max-width: 886px) 100vw, 886px\" \/><\/figure>\n\n\n\n<h2>Crawl budget optimization<\/h2>\n\n\n\n<p>The search engines assign to website a crawl budget primarily based on its <strong>authority<\/strong> (link portfolio) and a volume of unique and <strong>quality content<\/strong> that they are able to obtain. In the matter of web crawler step, a <strong>crawl waste<\/strong> has to be considered. Crawl wastes are the comments and queries that go on non-existent websites or the ones we don&#8217;t want to crawl. Following are the most common problems that occur in log analysis:<\/p>\n\n\n\n<ul><li>URL with error response<\/li><li>Non-indexable websites<\/li><li>Website with \u201cthin content\u201d<\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"<p>A Crawl Budget is a volume of URLs that a crawler can crawl in a certain period of time. Its amount is most commonly stated in the quantity of URLs scanned by a specific robot (crawler) in one day. It depends on several factors and the most crucial ones are content quality, website load speed &#8230; <a title=\"What is a crawl budget\" class=\"read-more\" href=\"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/\" aria-label=\"More on What is a crawl budget\">Read more<\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","meta":[],"kbtopic":[36],"kbtag":[87,84],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v20.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>What is a crawl budget - Marketing Miner Knowledge Base<\/title>\n<meta name=\"description\" content=\"A Crawl Budget is a volume of URLs that a crawler can crawl in a certain period of time. Its amount is most commonly stated in the quantity of URLs\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is a crawl budget - Marketing Miner Knowledge Base\" \/>\n<meta property=\"og:description\" content=\"A Crawl Budget is a volume of URLs that a crawler can crawl in a certain period of time. Its amount is most commonly stated in the quantity of URLs\" \/>\n<meta property=\"og:url\" content=\"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/\" \/>\n<meta property=\"og:site_name\" content=\"Marketing Miner Knowledge Base\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/marketingminer\/\" \/>\n<meta property=\"article:modified_time\" content=\"2020-10-13T08:03:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/help.marketingminer.com\/wp-content\/uploads\/2020\/10\/what-is-a-crawl-budget.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1354\" \/>\n\t<meta property=\"og:image:height\" content=\"761\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/help.marketingminer.com\/wp-content\/uploads\/2020\/10\/what-is-a-crawl-budget.png\" \/>\n<meta name=\"twitter:site\" content=\"@marketingminer\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/\",\"url\":\"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/\",\"name\":\"What is a crawl budget - Marketing Miner Knowledge Base\",\"isPartOf\":{\"@id\":\"https:\/\/help.marketingminer.com\/en\/#website\"},\"datePublished\":\"2020-06-24T14:53:27+00:00\",\"dateModified\":\"2020-10-13T08:03:41+00:00\",\"description\":\"A Crawl Budget is a volume of URLs that a crawler can crawl in a certain period of time. Its amount is most commonly stated in the quantity of URLs\",\"breadcrumb\":{\"@id\":\"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/help.marketingminer.com\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What is a crawl budget\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/help.marketingminer.com\/en\/#website\",\"url\":\"https:\/\/help.marketingminer.com\/en\/\",\"name\":\"Marketing Miner Knowledge Base\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/help.marketingminer.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/help.marketingminer.com\/en\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/help.marketingminer.com\/en\/#organization\",\"name\":\"Marketing Miner\",\"url\":\"https:\/\/help.marketingminer.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/help.marketingminer.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/help.marketingminer.com\/wp-content\/uploads\/2020\/06\/cropped-Logo-01@3x.png\",\"contentUrl\":\"https:\/\/help.marketingminer.com\/wp-content\/uploads\/2020\/06\/cropped-Logo-01@3x.png\",\"width\":4098,\"height\":819,\"caption\":\"Marketing Miner\"},\"image\":{\"@id\":\"https:\/\/help.marketingminer.com\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/marketingminer\/\",\"https:\/\/twitter.com\/marketingminer\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is a crawl budget - Marketing Miner Knowledge Base","description":"A Crawl Budget is a volume of URLs that a crawler can crawl in a certain period of time. Its amount is most commonly stated in the quantity of URLs","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/","og_locale":"en_US","og_type":"article","og_title":"What is a crawl budget - Marketing Miner Knowledge Base","og_description":"A Crawl Budget is a volume of URLs that a crawler can crawl in a certain period of time. Its amount is most commonly stated in the quantity of URLs","og_url":"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/","og_site_name":"Marketing Miner Knowledge Base","article_publisher":"https:\/\/www.facebook.com\/marketingminer\/","article_modified_time":"2020-10-13T08:03:41+00:00","og_image":[{"width":1354,"height":761,"url":"https:\/\/help.marketingminer.com\/wp-content\/uploads\/2020\/10\/what-is-a-crawl-budget.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_image":"https:\/\/help.marketingminer.com\/wp-content\/uploads\/2020\/10\/what-is-a-crawl-budget.png","twitter_site":"@marketingminer","twitter_misc":{"Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/","url":"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/","name":"What is a crawl budget - Marketing Miner Knowledge Base","isPartOf":{"@id":"https:\/\/help.marketingminer.com\/en\/#website"},"datePublished":"2020-06-24T14:53:27+00:00","dateModified":"2020-10-13T08:03:41+00:00","description":"A Crawl Budget is a volume of URLs that a crawler can crawl in a certain period of time. Its amount is most commonly stated in the quantity of URLs","breadcrumb":{"@id":"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/help.marketingminer.com\/en\/article\/what-is-a-crawl-budget\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/help.marketingminer.com\/en\/"},{"@type":"ListItem","position":2,"name":"What is a crawl budget"}]},{"@type":"WebSite","@id":"https:\/\/help.marketingminer.com\/en\/#website","url":"https:\/\/help.marketingminer.com\/en\/","name":"Marketing Miner Knowledge Base","description":"","publisher":{"@id":"https:\/\/help.marketingminer.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/help.marketingminer.com\/en\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/help.marketingminer.com\/en\/#organization","name":"Marketing Miner","url":"https:\/\/help.marketingminer.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/help.marketingminer.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/help.marketingminer.com\/wp-content\/uploads\/2020\/06\/cropped-Logo-01@3x.png","contentUrl":"https:\/\/help.marketingminer.com\/wp-content\/uploads\/2020\/06\/cropped-Logo-01@3x.png","width":4098,"height":819,"caption":"Marketing Miner"},"image":{"@id":"https:\/\/help.marketingminer.com\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/marketingminer\/","https:\/\/twitter.com\/marketingminer"]}]}},"_links":{"self":[{"href":"https:\/\/help.marketingminer.com\/en\/wp-json\/wp\/v2\/kb\/2403"}],"collection":[{"href":"https:\/\/help.marketingminer.com\/en\/wp-json\/wp\/v2\/kb"}],"about":[{"href":"https:\/\/help.marketingminer.com\/en\/wp-json\/wp\/v2\/types\/kb"}],"author":[{"embeddable":true,"href":"https:\/\/help.marketingminer.com\/en\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/help.marketingminer.com\/en\/wp-json\/wp\/v2\/comments?post=2403"}],"version-history":[{"count":2,"href":"https:\/\/help.marketingminer.com\/en\/wp-json\/wp\/v2\/kb\/2403\/revisions"}],"predecessor-version":[{"id":4098,"href":"https:\/\/help.marketingminer.com\/en\/wp-json\/wp\/v2\/kb\/2403\/revisions\/4098"}],"wp:attachment":[{"href":"https:\/\/help.marketingminer.com\/en\/wp-json\/wp\/v2\/media?parent=2403"}],"wp:term":[{"taxonomy":"kbtopic","embeddable":true,"href":"https:\/\/help.marketingminer.com\/en\/wp-json\/wp\/v2\/kbtopic?post=2403"},{"taxonomy":"kbtag","embeddable":true,"href":"https:\/\/help.marketingminer.com\/en\/wp-json\/wp\/v2\/kbtag?post=2403"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}