URL Indexability

Last modified: 17.10.2022
Estimated reading time: 2 min

In order for your site to appear in search results, it has to be indexed in the index of search engines. If it isn’t, you need to find a reason why. If the reason is apparent, that’s good for you, and there is nothing else that you need to take care of.

But if you identified that you have a problem, but you don’t know the cause, you have to discover it. Marketing Miner helps you with this, no matter if you need to analyze one or a thousand URLs.

In practice

You can use checker for indexability in case of:

  • If you just got a new project and you need to take care of initial SEO audit. One of the main steps is to find out if the pages are indexable.
  • You noticed a drop in your ranking or lower / no traffic from an organic search to a specific page.
  • You are updating or creating a sitemap, and you only need to include URLs that are indexable.
  • You are checking if your sitemap consists only of indexable URLs.

Insert URLs or Sitemaps

Get started by clicking on Create report button and specifying URL at the input. Then write or copy a list of URLs to the clipboard to find out if they are indexable.

URL Indexability

If you need to analyze all pages on the web and have a sitemap available, you can insert its link into the tool. To do so, click on the Import From URL section in the left sidebar.

Bulk URL indexability Marketing Miner - sitemap

For a report to be easily identifiable, it is good to name it. Just click on Your Report Name field.

Then click on the flag to select the country for which you want to get the data and finish by clicking on Next Step.

Selection of the Miner

In the Index Checking section, click on Miner URL indexability. Nothing else needs to be set. Click on Get Data to start processing your inputs. Once the report is complete, the processed data will be sent to you by email.

URL Indexability

Output example

Column description

  • Indexability status: Overall summary, whether the page is indexable for search engines. If it is not, you’ll find the reason in this column.
  • Robots.txt: Information whether the URL is blocked in robots.txt or not.
  • Link Canonical: Canonical in the source code (if any).
  • Canonical Header: Canonical in the header (if any).
  • X-robots-tag: If an x-robots-tag is sent in the header, it can block the robot from browsing the page. If the column is empty, everything is OK.
  • Meta robots: The meta tag robots you have on the page. If you don’t have them on the page, there will be a description “missing” in the column value.
  • rel = “next”: Content rel = “next” in pagination.
  • rel = “prev”: Content rel = “prev” in pagination.

Output analysis

You already have an output and now what? Let’s take a look at a few necessary things that you should not overlook:

  • First, check out the indexability status column and see if your page is indexable. If it is not, find out why. When you are aware of this situation, it’s okay. But it is often the case that you are not and someone unknowingly made the URL non-indexable. Unfortunately, it happens very often within the IT department.
  • You can also see in robots.txt whether the page is blocked for search engines or not. Or you can review the Meta robots column where are the tags as indexnoindexfollownofollowNoindex means a non-indexable page. If this is a situation that happened by mistake, change the noindex attribute to index or delete the URL from robots.txt.
  • If you don’t have a large or very complicated site, we don’t recommend to experiment with index and no index. Try to have each URL indexable, unless you have a very good reason to do the opposite.
  • If you have URLs on the web that you don’t want to index and Marketing Miner discovered these URLs in your sitemap, it is good to clean your sitemap. It should not contain URL’s that you don’t want to have in the search index.
Was this article helpful?

Continue reading

Previous: Ahrefs
Next: Contact Finder
Have questions? Search our knowledgebase.