Skip to main content

Apply meta robots tags to improve indexing

Written by Kristina Bethea
Customer Success Engineer at MindTouch
This page applies to:MindTouch Responsive

robot 2.jpg
Replacing meta.robots with robots.txt will provide faster and more accurate indexing for your end users. Below are a few examples of some benefits when using the meta tag:

  • The meta tag tells robots to de-index individual files, whereas robots.txt is used to de-index entire directories.
    • Since robots.txt can be applied to an entire directory, this will save you time and effort to be able to manage your content with various search engines.
  • The meta tag is superior to the robots.txt when excluding pages using "noindex."
    • As robots.txt can block out entire directories, it can become a problem when it is searched in Google when the pages accumulates links because the links never gets crawled, whereas just using <meta name = "robots" content = "noindex">, it will exclude individual pages from the search.
  • In robots.txt, crawlers are asked which pages you have to crawl and which pages you have to exclude, but the crawler is not being asked to not index the excluded pages.
    • If your customers wish to exclude pages that need to not be indexed in the results, using the meta tag will accomplish this with one line of code applied to the page.

For example: Let's say you have a directory labeled /library/files/search. With robots.txt you will use "disallow" to block the directory. When the URL is searched in Google regarding your site (, Google may index thousands of pages from that disallowed directory. Since the search engine has not crawled the URL's, they will appear as a URL when indexed.

What is robots.txt?

Robots.txt is a text file that webmasters can create to instruct robots (mainly search engine robots) how to index pages on their site. Meta tags (<meta>) are used to tell the robots not to index the contents of a page. Meta tags should always be placed in the head section of the HTML. The example below shows how the <meta> tag is used to not index a page:

        <title>Meta Example</title>
        <meta name="robots" content="nonidex, nofollow">

What is meta.robots?

seo 2.jpg

Meta.robots, or the robots meta tag tells the search engine not to show a specific page in the search results, as stated above. By default, most search engines index pages in which they find.

When using the <meta> tag:

  • Note that robots can ignore the <meta> tag, especially malware robots that scan the site used by spammers.
  • The "nofollow" attribute only applies to links on the desired page.
  • Do not confuse "nofollow" with rel="nofollow" link attribute.
    • rel="nofollow" is an attribute that can be set on a HTML <a> link tag.
      • example: <a href="happy.jpg" rel="nofollow">Happy</a>
    • These links do not get credited when Google ranks sites in search results.

How are meta tags are used?

If you want to tell the search engine to index your pages, the meta tag will look like this:

<meta name=”robots” content=”index”>

You can also apply the "follow" command which will tell the search engine to first index the page, and then follow any links found on that page to other pages; which will look like this:

<meta name=”robots” content=”index, follow”>

If you do not want pages to appear in the search engine, the meta tag below looks like this:

<meta name=”robots” content=”noindex”>


Take a look at a list of various meta functions that you can place in a DekiScript block for your environment.

  • Was this article helpful?