Skip to main content
MindTouch Success Center

Apply meta robots tags to improve indexing

Replacing meta.robots with robots.txt will provide faster and more accurate indexing for your end users. Below are a few examples of some benefits when using the meta tag:

  • The meta tag tells robots to de-index individual files, whereas robots.txt is used to de-index entire directories.
    • Since robots.txt can be applied to an entire directory, this will save you time and effort to be able to manage your content with various search engines.
  • The meta tag is superior to the robots.txt when excluding pages using "noindex."
    • As robots.txt can block out entire directories, it can become a problem when it is searched in Google when the pages accumulates links because the links never gets crawled, whereas just using <meta name = "robots" content = "noindex">, it will exclude individual pages from the search.
  • In robots.txt, crawlers are asked which pages you have to crawl and which pages you have to exclude, but the crawler is not being asked to not index the excluded pages.
    • If your customers wish to exclude pages that need to not be indexed in the results, using the meta tag will accomplish this with one line of code applied to the page.

For example: Let's say you have a directory labeled /library/files/search. With robots.txt you will use "disallow" to block the directory. When the URL is searched in Google regarding your site (, Google may index thousands of pages from that disallowed directory. Since the search engine has not crawled the URL's, they will appear as a URL when indexed.

What is robots.txt?

robots.txt is a text file that webmasters can create to instruct robots (mainly search engine robots) how to index pages on their site. Meta tags (<meta>) are used to tell the robots not to index the contents of a page. Meta tags should always be placed in the head section of the HTML. The example below shows how the <meta> tag is used to not index a page:

        <title>Meta Example</title>

What is meta.robots?

meta.robots or the robots meta tag tells the search engine not to show a specific page in the search results, as stated above. By default, most search engines index pages in which they find.

Things to keep in mind when using the <meta> tag

  • Robots can ignore the <meta> tag, especially malware robots that scan the site used by spammers.
  • The "nofollow" attribute only applies to links on the desired page.
  • Do not confuse "nofollow" with rel="nofollow" link attribute.
    • rel="nofollow" is an attribute that can be set on a HTML <a> link tag.
      Example: <a href="happy.jpg" rel="nofollow">Happy</a>
    • These links do not get credited when Google ranks sites in search results.

Use meta tags

If you want to tell the search engine how to index your pages, choose the appropriate meta tag and place in the Header template of your MindTouch site:

  1. Navigate to Site tools > Dashboard Branding > Header template
  2. Open the Header template in Edit mode.
  3. Copy one of the following meta tags into the template:

To index pages:


To first index pages, and then follow any links found on that page to other pages:


To hide pages from search engines:



Review the list of various meta functions that you can place in a DekiScript block for your environment.

  • Was this article helpful?