Meta Robots Tag

The meta robots tag is an HTML element placed in the head of a web page that gives search engines page-level instructions about how to treat that page. It takes the form , where the content value holds one or more directives separated by commas. The two best-known pairs are index/noindex, which control whether the page may be added to the search index, and follow/nofollow, which control whether the crawler follows the links on the page. Other values exist too — such as noarchive, nosnippet, and max-snippet — for finer control over how a page is treated and displayed. Because the tag is read per page, it lets you set rules for individual URLs rather than the whole site, and you can target a specific crawler by naming it (for example, name="googlebot") instead of the generic "robots".

The meta robots tag matters because it is the primary page-level control over how search engines index a page and handle its links. It is how you keep a confirmation page or an internal search result out of the index while still letting link signals flow, how you tell engines not to show a cached copy, or how you cap the length of a snippet. For non-HTML files, and for setting these rules server-wide, the same directives can be delivered in an X-Robots-Tag HTTP header instead, which does the same job without editing page HTML. Understanding the meta robots tag is foundational to technical SEO, because so many indexing outcomes — a page missing from search, links not being followed, a snippet suppressed — trace back to what this small tag in the head is telling crawlers to do.

The relationship people most often muddle is between the meta robots tag and noindex. Noindex is not a separate mechanism — it is one of the directive values the meta robots tag can carry. The meta robots tag is the container; noindex, index, follow, and nofollow are instructions inside it. So "add a noindex tag" almost always means setting the meta robots tag's content to include noindex. Because the tag can hold several directives at once, it does more than any single value: index/noindex govern the index, follow/nofollow govern link handling, and the two are independent, which is why noindex, follow is a common and sensible combination — keep the page out of the index but still pass its link signals onward.

The meta robots tag is also distinct from robots.txt and the canonical tag, though all three touch how search engines treat pages. Robots.txt is a site-wide file that controls crawling — whether a bot may fetch a URL at all — and it cannot reliably remove a page from the index; worse, blocking a page there can stop the crawler from ever reading the meta robots tag it needs to obey. The canonical tag handles duplication by naming the preferred version among similar pages, but it is a hint rather than an instruction and does not remove a page. The meta robots tag, by contrast, gives explicit page-level directives the crawler is expected to follow. The clean division of labor: robots.txt for crawling, canonical for duplicates, and the meta robots tag for page-level indexing and link directives — and a page must stay crawlable for its meta robots tag to be read.

Using the meta robots tag well means setting the right combination of directives for each page's purpose — noindex, follow for a page you want out of the index but still passing link signals; noindex, nofollow when you want neither; index, follow (the default) for pages you want found and their links followed. It means keeping pages you want indexed free of an accidental noindex, keeping noindexed pages crawlable so the directive is actually read, and using the X-Robots-Tag header when you need the same control for non-HTML files or across the server. It means auditing which pages carry which directives, because the meta robots tag quietly determines whether pages appear in search, and a wrong value can hide a valuable page or expose one you meant to exclude.

The failures are treating noindex as separate from the meta robots tag rather than as one value it carries; combining a meta robots noindex with a robots.txt block on the same URL, so the crawler never reads the directive; leaving a staging noindex on production pages, which silently drops them from search; and confusing the tag's job with robots.txt (crawling) or canonical (duplication). The discipline is to use the meta robots tag as the precise, page-level control it is — choosing directive combinations deliberately, keeping directed pages crawlable, auditing them regularly, and reserving robots.txt and canonical for the different jobs they do — so the tag governs indexing and link handling exactly as intended.

Meta robots tag — an HTML head tag carrying page-level directives like index/noindex and follow/nofollow — controls indexing and link handling, with noindex being one value it can hold.

Etymology: source.

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Areas of marketing where meta robots tag is a core concern: