Meta Robots Tag Analyzer
Analyse every robots directive on any page — meta tags, X-Robots-Tag headers, bot-specific overrides, and advanced directives like max-snippet and max-image-preview.
Tool created by iNet Ventures
Every Meta Robots Directive Explained
| Directive | Effect | Severity |
|---|---|---|
| noindex | Removes page from search results | Critical |
| nofollow | Stops PageRank passing through links on this page | Warning |
| none | Equivalent to noindex + nofollow | Critical |
| noarchive | Removes cached copy link from search results | Warning |
| nosnippet | Removes text snippet and prevents caching | Warning |
| noimageindex | Stops images on this page from being indexed | Warning |
| max-snippet:-1 | No limit on snippet length (best for SEO) | OK |
| max-snippet:0 | No text snippet shown in results | Critical |
| max-image-preview:large | Allows large image previews in Discover | OK |
| max-image-preview:none | No image preview in results | Critical |
| max-video-preview:-1 | No limit on video preview length | OK |
| unavailable_after:date | Removes page from results after specified date | Warning |
| notranslate | Prevents Google from offering page translation | Info |
| noodp | Deprecated — no effect | Info |
How Meta Robots Tags Work
When Googlebot (or any other crawler) fetches a page, it checks two places for robots instructions before deciding whether to index or follow links:
HTML Meta Tag
Declared inside the <head> of your HTML. Only applies to HTML documents. Multiple bot-specific variants can coexist — e.g. <meta name="robots"> for all bots and <meta name="googlebot"> for Google-specific overrides.
<meta name="robots" content="noindex, nofollow">X-Robots-Tag HTTP Header
Set in the HTTP response headers by the server. Works on any file type — HTML pages, PDFs, images, videos. If the same directive appears in both the header and the meta tag, the most restrictive one takes precedence.
X-Robots-Tag: noindex, nofollowImportant: If Googlebot is blocked from crawling a page via robots.txt, it cannot read the meta robots tag at all — so noindex won't work. You need to allow crawling but set noindex in the tag. This is one of the most common technical SEO mistakes.
Bot-Specific Tags: How Precedence Works
You can set different robots directives for different search engines. Each bot reads its own named tag first, then falls back to the generic robots tag if no specific one exists.
<meta name="robots">All botsThe default fallback for any crawler that doesn't have a specific tag.
<meta name="googlebot">Googlebot onlyOverrides the robots tag for Google's crawler. Use this when you want Google to behave differently from other engines.
<meta name="bingbot">Bing's crawlerSpecific override for Microsoft's Bing crawler.
<meta name="googlebot-news">Google NewsControls Google News crawler specifically — useful for excluding old news articles.
X-Robots-Tag: googlebot: noindexHTTP header, GooglebotBot-specific X-Robots-Tag header — applies only to the named bot.
5 Meta Robots Mistakes That Kill Rankings
🚨Noindex left on after launch
The most catastrophic mistake in SEO. Staging sites are routinely set to noindex during development — and then the tag gets deployed to production. An entire site disappears from Google overnight. Always check your live meta robots tag after every major deployment.
🔒Blocking crawling AND setting noindex
Setting robots.txt Disallow AND a meta noindex tag is redundant — but the real danger is setting noindex on a crawl-blocked page expecting Google to remove it. If Googlebot can't crawl the page, it can't read the noindex. Remove the disallow, let Google crawl it, and noindex will do the rest.
📉Missing max-image-preview:large
Most pages don't explicitly set max-image-preview. Without it, Google defaults to standard-size previews in Discover. Adding max-image-preview:large to your robots tag is one of the easiest wins for content publishers — Discover traffic can be 5-10x search traffic on popular articles.
🔁Conflicting meta and HTTP header directives
If your server sends X-Robots-Tag: noindex but your HTML says index, Google gets conflicting signals. This usually happens when a CDN, reverse proxy, or plugin adds its own X-Robots-Tag header without the developer knowing. Our tool detects and flags both.
👻Nofollow at the page level instead of link level
Setting nofollow on the entire page cuts off ALL link equity flow — including to your own internal pages. This is almost never intentional. If you want to nofollow specific external links, apply rel="nofollow" to those individual anchor tags, not to the whole page.
The Optimal Robots Tag for Most Pages
For most indexable pages, this single meta tag gives Google maximum flexibility for rich results:
<meta name="robots" content="index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1">
- index — explicitly allow indexing
- follow — explicitly allow link following
- max-image-preview:large — enables large images in Google Discover
- max-snippet:-1 — no snippet length restriction
- max-video-preview:-1 — no video preview restriction
Complete Your Technical SEO Audit
Technical SEO Sorted — Now Build Links
Getting your robots tags right is the foundation. High-quality backlinks are what drive rankings. Our blogger outreach team builds real editorial dofollow links.
Explore Blogger OutreachFrequently Asked Questions
Everything you need to know about meta robots tags and directives