noindex

Definition

noindex is an instruction telling search engines not to include a page in their index. Common implementations are meta robots (<meta name="robots" content="noindex">) or the X-Robots-Tag HTTP header.

Why it matters

• Keep low-value pages (search results, test pages) out of SERPs
• Avoid diluting quality signals with thin/duplicate content
• Manage index scope alongside canonical and sitemaps
• Protect paywalled or member-only content from free search access
• Hide development pages or staging environments
• Manage pagination: prevent /page/2, /page/3 from consuming index quota

How to implement

• Add meta robots noindex (or X-Robots-Tag)
• Do not list noindex URLs in your sitemap
• If crawling is useful, use noindex,follow
• Don't block with robots.txt: bots won't see the noindex directive
• Regularly check Search Console's 'Excluded' report to confirm
• Auto-add noindex via code for search results, filters, pagination
• Test with URL inspection tool to verify what crawlers see

Examples

<!-- Basic noindex -->
<meta name="robots" content="noindex, follow" />

<!-- Target specific search engines -->
<meta name="googlebot" content="noindex" />
<meta name="bingbot" content="noindex" />

// React dynamic noindex
function PageHead({ shouldIndex = true }) {
  return (
    <Helmet>
      <meta
        name="robots"
        content={shouldIndex ? 'index, follow' : 'noindex, follow'}
      />
    </Helmet>
  );
}

// Usage: search results page
<PageHead shouldIndex={false} />

Related