What’s the difference between Disallow and noindex?

Disallow (robots.txt) blocks crawling; noindex (meta robots or X-Robots-Tag) blocks appearing in SERPs. For noindex to work, bots must be able to crawl the page/response to see the directive.

Search Console says “Crawled - currently not indexed”. What now?

Rule out technical issues first: status codes, noindex, canonical, robots.txt, and whether content is visible (no login wall, not hidden behind delayed JS). Then evaluate content: thin/duplicate content or intent mismatch.

Should I include noindex URLs in my sitemap?

No. A sitemap should only list indexable canonical URLs you want indexed. Listing noindex URLs increases confusion and wastes crawl resources.

Indexability Checklist

If you want to rank globally, step zero is always the same: make sure search engines can crawl, understand, and index your pages. This guide is a developer-friendly troubleshooting flow with links to our tools and glossary.

Start here: Crawl vs Index

Crawlable: can bots fetch the response and content? (robots.txt, login walls, status codes)
Indexable: will search engines show it in SERPs? (noindex, canonical, quality/duplication)

Glossary: meta robots, X-Robots-Tag, canonical URL

1) HTTP status: make sure important pages return 200

Indexing starts with fetchable content. Key pages should return 200. Long-term 3xx chains, or 4xx/5xx responses lead to unstable indexing.

http

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8

Glossary: 301 redirect, 302 redirect

2) robots/noindex: don’t block yourself

Most “the site disappeared from Google” incidents come down to robots/noindex. Remember:

robots.txt controls crawling, not indexing
meta robots / X-Robots-Tag controls indexing
For noindex to work, bots must be able to crawl the page/response to see it

html

<!-- HTML meta robots -->
<meta name="robots" content="noindex, follow" />

http

# HTTP header (useful for PDFs/non-HTML)
X-Robots-Tag: noindex, nofollow

Glossary: noindex, robots.txt

3) canonical: consolidate duplicates

If the same content exists on multiple URLs (parameters, sorting, tracking codes, locale paths), Google may pick the wrong one to rank. Canonicals plus consistent internal links help consolidate signals.

html

<link rel="canonical" href="https://seo.lucas-futures.com/en/tools/indexability-checker" />

Glossary: duplicate content

4) JavaScript rendering: ship per-route HTML for important pages

CSR/SPAs can work, but for top competitiveness your key content pages should have real HTML immediately (SSR/SSG/prerender). Otherwise bots must render JS, increasing delay and uncertainty.

Glossary: JavaScript SEO, prerendering, SSG, SSR

Fast debugging flow (in order)

Check HTTP: 200? redirect chains?
Check directives: meta robots / X-Robots-Tag accidental noindex?
Check canonical: self or correct preferred URL?
Check robots.txt: any Disallow for critical paths/assets?
Check rendering: is important content present in View Source?

Tool: Indexability Checker

FAQ

Common indexability troubleshooting questions.