cen-force.com — AI Crawler Readiness
Public AI crawler readiness report for cen-force.com: score, bot access, discovery files, and actionable recommendations.
Readiness summary
Score: 30/100 · Grade: F
Last checked:
AI crawler access
- CCBot — Allowed (Common Crawl)
- AI2Bot — Allowed (Allen Institute for AI)
- GPTBot — Allowed (OpenAI)
- YouBot — Allowed (You.com)
- Diffbot — Allowed (Diffbot)
- Timpibot — Allowed (Timpi)
- Amazonbot — Allowed (Amazon)
- ClaudeBot — Allowed (Anthropic)
- Cohere-ai — Allowed (Cohere)
- Omgilibot — Allowed (Webz.io)
- Bytespider — Allowed (ByteDance)
- Claude-User — Allowed (Anthropic)
- FacebookBot — Allowed (Meta)
- ChatGPT-User — Allowed (OpenAI)
- ImagesiftBot — Allowed (Hive / ImageSift)
- anthropic-ai — Allowed (Anthropic)
- DuckAssistBot — Allowed (DuckDuckGo)
- OAI-SearchBot — Allowed (OpenAI)
- PerplexityBot — Allowed (Perplexity)
- Google-Extended — Allowed (Google)
- Perplexity-User — Allowed (Perplexity)
- Claude-SearchBot — Allowed (Anthropic)
- Applebot-Extended — Allowed (Apple)
- Meta-ExternalAgent — Allowed (Meta)
- Meta-ExternalFetcher — Allowed (Meta)
Discovery and page signals
- llms.txt: ✕
- ai.txt: ✕
- JSON-LD: ✕
- FAQ Schema: ✕
- Meta description: ✕
- Open Graph: ✕
Recommendations
- The page could not be fetched — AI crawlers may also be unable to reach it.
- Live AI-crawler probe blocked/unreachable for: GPTBot (HTTP 526), ClaudeBot (HTTP 526), PerplexityBot (HTTP 526). Your server or WAF may block AI bots even when robots.txt allows them.
- Add JSON-LD structured data for AI comprehension
- Add FAQPage schema for AI answer extraction
- Add a meta description for AI summaries
- Add Open Graph tags (og:title, og:description)
- Optional: add /llms.txt to help AI systems understand your site
- Optional: add /ai.txt for AI usage policies
- Add a robots.txt file to control crawler access