sharing.tw — AI Crawler Readiness
Public AI crawler readiness report for sharing.tw: score, bot access, discovery files, and actionable recommendations.
Readiness summary
Score: 2/100 · Grade: F
Last checked:
AI crawler access
- CCBot — Blocked (Common Crawl)
- AI2Bot — Blocked (Allen Institute for AI)
- GPTBot — Blocked (OpenAI)
- YouBot — Blocked (You.com)
- Diffbot — Blocked (Diffbot)
- Timpibot — Blocked (Timpi)
- Amazonbot — Blocked (Amazon)
- ClaudeBot — Blocked (Anthropic)
- Cohere-ai — Blocked (Cohere)
- Omgilibot — Blocked (Webz.io)
- Bytespider — Blocked (ByteDance)
- Claude-User — Blocked (Anthropic)
- FacebookBot — Blocked (Meta)
- ChatGPT-User — Blocked (OpenAI)
- ImagesiftBot — Blocked (Hive / ImageSift)
- anthropic-ai — Blocked (Anthropic)
- DuckAssistBot — Blocked (DuckDuckGo)
- OAI-SearchBot — Blocked (OpenAI)
- PerplexityBot — Blocked (Perplexity)
- Google-Extended — Blocked (Google)
- Perplexity-User — Blocked (Perplexity)
- Claude-SearchBot — Blocked (Anthropic)
- Applebot-Extended — Blocked (Apple)
- Meta-ExternalAgent — Blocked (Meta)
- Meta-ExternalFetcher — Blocked (Meta)
Discovery and page signals
- llms.txt: ✕
- ai.txt: ✕
- JSON-LD: ✕
- FAQ Schema: ✕
- Meta description: ✕
- Open Graph: ✕
Recommendations
- Page is marked noindex (meta/X-Robots-Tag) — AI search indexes will skip this content.
- Unblock AI crawlers in robots.txt: GPTBot, ChatGPT-User, Google-Extended
- Add JSON-LD structured data for AI comprehension
- Add FAQPage schema for AI answer extraction
- Add a meta description for AI summaries
- Add Open Graph tags (og:title, og:description)
- Optional: add /llms.txt to help AI systems understand your site
- Optional: add /ai.txt for AI usage policies