robotstxt-ai
MCP ServerAnalyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
Install
claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcpWhat is robotstxt-ai?
Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
Reviews
Log in to leave a review.
No reviews yet — be the first.
Explore related
Other things in this space — across every part of the ecosystem, not just mcp servers.
MCP Serverssimilar to this one
All mcp servers →Skills
All skills →link-rot-scanner
Crawl and validate all internal and external links across an AEM Edge Delivery Services site. Uses the query index or sitemap to discover pages, extracts links from .plain.html renditions, checks HTTP status codes, and produces a prioritized report of broken, redirecting, and insecure links. Use whe
151 stars
extract
Crawl an existing website (capped, multi-page) and seed stardust/current/ with PRODUCT.md, DESIGN.md, DESIGN.json, a per-page inventory, and the consolidated brand surface — the captured design system, palette, typography, motifs, and voice of the live site. Use when the user wants to analyze an exi
151 stars
scrape-webpage
Use this when the page-import pipeline needs to fetch a source webpage and prepare it for import/migration to AEM Edge Delivery Services. Covers scraping content, extracting metadata, downloading images, and returning analysis JSON with paths, metadata, cleaned HTML, and local images. Do not invoke
151 stars
Plugins
All plugins →exa
Exa AI web search, deep research, and content extraction. Provides MCP tools and research skills for comprehensive web search, people discovery, company research, academic papers, and more.
4.8K stars
cli
CLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agents
549 stars
tavily
Build AI applications with real-time web data using Tavily's search, extract, crawl, and research APIs.
436 stars