ActableSite AI Crawler Monitor
MCP ServerCheck AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
Install
claude mcp add --transport http ai-crawler-robots-monitor https://actablesite.com/mcp/What is ActableSite AI Crawler Monitor?
Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
Reviews
Log in to leave a review.
No reviews yet — be the first.
Explore related
Other things in this space — across every part of the ecosystem, not just mcp servers.
MCP Serverssimilar to this one
All mcp servers →Skills
All skills →link-rot-scanner
Crawl and validate all internal and external links across an AEM Edge Delivery Services site. Uses the query index or sitemap to discover pages, extracts links from .plain.html renditions, checks HTTP status codes, and produces a prioritized report of broken, redirecting, and insecure links. Use whe
151 stars
extract
Crawl an existing website (capped, multi-page) and seed stardust/current/ with PRODUCT.md, DESIGN.md, DESIGN.json, a per-page inventory, and the consolidated brand surface — the captured design system, palette, typography, motifs, and voice of the live site. Use when the user wants to analyze an exi
151 stars
scrape-webpage
Use this when the page-import pipeline needs to fetch a source webpage and prepare it for import/migration to AEM Edge Delivery Services. Covers scraping content, extracting metadata, downloading images, and returning analysis JSON with paths, metadata, cleaned HTML, and local images. Do not invoke
151 stars
Plugins
All plugins →exa
Exa AI web search, deep research, and content extraction. Provides MCP tools and research skills for comprehensive web search, people discovery, company research, academic papers, and more.
4.8K stars
cli
CLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agents
549 stars
tavily
Build AI applications with real-time web data using Tavily's search, extract, crawl, and research APIs.
436 stars