About StackScopeBot

StackScopeBot is our crawler that analyses new product launches to detect their tech stacks.

User-Agent

StackScopeBot/1.0 (+https://stackscope.dev/bot)

What it does

  • Fetches the main page of each product launch (one HTML request)
  • Renders the page in a headless browser to capture a screenshot and the final DOM (this executes JavaScript)
  • Fetches up to 5 linked stylesheets to analyse CSS
  • Checks a few well-known files: robots.txt, llms.txt, security.txt, sitemap.xml, ads.txt, humans.txt
  • Looks up DNS records (MX, TXT, NS, CNAME, DNSKEY) for the domain
  • Performs a TLS handshake to read the SSL certificate
  • Looks up the domain's ASN (via Team Cymru DNS) and registration date (via RDAP)
  • Fetches privacy policy and terms of service pages if linked from the homepage

What it doesn't do

  • Does not spider your site, only the homepage and explicitly linked legal/well-known pages
  • Does not submit forms, click buttons, or interact with the page
  • Does not attempt logins or access authenticated areas
  • Re-crawls only on explicit user request, not automatically

Crawling etiquette

  • Default rate: at most 1 request every 3 seconds globally, never parallel requests to the same domain
  • Respects robots.txt rules for both StackScopeBot and wildcard (*) user agents
  • Honours Crawl-delay directives, if your robots.txt specifies a longer delay, we follow it
  • Does not attempt to crawl sensitive paths (admin, login, API endpoints)

How to control access

To block StackScopeBot entirely, add this to your robots.txt:

User-agent: StackScopeBot
Disallow: /

To request a slower crawl rate:

User-agent: StackScopeBot
Crawl-delay: 10

Removal requests

If you'd like your site removed from StackScope, or have any questions about our crawler, please contact us at [email protected].