# Stock Portfolio Pro - robots.txt # Site: https://stockportfolio.pro # Updated: 2026-09-11 # Contact: support@stockportfolio.pro # # Policy change 2026-09-11: default flipped from allow-all to deny-all. Only # Google and Bing are crawlable now. Render's Hobby-plan bandwidth (5GB/mo) # was being consumed primarily by bot/crawler traffic hitting the SEO page # surface — 68% of all site traffic was bots, and Meta's AI crawler alone # was 53% of ALL traffic. backend/bot-blocker.js enforces this same policy # server-side (403), since robots.txt is only a request a crawler can choose # to ignore. The two files must agree agent-by-agent — drift is caught by # backend/test/robots-policy.test.js. Full rationale in # docs/growth/bot-crawler-situation-2026-09-05.md. # # Denied here does not mean unavailable. Licensed bulk data (SEC as-filed # fundamentals plus filing-change deltas, every row carrying a sourceUrl back # to EDGAR) and MCP/API access for agent builders are offered at # https://stockportfolio.pro/licensing. That page, /llms.txt and this file stay # fetchable for every agent, allowed or denied. # # /login and /news carry and are deliberately left # crawlable for Google/Bing below so the noindex itself can be read and # honored (blocking them here caused a "blocked by robots.txt" Search # Console warning instead of a clean drop). /register stays crawlable too — # it's a conversion landing page. # --------------------------------------------------------------------------- # Default rule — deny by default; only Google and Bing are allowed below # --------------------------------------------------------------------------- User-agent: * Disallow: / Disallow: /v1/ Disallow: /v2/ # --------------------------------------------------------------------------- # Allowed — Google # --------------------------------------------------------------------------- User-agent: Googlebot Allow: / Disallow: /api/ Disallow: /dashboard Disallow: /dashboard.html Disallow: /fundamentals Disallow: /fundamentals.html Disallow: /v1/ Disallow: /v2/ User-agent: Googlebot-Image Allow: / Disallow: /api/ User-agent: Google-InspectionTool Allow: / Disallow: /api/ User-agent: Storebot-Google Allow: / Disallow: /api/ # Google's AI-training crawler — a DIFFERENT agent from Googlebot above. Kept # as its own explicit stanza so a future "let's allow Google" edit cannot # silently re-admit it. User-agent: Google-Extended Disallow: / # --------------------------------------------------------------------------- # Allowed — Bing # --------------------------------------------------------------------------- User-agent: Bingbot Allow: / Disallow: /api/ Disallow: /dashboard Disallow: /dashboard.html Disallow: /fundamentals Disallow: /fundamentals.html Disallow: /v1/ Disallow: /v2/ User-agent: BingPreview Allow: / Disallow: /api/ # --------------------------------------------------------------------------- # Refused — other search engines # --------------------------------------------------------------------------- User-agent: DuckDuckBot Disallow: / User-agent: YandexBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: Applebot Disallow: / User-agent: Applebot-Extended Disallow: / # --------------------------------------------------------------------------- # Refused — AI / LLM crawlers # --------------------------------------------------------------------------- User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: meta-externalagent Disallow: / # --------------------------------------------------------------------------- # Refused — backlink / SEO scrapers # --------------------------------------------------------------------------- User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / # --------------------------------------------------------------------------- # Refused — social-preview bots # Cost of this change: link previews/cards no longer render on these # platforms (Facebook, X, LinkedIn). Accepted tradeoff — see rationale above. # --------------------------------------------------------------------------- User-agent: FacebookBot Disallow: / User-agent: facebookexternalhit Disallow: / User-agent: Twitterbot Disallow: / User-agent: LinkedInBot Disallow: / # --------------------------------------------------------------------------- # Sitemaps # --------------------------------------------------------------------------- Sitemap: https://www.stockportfolio.pro/sitemap.xml