What is Robots.txt?
Robots.txt is a text file placed at the root of your website that tells search engine crawlers which pages or sections they can and cannot access. It's the first file crawlers read when visiting your site.
Why Robots.txt Matters
A well-configured robots.txt prevents crawlers from wasting time on admin pages, API endpoints, and utility routes, preserving crawl budget for your important content pages.
Advanced Uses
- Block AI training bots while allowing search crawlers
- Reference your sitemap location
- Prevent indexing of staging environments
How FlowStrata Configures Robots.txt
FlowStrata's robots.txt allows full access for Googlebot and Bingbot while blocking AI training bots (GPTBot, CCBot, Google-Extended), we want search indexing, not free training data.