Understanding Robots.txt Crawl Directives
A **robots.txt** file resides in the root directory of your server. It instructs search engine crawlers (such as Googlebot, Bingbot, or DuckDuckBot) which sections of your site they should avoid requesting.
If your robots.txt file misconfigures disallow routes (e.g. including `Disallow: /` which blocks all indexing), search engines will stop indexing your site pages, causing search ranking drops.