A Web Robot (also known as a crawler, spider or search engine bot) is a program that traverses the Web automatically. 1 things they are used by is search engines to download and index web content. A website's robots.txt file communicates with these robots using the Robots Exclusion Protocol. It can be used to inform the robot which areas of the site should and shouldn't be scanned. A robots.txt file need to be located the site's root directory.

Example

The robots.txt file below communicates:

• ▪ block OpenAI, Google Bard and Common Crawl bots from crawling,
• ▪ all other bots are allowed to crawl the entire site and
• ▪ the site's sitemap file is located at https://garden.bradwoods.io/sitemap.xml.

User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://garden.bradwoods.io/sitemap.xml

Tools

• robots.txt Tester

References

• How to block AI Crawler Bots using robots.txt file
• Introduction to robots.txt
• The Web Robots Pages
• What is a web crawler? | How web spiders work

Feedback

Have any feedback about this note or just want to comment on the state of the economy?