Left ArrowBack

Notes

robots.txt

Table of Contents

The icon of the Adeptus Mechanicus. A cog with a skull at the center.
A sci-fi octopus-like robot.

robots.txt

Planted: 

Status: decay

A Web Robot (also known as a crawler, spider or search engine bot) is a program that traverses the Web automatically. 1 things they are used by is search engines to download & index web content. A website's robots.txt file communicates with these robots using the Robots Exclusion Protocol. It can be used to inform the robot which areas of the site should & shouldn't be scanned. A robots.txt file need to be located the site's root directory.

Example

The robots.txt file below communicates:

  • ▪ all bots are allowed to crawl the entire site &
  • ▪ the site's sitemap file is located at https://garden.bradwoods.io/sitemap.xml.

/robots.txt

User-agent: *
Allow: /

Sitemap: https://garden.bradwoods.io/sitemap.xml
A space ship's docking bay with various equipment.

Resources

Tools
robots.txt TesterBy Google

Where to Next?

A sci-fi robot taxi driver with no lower body
Notes
    
  │
  └── Web Dev
        
  │
  └── Web
            
  │
  ├── Accessibility
            
Arrow pointing down
YOU ARE HERE
            
  │
  └── robots.txt