Gopher
TEMPLATES

Robots.txt File

Hugo can generate a customized robots.txt in the same way as any other template.

To create your robots.txt as a template, first set the enableRobotsTXT value to true in your configuration file . By default, this option generates a robots.txt with the following content, which tells search engines that they are allowed to crawl everything:

User-agent: *

Robots.txt Template Lookup Order  

The lookup order for the robots.txt template is as follows:

  • /layouts/robots.txt
  • /themes/<THEME>/layouts/robots.txt

Robots.txt Template Example  

The following is an example robots.txt layout:

layouts/robots.txt

User-agent: *

{{range .Pages}}
Disallow: {{.RelPermalink}}
{{end}}

This template disallows all the pages of the site by creating one Disallow entry for each page.