Configure WordPress robots.txt

Last Revised: October 2, 2021

The robot file.txt that must be in the root folder of the hostname of the installation (even if your WordPress is in a folder), is a file that does not come by default in WordPress, although if you call it the system generates a simple one, and that you should create.

This text file tells search bots (such as GoogleBot, BingBot, YandexBot, Slurp!…) what they should and should not analyze about your site.

What should we block from WordPress in this file? In principle nothing. Nowadays you can get to block many elements through the internal configurations of security plugins, so if you want to have a secure site it is best not to give information in this file.

In this way, the file will have a simple content:

User-Agent: *

By default WordPress generates a virtual file with the following content:

User-Agent: *
Disallow: /wp-admin/
Disallow: /wp-admin/admin-ajax.php

Although this exposes is not serious, taking into account that access to /wp-admin/ the must previously go through a user access, it is still a very clear clue that this site is built with WordPress.

Seguir con Seguridad para WordPress

About this document

This document is regulated by the EUPL v1.2 license, published in WP SysAdmin and created by Javier Casares. Please, if you use this content in your website, your presentation or any material you distribute, remember to mention this site or its author, and having to put the material you create under EUPL license.