What is a robots.txt file and how do you add an entry to this file?

What is a robots.txt file?

The robots.txt file is a simple text document that serves as an official instruction for search engines (such as Google). 

Before the algorithms of these search engines begin scanning a website, they first check this file to find out which sections they may index and which they should skip. Thanks to this, the publisher gains control over what appears in public search results. 

Using the robots.txt file, you can block access to selected subpages, such as the admin panel, draft versions of articles, pages under construction, or private website files.

Robots.txt is based on two basic directives:

• User-agent: Specifies which search engines a given rule applies to. The asterisk (*) means that the instruction applies to all indexing bots on the internet.
• Disallow: Indicates a specific path (subpage or folder address) that should be excluded from scanning.

How do you add an entry to the robots.txt file?

Ads4media → cms4media portals → Edit → Settings → Rules for search engines - robots.txt file

In the ADS4media panel, go to the [cms4media portals] module. Find the portal in the list and go to its edit page. 

Find the Settings tab. There you will find the Rules for search engines - robots.txt file field.

Enter the rule and click Save.