I do not possess a robots.txt file during my root folder however i can can get on through example.com/robots.txt , it's shows the information:
User-agent: * Disallow:
After a little research on the internet I learned that it's produced quickly as my website is within wordpress. Can somebody assist me to preventing robot.txt quickly creation in wordpress?
This really is produced in wordpress-includes/functions.php, line 1845 (in 3.3.1) and you will filter the end result:
echo apply_filters('robots_txt', $output, $public);
So, if you do not would like it to contain anything, are you able to give a filter and return a clear string. WordPress uses this in reaction towards the blog/site's privacy setting (and $public will get set to '0' when the setting would be to block search engines like google). Hope this can help!