What is Robots.txt?Every blogger should have to add robots.txt file on their blogger blog to manage their content on search engine. If you don't manage your blog through robots all the peoples can get access to your private pages, which you don't have to show to all people.
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: http://yoursite.blogspot.com/feeds/posts/default?orderby=UPDATEDIf you have added custom domain on your blogger blog then you can replace sitemap section with your current domain name. i.e http://yoursite.com/feeds/posts/default?orderby=UPDATED,
If you have XML site map then again you can replace your sitemap section i.e http://yoursite.com/feeds/posts/default?orderby=UPDATED with http://yoursite.com/sitemap.xml thats it.
Explanation of Robots.txt Terms,User-agent: Mediapartners-Google – This terms for Google adsense that is used to server better relevant ads on your site based on your content.
How to Add Custom Robots,
- Go to blogger Dashboard, Now go for Setting and Click on Search Preference, Now its time to scroll and you will get Crawlers and indexing section, There you will fine Custom robots.txt Go for edit and click on Yes button to add custom robots txt.
- Now copy above robot.txt code and paste it on blank space. Then you just have to change at the sitemap section with your sitemap file. http://yoursite.com/feeds/posts/default?orderby=UPDATED that's it.