This Blog will give you information on Digital Marketing Tips.

Friday 2 August 2019

What is Robot.txt file in SEO


What is Robot.txt file in SEO

Robot.txt is a text file that tells search engine robots which pages on your sites to crawl.
Its also tells web robots which pages not to crawl.


Steps to create Robot.txt file for blog
1.) Open website https://ctrlq.org/blogger/
2.)  Put website in generate sitemap option and press enter
What is Robot.txt file in SEO







                                                                                                                                                                                                                                                                                                                      

3.)  After clicking enter your robot.txt file will generate




What is Robot.txt file in SEO
Add caption
















4.)  you can copy  all that blue portion and paste in blogger and click on save option 
      go in blogger setting option - Search Preference - Click on custom robot.txt   Edit
      and paste all that blue portion text there and click on save button. 


What is Robot.txt file in SEO














Steps to Submit Updated robot.txt to Google
1.Put in google robots.txt tester and press enter

2. After press enter you will see interface like this
3. Click on 2 option which marked in blue after you will see this interface
What is Robot.txt file in SEO















4.Paste in that box generated robot.txt data and click on submit.This submit function of robot.txt tester allow you to ask google to more quickly crawl and index a new robot.txt file for your site.

5.you can also test your website weather its block or not by putting your URL below 
and click on Test.
What is Robot.txt file in SEO














Blocking all web crawlers from all content
User agent : *
Disallow : /

Allowing all web crawlers to access all content
User agent : *

1 comment:


  1. Thanks for sharing such a piece of wonderful information. Your knowledge about the topic is great please keep sharing.
    Digital Marketing Course in Delhi




    ReplyDelete