Question :- What Is Robots.txt File ?
Answer :- Robots.txt is the plain text file which is human readable lying in the “wwwroot” directory of any website.In there some protocols / Rules are used that user is allowed to see all contents of the website or user is disallowed to see all contents of the website.Manually web developer can also manipulate it. In that they put some directory of the website and set a protocol in such manner that users are disallowed or allowed below shown folders of the websites.
Some Basic Types Of “Robots.Txt” Files
(1) Default / Basic Robots.Txt File :-
User-agent: *
Disallow: /
(2) This will disallow users to surf this pages from the websites or directly through the google with the help of the google dork.
User-agent: *
Disallow: /admin/
Disallow: /admin/images/
Disallow: /cgi-bin/
http://twitter.com/robots.txt
http://facebook.com/robots.txt
http://indishell.in/robots.txt
http://www.youtube.com/robots.txt
http://www.demonoid.com/robots.txt
http://www.myspace.com/robots.txt
0 comments: