Need Help - Robots.txt File in indexu5.4.0

[bugsdirectory]

Hello,

I put google adsense in my indexu5.4.0 directory sites and I keep getting the google crawl error message ( some page blocked, can't be crawled by google ) because of the Robots.txt File. My sites using the older version of indexu don't get this error message.

May I know if I delete the Robots.txt File from indexu 5.4.0, what will happen?
Any security issue to the site?

Please help.

Thank you.



Error message below:


Blocked URL Reason Blocked [?] Last Crawl Attempt

add. php Robots.txt File Dec 2, 2007
add. php? cat= 316 Robots.txt File Dec 5, 2007
add. php? cat= 47 Robots.txt File Nov 30, 2007
login. php? f= 1&b= %2Fbecome_editor. php%3Fcat%3D39 Robots.txt File Nov 29, 2007
login. php? f= 1&b= %2Fcp%2F Robots.txt File Dec 5, 2007
review. php? id= 1253 Robots.txt File Dec 2, 2007
review. php? id= 42864 Robots.txt File Dec 2, 2007
review. php? id= 62268 Robots.txt File Dec 4, 2007
review. php? id= 7612 Robots.txt File Nov 29, 2007
tell_friend. php? id= 59182 Robots.txt File Dec 5, 2007
upgrade. php Robots.txt File Dec 2, 2007

[Bruceper]

You do not want to remove the robots.txt, here's why.

All those pages that are listed in robots.txt are "useless" pages that contain no real information (such as web forms which search engines don't care about) or duplicate information which search engines absolutely hate.

Duplicate information will get your site penalized and you could actually get dropped from some search engines.

It's better to leave robots.txt as it is.

However, if you really want to, you can delete it, and it is not a security issue.

[bugsdirectory]

Thanks for the advise, Bruce