Robots.txt from IndexU.co.uk

[FSGDAG]

I hope its ok that I'm posting this here, but there really isn't a place on Inspireme's site to discuss this at all....

Under the tips and tricks section this is posted:

Robots.txt

IndexU uses a different style of link to get to the same pages on the hit new and top.php pages. Its important to add these to your robots txt file to avoid duplicate content and keep page rank concentrated on important pages.

User-agent: *
Disallow: add.php
Disallow: top_rated.php
Disallow: new.php
Disallow: pick.php
Disallow: tell_friend.php
Disallow: review.php
Disallow: bad_link.php
Disallow: /cp/

What struct me about this was the "to avoid duplicate content and keep page rank concentrated"... What about when a link is in multiple categories?? Is this the same as whats mentioned above?

Also, will Google penalize us is we dont have these in robots.txt??

[inspireme]

Its only on new.php you get links like this:
http://www.bigfreeguide.com/camping/...&dt=1203206400

google indexes them - you end up with 1000s of "pages" all with different date ids, and your page rank for the site isnt concentrated on the pages you want htem to but spread around to "chaff" like the above. Block it, youll do your site a favour. If you dont have it blocked have a look at google webmaster, when i didnt have it blocked it thoguth i had 25,000 pages! (i have about 2,000 )

tell friend, review, and add.php are pointless pages to have indexed, also you get links like add?cat=1 add?cat=2 etc, hundreds of pages the same.

It wont do your site much harm to let google index them, but in my opinion youll do yourself a favour to block them.

[inspireme]

by the way makes no difference to your browse pages, your link can be in any category... all the main "content pages" will be indexed with the above.

[seveneth]

For me, the robots.txt is to blocks pages that I do not want to see in search result. I have similar robots file, but I allow the main add.php and new.php file.

[Bruceper]

robots.txt is to block pages or directories that you do not want to be indexed. But it can also push search engines like Google to assist you with your SEO by removing duplicate information and ignoring information that isn't relavent and contains no real useful info or duplicate info such as add, new, upgrade, /cp/, editors pick etc.

[FSGDAG]

What about /user_details.php and /user_search.php?? Are they safe to add to the 'disallow' also??

[Bruceper]

user_details yes, because you dont want others to benefit from your site by having their profile indexed. for user_search it's the same thing, there is nothing usable by a search engine there other than a form.

Now this brings up a different way of using (abusing) search engines which is published search url's, otherwise known as SEO stupidity trick #1. If you placed a link on another site to the following URL http://www.your.com/user_search.php?...&username=NAME you can actually get that address picked up by search engines to index your own URL from your own profile!

So you could technically add links to other sites on your site that linked to your own profile so that you can benefit from it! It boggles the mind doesn't it? Of course others can do this to you as well unless you add user_detail to your robots.txt

[FSGDAG]

Just to update this thread for future visitors that read it... If you have Zubby's YouTube block installed, I would also suggest adding /video.php to your list of disallows. I found a ton of these in Google Webmaster.

[jromano]

If you have more than 10 listings per page, you will carry the same title and description on the following page when you click on 'Next'. Anyone has tried making the titles unique?

[Bruceper]

Already done, search the forum or take a look at indexusupport.com

[esm]

If you have more than 10 listings per page, you will carry the same title and description on the following page when you click on 'Next'. Anyone has tried making the titles unique?

are you referring to this thread for v5.4? Different Titles for Category Pages


.

[jromano]

Yes I was referring to that thread. You guys are unbelievably fast in replying on this forum! However, due to my numerous customizations on this script, I think I will need to make some modifications to the above recommended code. Thank you.

[FSGDAG]

You guys are unbelievably fast in replying on this forum!

Really its that we are all bored!!! When we get REALLY bored, we start talking about the license issues again!!!!

[esm]

...When we get REALLY bored, we start talking about the license issues again!!!!

I can see brucepr now beating the bushes for some new active members so the rest of us won't be bored...!


.

[Bruceper]

It's funny you say that. I need to confirm some things but I think someone made a big change on me today. A change for the better I think.