Sitemap issue

[mediaone]

Hi,

I have configured indexu as a product directory (using version 5.4).

I have modify all templates and use only about half of the indexu features (we don't use new, hot, pick and others).

I receive merchant catalogs (csv files) and we upload them directly in database in table idx-link with some customized field.

Site is running from 20 days and we have upload about ten catalogs: now there are 450 categories and 113,000 links (products). We think to reach 300,000 links in few time.

Indexu run well and it is rather quickly.

Sitemap is update every day with cron and we have submitted indexu sitemap to Google. All OK until about 90,000 links.
I have noticed that number of links read from Goggle in sitemaps are more than the double of links in idx-link, but I have not worried myself.

Yesterday we have an error in Google (webmaster tools): two of the six sitemap files exceeded the 50,000 links.
As I think that sitemaps from 2 to 7 would have to contain the pages detail and therefore the links (idx-links) that are 113,000, I don't understand why Google read over 230,000 links in sitemaps.

If sitemap.php run well, every sitemap file would have to contain approximately 18.000 links. It seems that links are not subdivided proportionally between the sitemaps and moreover I do not understand because they are thus many (the double).

Any idea?

Excuse my bad english.

[Bruceper]

sitemap1 contains pages (hot, pick, browsepr etc), categories and tags

This is probably your first error because I'm sure if you have that many listings that you have more than 50,000 tags in total

sitemap2 through 7 contain links (the detail pages). I do not know why they are not evenly distributed. However there should be a maximum of 50k per file as this is what sitemap.php specifies. You can add more sitemap files to distribute the links across more files.

Also if you take a look at sitemap.php, you can remove pages such as hot, pick etc from being indexed. Even if you do not link to them, they are still being indexed by Google because they are in your sitemap.

You could also remove the tags from the sitemap too.

[mediaone]

Thanks for answer.

I have checked file size with ftp in the server and I read number of link in detail of Google Sitemap:

We have:

sitemap 1 - file size 24 kb - Google n. link 5,973
sitemap 2 - file size 133 kb - Google link 18,959
sitemap 3 - file size 288 kb - Google link 37,918
sitemap 4 - file size 603 kb - Google link = Error (more than 50,000 links)
sitemap 5 - file size 732 kb - Google link = Error (more than 50,000 links)
sitemap 6 - file size 519 kb - Google link = 37,914
sitemap 7 . file size 281 kb - Google link = 18,955

I have downloaded and unzipped sitemap 1, 2 and 7 to read their content.

Sitemap 1 contents hot, pick, etc (only one row for page), and 453 categories and subcategories and pages shows in browse (more 1, 2, 3 and so on). We have a lot of categories in browse where links are displaied in more pages.
So 5,973 links that Google read in sitemap 1 may be correct.

In sitemap 2 there are first 18,959 links (idx_links from 1 to 18,959)
Sitemap 3, content other 37,918 links.

I am not able to read sitemap 4 and 5 (as rss file) becouse is too big (unzipped) and I have had a memory overflow but sitemap 4 begin with links id 57,876 and so on.

Sitemap 6 and 7 seems right.

So problems seems to be in sitemap 4 and 5 but I am not able to read and check all links and what links are in this sitemaps.

Before of Google alert (when links are about 95,000 links) I have notice that Google wrote: "links in sitemap (about) 200,000" but I thought, 95,000 links (page links) plus 95,000 links in browse pages, plus categories in sitemap1, is quite right.

I notice that also sitemaps that are containing in indexu software (as example) follow the same sequence of size:
sitemap1= kb 4
sitemap2= 9
sitemap3= 16
sitemap4= 23
sitemap5= 22
sitemap6= 15
sitemap7= 9 kb.
So seems that links are concentrated in sitemap4 and 5. But what links?

Before my first post, I have reload again sitemap.php and all sitemaps in original script and run sitemap.php with the same results I have now.

I have add a sitemap 8 changing number of sitemap in sitemap.php and run it to see if the dimensions of the files became smaller, but on the contrary with map 8 all sitemaps size are increased.

I know that 113,000 links are many (and I thought about being able to arrive to 250,000-300,000 in 450-500 categories) but I don't understand what does not work.

Thanks again for the your aid.

[Bruceper]

Adding more sitemap files should solve the issue, try adding a few more and see what happens.

[FSGDAG]

My suggestion would be to get a membership to IndexUDirectory.com. Zubby has a MOD that will fix this issue. I ran into the same problem on my Web4Pennsylvania.com site.

[mediaone]

Adding more sitemap files should solve the issue, try adding a few more and see what happens.

As I said, we have add a new sitemap8 (and correct numer of sitemap in sitemap.php) but when we have run sitemap.php, all sitemap 2-8 (detail) are increased of about 500 kb in total so we have had also sitemap6 not only 4 and 5 over size.

We will try to increse again number of sitemap to ten and run again sitemap.pgh.

Thanks

[mediaone]

My suggestion would be to get a membership to IndexUDirectory.com. Zubby has a MOD that will fix this issue. I ran into the same problem on my Web4Pennsylvania.com site.

Thanks for your suggestion. I think to see IndexUDirectory.com in few hour.

Thanks again

[mediaone]

Hi,

I have work during this week, compatibly with my time and my knowledge (I am not a programmer), on issue I have explained in my first post.

I have added further sitemaps but numbers of links in sitemap2 to 7 continue to increase if they are added others sitemaps.

So I have tried on an other site that I am preparing and that contains 2021 links where suspended = 0 (detail links) only.

Problem is in script (sitemap.php).

Dividing number of the links by the 6 sitemaps (2-7) we have about 338 links for each sitemap.
Runnig sitemap.php, the six sitemaps are as follow:

sitemap2 from link_id 2 to 339 (=338 links)
sitemap3 " 340 to 1014
sitemap4 " 678 to 1688 <-- doubled from 678 to 1014
sitemap5 " 1015 to 2173 <-- doubled from 678 to 1014
sitemap6 " 1352 to 2173 <-- doubled from 1352 to 2173
sitemap7 " 1689 to 2173 <-- doubled from 1689 to 2173

So initial counter for each sitemaps is right but from sitemap 3 to 6 number of links in each sitemap is more or less a multiple of 338:
644 in sitemap3, 1010 in sitemap4, 1058 in sitemap5, 821 in sitemap6 and 484 in sitemap7.
It is not exacty double-triple-triple-double and again once (about 338) in sitemap7, becouse 150 links are suspended.

Obviously, also size in kb for sitemap3 to 6 increase becouse links are double or triple and when number of links is big (in my case 113.700) sitemap4 and 5 run over 50,000 links that Google require.
If more sitmaps are added, the problem increases and can go beyond the limit of the 50,000 links also sitemap 6 and, adding olther sitemaps, sitemap 3 and 8 too.

So I think that something does not work in dividing the number of the link for every map.
Moreover I have not found in script a limit of 50,000 links per sitemap but, as I have said above, I am not a programmer and I do not know where to take part.

As you know, submit sitemaps to Google is important.

I hope to have explained the problem well and hope that someone can resolve it.

Thanks

[mediaone]

Thanks to all for the help.

I have resolved the problem changing the script so this thread is closed.

Thanks again.