Bug with "Spider Link CheckUrl"

[Mickey]

I use to lauch from time to time the "Spider Link CheckUrl", and it never gave more than 300 results (my directory has more than 5000 links).

But I had an idea about the possible bug

In my directory, I have links to download a pdf document.

So, I suppose that the links ending with "document.pdf" don't react like "normal" links for html or php pages. And I suppose that the script is stopped by the first ".pdf" link, because I have not seen such links in the 300 first (and last) results.

Dody, have you tested the Spider Link CheckUrl with such links (links to download .pdf, .ppt or .exe documents) ?

[dody]

It doesn't matter what file extention you have. Indexu will crawle the url and will ONLY read the http header. Technically it will find '200 OK' message.

I think the url of pdf may do not have 200 OK message and refer to another url. It could be that to download pdf need to register first at the website.

If you can provide which url that indexu unable to crawle, then it will help us a lot. We can try to examine the header data.