MSN Blog: Why Didn't MSN Crawl My Site?

MSN's blog is openly answering some questions about crawling, etc. The post at http://blogs.msdn.com/msnsearch/archive/2004/11/18/266087.aspx gets into why a page would or would not have been crawled by MSN's search spider:

"Thus, URL?s with many (definitely more than 5) query parameters have a very low chance of ever being crawled. Another thing to consider is whether we can find your page. If we need to traverse through eight pages on your site before finding leaf pages that nobody but yourself points to, MSNBot might choose not to go that far. This is why many people recommend creating a site map and we would as well."

Posted to Search Engines-MSN | TrackBack