Yahoo Announces Big Honking Web Index

(Yes, I know, this was several days ago, I'm still catching up). Yahoo has announced in their blog a new index. The announcement's at http://www.ysearchblog.com/archives/000172.html .

From that page: " As it turns out we have grown our index and just reached a significant milestone at Yahoo! Search – our index now provides access to over 20 billion items. While we typically don't disclose size (since we've always said that size is only one dimension of the quality of a search engine), for those who are curious this update includes just over 19.2 billion web documents, 1.6 billion images, and over 50 million audio and video files. Note that as with all index updates we are still tuning things so you’ll continue to see some fluctuation in ranking over the next few weeks."

I remember when I flipped out that Google had a billion items. At the moment Google's Web page says they're indexing 8,168,684,336 pages, though of course that doesn't include their over two billion images. It also doesn't include their news stories, etc.

Of course how many pages are indexed by a search engine is totally irrelevant if the ranking system is flawed. The number of pages indexed by a search engine should only be of deep concern if a) the data pool is absurdedly small (nowadays a general all-Web search engine indexing only a hundred million pages is going to miss a lot), b) the pages were all added to the index as a result of payment, or c) the pages are exclusively drawn from another resources (something like a search engine searching nothing but Open Directory Project links. There are reasons for using these kinds of sites, but they usually involve technology demonstrations.)

I have two general comments to make about Yahoo's index. First of all, congratulations, good to see you back in the search engine wars. Second of all, when are you planning to do a similar upgrade for the YAHOO DIRECTORY??!

God's bread! It makes me angry. Day, night, hour, time, tide, work, play, Alone, or in company, still my main care has been to have people realize that searchable subject indexes are still viable and useful and should not be abandoned completely in favor of full-text search engines. (With apologies to Capulet.)

And Yahoo's is so disappointing. Do a Yahoo Directory search for autism. You'll get 192 results in the Yahoo Directory vs. 948 in the Open Directory Project. Do a search for "Dance Dance Revolution" (with quotes). You'll get twelve results, vs. 27 in the Open Directory Projects. There are more ADS FOR DDR ON THE FIRST PAGE OF YAHOO'S RESULTS THAN THERE ARE TOTAL LISTINGS.

Now I just said that result counts are not the be-all and end-all of everything. And they aren't. But when numbers are this different between the two major directories, what is happening? Is Yahoo going to abandon their directory? I hope not; there are indicators that they want to make it more useful and visible -- the RSS feeds for one (I'd love one for every category!) Is it because of the fees to get included in the directory? Is it just not worth the human time and effort? Is all the search mojo and expertise at Yahoo going into the full-text search?

The Open Directory Project has its own problems, and I've been frustrated with them in the past. But they do some very cool things, too-- the RSS listings in the categories. The articles listed with date and source. Now that Yahoo is offered Creative Commons search as well as paid content search, perhaps it would behoove them to do at least a little extracting from those sources and add more useful, article-based content to their directory? And they have a listing of feeds stashed somewhere; they use it for My Yahoo. Are those feeds listed in their appropriate categories in the Yahoo Directory.

Yahoo was made on the directory; I hate to see it not getting more attention.

Posted to Search Engines-Yahoo | TrackBack