Search Wikia open sources Grub

Stephen Withers
Monday, 30 July 2007 05:45

Your IT - Home IT

Wikia's web search project will use the Grub distributed search engine it has acquired from LookSmart and released as an open source project.

"The desire to collaborate and support a transparent and open platform for search is clearly deeply exciting to both open source and businesses," said Jimmy Wales, co-founder and chairman of Wikia.

Conventional search engines crawl the web in house. Grub distributes this task across volunteer systems.

A Grub client receives a set of URLs from the server, then checks whether any changes have been made to them. If so, the data is compressed and sent to the server.

The Windows Grub client is currently running in test mode, while the Linux client probably "isn't going to work at all" in its current state, according to the project's web site.

But crawling (or spidering) only collects the data. What's important is that a search engine returns the most appropriate pages for a given query. Search Wikia's plan is to combine Grub "with the power of a wiki to form social consensus," according to Wikia officials.

Adding human collaboration means there is a chance to reduce the opportunity for unscrupulous operators to fool search engines into giving high rankings to pages containing little or no information that is genuinely relevant to the search query.

"Look for other exciting announcements in the coming months as we collectively work to free the judgment of information from invisible rules inside an algorithmic black box," added Wales.

Search Wikia is expected to go live, if only in limited form, later this year.

Wikia recently licensed LookSmart's 'white label' text and display ad serving platform.