Search: Wales of Wikipedia Battles Google with Open Source

Over this past year’s Christmas and New Year’s celebrations James Wales, co-founder of Wikipedia, announced that ‘Search is Broken‘, referring to Internet search options like Google, Yahoo! and MSN. (Wales also says that his own creation Wikipedia has spun out of control and is irreparably broken.) While often the search results are good, many times results will contain SPAM and irrelevant content. The results are also easily gamed. And the search rank algorithms are secret.

Wales’ announcement in December was that his solution to the problems was a fully-scalable user-controlled Open Source search engine. Open Source projects like Lucene and Nutch were floated as the time as providing some of the components for the new system that Wales envisioned. Wales tentatively named his new Open Source project Wikiasari.

This past weekend Wales announced that Wikia had gained control to the rights of search technology from LookSmart that would form the core of the new project. The technology acquired is called Grub, and it is unique because of its distributed web crawling capabilities. Unlike Google and the other major search engines that spider and consolidate information centrally, Grub sends out indexing tasks to a distributed set of computers on the Internet. Anyone can contribute machine cycles to the project by downloading the Grub client to their computer and letting it run while their machine is idle.

Wales announced at a recent O’Reilly Open Source conference that Grub is to go Open Source and that the source code would be made downloadable later in the year. Gil Penchina, Grub’s CEO, said that their target is to someday to be able to capture at least five percent of the public search market. But with the nature of Open Source licensing, the product has the potential to become fragmented with multiple competitors going after a very lucrative market.

“In looking at the overarching industry, it has become clear that open is the business model of the future,” said Michael Grubb, Senior Vice President, Technology, and Chief Technology Officer, LookSmart. “We are pleased to collaborate with Wikia and believe that Grub will thrive under an open source license. We are happy to be able to assist in the movement to make search a more open proposition and look forward to seeing things progress from here.”

Wales feels that getting a user community behind the search project has the potential to lead to better and less biased search results and for building what could become a challenger to Google and Yahoo!. This is just another example of Open Source changing the business of software.