Have you ever heard of an Okonomiyaki recipe? Neither have I.
But now that you’re curious, it is very likely that you will go and type that into Google.
Google, the rockstar of search
For years, hundreds of thousands of Google servers have been laboriously crawling, organizing, and categorizing information, so that when you type “Okonomiyaki r--" Google can bring you the “best recipe” before you even type the “e” in “recipe.”
But search is a hard problem. And it’s no secret:
Udi Manber on Why Search is a Hard Problem
Seth Godin's Search is Hard post
I know about the search problem first hand. I was an engineer at Google for over seven years, working on search ranking and scaling Google's search index to support billions of searches a day.
There was once a Quora question: What inventions made you think "Whoa, this changes everything!" when you first saw them?
Answers ranged from pocket calculators to iPhones and 3D printers. My personal favorite is Google. Few people realize what a technological marvel it is. You can type a couple of vague (even misspelled) words into Google, and of the billions and billions of pages on the Internet, Google can bring you the right one most of the time, in a fraction of second.
“Don’t be evil” begets open source begets Big Data
In 1998, there were quite a few search engines out there. But Larry and Sergey had a different approach to search with their PageRank algorithm. That difference made Google miles ahead of other search approaches back in the day.
Still it took people a while to realize what was going on in the small office at 165 University Avenue in Palo Alto.
But the Google of 2014 is light years ahead of Google of 1998. Some of the best minds in computer science have toiled for over a decade to make this happen. The whole Big Data space exists today because engineers at Google needed to to build large scale distributed systems to handle hard search problems.
They chose to share the details of their work with the world with the 2003 Google File System paper and the 2004 MapReduce paper, so that some of their hard work could be replicated in open source.
The Starbucks-ization of search
The average number of searches per day is over 3.5 billion on Google alone. That’s one search for every two people (including babies and grandmothers, but excluding zombies) in the world.
That doesn’t even include the number of searches on Amazon, LinkedIn and Facebook. We use search for everything.
Oh, except your company data. We still use BI analysts, data scientists, specialized tools, and SQL for that.
Now you know why search is hard. You couldn't stop reading now without knowing why relational search is harder, could you?<br>
