The Engine

If you stare at the lines in the road, you’ll never see where you’re going.

While the Devil might be in the details, if you want to know where he’ll be you need to look for the pattern. Pattern-recognition is one of the trickiest pieces of programming, partially because we know very little about how our brains work, and partially because humans make it look so damned easy. Our brains are designed to spot and record patterns.

Sure, it takes a long time. Evenutally, enough memories gel to enable us to look back and assign confidence of some correlation (if not causation.)

Red sky at night, sailor’s delight. Red sky at morning, sailor take warning.

That won’t hold up against the very best prediction engines we have, but it got us down that path. You go back and examine what happened before an event to determine what contributed.

Meanwhile, 200,000 people died at the end of 2005 because no one made the correlation that a big underwater earthquake might pose a problem. There was no warning for the tourists who were trapped by the wall of water. There was a warning for the Indonesian natives who knew that a sudden and unexpected ebb tide was a sign of a huge wave coming in. Many of the animals knew it, too.

It always seems far simpler when you know what the trigger is, instead of drowning yourself in a myriad of probably inconsequential details.

Mapping Ourselves

This post is more straightforward than I wanted it to be. It was originally envisioned as a short story describing the greatest effort in the fictional history of programming. Only I am not so sure it is fictional. It is too possible to not be probably, and world domination is potentially at stake.

I rush this essay into publication because of a fractured discussion with Steve Rubel, who tracks all things technology. Google has announced it is shutting down several of its free webservices, and Rubel mentioned in passing that Google Reader might not survive the next round without showing some value.

Reader is an RSS feed reading application. You ‘subscribe’ to blogs and content that you like online, and Reader ‘delivers’ it to you. Instead of clicking links and folders and bookmarks, the Web you like comes to you in one quick and tidy place. You can also subscribe to feeds of news searches, or mix and match your own sources.

Google Reader offers a couple of very useful features. One, all the feeds I subscribe to are searchable using the Google indexing and algorithm. I don’t have to ‘file’ things in my Reader, I can always find them later. If I choose, I can add whatever tags and descriptors I deem relevant. Second, I can share links with people I know. Real-life friends, business acquaintances, or those I network with online. I actually get ‘feeds’ of what my network of people found interesting, and we can add notes to each other pointing out key facts or summaries.

Soup to Nuts

What makes this interesting is Google’s role throughout. In the course of the content, Google plays several key roles. Not a monopoly, but it is a player in:

  • Content creation (Blogger, YouTube)
  • Content delivery (Feedburner)
  • Content aggregation (Reader)
  • Content discovery (Search)
  • Content sharing (Shared links in Reader)

No one else is a significant-enough player in all these aspects of the Information Age to track what is said to whom, and when it happens.

If you’re Google, you have Willy Wonka’s Golden Ticket: the map to the influencers. There’s a huge debate raging among the marketers and the public relations people and the politicos over just who has influence, and how you locate them.

Finding Influencers is important, because it allows you to target your message or your plea to only those people that really matter for a given function or moment. Those Influencers can change over time or given a different objective, but locating them is the key.

From the instant someone creates a video or a blog post, Google knows what is in it. (Again, there are other services, but Google gets enough of the video and blog business to make this scale.) Google also knows:

  • when it arrives in your Reader
  • when you read it
  • how you mark it
  • when you share it
  • when your friends read it
  • when they act on it…

…and the cycle continues. From Soup to Nuts, Google can know which people start the online tremors that lead to popularity of content. ‘Viral’ is no longer a marketing mystery – Google has the data to find the epidemiology.

The Ticket is the Beginning

Willy Wonka’s Golden Ticket wasn’t the prize itself – it was the step you had to clear to get to the prize.

Sure, Google could sell some of those results. It could offer up premium information to advertisers, or even offer direct targeted ads at the highest of most high Influencers. We’re not talking about the Pete Cashmores or Steve Rubels of the world – we want the people who are more likely to seed them with inspiration and information.

But even that isn’t the prize. Stay with me here.

Get several million people on Blogger, a large contingent of content producers. Get a couple of million more on Google Reader, and then sit back for about five years while they share data like no one’s business.

Except it is your business. You need to understand who the Nodes are, and how much time elapses between certain events. You need to learn how to look beyond the tiny pieces of data as individual bits, and instead look at the whole. Big picture, a bunch of water droplets becomes a cloud. And under certain atmospheric conditions, that cloud looks red.

Google isn’t going to drop Reader, because it needs us to keep feeding the data beast. It will take a good five years of collection (and maybe a couple of more concentrating on the data visualization to make it feasible, but isn’t that why Google hired all those engineers and algorithm people?)

Once you know what a ripple looks like, and the content of that ripple, you can track it. And you start to see the others. And eventually, you start to identify the ripples that preceded a discrete event instead of the ones that followed.

Making Waves

Google is building the world’s largest prediction engine. It’s now in a learning phase, and an early one at that. It’s building a new Vocabulary of Influence, not to sell us products but instead to tell the future. All you need is a series of similar events that you can compare, and look for correlating ripples that came before. Certain punctuated events would have no meaning, like outcomes of Super Bowls. But something like, say, a quarterly stock report, would be easy to parse.

It would be regular enough (and have a large enough data mine of its own) that you could put the most powerful computers to work just looking for the pattern. And as the owner of the ONLY data set that traces complete ripples of influence, there is no break in the chain to cloud the data.

Maybe the predictions will come with just a few hours notice, like a tsunami warning system. Maybe it will evolve into a longer-range forecasting tool for economics or finance. What could Larry and Sergey do with the Ginormous Google Gigawatt Crystal Ball? Other than promise us they Won’t Be Evil?

The truth is right there in your Google Reader, and the Devil is in your details. And that is why the High Holy Priests of Mountain View will never bring Reader to the sacrificial altar.

Share Button

Comments

  1. Google is certainly set on connecting the dots on information.

    I have no doubt that the head Googlers wake up every morning and ask, “What would God do?” ~@jesseluna

  2. I can’t be the only one who has had multiple less-techie friends ask about Google Reader just because I send them articles using the “email” feature. I’m also thinking that as social media marketing matures, more and more companies will use RSS to track conversations. I’ve trained half a dozen people at our nonprofit on how to set up G Reader, and they all exclaimed how easy it was and how cool it was. RSS still confuses and intimidates people, but I think that will change relatively quickly (next 2-4 years). G Reader has the viral tools built in to take advantage of RSS when it does go mainstream, and I can’t imagine that they’ll abandon it before then.

  3. So is it your estimation that Google Reader is the conclusion to Graham Chapman and John Cleese’s unfinished masterpiece?

  4. Those of us who saw Terminator 2 knew this day would come.

  5. Nice post. I agree that the data Reader provides is much too valuable for Google to give up on it. Just interesting that they haven’t done more to push it to a larger user base. Could be that more data does not necessarily mean better data.

  6. Thanks Beau…

    There does come a point where you get diminishing returns with more data, but I think the bigger issue is the “purity” of the data and the intent.

    Right now, there is just a tool, and people are sharing and clicking and storing and reading what they want.

    For Google to accelerate the adoption of Reader, it would have to add and/or promote a purpose. The “Killer App”, as it is often called. To promote the technology to new users in that manner would tend to skew their initial perceptions of what it is and how they use it.

    I could promote the use of Google Reader for fantasy football players to consume a pack of feeds relating to player news, weather conditions, and injury updates. I could promote it and give them a way to auto-subscribe to an OPML file. But in doing so, I have turned off a number of others who could have used it in more interesting and flexible ways.

    To promote it to the masses, you risk killing the individuality and the authenticity of the data. You skew the results, and once you do it’s hard to unskew. Let the relationships, the link networks, and the nodes identify themselves — the real value will emerge. There is no shortcut to organic growth without a trade off in the integrity of the data set.

  7. Good technology w/efficient applications always come at a price! I love google reader and of course google is collecting data….hopefully good over evil will prevail as they hash out what it all means. Great writing, content rich and timely. Thanks!

  8. Interesting idea. A lot of useful wiretap information comes just from watching to see who calls whom.

  9. Google has been working on predictions for a while now. Just because they are currently have “the betterment of society” in their public face ( http://www.google.org/predict.html), doesn’t mean that we should ignore the potential for massive control laying not quite dormant.

Trackbacks

  1. Twitter Comment


    RT @ikepigott: Occamtude: The Engine: [link to post] VERY NICE ike

    – Posted using Chat Catcher

  2. Twitter Comment


    RT @ikepigott: Occamtude: The Engine: [link to post]

    – Posted using Chat Catcher

  3. Twitter Comment


    Retweet @ikepigott: @steverubel – My answer: [link to post]

    – Posted using Chat Catcher

  4. Twitter Comment


    Re-T @ikepiggot Occamtude for you ===> The Engine [link to post]

    – Posted using Chat Catcher

  5. Twitter Comment


    There’s a lot to this @ikepigott post – [link to post] , all really interesting and dense.

    – Posted using Chat Catcher

  6. Twitter Comment


    @ikepigott Sheer brilliance! [link to post]

    – Posted using Chat Catcher

  7. Twitter Comment


    Ike explains 21st Century mining. Should we hail or fear The Engine? RT @ikepigott: Occamtude: The Engine: [link to post]

    – Posted using Chat Catcher

  8. Twitter Comment


    A brilliant post from @ikepigott on Google’s “future”: [link to post] Via @chrisbrogan

    – Posted using Chat Catcher

  9. Twitter Comment


    Very interesting commentary on a different side of the Google coin. RT @ikepigott: Occamtude: The Engine: [link to post]

    – Posted using Chat Catcher

  10. Twitter Comment


    Occam’s RazR » Blog Archive » The Engine: If you stare at the lines in the road, you’ll .. [link to post]

    – Posted using Chat Catcher

  11. Twitter Comment


    @ikepiggott is bordering on Chaos Theory with this post! I agree… Google is trying to predictfuture based on patterns [link to post]

    – Posted using Chat Catcher

  12. Twitter Comment


    Deep thinking. Read back to back: @ikepigott [link to post] & @dtunkelang http://twurl.cc/cho. Which is right/can both be?

    – Posted using Chat Catcher

  13. Twitter Comment


    Morning, twitterverse. Have this with your coffee. RT @technosailor @ikepiggott Google predicting future based on patterns [link to post]

    – Posted using Chat Catcher

  14. Twitter Comment


    Interesting even without coffee. RT @mensan98th RT @technosailor @ikepiggott Google predicting future based on patterns [link to post]

    – Posted using Chat Catcher

  15. Twitter Comment


    RT @ikepigott So, who believes in conspiracies? [link to post]

    – Posted using Chat Catcher

  16. Twitter Comment


    Retweeting @ikepigott: So, who believes in conspiracies? [link to post]

    – Posted using Chat Catcher

  17. Twitter Comment


    Very interesting post about why Google will not likely drop Reader like it did with several of its other free webservices [link to post]

    – Posted using Chat Catcher

  18. Twitter Comment


    Wow. Chat Catcher added 16 comments from Twitter to @ikepigott’s blog post – [link to post] – I would have missed them on Twitter alone.

    – Posted using Chat Catcher

  19. […] Erg interessant artikel dat een goede kijk geeft op de werking van Google. Must read. […]

  20. Twitter Comment


    This blog post from @ikepigott is way over my non-techy head, but interesting: [link to post]

    – Posted using Chat Catcher

  21. Twitter Comment


    Check out @ikepigott’s post, if you’d like some insight into what Google is up to. Prediction is the future 😛 [link to post]

    – Posted using Chat Catcher

  22. Twitter Comment


    @C_Collins @ikepigott’s RT if you’d like some insight into what Google is up to. Prediction is the future 😛 [link to post]

    – Posted using Chat Catcher

  23. Twitter Comment


    Damn… check out the big brains on Ike! Best blog post I have read in a while. [link to post]

    – Posted using Chat Catcher

  24. Twitter Comment


    Rather: check out the big brains on @ikepigott! Best blog post I have read in a while. [link to post]

    – Posted using Chat Catcher

  25. […] the meantime, a couple of notes about connections. The Google Engine post got a lot of traffic, and while tracing those messages back I saw a familiar name on the […]

  26. Twitter Comment


    Liked “Google is building a giant crystal ball with your Reader” [link to post]

    – Posted using Chat Catcher

  27. Twitter Comment


    Reading @ikepigott “Google is building the world’s largest prediction engine. It’s now in a learning phase, and an ….” [link to post]

    – Posted using Chat Catcher

  28. Twitter Comment


    Reading: “Occam’s RazR » Blog Archive » The Engine” ( [link to post] )

    – Posted using Chat Catcher

  29. Twitter Comment


    If you haven’t read @ikepigott’s post on Google & data aggregation, read it now: [link to post]

    – Posted using Chat Catcher

  30. Twitter Comment


    RT ikepigott: Occamtude for you ===> The Engine [link to post] …

    – Posted using Chat Catcher

  31. Twitter Comment


    RT ikepigott: Occamtude: The Engine: [link to post] …

    – Posted using Chat Catcher

  32. Twitter Comment


    @IkePigott’s brilliant post: The Engine (about Google) [link to post]

    – Posted using Chat Catcher

  33. Twitter Comment


    Google as a gigantic influence prediction machine. [link to post]

    – Posted using Chat Catcher

  34. Twitter Comment


    Behold, the power of Chat Catcher: [link to post] Turns links from Twitter into blog comments. Great for monitoring & tracking discussion

    – Posted using Chat Catcher

  35. Twitter Comment


    RT @mjkeliher Behold, the power of Chat Catcher: [link to post] Turns Twitter links into blog comments. Great 4 monitoring & tracking

    – Posted using Chat Catcher

  36. Twitter Comment


    RT @mjkeliher Behold, the power of Chat Catcher: [link to post] Turns Tweets into blog comments. Great for monitoring/tracking discussion

    – Posted using Chat Catcher

  37. Twitter Comment


    A good read: @ikepigott’s Neal Stephenson-like post on Google’s (future?) ability to visualize data is compelling: [link to post]

    – Posted using Chat Catcher

  38. Jeremy Day says:

    Google as a gigantic influence prediction machine. http://bit.ly/YvCd

  39. cloudspark says:

    RT @mjkeliher Behold, the power of Chat Catcher: http://is.gd/fXdY Turns Twitter links into blog comments. Great 4 monitoring & tracking

  40. Twitter Comment


    Occam’s RazR ” Blog Archive ” The Engine [link to post]

    – Posted using Chat Catcher

  41. Tim Walker says:

    A good read: @ikepigott’s Neal Stephenson-like post on Google’s (future?) ability to visualize data is compelling: http://is.gd/fXdY