Where’s the Money in Big Data?

So asks NY Times writer James Glanz in his recent article, “Is Big Data an Economic Big Dud?”. The short answer to his question is no. Now for the slightly longer answer.

What’s So Great About Big Data?

It may not strike you as immediately obvious why more data should be a good thing. Does the world need more YouTube videos of cats, or Facebook posts, or tracking data from your web clicks? How does that create economic value?

Simplistically, of course, the companies collecting the data, and convincing users to generate it, are doing pretty well. Facebook stock has risen back to its already phenomenal IPO valuation. YouTube turns out to have been a bargain for Google at only a $1 billion price tag. And online advertising continues to grow at double digits.

Looking deeper, the data craze builds on a more fundamental observation, brought into the ranks of corporations by the likes of Peter Drucker in the 1940s and 50s: business decisions ought to be based on data. Or at least not to contradict the data.

In the sciences, the turn to data and quantification and mathematics has been on a centuries-long march that has recently penetrated into the social sciences, including psychology and marketing. One way to look at the Big Data hoopla is simply as the logical extension of this: if you have data, and you can improve business decisions — like the choice of advertising to show to users — you can make more money.

The premise is simple enough, and the results are indisputable in fields like advertising.

What’s Not So Great

Small businesses can’t benefit from data in the same way that larger ones can, because they don’t have enough customers of their own to extract trends, and improving their decisions on average may not help if they only have one store location to the average across.

Big Data will benefit big businesses first. From big box stores to Big Data to mega-airline mergers, the trend is toward bigger. That’s not so great if you care about the idiosyncrasy of the small if you like the autonomy of the mom and pop store.

As a result, small businesses have little choice but to rely on big companies to provide the data they can use. They must advertise using the aggregators, play the game of rankings and search engines and social media, and find a way to stand out in the noise.

They depend on the Amazons and Apples of the world to be their distribution channel, knowing that larger companies can copy their ideas and produce them at the larger scale and lower cost if they’re successful.

Drowning in Data

Data has become too easy to collect and store. Our memories are fallible and impermanent. Our photos on Instagram reveal more than we can recall.

With inventions like Google Glass, a future where every human’s every waking moment is recorded in high fidelity 3D video is not so far-fetched.

Companies, too, can collect more data than ever before, and they can actually search through it for trends, and adjust their product strategies, advertising, and pricing based on what they find.

It’s pretty obvious that there’s money to be made from not just collecting the data, but analyzing it and acting on it as close to instantaneously as possible. In economic terms, more accurate data can lead to greater efficiency, by detecting mis-pricing and missed opportunities.

The Big Data industry has risen up to provide the technology tools so that every company, in every industry, can harness, store, search, mine, learn from, and act on the data that can help it to perform better.

Why Big Data Might Not Pan Out

With such clear benefits, why would anyone doubt the economic contribution? Here are a few legitimate concerns.

Maybe Big Data will not primarily create new value, but mainly shift the value to those who collect and use the data better than the competition, in the process enriching the companies that develop the tools.

Maybe more data will face diminishing returns very quickly, rather than creating new sources of value. Smoothing out some suboptimal pricing, and luring in a few extra shoppers with better-targeted offers, could provide a marginal benefit, not a transformative and sustainable one.

Big Data could turn into an arms race, similar to high-frequency trading. Or, well, literal arms races. Big Data could be a more powerful weapon, and the best-armed companies will destroy the competition. Free market zealots will hail the Schumpeterian “creative destruction”.

In the past, the destruction has tended to create more opportunities than it has lost, over the long term, despite the undeniably negative effects on those immediately impacted, like the mill workers unemployed, or the auto workers whose salaries have halved.

With the rise in income inequality to levels last seen at in the gilded age, and with the “hollowing out” of the global workforce, the past may not be as reliable a guide.

Even if one of these skeptical scenarios come to pass, Big Data will not be a Big Dud, at least not for all.


In a more optimistic outcome, which I consider to be more likely, Big Data will be a net win, keeping the energy of economic growth thrumming at a pace to lift hundreds of millions more around the world out of poverty in the coming decades, bringing a higher standard of living, longer lives, and less misery to the world than at any time in history. Big Data can hardly take credit as the only or even the main cause. It will be but one minor contributing factor. But I believe it will contribute positively.

Ignore the data — like ignoring the facts — at your own peril.…

Read more

Cities of the Future

Well-heeled liberals tend to assume that everyone should live in cities, and the world would be a better place for all. People would consume less energy for transportation and temperature control. They’d get more exercise from walking and taking public transit, extending their lives. Quaint Jane Jacobs-style neighborhoods would improve people’s daily routines with friendly local businesses and amenities.

All of that’s true. But here’s why it could go a different way.

The City Is Dead
Given technology trends, there will soon be fewer reasons for cities to exist.

  • Video conferencing improves so much that physical proximity becomes irrelevant — for socializing as well as working
  • 3D nano-printers one day can create any food near instantaneously, driving most restaurants out of business, because we’ll all be able to Jetson our dinners on demand. Assuming that virtual reality doesn’t render the physicality of food irrelevant first.
  • Online shopping continues along its trend toward beating the real thing.
  • Online dating utterly replaces the in-person variety, as the recent marriage stats suggest.

If all these come to pass, what reason will be left for living in a city? It won’t matter where anyone lives.

Long Live the City
And yet, some of the benefits of cities will remain.

  • Energy efficiency
  • Greener living
  • Lower cost delivery of goods
  • Virtual reality may one day provide an adequate experience of nature

In an especially science fictional future, people might end up living in Matrix movie style pods, plugged into virtual reality, living at a density to make Tokyo seem spacious.

What do you think will happen?…

Read more

Can Computers Compose a Hit Music Single?

Computers can beat us at chess. They can beat us at Jeopardy-style trivia. They can out-calculate us and make fewer mistakes on rote problems.

Can computers write better music? What would it take for them to compose a #1 hit music single?

For ideas, I finally got around to reading Raymond Kurzweil’s book The Singularity Is Near, a touchstone of futuristic thinking.


Consciousness in the Cloud
In Kurzweil’s vision, the Singularity is the time when computers surpass people in intelligence — and not just intelligence, but effectively all ways. He believes that people will choose to merge with computers, and replace their biological bodies with other physical forms, or with pure software, uploading their consciousnesses into what we now call “the cloud” of computer systems distributed around the world.

Naturally, in Kurzweil’s vision, computers will be better than humans at art and music too.

But what would it take?

The Computer Music Challenge
Suppose that IBM were to build a supercomputer called Golden Ear, whose mission is to top the global popular music charts, beating out the Los Angeles music industry as well as the music industries in Korea, Mumbai, Tokyo and everywhere else.

IBM already developed Blue Gene for chess and Watson for Jeopardy. Now Watson is taking on other challengs, like medical diagnosis.

Is Golden Ear the successor to Blue Gene or Watson? Or a new class of computer?

Designing a Musical Computer
How can a computer know what melody or chorus sounds good to people?

By analyzing trends in music, popular culture, news, celebrities, vernacular language. Then choosing sounds and lyrics that capture the moment.

Consider that mobile phone apps like Shazam can already recognize music from a few notes and retrieve the title, artist, lyrics and more. IBM’s Watson can mine vast troves of text and guess about puns and trivia.

Is it a coincidence that the Lady Gaga hit “Poker Face”, released on Sept 2008, arrived not long after the peak of the poker mania? Look at the plot of Google searches for poker and you’ll find that the mania peaked in Jan 2007, close to the height of the housing bubble. “Poker Face” the song consists of upbeat music and that catch phrase, repeated often. The song title itself peaked as a search term in April 2009, 7 months after the album’s debut.

Could a computer have identified the poker trend, selected a catchy phrase timed to the bursting poker bubble, and produced and launched a song on iTunes earlier than Lady Gaga did? Could such a computer catch the next trend?

Would the music industry pay royally for computer software that wrote songs for a stable of bankable performers and generated a higher hit rate than human composers?

The Golden Ear Challenge
Previous efforts at computerized music composition have not met with much success. That’s an understatement, as a matter of fact.

But computers have improved rapidly, and continue their exponential improvements. As they did with chess and trivia, computers could overtake humans in this new challenge.

Kurzweil would not be surprised. The Singularity may be near for artificial intelligence. Is the #1 Single nearer?…

Read more

A Modest Proposal for Silicon Valley Housing

A modest proposal to fix the Bay Area housing crunch: Fill in the southern part of San Francisco Bay with landfill and build a 21st-century city of earthquake resistant glass.

All of the land from the Dumbarton Bridge south could be converted from salt marshes and foul-smelling sludge to a glorious city of the future. That could solve the housing shortage and bring prices back to earth, simultaneously allowing Silicon Valley to expand and grow still more.

It’s no secret that Silicon Valley has the highest real estate prices in the US. The reasons are many, including high salaries, pleasant climate, beautiful geography, jobs — and a dire lack of houses.

Yes, Silicon Valley real estate prices are the highest in the nation for many reasons. And all of them could go away if governments just let developers build enough more, and limited the ability to existing property owners to block what other people can do on their own land.

Local communities exercise near veto rights over development near them, and city planners and developers find it difficult to win many small battles over tiny plots of land. Broad expanses of land abound in the ridge of hills and mountains between the bay and the ocean. But those are widely considered beautiful, and it’s near inconceivable that such stunning parkland will be developed anytime soon.

Meanwhile, the southern part of the bay is not generally considered beautiful. It’s filled with famously smelly salt marshes that serve the only modest ecological value. Their benefits could readily be offset with eco-friendly preservation somewhere else.

Even better, the water comes within walking distance of Facebook, Google, Yahoo, Cisco, and Tesla. It’s a prime location, much more so than San Francisco, so many miles further north, away from the high tech center of the Bay Area.

A new city, literally in the bay, would provide expansion area and take pressure off of housing prices. Better yet, an inchoate city would allow the vaunted technology visionaries of Silicon Valley to experiment in urban design, engineering the public transit, housing, and earthquake resistance of the future, supporting their own high tech economy and showing an example for the world.

And no human infants need be eaten or sold.

Why not? Please share your reasons.…

Read more

What’s the Recipe for Innovation?

A growing body of research shows where creative innovation comes from. Here are 9 tips:


1. Look outside your field. Innovation comes from outside your field. Mix with experts in a range of specialties, a strategy cultivated at places like Bell Labs according to the recent NY Times article. Study widely.

2. Find the Eureka. The moment of innovation often feels like a eureka moment, according to Columbia professor William Duggan. It will never come if you’re not knowledgeable and prepared.

3. Get physical. Use your physical body as well as your mind, based on new research at University of Michigan and NYU.

4. Spend time alone. Brainstorming is best done alone, according to this New Yorker article, contrary to how most people have tried it for the past several decades. There’s truth to the lone inventor idea.

5. Engage with critics. Test your ideas on others — who are critical. Convince a skeptic like your sibling, not a loving fan like your mom. Talk to unhappy customers, who are possibly the single best source of new business ideas. A non-judgmental environment fails to bring out the best.

6. Start with what you hate. Find a product or experience that frustrates you. Figure out why it sucks and make it better. The iPod was not the first portable digital music player. But Steve Jobs the perfectionist found all of the ways the experience could be better, and created an iconic product. He may similarly succeed with the new Apple TV set.

7. Coffee and alcohol help. Especially in combination, I find. If you look at the habits of the most famous writers and artists, you’ll find ample evidence of legal and illegal substances, often pushed to life-damaging extremes. Don’t go that far. But there is a reason why many of the leading high tech companies in Silicon Valley do not perform drug tests.

8. Keep your day job. Drop out of college — or quit your job — after you’ve already created an amazing, growing business idea, not before. University life and the workplace can be rich sources of ideas — and recruits, which is why Facebook’s Mark Zuckerberg visited Harvard again in November.

9. Make your workplace innovative. Companies can create innovation-friendly environments by following a few tips from McKinsey.

What other tips do you have for being creative?…

Read more

iTV or Apple TV 3: A Perfect TV Set?

Can Steve Jobs reinvent the TV set and bring it closer to perfection, just as he helped perfect the tablet computer, the phone and the music player? From the recent revelations in the Steve Jobs’ biography and articles in the Wall Street Journal, as well as leaks from suppliers in Japan and Australia, the rumor mill is pounding the drums. Below are my own speculations about what the Apple TV set will be like, based on taking seriously Steve Jobs’ obsession with perfection.


Full disclosure: I have a personal interest in the outcome. Back in the summer of 2010, I placed a $1 bet that Apple would announce a TV set during Mac World of Jan. 2012.

On July 24, 2010, I struck up an animated debate with classmates in a technology strategy course at Berkeley by posing the question, “When will Apple finally release a real TV?” Here’s what I initially wrote:

When will Apple finally launch a real TV? By that I mean an actual TV set, not a set top box. This is the one irresistible question I had…

Apple of course already sells Cinema display with screens up to 30″ and iMacs with screens up to 27″. Samsung makes the LCDs, if I recall correctly. Apple also sells the Apple TV set top box for streaming video and music. They have a remote control for Front Row, their media center software. They have a huge installed base of iPhones and iTouches and iPads that could be pushed a remote control app as part of the core OS. Why not roll it all up together and make an integrated TV? Does selling a TV solve any of the problems that have limited the success of the Apple TV set top box, which Steve Jobs calls a “hobby”?

What features would the TV set include? How would it integrate within the Apple ecosystem? Could Apple leverage its iOS without a touch screen on the actual TV? Would remote control navigation be acceptable? How should Apple price a TV set to gain widespread acceptance? Is there a magic price point, and at what premium to commodity TV sets?

Our debate drew more comments than any other class discussion. Ultimately I emailed a transcript of the discussions to Steve Jobs himself on Sept. 24, 2010. (I received no response back, in case you were wondering.)

Naturally, the fiercest detractor against my argument works for Google. More about that later. Also, a high level executive at Facebook replied, “I fail to see how any of this leads to Apple making an actual TV. I just don’t see the need… You are wrong.”

Preliminaries out of the way, what will the Apple TV, or iTV, be like? The supplier news suggests three screen sizes, including 32″ and 55″, as well as an intermediate size, most likely 42″ or 46″. It will run a version of iOS and support voice control using Siri from Apple devices including the iPod Touch, iPhone and iPad. It may include DVR functionality. Chances are that the interface will look similar to the current Apple TV set top box. These are all mere features, and don’t yet get to the core of what made Steve Jobs’ greatest hits so great: perfectionism

After all, Google TV could copy any of the list of features. In fact, based on what we saw with Android, they’re likely to rapidly copy all of Apple’s features and win the spec war. And that won’t be enough, because consumers don’t care about the specs, as a hilarious YouTube video pointed out.

Rather, they care about the sense of design perfection. “It just works,” as people like to say about Apple products. But how will the Apple TV set just work?

Here’s what I predicted in an email dated Sept. 17, 2010:

And I think we all recognize that set top boxes suck. Just because viewers are used to having 5 remote controls and a tangle of interconnects doesn’t make it a good solution.
There’s no reason why there can’t be a single display panel with a power cord and a wired or wireless network connection. It can transmit audio channels wirelessly and people can buy powered speakers.
If Apple bundles it all together with a content delivery deal with Comcast or ATT it would certainly be a radical rethinking of the TV.
I also offered some longer term predictions on July 27 that year:

And eventually the future holds:

  • A camera on every TV set
  • Video calls and conferencing from the TV
  • Interactive games using TV cameras
  • Audience participation in televised events via the cameras
  • Motion sensing technology for games and interactivity
  • TVs being replaced by computers, effectively
  • Set top boxes fighting to stay relevant as the TV-computer subsumes their functionality
  • All content delivered on demand (sorry, TiVo)

More recently, on October 25 of 2011, around the launch of the iPhone 4S, I wrote:
With Siri as the interface, a user can simply say, “Watch Delhi Belly”, and who cares where it came from, it will just play. If the user has access to the title through a monthly subscription, it will start instantly. If not, a payment screen will appear and the user will confirm. If it’s on YouTube or Netflix or Comcast, it’s all fine — as long as Apple can avoid getting blocked the way Google’s failed TV did — it will play in the same way, with the same seamless experience. People might not like the expense of having multiple sources behind the scenes; they’ll love the convenience. And Apple will gain ground against Netflix and Unbox.
It will be marketed as Steve Jobs’ creation, the product he had secretly worked on for years, the achievement of his final vision, a “hobby” that finally became a product. And even if it’s not as big a success as the iPad, it will still do well enough to create a market that previously did not exist.
My Google friend didn’t like the negative comments about Google’s foray into television, or the suggestion that content may cost money or be less available.

In another email from the same day, titled “and one more thing about the Apple TV set”, I added
And one more thing: the TV has no video inputs or outputs. The remote control has no number keys and no input selector. And as if that’s not bad enough, the TV has no audio outputs either.
Sounds like a terrible product… right?
Here’s what it does have. Its connections consist of a power cord and an optional network cable. All video content comes in from the network connection or wireless. The TV includes speakers and a camera for video conferencing and interactive games.
The interface is controlled by a tiny, almost button-less remote and by voice with Siri. An iPhone, iPad or iTouch app will also work.
For surround sound, it offers wireless audio. You can buy Apple speakers that consist of a power cord and a dial on the back to set what position the speaker is in (front left, right rear, subwoofer, etc). They’re powered speakers that get the signal and volume via wireless. Then it just works.
In response to this, my Google friend almost sounded interested, provided that the television set offered a DVR. Also, he worried about whether it would have enough content, perhaps of the bootlegged variety available on BitTorrent.

Regardless of the specifics, the key story of Steve Jobs’ life was the story of seeking perfection. He recognized that narrowly defining perfection based on a single product specification was doomed to failure, because a competitor could always beat the spec.

Instead, Jobs appreciated that perfection has an aesthetic side. Successful perfectionism requires meticulous attention to detail, of the sort that might satisfy the clinical definition of obsessive compulsive disorder. As he told his biographer,
“I’d like to create an integrated television set that is completely easy to use. It would be seamlessly synced with all of your devices and with iCloud.” No longer would users have to fiddle with complex remotes for DVD players and cable channels. “It will have the simplest user interface you could imagine. I finally cracked it.”
Perhaps he has done it again, conceiving and creating a carefully perfected product that becomes a category leader.

Regardless, Jobs possessed a rare ability. He could step back from an existing product and see all of the ways in which it failed. He knew when a product sucked, and he would say so bluntly, tactlessly, insensitively.

What about the current television experience falls short of perfection? It’s easy to start the list:

  • Multiple mismatched and often ugly devices
  • Multiple remote controls bubbling like pox with every imaginable button
  • Tangles of interconnect cables
  • The complexity of changing from one source to another, and bringing all devices into sync
  • Operating remote controls in the dark
  • Finding the channel you want to watch even when you know what show you’re looking for
  • Content comes from too many different places, too many devices and input sources

For comparison sake, which of these does the Google TV solve? None, really. That’s right. Read drill preess reviews through the list again. Consider the remote control for the Google TV. It actually made the situation worse. As if existing remote controls did not already have too many tiny buttons with unclear meanings, the Google TV added an entire alphanumeric keyboard. That’s 26 more reasons to reject it.

What could have been Google’s greatest contribution was bringing Google search to the television. Easy and fast search could overcome the nuisance of finding content among hundreds of stations and the YouTube catalog. However, it simply doesn’t work well on a television set using a minuscule keyboard.

That’s the single biggest reason the initial product was doomed to fail, even before the networks blocked Google TV from accessing their content. It didn’t solve any of the biggest shortcomings of our living room television viewing experience.

By contrast, voice command using Siri directly addresses the imperfection of remote controls. With voice control, a remote might not be needed at all, if a sensitive enough microphone can be embedded in the television

A closed system, Apple’s famous walled garden, does offer potential respite from the onslaught of set top boxes, including the existing Apple TV box. Eliminating all video inputs and outputs, as I proposed, is an extreme move. Ultimately, it’s what we will all have, whether with the first generation iTV or a subsequent one. Why? Because it’s more perfect.

Likewise, having a separate stereo receiver and remote control and interconnects and speaker cables is something only an audiophile could love. Eliminating all of them in favor of wireless powered speakers would be a more elegant solution. Ultimately, it’s what we will all have, whether with the first generation iTV or a subsequent one. Why? Because it’s more perfect.

For any audio lovers who are reading, no, sound quality need not suffer. High end audio manufacturers are already heading in the direction of simplicity with products like the THIEL zoet 3.7D, which brings one of Stereophile magazine’s highest ranking speakers into the wireless world. It’s easy to imagine wireless add-ons that would enable any high-end speaker to work with the iTV, bringing the best possible sound quality.

Regardless whether any of my specific predictions are correct, the iTV, or Apple TV set, will bring the television viewing experience that much closer to perfection. Of that we can all be sure.

Will it be a successful product? Will it escape the Perfection Paradox? We can only wait and see.…

Read more