Big data is undergoing big change, but most companies are missing it or just grasping at the edges. My colleague Fatemeh Khatibloo and I have just completed an exhaustive study of the big data phenomenon. We found a familiar pattern: business confusion in the face of stern warnings about the dangers of big data and vendor-sponsored papers extolling its benefits. Here’s what we found hidden beneath the buzz:

As data explodes, so do old ways of doing business.

Everywhere we look, we find businesses using more diverse, messier, and larger data sets to stay competitive in the age of the customer — like the consumer goods firm that allocated marketing dollars based on flu trend predictions and the oil and gas companies that used weather data to predict iceberg flows and extend their drilling season. Savvy businesses find ways to turn more data into a competitive advantage. If your firm doesn’t get this, it won’t be pretty — starting in the not too distant future.

Technology managers and architects can’t afford to sit back and think that their Hadoop project will deliver everything the business needs. Nor can you afford to think that big data isn’t for you because you don’t have that much data. Why? Because “big data” is really the practices and technologies that close the gap between the available data and the ability to turn that data into business insight — insight that your firm needs to survive and thrive in the age of the customer. Four things to understand:

  • It’s not the amount of data, it’s what you do with it.Big data provides a high-definition view of your business and your customers’ behavior and lets you deliver personalized experiences at scale. Big data is finally making data a strategic asset — so what is your firm doing to build on it? How are you going to use this new asset to delight customers and squeeze competitors?
  • You need to take action and challenge the status quo. Big data success requires overcoming any momentum in the wrong direction — such as executives’ tendency to hoard data or lack business curiosity about what quantitative insights are possible. It also means that business must take accountability for data and stop making data issues “an IT problem.”
  • You face risks that you don’t understand. Big data gives your firm the power to understand customers in ways not before possible. But just because you can doesn’t mean you should. Even the US federal government recognizes that big data means that policy has to change to protect citizens. How will your firm navigate the risks and ethical situations presented?
  • Describing big data with Vs is stale and not actionable. The Vs of big data have run their course; they were useful when we knew little about big data, but they no longer inspire action. Instead, refocus your change effort on keeping the three Cs in balance: a data-driven culture; the competency to leverage data that is more diverse, messier, and larger than you’re used to; and the technical capabilities to put the right insights into the hands of your customers and employees.

In the coming months, Fatemeh and I will be doing events and webinars on big data based on our research. Check out our reports for CIOs and for marketing leaders, as well as Fatemeh's blog post, and please reach out to us if you want to talk more about what this means for your firm.

Next up on my agenda: Noel Yuhanna and I will be publishing a TechRadar™ assessment on big data to help architects get a handle on the complex and quickly evolving landscape.