Just a few years ago, when big data was associated primarily with Hadoop, it was like a precocious child…fun for adults, but nobody took it seriously. I’m attending Strata in San Jose this February, and I can see things have changed. Attendance doubled from last year and many of the attendees are the business casual managers – not the blue jeaned developers and admins of days gone by. Big data is maturing and nobody takes it lightly anymore.

Most impressive was the data science focus and tracks, with experts like Joe Hellerstein from Trifacta and UC Berkley, and DJ Patil, the United States’ first Chief Data Scientist. Lisa Hammitt from Salesforce gave the best keynote where she posited that emerging systems of engagement would eventually become systems of intelligence through analytics. Brilliant! While I’m gushing, I have to call out to the extraordinary amount of vendor innovation going on in data science, preparation, and real time big data, mostly from startups. Firms like Paxata, Trifacta, Skytree, Dato, WebAction, and Snowflake Software are doing amazing things.

It’s easy to get caught up in the excitement, but then I remember to swallow a reality pill – as an analyst, I’m paid to be a bit skeptical.

The reality is that big data has hit those awkward teenage years, where it is both full of promise and pimples. I just published a report, “Brief: Turning Big Data Into Business Insights, 2015”. It gives my outlook for big data in 2015, but for those that don’t have a subscription, here are the high points – with some color added from my observations at Strata.

In 2015, The Need To Understand Customers Will Take Precedence

Early big data implementations focused on solutions that were easy for technology management to justify like IT security, fraud recovery, or data warehouse augmentation. In 2015, the drive to understand and serve powerful customers will move big data investments toward customer engagement analytics and toward quickly turning insight into customer action.

Strata observation: The theme of using more data to drive customer insight and engagement was missing except from Lisa Hammitt of Salesforce. Forrester thinks this will be a huge force for big data, especially when you think about the impact that Internet of Things analytics and engagement is going to have. I expect that next year, we will hear much more about this.

Data Hubs Will Evolve To Platforms For Digital Insight

In 2015 the variety of technology and data required for customer engagement analytics, the drive to work in real time, and the explosion of cloud technology will force an evolution in the big data architecture. Instead of a hub, architects will recognize the need for a platform that supports seamlessly turning data to insight and insight to action. Specifically action to engage customers to win, serve, and retain them.

Strata observation: One of my biggest takeaways is the clear lines now drawn between vendors who want you to build everything on Hadoop and those pushing a federated approach.  Really interesting were the keynotes on the new open data foundation initiative from IBM and Pivotal Software, and Cloudera’s public decision not to participate. Clearly the former must land on the side of federation and Cloudera (and Hortonworks) are on the data hub/lake-side. Interestingly, MapR Technologies’ Myriad project, bringing YARN and Mesos together, puts them on the federated side as well.

Personally I think the federated approach will win, but it will be more like a virtual data hub, with virtualization and distributed data flow orchestration tools pushing and pulling data and materializing views or feeding hubs as close to real-time and on-demand as the technology allows. Another interesting observation from BMC Software – developers want the federated approach where they can isolate what they are doing through service APIs and data scientists want the hub approach were all the data is available for experimentation. It remains to be seen were managers with the money will land.

The Big Data Market Will Stay Hot, But Forces Will Exert Pressure On Revenue

About a quarter of firms invested in big data in 2014. However despite this 'smaller than you might think' number, software vendors saw huge growth in their customer bases, and demand for big data services became strong. In 2015, expanding service ambitions, scarce talent, open source, and revenue pressure from the cloud will squeeze vendors who have set high expectations with their investors.

Strata observation: Here is where I need to be a bit of a ‘Debbie Downer.’ In a poll in the data driven business day, a speaker asked how many folks were involved in a big data initiative. About a third of the room held up their hands. He then asked how many folks were getting the benefits they expected, and everyone’s hands went down. I think 2015 is going to be a rough year for big data, but that no one will admit it until 2016. Scarce talent will be made worse by the explosion in technology.  This will lead firms to try it via DIY using cheap cloud and open source, which will result in enterprise issues with reliability and security. Meanwhile VCs, who have been pouring money into big data startups for years, will start expecting exits. Add to this the fact that ESVs like IBM, Microsoft, and Oracle have jumped into big data in the cloud, but I suspect their pricing models for pay as you go services are still not completely baked. This means revenue issues for them or unhappy customers who find that it’s not as cheap as they thought.

Welcome to those awkward teenage years of big data, may they end soon so the fun of college can begin.

Got a good story to tell that agrees or disagrees with this? Take our brief online survey.