Big announcement at Dataworks today — IBM is transitioning Big Insights into a partnership with Hortonworks. This seems not unexpected and only mildly interesting until you think deeply about it. Then it hits you: Hortonworks, after digesting Microsoft, is now eating IBM. It’s a total reversal from the past, when big fish like IBM would eat little fish like Hortonworks. See my blog post from yesterday for more on this.

The tables are turned. Let me explain:

  • Working on distributed open source is new and complex. The young developers with the new skills and energy to work on this stuff don’t go generally to work for IBM or Microsoft. They go to work for valley startups who are challenging the big boys. This has been a problem for enterprise software vendors who have had issues developing and building a business around their open source work. Need proof? Microsoft got out of the Hadoop business first, then EMC did the same (through Pivotal, which then gave up), then Intel, and today IBM with their announced partnership with Hortonworks at Data Works 2017.
  • Proprietary data analytics tools are where the growth is. IBM isn’t  growing Infosphere; Teradata cut its prices in half to boost sales; and Oracle doesn’t even want to talk about Exadata in favor of “all cloud, all the time.” The future of these software giants is tied to how well they can navigate the disruption to their legacy business model, which is on-premises, scale-up, and centers on enterprise deals.
  • Software giants now rely on newcomers for growth opportunities. By becoming the open source data vendor to IBM and Microsoft, Hortonworks is turning the tables on these giants. Why? Because the big software vendors are now dependent on a relative newcomer for the biggest growth opportunities in the market — machine learning in the cloud using the open source big data ecosystem of tools as infrastructure. Hortonworks built HDInsight for Microsoft. It will eventually build the infrastructure underneath IBM’s cloud big data fabric, too.

With today’s announcement, Hortonworks’ nonpublic strategy becomes more clear. As big enterprise vendors eventually realize that open source, subscription-priced big data technology is hard and different, they will turn to smaller firms that can do it by partnership, not acquisition, because of the talent problem. As soon as Microsoft or IBM buys Hortonworks, they’ll start bleeding talent. Do not get me wrong big enterprise software vendors can code open source, IBM for example, has invested plenty in open source, it just that talent young open source innovators don’t tend to stay with the big boys long. 

What does this mean for the industry in general?

  • It attempts to drive a stake in the heart of Cloudera. This is true for a lot of reasons I don’t have space to discuss. Most important, however, is the potential for Apache Atlas and Ranger to unify security and governance for the big data ecosystem. These can’t be proprietary because too many big software vendors like IBM and Microsoft don’t want to lock into a startup’s IP without acquisition, which, as previously discussed, they can’t do. They’d rather bet on the open source ecosystem. Which is why Hortonworks’ strategy is the winner.
  • It puts pressure on other big vendors.  Oracle, HPE, SAP and Teradata have not committed to an open source big data vendor, choosing instead to play the field and keep their options open. If this IBM/Hortonworks partnership works out well, one or more these could be next in line for Hortonworks, or they might be wooed by Cloudera if they starts to follow Hortonwork’s lead for once. We’ll see, this could kick off a piranha feeding frenzy as the big vendors give in to letting the Hadoop experts run their cloud infrastructures.

This brings me to my final thought – the more of the enterprise guys Hortonworks eats, the better an acquisition target they do become. Since they went public early, the big vendors felt more comfortable with a partnership. What happens next year if Oracle buys Hortonworks, given that IBM and Microsoft are dependent on Hortonworks’ product? Holy competitive move, Batman. Ah, maybe that is the master plan after all?