December 11, 2013
Rather than going with the usual, ubiquitous, and often (yawn) repetitive “top 10 BI predictions” for the next year, we thought we’d try something different. After all, didn’t the cult movie Highlander prove beyond the shadow of a doubt that “in the end there will be only one”? And didn’t the Lord Of The Rings saga convince us that we need one prediction “to rule them all”? The proposed top BI prediction for 2014 rests on the following indisputable facts:
- Business and IT are not aligned. Business and IT stakeholders still have a huge BI disconnect (after all these years — what a shocker!). This is not surprising. Business users mostly care about their requirements, which are driven by their roles and responsibilities, daily tasks, internal processes, and dealings with customers (who have neither patience nor interest in enterprises’ internal rules, policies, and processes). These requirements often trump IT goals and objectives to manage risk and security and be frugal and budget minded by standardizing, consolidating, and rationalizing platforms. Alas, these goals and objective often take business and IT in different directions.
- Requirements are often lost in translation. Business and IT speak different languages. Business speaks in terms of customer satisfaction, improved top and bottom lines, whereas IT speaks in metrics (on a good day), star schemas, facts, and dimensions. Another consideration is that it’s human nature to say what we think others want to hear (yes, we all want our yearly bonus) versus what we really mean. My father, a retired psychiatrist, always taught me to pay less attention to what people say and pay more attention to what people actually do — quite handy and wise fatherly advice that often helps navigate corporate politics.
- Shadow IT rules. As a result of business/IT disconnect, as much as 80% of all BI content is authored in shadow IT applications, mostly based on spreadsheets.
Now, here’s a dose of a sobering cold shower: A single enterprise BI platform based on a single version of the truth — a single enterprise data warehouse — will not solve this challenge. We all fought this battle in the 1990s and largely lost it. Time to embrace the pragmatic reality of the 21st century. Here it is in a nutshell:
- Rule #1. IT needs to embrace — not fight — the fact that majority of BI content will always be in the hands of business users, not IT.
- Rule #2. IT needs to regroup and reorient itself to monitor what business users do rather than ask for requirements, and use what Forrester calls BI on BI as input into and requirements for enterprise BI architecture and applications.
- Rule #3. If you disagree, see Rule #1.
So how do you have the BI cake and eat it too? How do you balance business users’ need to produce their own content with little dependence on bureaucratic IT processes while at the same time minimizing enterprise risk, achieving economies of scale, and getting rid of silos? We think we have an answer — maybe the answer. This is how Forrester proposes to address the dilemma (we already see this approach at a handful of customers with win-win for business and IT success stories). It’s a three-tier environment based on:
- Individual BI sandboxes
- Business users rule. Here they author and use about 50% of all BI content with no constraints or limitations. This is where data exploration, discovery, and what-if analyses happen.
- Tools and technologies: desktop BI applications, in-memory BI applications, spreadsheets, cloud/self- provisioned sandboxes, and server-based sandboxes.
- IT involvement is strictly limited to infrastructure and tools support plus monitoring to identify patterns, commonalities, and opportunities (using BI on BI) for productionalization. As IT succeeds in moving sandbox-based applications to production, that 50% number goes down.
- Content produced here is used in individual tasks and low-risk applications.
- Collaborative, shared, quasi-production BI environment
- Business users share and collaborate on BI content — about 30% of all BI content — with their colleagues.
- Tools and technologies: shared folders, portals, and data virtualization.
- IT steps up its monitoring and now watches for red flags (too much data, too many users, too critical or risky applications, etc.) and opportunities (using BI on BI) to productionalize BI content. As IT succeeds in moving business user-authored BI applications to production, that 30% number goes down.
- Content produced here is shared within departments and workgroups. Low-risk, low-criticality decisions can be made based on this content.
- Production environment
- Business uses and authors BI content — about 20% of all enterprise BI content — within the limitations and constraints of the enterprise data model, standards, policies, rules, guidelines, etc.
- Tools and technologies: EDW and enterprise BI platforms.
- Owned, run, and managed by IT. As IT succeeds in moving sandbox and shared BI content to production, that 20% number goes up.
A step-by-step process would look like this:
- A business user builds a spreadsheet (or a model using a desktop in-memory BI tool) that pulls data from multiple data sources (via direct database connections and/or by exporting data from operational apps), creates a pivot table, refreshes it on a daily basis, and uses the model in daily operational tasks.
- IT flags that spreadsheet as part of the BI ecosystem, but does not take any action yet, as it’s still a standalone, single-user application that is not setting off any red flags.
- Colleagues notice the application and ask the business user to share it with them. The business user saves the spreadsheet/model to a shared folder and his/her colleagues are now leveraging the results in their daily routines too.
- IT monitors the newly shared content for patterns such as data complexity, frequency of usage, number of users sharing the content, etc.
- When/if some of the established thresholds are reached, IT offers to help automate the process. As a first step, rather than manually importing data sources and manually refreshing the pivot table, IT hooks up the model to all relevant data sources via data virtualization.
- When/if the usage patterns of the application indicate that it’s a candidate for production, IT steps into action. It documents the model and the process, and puts it into the request queue. When its time comes, IT moves and integrates data sources into the EDW and gives the model a production look and feel.
- One morning, without having been dragged into endless requirements gathering meetings, the team of business users is pleasantly surprised that their homegrown app is now part of the production enterprise BI environment.
Didn’t we just have our BI cake and eat it too? Yes, we did, because:
- A business user created BI content on his/her own schedule without any constraints and limitations. He/she did not have to waste their valuable time (often aware from customer interactions) in requirements-gathering meetings, steering committees, and building permit processes.
- IT productionalized the content based on the actual model that’s been battle tested and not on vague and imprecise requirements.
Brilliant, huh? But this is our first attempt to document and understand this process — we don’t yet know very well all of the possible ramifications and implications. We welcome all constructive criticism, suggestions, best practices, lessons learned, etc.