Note: Don Reinertsen is one of the most influential voices in agile methods. While Don is not a software professional himself, he is widely respected and cited by many authors writing on agile, DevOps, and related topics.

Don Reinertsen
Don Reinertsen

Don’s major books Managing the Design Factory and The Principles of Product Development Flow remain go-to reading for product managers worldwide in both digital and physical domains. Covering critical topics like flow, work in process, variation, options, queuing, and decentralization, Don brings a precise vocabulary and powerful set of analytic tools to bear on the challenging problems of creating a next-generation operating model supportive of agile and DevOps.

Don generously gave of his time last summer in my research for the report “Your Operating Model Must Enable Innovation,” and there was much valuable material in the interview, some of which (with his assent) I am reproducing here, lightly edited for content and flow. Below, Don and I talk about the problem of expertise and how shared services organizations have failed in managing expertise as a service. 

Charlie: In the inquiries I get here at Forrester lately, if they are about cross-functional teams, we frequently encounter some staffing questions that you’ve touched on in some of your work. For example, we hear repeatedly, “We haven’t got enough database administrators (or network engineers, or security architects) for every team to have one. We have to either use a shared service or live with a staffing shortage.” We have many teams needing quick responses from somebody with high expertise, but they don’t need this all the time.

In your book, The Principles of Product Development Flow, you mentioned everyone on a Navy ship being trained in a damage control skill like firefighting. However, having a basic damage control skill is very different from turning everybody on a development team into a skilled database administrator.

In development, we often require skills that are both deep and scarce. When such skills aren’t available, we can create expensive work queues. I’m wondering if you have any insights on how to deal with the problem of scarce skills in a world where the operating model is moving more towards semi-autonomous, cross-functional teams, increasingly populated with generalists rather than specialists.

Don: This sounds like the 1980s all over again. In 1981, IBM amazed the world by quickly designing the original IBM PC using a completely autonomous team down in Boca Raton. I talked to one of the employees involved in this project and he said, “If we had used the standard IBM organization and procedures, it would have taken us three years to ship an empty box.” This success caused everybody to believe that the best and fastest way to do development was with an autonomous, cross-functional team. There was a huge movement to pursue the benefits of autonomous teams, and later, a huge surprise when people discovered such teams have important disadvantages along with their advantages.

I remember a conversation I once had with the manufacturing engineering manager in a medical device company that used relatively autonomous cross-functional teams. He told me, “We just had a quality problem with one of our products last week. Our products use molded polycarbonate parts that are attached to vinyl hoses. The sterile hoses are normally attached immediately before use and disposed of at the end of an operation. The parts have a long shelf life, and we’ve never had quality problems with this approach. This time, the team decided to preassemble the vinyl hose to its connection point and experienced significant quality problems.

“There is a subtle failure mode which occurs when a vinyl hose is left connected to some polycarbonate plastic parts for an extended period. The plasticizer from the hose leaches into the polycarbonate part and weakens it. If the polycarbonate part has residual stresses from its manufacturing process, it eventually cracks. So some of us have learned that any polycarbonate part left in extended contact with a vinyl hose must be stress relieved.”

He pointed out that this particular failure mode was being rediscovered over and over again because each team is only aware of what happens on their own design. He said, “Because I am exposed to the work products of all of the teams, I have become the organizational memory on this issue. I seem to be the only one who remembers that we must stress relieve these parts under these conditions.”

His point, which I think is extremely important, is that your ability to build deep expertise is proportional to the number of exposures you get to problems, particularly low-frequency problems with important consequences. If you’re working 40 hours a week in an area, you are going to gain experience at a faster rate than if you’re working 1 hour a week in the same domain. Many companies with cross-functional teams have created environments that reduce the rate at which people with specialized skills are acquiring expertise, and this weakens the skill base of the company.

Ultimately, your organization’s structure will affect the speed at which you can grow specialized expertise. This may not be an issue in domains where technology and market needs change at a glacial pace, but it is extremely important when you are producing advanced systems with leading-edge technology for rapidly evolving market requirements. While autonomous, cross-functional teams bring important benefits, it is critical to be aware of what they weaken and to put in place countermeasures for their disadvantages.

We often err on the side of thinking generalists will solve all our problems. Generalists are necessary, but complex systems always combine this with specialization. The design of a multicelled organism like the human body is a good analogy. In the human body, every cell doesn’t have to do everything. Instead, the body is organized into specialized organ systems because organs operate with different chemistries and often accomplish tasks that are scale-dependent. The low pH of stomach acid wreaks havoc if it is aspirated into the lungs.

Most complex systems have an important role for specialization. Sometimes we can be naïve about this in product development. Almost every time someone tells me that they are using a completely autonomous cross-functional team, when I dig deeper I find they have had to put in place a mechanism to create infrequent but quick access to deep expertise. While generalists with broad skill sets are very valuable, the idea that if everybody does everything then we won’t need experts is clearly going to fail. And we know this because it is already failing for people, and they are already coming up with intelligent workarounds for the disadvantages. The problem is that they are still articulating doctrines that don’t match what they really do and what really works.

It is worth noting that one of the biggest problems with the shared service model has been around for at least the last 40 years. People simply don’t understand how to measure the performance of a shared service. They naïvely assume that since they spend money to produce output, they should measure the cost per unit of output. Well, if you measure shared services on cost per unit of output, that leads you to minimizing the amount of idle time in the service, so you move to high levels of utilization. Unfortunately, if you combine high utilization and fluctuating demand, you will create serious queuing problems, which then lead to long and unpredictable response times.

Unfortunately, because queuing problems coincide with the presence of specialists, companies often assume that specialization alone causes the queuing problem. In fact, the queuing problem is not caused by the fact that you’ve got specialists. It occurs when you have specialists, and you simultaneously measure and incentivize them with the wrong metrics.

Poorly chosen metrics will lead you, either consciously or unconsciously, to load the shared resource to too high a level of utilization. Over the years I have found that teams react to the long latency and slow response times of overutilized shared resources by requesting direct control over these resources. They ask to have the specialists put on their team. Curiously, whenever a shared resource has high capacity, consistently low latency, and fast response times, nobody wants the headache of managing a smaller, local resource to replace a satisfactorily performing shared resource.

Thus, I think people are actually misdiagnosing the problem as an issue of organizational structure when it is, in fact, a problem of not being smart about how you operate a shared group of specialists.

Charlie: Right. It’s the operating model for the specialists and its associated measures of performance.

Don: Yes, people keep coming to me saying that in order to get the desired outcome for their program, they need direct control over everybody who affects this outcome. They say, “I need to have purchasing on my team; I need to have manufacturing engineering on my team; I need to have quality assurance on my team.”

So, I ask them, “Well, why do you need them?” They say, “Because their work is important, and it affects my ability to get a good outcome.” Then I say, “Do you need to have a payroll function on your team?” They would say “No, why would I need that?” I say, “Well, isn’t payroll important? Don’t people like to receive their checks on time?” And they say, “Yeah, it’s important, but payroll does their job — they do what they’re supposed to do.”

So I say, “At the end of the day, you do not need a dozen extra skill sets for you to manage on your team. You need to make those shared resources responsive to what you really need. When you work with shared resources that are responsive and meeting your expectations, and your real measure of service is being satisfied, then you have no desire for the extra headache of managing another group of specialists. You need to focus on how you measure the performance of your shared resources and how to enable people to achieve this performance. Fix that problem, and you’re not going to be interested in putting them on your team.”

Don can be reached via Reinertsen and Associates.