It happens often in conversations with clients that I realize they have disjointed initiatives going on to support their digital transformation. The most dangerous parallel initiatives are those where, on one side, they are changing their development teams to become more Agile, but a separate initiative in the same enterprise exists where their Operations folks are running a development and operations (DevOps) transformation. The first thing I recommend to those clients is to unify or tightly connect those programs with an underlining common lean strategy. But I don’t want to dig in here about Agile+DevOps and how overused and abused the term “DevOps” is. I will just recommend to you some reports we’ve published explaining how “Agile” and “DevOps” are two sides of the same coin (see, for example, “Faster Software Delivery Will Accelerate Digital Transformation”).  The Modern Application Delivery playbook I’ve co-authored for years is all about what it means to adopt Agile+DevOps. Check that out too.

But the second and equally important thing I realize with these clients happens when I start querying them about their testing capabilities and approach during those journeys towards more agility and DevOps. And that opens the next can of worms. Why? Because if Agile disrupts how we test applications, continuous delivery, which DevOps is a core enabler of, represents unprecedented disruption of testing. I just published a report on the  continuous testing (CT) services providers landscape, where I provide my definition of what continuous testing means and is. I think the figure here makes it very clear. 

I have been evaluating testing tools for a few years now as part of the continuous testing framework, but I’ve also been meeting and working with the testing services providers and thought I’d spend some research time on the latter. In fact, the market-spend estimation we made on testing services of $20 billion in 2016–2017 shows relevant market spend in the area. So, we better make sure that money is spent well.

I scanned over 47 organizations, asked them about their software accelerators/solutions how they complement or integrate existing testing vendors’ tools and add additional value to them, asked them about their automation engineering skills and ratio compared to manual testers, their top CT services across a large sample list of what I believe contributes to continuous delivery, about their geographical presence, their payment models, and more. The result? It’s clear that providing strong CT services is a huge business model and organization change for the testing service providers, not just for you as consumers of CT services. Most vendors are in the process of jumping on the CT bandwagon and leveraging some old capabilities, but the impact is so huge that the large, multibillion-dollar testing services players have a long way to go. You will also find an interesting bunch of smaller players listed there who are more dynamic and fast growing. However, it’s hard to navigate the space. I am hoping to help you with that research, but also provide you with some guidance on how to short list your preferred continuous testing services partners going forward.

Download the doc here if you want to learn more. Stay tuned for a Forrester wave I am working on for continuous testing and later in the year how Artificial Intelligence can make our testing better, smarter, and faster. I’ll be happy to hear about your experiences with continuous testing and service providers as this interesting market matures.