A few months ago, I blogged about testing quality@speed in the same way that F1 racing teams do to win races and fans. Last week, I published my F(TA)1 Forrester Wave! It examines the capabilities of nine vendors to evaluate how they support Agile development and continuous delivery teams when it comes to continuous testing: Borland, CA Technologies, HP, IBM, Microsoft, Parasoft, SmartBear, TestPlant, and Tricentis. However, only Forrester clients can attend “the race” to see the leaders.

The market overview section of our evaluation complements the analysis in the underlying model by looking at other providers that either augment FTA capabilities, play in a different market segment, or did not meet one of the criteria for inclusion in the Forrester Wave. These include: 1) open source tools like Selenium and Sahi, 2) test case design and automation tools like Grid-Tools Agile Designer, and 3) other tools, such as Original Software, which mostly focuses on graphical user interface (GUI) and packaged apps testing, and Qualitia and Applitools, which focus on GUI and visualization testing.

We deliberately weighted the Forrester Wave criteria more heavily towards “beyond GUI” and API testing approaches. Why? Because:

  • GUI-based testing won't help you catch everything that’s going on behind the scenes of your web or mobile app.Modern applications require a layered and decoupled architecture driven testing approach; In fact web, mobile, connected devices (IoT), cloud, and third-party services demand a multitier architecture driven testing approach.
  • Fast feedback loops frequently break UI-only test suites.Agile development encourages change. When developing in short iterations, the automated UI-based test suites often break as soon as change happens, even for simple changes to the GUI layout or unexpected changes to the behavior of back-end services. Not all tools suffer from this, but most of the traditional ones do.

So what objectives and strategy should you have when using these tools? To be Agile and implement a continuous testing strategy, you should:

  • Target at least 80% automated testing and maximum 20% manual testing.To reach these target levels, Agile teams test at the API level.
  • Ratchet down UI test automation in favor of API test automation.Currently, 80% automation  is UI testing; modern applications require cutting this to approximately 5% to 10% and replacing much of that test automation with API test automation.
  • Fully automate regressed user acceptance testing (UAT).While the business should perform UAT manually the first time, you can then 100% automate the regression tests.

If you don't reach those automation levels, and/or fail to automate nonfunctional tests and the test process itself, it will be very hard for your team to deliver software application features as quickly as your business wants. That’s why testing quality@speed has become so important. The FTA Forrester Wave shows that of the nine vendors we assessed, two are Leaders while seven are Strong Contenders — there were no Risky Bets this time around!

Another takeaway: Tools alone won't meet all your test automation needs. I have written a brief on how testing services companies can augment your automation capabilities with automation services, but you must navigate the vendor landscape carefully to select the right partner for the right service. Forrester clients can read my brief for more details.

Your comments are always welcome!  Happy reading.