What is a test management tool?
A test management tool is the central tool of your QA process. It provides support for managing test cases, test sets and test cycles. Other than providing full traceability between all test ware, it assists users in generating reports, logs and in-dept analysis.
A good test management tool provides support for everyone in the software development life cycle (SDLC) ranging from QA to dev and any stakeholder involved. But how do we recognize a good test management tool and are there really that many differences?
Well for starters, let’s look at one of the most important tasks for a test management tool: integration possibilities. Because everyone in the SDLC needs to benefit from the test management tool, it’s important that all used tools can interact with this central tool. For this reason, a popular choice for test management tools is one that is built into tooling which everyone already uses such as the popular issue and project management tool Jira. In this case an addon such as Xray or Zephyr (add-on version) is used to provide a toolset for test management in Jira. Another option for test management tooling is to purchase a standalone application such as TestRail.
Jira integrated tooling or standalone?
When facing a question like “what do we choose and why?”, often a simple answer is just not possible. Let’s dive a bit deeper in the differences between Jira integrated tooling such as Xray, Zephyr and a standalone application such as TestRail.
First of all, let’s talk functionality. Even though all tools provide support for test management, there are slight differences. Below you can find an overview of differences in functionality between TestRail, Xray and Zephyr. Some of these differences such as the lack of BDD support for Zephyr can be dealt with potentially paid addons, for the sake of comparison, these have been marked with an asterisk*.
Look and feel
Each of the three tools discussed provide support for creating and executing test cases, as well as assigning tasks to users and reporting, but each tool has a different approach to these functionalities resulting in a different look and feel.
Xray uses Jira tickets to implement test cases, pre-conditions, test sets, test executions and test plans. Because everything occurs within Jira, there is no problem linking all of them together providing full traceability from requirement to test result and everything in between. Reporting is done by Jira Dashboards and on the test execution Jira tickets.
Zephyr uses Jira tickets solely to create test cases. Zephyr also provides with an additional view where test cases can be grouped to test sets and test cycles. This additional view also provides users a way to execute test cases and report them. Since Zephyr also uses native Jira tickets, traceability and passing information within the Atlassian stack is automatically done when linking items such as tickets and dashboards.
TestRail doesn’t have issue management capabilities by itself but implements a seamless two-way integration with Jira. All tickets in Jira are linked to TestRail and test cases and progress on those test cases is linked to Jira. Projects, test cases and test runs are created within TestRail and everything can then be reported to Confluence, Jira, mail or simply downloading a report.