What is a test management tool?

A test management tool is the central tool of your QA process. It provides support for managing test cases, test sets and test cycles. Other than providing full traceability between all test ware, it assists users in generating reports, logs and in-dept analysis.

A good test management tool provides support for everyone in the software development life cycle (SDLC) ranging from QA to dev and any stakeholder involved. But how do we recognize a good test management tool and are there really that many differences?

Well for starters, let’s look at one of the most important tasks for a test management tool: integration possibilities. Because everyone in the SDLC needs to benefit from the test management tool, it’s important that all used tools can interact with this central tool. For this reason, a popular choice for test management tools is one that is built into tooling which everyone already uses such as the popular issue and project management tool Jira. In this case an addon such as Xray or Zephyr (add-on version) is used to provide a toolset for test management in Jira. Another option for test management tooling is to purchase a standalone application such as TestRail.

Jira integrated tooling or standalone?

When facing a question like “what do we choose and why?”, often a simple answer is just not possible. Let’s dive a bit deeper in the differences between Jira integrated tooling such as Xray, Zephyr and a standalone application such as TestRail.

Major differences

First of all, let’s talk functionality. Even though all tools provide support for test management, there are slight differences. Below you can find an overview of differences in functionality between TestRail, Xray and Zephyr. Some of these differences such as the lack of BDD support for Zephyr can be dealt with potentially paid addons, for the sake of comparison, these have been marked with an asterisk*.

Look and feel

Each of the three tools discussed provide support for creating and executing test cases, as well as assigning tasks to users and reporting, but each tool has a different approach to these functionalities resulting in a different look and feel.

• Xray
Xray uses Jira tickets to implement test cases, pre-conditions, test sets, test executions and test plans. Because everything occurs within Jira, there is no problem linking all of them together providing full traceability from requirement to test result and everything in between. Reporting is done by Jira Dashboards and on the test execution Jira tickets.

• Zephyr
Zephyr uses Jira tickets solely to create test cases. Zephyr also provides with an additional view where test cases can be grouped to test sets and test cycles. This additional view also provides users a way to execute test cases and report them. Since Zephyr also uses native Jira tickets, traceability and passing information within the Atlassian stack is automatically done when linking items such as tickets and dashboards.

• TestRail
TestRail doesn’t have issue management capabilities by itself but implements a seamless two-way integration with Jira. All tickets in Jira are linked to TestRail and test cases and progress on those test cases is linked to Jira. Projects, test cases and test runs are created within TestRail and everything can then be reported to Confluence, Jira, mail or simply downloading a report.

User friendliness

Another thing to consider is the user friendliness. Who needs to work with this tool? Are my users used to working in Jira so will it be easy for them to adapt to Zephyr or Xray or have they never worked with any of those? If you were to ask users of Zephyr, they would prefer to keep working with Zephyr rather than Xray or TestRail and vice versa. In the end it usually comes with personal preference.

However, there is still something very important to keep in mind. Whatever tool you select, ensure that there is an incremental roll out and that there is plenty of training. This provides your users with context and gives them time to adjust to the new tool. This maximises success rate of any tool, not just a test management tool.

Whatever tool you select, ensure that there is an incremental roll out and that there is plenty of training.

Pricing

Off course, a very important part of any decision is and always will be: pricing. If a tool is too expensive, it’s not taken as a viable option because it wouldn’t be feasible to implement this. If we look at the current pricings of TestRail, Xray and Zephyr we immediately notice a big difference. However, it’s not as easy as it might seem at first glance. Please note that these are average prices, taken from the most recent pricing on Atlassian Marketplace and the website of TestRail in May 2020. Actual prices may vary.

If we look at these prices, we can immediately see that TestRail is far more expensive, but that might be deceptive. In the case of a standalone tool, only the collaborators such as the QA staff need to have a paid licence. Whereas if you install an addon for Jira, you will have a licence cost for every Jira user, even though they might not even use the test management tool. This could result in a potential huge hidden cost.

Automation

Automation nowadays is impossible to ignore in any QA implementation. This is also the case when selecting a test management tool. Each of the three tools described can be used in combination with test automation. Some are just a bit more flexible than others and some require a bit of creativity of your test automation engineers.

• Xray
Xray uses Jira tickets to implement everything. This means we can use Jira’s native REST API to create or update the Xray entities. Test results are also stored in Jira tickets, so extracting this information is also done with Jira’s REST API. An additional benefit of Xray is that it provides native support for CI tools such as Bamboo and Jenkins.

• Zephyr
Since Zephyr uses a custom view to execute tests and reporting, we can’t use the Jira API to get or set information to the related Zephyr tickets. Fortunately, Zephyr’s creators Smartbear provides a solution for this problem in the form of the Jira addon “ZAPI”. ZAPI enables users to use an endpoint provided by Smartbear. This endpoint communicates with the Jira cloud to query data or to add and edit testware in Zephyr.

• TestRail
→ The creators of TestRail provide several open source libraries in various languages which can be implemented with your test automation framework to directly interact with all your test cases in TestRail. Results can be pushed immediately to TestRail and results are accessible via its API.
→ Testruns are created from the framework
→ Testresults with comments and linked defects can be pushed directly into the created testrun
→ Testruns can be closed, milestones can be added….

Conclusion

Choosing a tool is not easy and there is no simple answer to the question: ” which is the best test management tool?” It all depends on various factors and every tool has certain benefits and downsides. If there was one tool which was perfect, no other tool would exist.

So how do we proceed then? Well, start by making a list of must haves and nice to haves. Once you have found a tool which has all the must haves and enough nice to haves, you can start with a proof of concept. In the end, all vendors will promote their own tool to be the best, but often a good proof of concept can quickly determine whether it’s indeed a good solution for your needs.

Afterwards set up pilot project and take time for a lengthy evaluation. This ensures that all strengths and weaknesses are known and tested against your requirements. After this pilot project you will have a solid look and feel of everything and you can define a good way of working with this tool. Following this pilot project, roll out the tool incrementally in your organisation and provide training where needed. During this phase you will also learn more about the tool, maybe even hidden flaws or even strengths. It’s never too late to tweak things which could benefit your process.

It’s however never easy to make a solid choice, especially since this choice impacts so many areas of the SDLC. Our professional team at Brightest is ready to answer any question or provide coaching  that fit your need. Feel free to contact us. We  look forward hearing from you and help you with our expertise.