Tooling

Andere artikels

The burden of choosing the best Test Management Tool

16 Feb

Posted by: Eva

Category: Business

No Comments

A blog about Test Management Tools… one of many you might think. However, this is not a blog about which Test Management tool is the best, but a little insight on the usage of Test Management tools by our own Brightest consultants. Which ones are popular? What tips and tricks do they have to share with you to make the use of a particular tool just a bit more efficient or fun?

This is the overview of Test Management Tools currently used by our Brightest consultants:

Test Management Tools used by Brighters

→ TestRail was the al time favorite, both in usage % as in preference.
→ xRay was the least popular tool, although it isn’t the least used tool at our customers.

Knifes or screwdrivers?

As a tester – gatekeeper of qualitative software – we don’t always have the luxury to choose the best Test Management Tool for the environment and tooling eco system we work in. Of course, you can tighten a screw with a knife, but it’s not the best option. Often, our consultants must use that knife instead of the correct screwdriver. So, they must be inventive and bend the Test Management Tool to adjust it to their needs.

Automation

With the evolutions in software development, testing can’t stay behind. More and more tests must run more frequent and faster than ever to ensure consistent quality in continuous evolving software. That’s why managing and automating your tests becomes more important than ever.

This means, good integration with (most) test automation tools to follow up their automatically triggered executions, is almost inevitable nowadays. Since this gives you direct insights in what’s going on with your software before any change is deployed into production at any time.

Of course, you can tighten a screw with a knife, but it’s not the best option.

Reporting

Automation is only one part of the puzzle. We must schedule, follow-up and evaluate the results of our test efforts. This brings us to the importance of decent reporting and the possibility to tweak these reports to the needs of the different stakeholders. Gathering all information about all kinds of testing and test phases, in a transparent way ànd in one place, will enable your team to make better decisions and improve their way of working. Which immediately impacts the ROI of software development.

Integrations

When developers create new features, they want to know about the status and progress of testing these new features. The follow-up and building of those features are often done in other tools then Test Management Tools. So, your Test Management Tool must be able to ‘speak’ with the development tools to enable the best teamwork and get all the benefits out of your tooling.

Here’s an overview of the top 5 tools used by our consultants and their ability to integrate seamlessly with the most popular development tools:

Test Management tool	Automation tools	Azure DevOps	JIRA
TestRail	V	V	V
MicroFocus ALM	V	V	V
xRay	V	X	V Plug-in of JIRA
SmartBear Zephyr	V	X	V Plug-in of JIRA (EE: 2-way integration)
Azure DevOps TestPlan	X	V Part of Azure DevOps	X

Features

When our consultants are asked about what they expect from a good Test Management Tool, these are their top 5 answers:

User friendliness: not too many clicks and complexity to do simple things
Reporting and dashboarding: having a clear overview of what’s going on!
Seamless built-in integration with other (dev or automation) tools:
→ good traceability
→ a central overview
……..→ easy follow-up on progress of requirements (user stories / features), (automated) test cases, and bugs / incidents

Structured overview of :
→ tests cases and test plans: reuse and maintainability
→ test runs and timestamps: debugging and when / what is done by whom?

Test case, test run and bug versioning: history, maintenance and improvements

.
So, how do the top 5 tools used by our consultants rate on these features? ↓

Test Management Tool	Like	Dislike
TestRail	▪ Meets the top 5 features: User friendliness, reporting, integrations, structured overviews, versioning	▪ Non-standard markdown
MicroFocus ALM	▪ Auto complete feature ▪ Filtering	▪ Reporting is difficult ▪ Complex
Azure DevOps Test Plan	▪ Seamless integration with Azure DevOps (easy teamwork with developers) ▪ Shared steps ▪ Test status visible on user stories and development boards (reporting)	▪ Export is cumbersome ▪ Focus on unit and user story (exploratory) testing (reuse is difficult) ▪ Complex
xRay	–	▪ Complex handling for simple tasks ▪ Focus on unit and user story (exploratory) testing (reuse is difficult)
Smartbear Zephyr	▪ Seamless integration with JIRA (easy teamwork with developers) ▪ Automation ▪ Reporting	▪ Focus on unit and user story (exploratory) testing (reuse is difficult)

Tips

Consultants can’t always choose the tools they work with, which is a big advantage, because they learn to use many different Test Management Tools integrated with various development or automation tools. This way we discover new gems and / or get creative in ways to bend the toolset our way.

Here are some tips and tricks of our experts to help you when you come across one of these tools:

Tool	Tip
TestRail	▪ Learn their many keyboard shortcuts and its specific markdown
MicroFocus ALM	▪ Learn basic SQL since it’s needed for decent analysis and reporting
Azure Test Plan	▪ Create a separate test-plan for each application / project as a (master) regression set, which you don’t use to execute your tests. ▪ Select from this master set the specific test cases for each user story (linked via requirement-based suites) which you want to execute and follow-up in a separate test plan (for each cycle/sprint or for 1 application/project).
xRay	▪ Use the term ‘run status’ for the execution status in the current run
Smartbear Zephyr	▪ There is no way to group tests and link that group to a feature. You must do this test by test or work in cycles per epic/features.