Have you ever heard of Test.ai? Well, it’s a company that is trying to change the way mobile apps are currently being tested. Testing is critical nowadays, definitely for mobile apps, once a mobile app is released it gets immediately reviewed by the users. This review will either mean the success or failure of this app. Testing these apps often becomes a bottleneck within your organization. In mobile test automation Appium is one of the more popular tools. Within Appium, you typically identify an element by specifying a selector (using the id, css, xpath, …). Because the application-under-test can change frequently, the maintenance of these selectors is the reason why testing can become a pain. But what if Appium was able to ‘think’ and recognize elements visually instead of using these common selectors …?

Test.ai, together with Jonathan Lipps, the lead contributor of Appium, tried to add a bit of their AI-power into Appium. They created a plugin which incorporates a machine learning model from Test.ai designed to classify icons (for those who are interested in the training data for this model, it is open-sourced). This model is capable of telling us what a specific icon represents (a shopping cart, a black arrow, …). By using this plugin we can find icons on the screen based on their appearance rather than using the internal identifiers.

How this works? The model that is used by the Appium plugin is trained to recognize some commonly used mobile icons, i.e. shopping cart, an edit button or even the twitter logo. The plugin will automatically take screenshots of each element on the screen of the mobile app and send it through the model. This will result in the plugin ‘recognizing’ the element you’re looking for. We will not go in too much technical details, but at the end (meaning after configuring Appium to use the custom plugin) you will be able to write a simple line of code to identify the element using the AI plugin : driver.elementByCustom(‘ai:cart’); In short, this line of code will tell Appium to use a custom locator instead of the supported locator strategies (like id, css, xpath, …). We specify it to use the “ai” custom locator and we ask it to search for a shopping cart icon. And that’s it, plain and simple.

In many cases this approach is more flexible than the existing locators because the model is trained to recognize icons without any context and without the requirement to match only one precise representation of a shopping cart icon. This means that using this plugin to find a “cart” icon will work across different apps and different platforms, without worrying about the minor differences between these icons. It results in a faster creation of our automated tests since we no longer have to look for a unique way to identify the element. And if for some reason, the icon has changed, our tests will still work as long as the new icon still looks like a shopping cart. This will lead to fewer broken tests and will significantly improve the maintenance of our automated tests.

This all looks amazing and promising, but unfortunately we’re not there yet. This is only the “Hello World” for AI-driven automated testing. The model still needs to be trained more and there are also some significant limitations that have to be taken into account. The plugin is only able to recognize icons. It isn’t suitable to identify a text box, a drop-down list, a button with text, or any other element that isn’t represented by an icon. Also, the model can only reliably identify elements for which the model has been trained. On top of that, the process of retrieving all elements on screen and sending them trough the model to find out which one is the element we need, is fairly slow. All of these areas will probably see major improvements in the near future. As for now, this plugin is probably not yet useful for your day-to-day job, but it demonstrates that concrete applications of AI in testing are not only possible, but actual.

For those who are interested in the Appium plugin, you can find it on github. If you are interested in a more in-depth explanation of how the trained model works, check it on David Paul Röthlisberger’s blog. Other useful information can be found on Test.ai and Jason Arbon’s pages, or on the Appium Pro blogpost. And you can also watch the entire presentation of Jason Arbon on SeleniumConf.

Enjoy!

frederique.dewinter@brightest.be