NelsonHall continues to explore how cognitive is reshaping software testing services, and here I look at how Amdocs is automating chatbot testing.

The market is shifting from continuous testing to cognitive

Over the last year, for many testing services providers, the focus evolved from creating continuous testing platforms (in the context of agile and DevOps adoption), to incorporate AI with the intent of automating testing beyond test execution.

The next priority of the testing industry is going beyond the use AI to automate testing to testing AI systems and bots – which brings new levels of complexity. Some of this comes from the fact that AI and bot testing is new, with methodologies to be created and automation strategies to be formulated.

Chatbot testing is one new area where the industry is getting organized, with initiatives such as making sure the training data is not biased and creating text alternatives (“utterances”) to the same question to a chatbot: whatever way the question was asked, the response must consistenly be the same.

Amdocs’ approach to chatbot testing

To most readers, the name Amdocs is probably reminiscent of OSS/BSS and other communication service provider-specific software products, and rightly so. But Amdocs has also become an IT service company, providing services around its own products, non-Amdocs products, and custom applications.

Amdocs recently briefed NelsonHall on the work it does in chatbot training and testing. Its approach to chatbot training and testing relies on several priorities:

• Understanding what the customer wants to achieve (the “intent”), e.g. to check the balance of its data plan
• Finding out the context, or reason of a call, e.g. to understand what current mobile data plan the customer has
• Keeping the “flow” of the chat relevant, e.g. making sure the chatbot’s response remains related to the context and the intent.

Automating chatbot training and testing

Amdocs uses several technologies to help automate chatbot training and testing, for example NLP for word clustering, and ML for classification to help in understanding the intent of a customer’s interaction with a chatbot. Amdocs highlights that it can achieve an accuracy level of ~96%.

Amdocs relies on integration with other applications and APIs for its context needs.

For its chat flow needs, Amdocs is using the BDD approach. Under BDD, testers or business analysts will write test cases (named feature files) in English (and then in the Gherkin language) and translate them into test scripts. With this approach, Amdocs creates series of scenarios guiding the bot step-by-step on how to react to the customer interactions.

Amdocs also uses open source software Botium, which relies on the same principles as BDD. The integration of Botium helps it testing chatbot technology from vendors, including Facebook, Slack, Skype, Telegram, WeChat, and WhatsApp.

Amdocs has integrated the BDD approach in its main testing framework, Ginger. Ginger integrates with CI tool Jenkins, which means the BDD scripts can be run in a continuous testing mode as part of an iterative approach to training and testing. The integration with Ginger also provides back-end integration testing, including API testing, notably for its context needs.

Testing AI systems

Amdocs’ approach brings some priority on what to test in chatbots, and some level of automation. This is the beginning: chatbots are currently relatively simple and, as their sophistication grows, so will their testing needs.

This, of course, raises further questions:

• Further methodologies and governance. Testing data is usually scarce, and vendors sometimes use training data for testing purpose. Also, now, training and testing activities tend to be similar. This raises the question of the role of testing’s independence
• Chatbots will evolve into voice bots. This raises the question of accents and local expressions
• Chatbots are simple enough as they are deterministic, which means their output can be predicted. But how do you test AI systems where you cannot predict the output?

NelsonHall will be publishing its latest report on software testing services soon, focusing on next-gen services, including the role of AI and RPA in testing, along with mobile and UX testing.