The QE/testing industry has quickly adopted GenAI, and the nature of GenAI use cases has significantly changed in the last 18 months. Initially, the industry went through a discovery phase, identifying how to use LLMs for QE. The first GenAI use cases were test authoring (generating test scenarios from user stories/requirements, test cases, and test scripts), test data management, and knowledge management.

GenAI is still a new tool in the QE industry, but we are starting to see the industry ask the right questions; for instance:

• How will GenAI co-exist with AI, whether ML/DL, NLP, or even machine vision, and does Agentic AI have a role to play?
• Will GenAI replace the many tools used in software testing, resulting in tool fragmentation, inhibiting automation?
• How can we bridge the QE and software development worlds with GenAI?

We recently talked to Infosys QE (IQE), Infosys’ testing unit, about how it applies GenAI to QE/testing. Infosys’ QE unit is the second largest practice within the firm, and its word carries weight in the industry. The discussion with Infosys showed how much the firm has advanced in a very short time. Below, we discuss some of our takeaways.

Where to replace Machine Learning with GenAI

Infosys QE has started assessing when to use LLMs rather than ML or NLP. The approach is essential, as LLMs bring immediate benefits at limited costs; ML is less immediate (it requires data training for three to six months) but brings higher accuracy than LLMs. Infosys’ experience is that GenAI can replace ML in analytics (for instance, test defect classification for identifying duplicate defects) and for knowledge management (replacing bots with LLMs).

Infosys points out that other AI use cases, such as logs analytics and test suite optimization (reducing the number of test cases), will rely on ML rather than on LLMs. Log analytics, for instance, requires significant processing, and commercial LLMs have token limitations to this processing. Eventually, this token limitation will fade away.

Infosys rolls out Agentic AI concept for testing

IQE has already deployed Agentic AI to QE (‘Agentic AI Defect Engineer’), decomposing activities into tasks. Its first scenario is:

• One agent identifies similar test defects (rather than using NLP) and finds if an existing test case is relevant to the defect
• If required, another agent will create a test case/script
• A third will conduct defect RCA through defect classification.

Three requirements will favor the emergence of Agentic AI:

• Access to tools; for instance, for executing functional testing
• Knowledge, using RAG
• Memory, thanks to the recent reasoning capability found in several LLMs.

Next in line is unattended functional test execution and solving the issues (e.g., lack of synchronization between the application under test and the testing execution engine) that make testing batches fail overnight.

Alongside Agentic AI, Infosys is also using its 1m test case repository to build knowledge management in the form of RAG, in support of LLM-based activities such as test case generation, targeting industry-specific applications.

Testing tool fragmentation: where GenAI plays

One of the challenges of QE has been around tools. Testing requires many different tools, some of which are COTS, others open-source. Even in functional test execution, many engines co-exist, such as Open Text, Selenium, and, increasingly, Playwright. The challenge is not only about the tool fragmentation but also that each tool has its own script format, and script migration was not easy in the past.

Infosys does not necessarily think that GenAI will replace existing tools. For now, it has positioned GenAI to augment the capabilities of the main tools in the market. An example is test data management. GenAI is now used to generate synthetic test data at little cost. However, the industry still needs specific tools to address complicated scenarios; e.g., creating tables in SAP.

Another example is static code analysis to evaluate the quality of an application under development. Infosys is finding that LLMs bring better results than current tools. If anything, GenAI will force testing ISVs to specialize their tools further, while LLMs will replace mainstream usage. NelsonHall believes that GenAI will lead to streamlining of the tool ecosystem.

Deploying GenAI for testing on development tools

While close, application development and testing remain very different and rely on different skills, tools, and processes. To bring GenAI for QE to application development teams, Infosys launched its ‘Pair Programming’ offering (an analogy is in application development, where one engineer develops code while another reviews the code in real-time).

Rather than deploying the ChatGPTs of this world in testing, Infosys decided to use development tools already commonly used by developers. Infosys has worked on several tools, mostly GitHub, using Copilot, and also GitLab Duo, Amazon Q, and Google Duet AI.

Initial use cases developed by Infosys are like the first use cases Infosys formerly developed for testing and include (BDD) feature file generation, test script generation, test script completion, and test script conversion. The approach relies on a repository of prompts developed by Infosys to standardize output generation (LLMs are non-deterministic models, and the output of prompts varies even more if the prompts are slightly different).

Still the start of the journey for GenAI in testing

We think the QE industry’s holy grail will be moving from a greenfield approach to brownfield. The industry currently uses GenAI to generate test artifacts. However, the reality is that GenAI must consider existing investments that organizations have already made: some clients have tens of thousands of test cases and scripts.

Infosys has started to address brownfield requirements. It has developed an LLM use case for test script migration from one language to another, and the migration is as simple as moving from one syntax to another. Each testing tool has specific functions and components that require heavy human intervention. Infosys QE believes it has good results in test script migration.

Other brownfield use cases are emerging, such as creating test scenarios from RPA tools and documenting processes out of test scripts in reverse engineering approaches. We expect brownfield to drive most GenAI activity for QE in the short term, and Infosys will be part of this momentum.

NelsonHall recently attended Dreamforce, the annual Salesforce flagship event in San Francisco. Billed as the largest AI conference in the world, Dreamforce 2024 saw a marketing push like none other, even by Salesforce’s standards, towards Agentforce, its new platform for creating autonomous AI agents capable of analyzing data, making decisions, and taking action. Agents built on the Agentforce platform can, for example, be deployed to answer customer service inquiries, qualify sales leads, and optimize marketing campaigns.

From the keynote, speaking sessions, and our conversations with Salesforce experts, it was evident that Salesforce’s future is now spelled completely as Agentforce. So much so that the rare moments when Agentforce was not mentioned were when Mark Benioff took swipes at competitors and what he calls the DIY culture, where enterprises try to do AI on their own.

What is Agentforce and why Salesforce claims it is different

The difference between Agentforce agents and AI copilots, according to Salesforce, is the ability to go beyond generating content to act autonomously without the need for human supervision. For example, when most of today’s virtual assistants get a query, they scrounge through a data set and throw up a response in the form of a link or a next step to do. But an agent, according to Salesforce, will initiate a purchase instead of merely suggesting the ways to make a purchase.

This capability is enabled by Salesforce agents from Agentforce and custom-made agents that enterprises will be able to put together using the Agent Builder, a low-code tool available in the upcoming Agentforce October release. These agents are powered by the new Atlas reasoning engine, which, according to Salesforce, improves itself based on the outcomes from previous interactions and not necessarily human feedback. Salesforce pitches itself as having a unique advantage over open-source models, claiming a comprehensive customer view from its multiple clouds, and better data quality orchestrated through its Data cloud. This, according to Salesforce, results in fewer hallucinations.

Salesforce service providers respond

NelsonHall spoke with more than 15 service partners at the event. The general tone was one of guarded optimism. Most agree with the perception that Salesforce had catching up to do in the AI race but that now, true to style, Salesforce has pitched itself right into the centre of the AI world. Clients are impressed with early Agentforce examples like OpenTable, Wiley, Bombardier, Wyndham and Saks Avenue, where agents are used to offer customized service to a customer, suggesting and executing a purchase and any subsequent return or re-delivery without human intervention. The initial hesitancy expressed by clients is to be expected for a release of this scale and significance, given that clients continue to grapple with the perennial issue of tech debt and messy subscriptions to SaaS platforms. However, what Benioff and his execs did well was build a compelling case for Agentforce and showcase possibilities across industries which will play out positively as adoption grows in a few months from now.   

Vendors are keen to get their hands on the upcoming release while awaiting an improved understanding of the commercial promise of $2 per transaction. The next six to twelve months will see a flurry of building agents across horizontals and industries. Initial adoption is likely to be faster in industries like retail that deal with seasonal surges, and for use cases involving personalized customer service.

The agent lined road ahead

Salesforce states that Agentforce represents the third wave of AI, the first being the predictive wave back in 2016 with Einstein, and the second wave involving co-pilots. The fact that Salesforce claims to have rearchitected the entire Salesforce product portfolio, including applications like Tableau and Slack, is a strong indicator of intent and future direction.

Salesforce is not alone in the agent gold rush, even if it sits atop by the sheer scale of its ambition. Oracle has spoken about having ~50 agents inside its Fusion applications by next year, and ServiceNow before Dreamforce announced use cases for agents across ITSM and customer service. With others also moving in this direction, Agentic AI is set to become more mainstream and build on the success of GenAI.

Tech Mahindra (TechM) recently briefed NelsonHall on TechM AppGinieZ, its GenAI solution for software engineering and SDLC.

Recent times have seen all major IT services providers release GenAI-powered solutions, targeting two broad scenarios: those that help developers build, test, and support applications more efficiently and those that enable capabilities like virtual assistants to help clients improve or even transform business processes.

Identifying GenAI opportunities internally

TechM’s AppGinieZ GenAI solution falls into the first category. AppGinieZ assists TechM’s teams in application services, including development, QE/testing, and support. AppGinieZ and other investments in AI/GenAI are a part of TechM’s strategic initiative, ‘Scale at Speed,’ where TechM promises clients accelerated delivery. This gives AppGinieZ senior management’s sponsorship and investment focus.

For now, TechM has taken a measured approach with AppGinieZ. It has been built by TechM’s ADMSNXT COE (application development and maintenance services), focusing first on the SDLC stages that provide opportunities for automation and then expanding into other use cases depending on client interest and GenAI’s evolving capabilities.   

Broadly, TechM AppGinieZ has two sets of capabilities.

• GenAI: generates text and code from different inputs like text, code, image, etc.
• Predictive AI: analyses data to perform activities like defect triage, risk-based testing, and log analysis.

TechM AppGinieZ supports the following use cases in the software development lifecycle, with code snippet generation, log analysis, and unit test generation seeing higher adoption.

• Requirements refinement: generates or refines stories based on requirements artifacts, simple text
• Code snippets: generates code snippets from text and image prompts
• Code documentation and commenting: generates comments for multiple languages and synopsis of the code functionality in text
• Unit tests: generates unit tests for the input code
• Log analysis: reviews logs and generates reports in multiple formats
• Code conversion: converts code from one language to another, like Java to Python
• YAML: generates YAML code for automation tools like Ansible.

To date, TechM has trained around 25,000 employees in AI pair programming. It claims that in some DevOps implementations using TechM AppGinieZ there was a 25% effort saving. NelsonHall believes the effort and cost savings will be more determinable and subject to further improvement once working with GenAI becomes institutionalized. Initial engagements also require more effort towards training, familiarisation, oversight, and human-led reviews, which, with time, will get faster for all vendors with a GenAI play.

Client case study

TechM highlights a North American client success story. Taking the traditional Three Amigo concept of business, development, and testing perspectives in Agile development further, TechM added AppGinieZ as a GenAI assistant, which it claims helped delivery teams perform story reviews and rewrites faster and efficiently generate test cases from refined stories. Encouraged by the engagement's success, the client and TechM have jointly filed for a patent for the solution.

QE/testing activities have been early adopters across the STLC lifecycle in implementing automation and AI, and now GenAI. TechM AppGinieZ is used in QE across: 

• Test strategy creation: converts requirement documents/user stories to a test strategy
• User story refinement: takes rough user stories from Jira and other sources and generates detailed user stories
• Test automation: generates test scripts from test cases
• Test data generation: generates synthetic test data in multiple formats
• Test case generation: generates scenarios and test cases based on inputs like requirement documents/user stories and images.

Test cases and script generation are currently the most popular QE use cases. In early deployments, TechM claims savings of 20-30% in the end-to-end test life cycle when using AppGinieZ. 

Overall, TechM feels that AppGinieZ and AI-driven development will have a positive and meaningful impact on margins in the future.

The road ahead

TechM showed us a demo of TechM AppGinieZ in action across QE and ADMS use cases. Based on the scenario, it can be connected to LLMs such as Gemini, OpenAI, Llama, and others. Its ability to be integrated with an increasing number of tools gives it flexibility and more acceptance into existing client landscapes.

Constant oversight and reviews are necessary when using GenAI, as the output can only be as good as the data quality and LLMs involved. This necessitates the infusion of client-specific rules to create a contextual layer to improve the accuracy of the response generated.

NelsonHall believes that the TechM AppGinieZ roadmap is pragmatic and will see the addition of more predictive AI, compatibility with more LLMs, and increased granularity of use cases across the SDLC.

We recently spoke with Datamatics about its $17m acquisition of Dextara, which was announced on April 1st.

Datamatics is a Mumbai-headquartered IT services and BPS provider with revenues of INR 1,550 Crore ($187m) for FY24, which ended March 31. It has three lines of business:

• Digital Technologies (IT services: 39% of revenues)
• Digital Operations (primarily F&A, plus intelligent process automation: 45% of revenues)
• Digital Experiences (CX, research & analytics: 16% of revenues).

The company generates 54% of its revenues from the U.S., primarily the SMB segment, and 24% from India. It has 300 clients globally, with the top 5 accounting for 23% of total revenue, the top 10 contributing 35%, and the top 20 over 50%.

Details of the Deal

Datamatics has expanded its capabilities with a series of acquisitions, including those of TechJini (mobile and web app development, 2017) and RJ Globus Solutions (voice-based BPS, 2018). Now, with the acquisition of Dextara Digital, a Salesforce Summit (Platinum) Consulting and ISV partner, Datamatics is boosting its fledgling Digital Technologies business (0.5% growth in FY24, declining by 12.4% y/y in Q4). Key drivers for Datamatics acquiring Dextara included:

• Expanding its Salesforce capabilities and improving the Digital Technologies business performance
• Augmenting its existing leadership with Dextara’s experienced management team.

Datamatics highlights that 25% of its clients are Salesforce users, an opportunity that it could not effectively target before the Dextara acquisition. In a previous effort to address this gap in its portfolio, Datamatics formed a JV with Cloud Route in 2022 with a view to start building a Salesforce services practice. These capabilities are now consolidated with Dextara, which is now the face of Datamatics Salesforce and leads any GTM initiatives. Datamatics and Dextara combined have around 150 certified Salesforce resources (130 in India and 20 onsite in the U.S.), serving around 80 clients.

Along with the Salesforce capabilities, Dextara brings to Datamatics an experienced management team led by the founder, Sreekanth Lapala. Prior to starting Dextara in 2020, Lapala managed around 25,000 resources as the global delivery head at Virtusa. This experience in building and leading large sales and delivery organizations that Lapala and his management team bring will help Datamatics beyond Salesforce as it aims to re-energize its technologies business and compete for bigger deals.

Dextara has an existing client base of around 50 American SMBs in the manufacturing, healthcare, professional services, high-tech, and BFSI sectors. Its core capabilities include CPQ, CLM, LWC, and Integrations, along with Einstein Analytics. It has developed two AppExchange-listed applications:

• Dextara CPQ, a customizable solution that clients can tailor to specific processes and industries. Features include product-attributed-based pricing. It was launched in FY24 and has two clients

• DXHealth+, a patient management product that targets small and medium elective healthcare providers whose services, like cosmetic surgery, are typically not covered by insurance. The product has 12 clients.

A Growth Engine for Datamatics

The existing Salesforce users among Datamatics’ clients are prospects for cross- and up-sell opportunities across Salesforce services, products, and Dextara’s IP. More importantly, the improved scale now makes Datamatics eligible for larger Salesforce deals (greater than $5m) and also positions it to convert some of its existing sales pipeline.

Datamatics has been busy on this front. It claims to have already introduced most of its clients to Dextara for consideration in Salesforce engagements. NelsonHall expects more account mining and cross-selling to Datamatics clients and an evolution of Dextara’s existing client profile to the larger Datamatics client base.

The Datamatics sales engine will also leverage Sreekanth’s leadership team’s delivery and sales experience to help manage large deal pursuits.

Datamatics has guided revenue growth of 7-8% for FY25, of which around 3-4% is organic and 4% (~$7m) from Dextara. Given this is the same as Dextara’s revenues last year, this is a conservative estimate: Datamatics looks to be factoring in time for the integration and client outreach completion over the next few months.   

Expect to see some investments in both Salesforce products and services. NelsonHall anticipates a particular focus on Salesforce Einstein and copilots in line with Datamatics’ corporate positioning as an AI-first service provider.