In the world of testing services, crowdtesting stands out. While efforts towards automation are accelerating in testing services/quality assurance (QA), the perception that crowdtesting is labor intensive and relies on communities of tens of thousands of testers seems at odds with this. This perception is no longer valid: the crowdtesting industry has changed.

Managing crowdtesting communities

A core element of the activity of crowdtesting vendors is around managing their communities. In this industry, one-off events (e.g. where participants join for an event like a hackathon) are uncommon. Quite the contrary, crowdtesting vendors provide their activities on an ongoing basis, increasingly as part of one-year contacts.

Crowdtesting vendors have focused their effort for the past two years on getting a detailed understanding of their communities, on the skills and capabilities of crowdtesters, and helping them enhance skills through online training.

Crowdtesting firms continue to invest in their communities, focusing on two priorities: to accelerate the identification of relevant crowdtesters for a given project’s requirements, and to increase the activity level of communities to make sure crowdtesters with specific skills will be available when needed. Speed is now of the essence in the crowdtesting industry, primarily because of agile development.

Agile & crowdtesting

Crowdtesting vendors have largely repositioned their offerings to accommodate fast testing turnaround times. This makes sense: most of the technology tested by crowdtesters is mobile apps and responsive websites, which have driven the adoption of agile methodologies.

Crowdtesting vendors are now often providing agile testing during the weekends, making use of the availability and distributed location of crowdtesters.

Automation increasingly takes a primary role in crowdtesting.

Automation is changing

Initially, the crowdtesting industry relied on the comprehensiveness of its software platforms, focusing on three personas: the crowdtester for reporting defects, the crowdtesting vendor for managing errors and providing analysis and recommendations, and the customer for accessing the results and analysis.

Crowdtesting vendors continue to invest in their platforms, but that investment is not enough anymore. AI and enhanced analytics have made their way into crowdtesting. An example of an AI use case is defect redundancy identification: cleaning the raw defect data and eliminating, for instance, duplicate defects. In the past, this defect analysis was done manually; increasingly, it is done using ML technology to identify bugs that have similar characteristics.

Another example is emotion analysis where, in addition to their defect reports, crowdtesters also provide videos of their activities. In the past, emotion analysis required human video analysis; in the future, AI will help identify the emotions, negative or positive, of the crowdtester. This will help both the crowdtesting vendor and the client in knowing where to look within a given video.

The crowdtesting industry is also pioneering other AI use cases. AI use cases have expanded from enhanced analytics to automation. The most advanced crowdtesting vendors are looking to create test scripts automatically, based on the gestures of the tester, or use web crawlers. Over time, the crowdtesting industry will combine manual activities and automation.

IP protection

This is for the future. For now, crowdtesting is still a niche activity and needs to gain scale. A significant challenge for scaling up is IP and privacy protection. Clients still fear IP may leak to competitors or consumers during the testing cycle, and the crowdtesting industry is trying to address this fear by relying on ‘private crowds’.

We think that the crowdtesting industry will become more successful when clients recognize that not all their mobile apps or websites are strategic differentiators and that the impact of an incremental new feature in a mobile app being leaked in the open will be limited. And clearly, agile is promoting this incremental approach, which makes crowdtesting more acceptable over time.