DEBUG: PAGE=domain, TITLE=NelsonHall Blog,ID=1469,TEMPLATE=blog
toggle expanded view
  • NelsonHall Blog

    We publish lots of information and analyst insights on our blogs. Here you can find the aggregated posts across all NelsonHall program blogs and much more.

  • Events & Webinars

    Keep up to date regarding some of the many upcoming events that NelsonHall participates in and also runs.

    Take the opportunity to join/attend in order to meet and discover live what makes NelsonHall a leading analyst firm in the industry.


Subscribe to blogs & alerts:

manage email alerts using the form below, in order to be notified via email whenever we publish new content:

Search research content:

Access our analyst expertise:

Only NelsonHall clients who are logged in have access to our analysts and advisors for their expert advice and opinion.

To find out more about how NelsonHall's analysts and sourcing advisors can assist you with your strategy and engagements, please contact our sales department here.

test IO: Strategy Update on EPAM’s Crowdtesting Business


We recently talked to test IO, the crowdtesting vendor that was acquired by EPAM Systems last April. We wanted to understand the crowdtesting positioning of the company, learn more about the User Story crowdtesting offering launched in December 2019, and understand how test IO fits within the larger EPAM organization.

Founded in 2011, test IO today has 200 clients, many in the retail, media, and travel industries, with a sweet spot around customer-facing applications.

test IO has positioned its crowdtesting portfolio in the context of agile

test IO has focused on functional testing (along with usability testing), targeting agile projects in recent years. Crowdtesting in the context of agile development projects (i.e. continuous testing) remains a priority and test IO recently launched an offering named User Story testing.

The crowdtesting industry has to date relied on two main offerings: exploratory testing and test case-based testing. Under the exploratory testing model, a crowdtester gets few instructions on what to test and goes through the application-in-test by intuition. With a test case-based approach, the crowdtester relies on detailed instructions on how to complete a task or a business process.

With its User Story approach, test IO is promoting a method that lies somewhere between exploratory testing and test case-based testing. In agile development methodologies, user stories are the instructions given to software developers in the form of “as a [user/admin/content owner/], I want/need [goal] so that [reason]”. The challenge from a testing perspective is to test these user stories by setting up acceptance criteria that are specific enough to be tested but still meet the spirit of agile that relies on loose requirements and iteration.

Speed is of the essence, as with agile projects. test IO argues it can achieve User Story testing in approximately two hours, from the moment it sends a mobilization campaign to its members to the moment it receives defects back from crowdtesters. The company highlights that working with a population of crowdtesters with the right level of testing skills helps in achieving speed. test IO believes it has enough members to react quickly to any project invite while providing the right defect coverage.

Exploring portfolio synergies with EPAM

test IO has also expanded its delivery model from public crowds and private crowds (e.g. employees of the client) to the EPAM internal crowd. test IO can rely on EPAM’s 6k testing practice members to expand its reach and bring to the client career testers with vertical experience while reassuring a client that its application-in-test will be exposed to EPAM personnel only.

User Story testing and internal crowds are just the beginning: test IO and EPAM intend, over time, to expand their crowdtesting capabilities to specialized services: performance testing will be the first of these.

AI is also on the agenda. One of its first AI use cases has been around crowdtester selection, based on the technologies and experience needed by the client. A current priority is defect review. For each crowdtesting project, test IO will review the defects logged by crowdtesters and remove duplicates. The company wants to automate most of its test duplicate activity to free up time and focus on data analysis.

In a later phase, test IO wants to run through the defect data it has built during its nine years of existence and identify crowdtesting best practices and application defect patterns.

test IO hints that EPAM is exploring how to best use its experience in crowdsourcing and expand it to other software services activities. We will be monitoring with interest how EPAM develops the crowdsourcing model of test IO. Despite claims of attracting young talent through different employment models, most vendors still rely on permanent or freelancer positions. With test IO, EPAM may be able to invent a new employment model that will expand from testing to other application service activities.

No comments yet.

Post a comment to this article: