Fast Data is the emerging hot topic of discussion for business leaders seeking to get ahead of the next wave of data utilization. But Fast Data isn't just an evolution of Big Data; it's a market force unto itself that's asking more of traditional and start-up vendors in both traditional DBMS and AI.

I spent a (surprisingly snowy) morning this week talking with AI and Big Data thought leaders at the Global Data Summit 2017 in Colorado. While there’s no shortage of topics to hold their current interest, none was a higher business priority than solving the challenge of managing Fast Data through the application of AI. The consensus is certainly that the organizations that can best address this challenge will also be those best positioned to compete and win overall. But how best to get their arms around this opportunity and move forward effectively?

First, it's important to distinguish between the challenges of leveraging Big Data and Fast Data. Big Data is generally data at rest; it's explored at (relative) leisure, and doesn’t change so quickly or accumulate so rapidly that offline analytics become impossible. AI has no shortage of applications in Big Data, but in that environment, it's more the ability of an AI platform to manage complexity and work at scale that offers value.

Fast Data, by contrast, accumulates quickly and can change substantively within the course of a day or even an hour. Think adtech here, or online gaming, or vendor pricing with commodity costs as an input; vast amounts of data need to be ingested, analyzed, and understood by the second in order to secure the right ad placement at peak value, or to manage complex MMO games, or to ensure that pricing continuously secures competitive advantage at acceptable margin.

Fast Data becomes Big Data quickly, just by nature of its accumulation rate, and while it's often valuable to query the Big Data that Fast Data becomes to understand trends and cyclicality, Fast Data will always yield its peak value at the millisecond level. It’s the freshest layer that offers the most insight. The Big Data value proposition to retailers, for instance, is looking for cyclicality of demand and regional demand preferences over time; the Fast Data value proposition is understanding the products a shopper is looking at right now and making real-time recommendations for, say, footwear and accessories to match. AI can accomplish both tasks, but often needs to be set about different tasks – with different priorities and ground truths – to succeed. The implications for every phase of the organizational data analysis and workflow management platform – from MDM and data hygiene to machine learning and AI application – are immense.

In response, expect to start seeing considerably more focus from major AI platform vendors not just on depth of understanding by their products, but speed of reaction as well. Organizations big and small in the traditional data sector, from Oracle to VoltDB, are developing and marketing smarter Fast Data solutions, while AI leaders – like IBM and Wipro – are building capabilities for faster data management within their AI platforms.

Servicing this rapidly-growing need for Fast Data management will be a convergent effort: the smart will get faster… and the fast will get smarter.

Dave Mayer is a Senior Analyst responsible for NelsonHall's RPA & Cognitive Services research program, covering the areas of robotic process automation (RPA), artificial intelligence, cognitive business, and machine learning. He is currently working on a major global project evaluating RPA & AI technology. To find out more about the project, contact Dave Mayer or Guy Saunders.