By Irene Aldridge
This article first came out on Huffington Post: https://www.huffingtonpost.com/entry/59dcff8de4b0b8751b878b3d
Understanding the capabilities and disruptive threats of big data is a must for all sophisticated finance professionals today. This article considers the development and current state of Big Data Finance, as it relates to innovation in investment management, intermediation (broking), insurance, risk management, and most other aspects of Finance.
Big Data Finance is a revolution as well as an evolution that has come prominence several decades. Chartists and other technical analysts since at least the 1920s have utilized market data to derive upcoming patterns of prices. Perhaps the first formal breakthrough in Big Data Finance occurred in the 1980s, when companies like Bloomberg began packaging and delivering market data in large sets to investment professionals. Big Data Finance 1.0 allowed for extensive data mining, yet with still limited mathematical tools. Computer terminals were expensive, clunky and slow. Still, the inferences were valuable and much sought after.
Big Data Finance 2.0 came with the explosion of the Internet in the late 1990s. Suddenly, it became possible to receive streaming real-time or near-real-time financial data. The innovation enabled the growth of financial technologies such as electronic trading, and market making. Efficient online communication facilitated deal-making, globalization of trading, and much more. The efficiencies of Big Data Finance 2.0 streamlined financial services’ processes, and brought down previously prohibitive costs of financial transactions by a factor of 100.
Big Data Finance 3.0 is presently upon us. Big Data Finance 3.0 is really about managing the scale of data and extracting the information within. Today’s big data is about faster, better analytics, an ability to extract that “needle from the haystack” using the latest data science inferences, and storing, managing and integrating ultra-large sets of streaming and historical data of all kinds: market data, social media data, news, regulations, announcements, and so on.
Irene Aldridge is a Visiting Professor of Financial Mathematics and Microstructure at Cornell University and President of AbleMarkets.com, a real-time streaming financial research company. She is the author and co-author of several best-selling books related to data in financial markets, including “Real-Time Risk: What Investors Should Know About Fintech, High-Frequency Trading and Flash Crashes” (2017, Wiley, with Steve Krawciw, translated into Chinese) and “High-Frequency Trading: A Practical Guide to Algorithmic Strategies and Trading Systems” (2013, Wiley, 2nd edition, translated into Chinese). She will discuss her latest research on clustering applications to intraday statistical arbitrage at the Big Data Finance conference in Miami on December 5, 2017.