Big Data = Big Jobs in Finance for Years to Come, but Things Can Go Wrong

By Irene Aldridge

S&P Global just announced acquisition of IHS Markit for a whopping US $44 billion, the largest acquisition in the financial data industry to date. The amount reflects the current reality on Wall Street: premium data brings on premium revenues and valuations follow. IHS Markit’s flagship offerings, like the ISM Manufacturing Index, has long been a beacon to selected hedge funds, who paid a fair share of their investors’ earnings for the opportunity to squeeze an extra edge out of the data. As shown in our book, “Real-Time Risk: What Investors Should Know About Fintech, Flash Crashes and High-Frequency Trading”, in many cases the consumers pay to obtain the market-moving data as early as possible, to maximize their profitability. In fact, timely data is so valuable to hedge funds that a persistent rumor claims Twitter only remains in business by selling their post volumes instantaneously to the social-media platform’s hedge fund clients. 

As we describe in the book “Big Data Science in Finance” (co-authored with Marco Avellaneda, published by Wiley and forthcoming Dec 2020) utilizing cutting-edge data science techniques on less competitive data helps produce higher profits in a much less contested field.  In the book, we show techniques and their applications that all but guarantee that Big Data Science in Finance is the future.

As with any new technology, Big Data brings along societal benefits and pitfalls, creating new headaches and job security for regulators. Since the days I used to serve on a CFTC Tech sub-committee, I developed a unique perspective on how to assess the impact of new technology on society and what actions are optimal for the mutual benefit of the techies and the broader population. Some of these ideas, outlined in this article, may be something to ponder for the incoming Biden administration. 

Let’s start with the positive ways Big Data benefits society. First, as we show in our book, “Big Data Science in Finance,” Big Data allows us to make decisions without bias by letting the data themselves reveal critical trends and dependencies. 

Second, Big Data jobs will be numerous, whether in Finance or in other fields. The opportunity to utilize Big Data is just too powerful and very inexpensive for business not to embrace. The resulting huge cost savings for corporations ensure swift adoption of the technology that will require many qualified practitioners. The potential of the technology is so broad that it is likely to have a very high endurance, lasting through the years with iterative improvements.  

On the flip side, however, Big Data needs a safe path and plan for growth thought through by the regulators. In its current implementation, Big Data is unchecked — its parts, AI and machine learning, can be harmful if left to their own devices. This is really where the current administration should be focusing its regulatory efforts, in the Financial sector and other industries. Being proactive with containing the Big Data beast may be a better approach than the sadly common knee-jerk reaction fueled by the latest crisis induced by another technological innovation. Asking a question of “What Can Go Wrong?” may be a good start. 

Irene Aldridge is a co-author of “Big Data Science in Finance” (Wiley, 2020), an internationally-recognized quantitative and Big Data Finance researcher, Adjunct Professor at Cornell University and President and Managing Director, Research, of AbleMarkets, a Big Data for Capital Markets company. She is also the author of “High-Frequency Trading: A Practical Guide to Algorithmic Strategies and Trading Systems” (2nd ed, 2013, Wiley)