Why Big Data Is the New Must-Have Skill in Finance

Why Big Data Is the New Must-Have Skill in Finance

By Irene Aldridge

The history of Finance is full of Mathematical and technological revolutions. With the expansion of computer technology, mathematical innovations are not only abstractly interesting, but also very fast and profitable. Most recently, mathematical innovations in Finance first generated opportunities in quant techniques that were followed by the revolution of high-frequency trading, Now, the opportunity expands as Big Data techniques are implemented within Finance.

Many people are still puzzled by what is so special about Big Data. After all, Econometrics and other data processing tools have been used in Finance for decades. It is true that Econometrics comprises a part of Big Data analysis, known as Supervised Learning. However, the much more extensive and powerful set of Big Data techniques has barely registered on the Finance radar.

The tools comprising the core of  Big Data analytics deal with extensive data tables, sparse or missing values, as well as data clustering techniques, to name a few, that often make Econometrics look like a set of exercises for kindergartners. Gone is the need to discard the data due to incomplete information. Big Data welcomes the data in all shapes and forms, and disjointed, irregular and often even partially corrupted or biased data sets can be processed with equal ease to extract true values and relationship.

One of the key properties of Big Data is speed: the techniques lend themselves to efficient, fast and powerful data processing and inferences. Move over, high-frequency trading, Big Data can really process financial data in real time.

What is the key difference between Big Data and traditional data processing techniques? As explained in our new research paper, “Big Data in Portfolio Management”, available on SSRN at: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3142880, Big Data extensively utilizes the capabilities of eigenvalues, or principal values. The eigenvalues, first developed in the 18th century as an aid to solve differential equations, have since been extensively studied. Many properties of eigenvalues have been researched during and after the WWII, and, most recently, during the social media advertising boom of the past two decades. Eigenvalues featured prominently in the original search algorithm of Google and helped propel Google to its present prominence. In Finance, however, eigenvalues are underutilized at best and completely unknown even in some of the most prominent shops. Lots of work remains to be done in the area of implementing Big Data concepts in the financial sector.

Eigenvalues are also key drivers of artificial intelligence (AI) and automation. While the concept of AI seems mysterious and ominous, the mathematical concepts are straightforward and well-developed. A great place to learn about the techniques, their adoption in Finance and key trends is the upcoming 6th Annual Big Data Finance conference, scheduled to take place on May 11, 2018, at the brand-new Cornell Tech campus in New York City (BigDataFinance.org/BDF). The mission of the conference is to bring together financial industry, academia and government to facilitate the exchange of information and key developments in the are of Big Data in Finance.

Financial regulators certainly stand to benefit from Big Data techniques. There is a real chance to catch up and even overtake financial professionals with Big Data capabilities. However, a fair amount of work remains to develop and implement the Big Data market surveillance capabilities – the available research applied to Finance is still scarce, and the field is wide open to new discoveries and applications.

Irene Aldridge is President and Head of Research of AbleMarkets, a Big Data for Capital Markets company. She is a co-author of “Real-Time Risk: What Investors Should Know About Fintech, High-Frequency Trading and Flash Crashes” (Wiley, 2017,http://www.realtimeriskbook.com) and author of High-Frequency Trading: A Practical Guide to Algorithmic Strategies and Trading Systems (Wiley, 2014).