What Can You Do With Big Data in Finance?

What Can You Do With Big Data in Finance?

By Irene Aldridge

In the Spring of 2013, a lively discussion on LinkedIn went like this:

– “If someone says ‘Big Data’ one more time, I am going to throw up”, declared Head of Marketing at a prominent software firm

– “Agree, ‘Big Data’ is such an annoying buzzword”, chimed in Head of Research at a mid-tier broker-dealer

– “Uh, it’s such a fad”, stated a well-funded hedge fund manager.

And so it went: “Big Data” is annoying, fleeting, and, by implication, useless. Fast forward to today, even though Big Data is a much more established term, eyes still roll when the subject comes up. And the key issue is still the same: okay, we’ve got data, so what’s the point?

It turns out that the point of Big Data is not about having a ton of data, although data is a prerequisite. Big Data comprises a set of analytical tools that are geared toward fast meaningful processing large data sets. Meaningful is an important keyword here: Big Data analytics are used to derive meaning from data, not just to shuffle the data from one database to another.

Exactly what kind of techniques do Big Data tools comprise? First and foremost, Big Data techniques are geared toward analyzing a more manageable amount of data that still represent the entire data set. While working with a huge data set can be fun and result in precise inferences, it can also be a very slow and painful process. Big Data helps researchers probabilistically identify the aspects of the data that are most important, allowing researchers to discard some data while sacrificing little or no accuracy. Since most financial researchers are still trained to study low-dimensional data (say daily, or very structured intraday bars), Big Data Finance techniques deliver an immediate gratification in reducing the amounts of data required to manipulate.

A second application of Big Data is finding the most important factor influencing a particular set of behaviors. For instance, we can use Big Data to evaluate the key drivers of stock price behavior common to all stocks. That driver turns out to be the movement of the aggregate market, which can be proxied by the price of the S&P 500 ETF or beta of each stock (for more details, see recent research by Financial Mathematics Professor Marco Avellaneda of NYU Courant, http://www.math.nyu.edu/faculty/avellane/). Due to the ever-sophisticated algorithmic linkages among various financial instruments of all asset classes and ETFs, a slightest perturbation in one stock may indeed be felt across the entire market and distributed back to each individual stock! Likewise, Big Data is used to automatically “read” (process) streaming news and figure out what is wheat and what is chaff as it pertains to stocks on the fly.

A third application of Big Data is finding natural affinity groups of similar stocks or potential substitutes. The concept of similar stocks is a key driver of trading strategies such as statistical arbitrage. With Big Data Finance, statistical arbitrage delves in one layer deeper, letting mathematics identify the stocks that should be tracked and arbitraged. Finding stocks that are potential substitutes is also critical for portfolio managers and execution traders alike: in low-liquidity conditions, it may be critical to find a liquid substitute to execute a quick trade, and then rebalance the portfolio to its target composition at the trader’s leisure. (Yes, Professor Avellaneda, http://www.math.nyu.edu/faculty/avellane/, has written on this subject too.)

Big Data is used for pattern recognition of trading strategies, for instance, by AbleMarkets (http://www.AbleMarkets.com) to detect participation of high-frequency traders and institutional investors in the markets. Big Data is also used in the reconstruction of lost data (say, vanished en route to a trader’s desktop during an Internet outage), even catching errors in algorithms – can all be done and is done today with Big Data Finance analytics. The techniques are numerous, established, trusted and true. To the surprise of many, Big Data can be traced back to 1901, when a noted statistician Karl Pearson developed Principal Component Analysis (PCA), a technique widely used in today’s data analysis.

Are you using Big Data in your financial analysis? If not, it is about time to start. In today’s reality, summarized in “Real-Time Risk: What Investors Should Know About Fintech, High-Frequency Trading and Flash Crashes” (by Irene Aldridge and Steve Krawciw, Wiley 2017, https://www.amazon.com/Real-Time-Risk-Investors-FinTech-High-Frequency/dp/1119318963), one thing is given: either you consume data, or the data swallows you.

Where can one learn how to use Big Data Finance techniques? A great place to do so is the annual Big Data Finance conference (http://www.BigDataFinanceConference.com/BDF2017), which will take place on May 19, 2017, at New York University. In its fifth year, the conference gathers together Big Data academics, practitioners, and even regulators to exchange ideas and latest techniques to benefit all. As Gene Schupak, PhD, Head of Research at Suite LLC, a company with cutting edge analytics for fixed income pricing and a sponsor of Big Data Finance, says, “The precision offered by Big Data techniques is critical for success in today’s exceedingly automated finance environment. We see a higher and higher demand for multi-dimensional analytics, such as Suite LLC software, especially in fixed income, our area of expertise. The conference is one of the very few events we support because of the timeliness and the value of the forum the conference creates.”

Irene Aldridge is Managing Director, Head of Research at AbleMarkets, a Big Data for Capital Markets company, specializing in real-time and near-real time Software-as-a-Service improving execution, portfolio allocation and risk management. She is a co-author of “Real-Time Risk: What Investors Should Know About Fintech, High-Frequency Trading and Flash Crashes” (Wiley, 2017, https://www.amazon.com/Real-Time-Risk-Investors-FinTech-High-Frequency/dp/1119318963), and an author of “High-Frequency Trading: A Practical Guide to Algorithmic Strategies and Trading Systems” (Wiley, 2nd edition, 2013, https://www.amazon.com/High-Frequency-Trading-Practical-Algorithmic-Strategies/dp/1118343506).


  1. This is a bleeding interesting post. Hinder my locate mambo jambo

  2. Attractive portion of content. I just stumbled upon your website and in accession capital to say that I acquire actually enjoyed account your blog posts. Any way I will be subscribing in your feeds or even I success you get admission to persistently fast.|

  3. Знаете ли вы?
    Альбом «Битлз», признанный одним из величайших в истории музыки, ругали за искусственность и «перепродюсированность».
    Врач на карантине спел созданную для фильма песню Высоцкого «Давно смолкли залпы орудий».
    Картина парада Победы, где руководство страны смещено на задний план, получила Сталинскую премию.
    Не удержавшись от писательства, Амалия Кахана-Кармон создала одну из важнейших книг в истории Израиля.
    Предок вождя революции участвовал в управлении долгами Российской империи.


  4. An impressive share, I just given this onto a colleague who was doing a little analysis on this. And he in fact bought me breakfast because I found it for him.. smile. So let me reword that: Thnx for the treat! But yeah Thnkx for spending the time to discuss this, I feel strongly about it and love reading more on this topic. If possible, as you become expertise, would you mind updating your blog with more details? It is highly helpful for me. Big thumb up for this blog post!

  5. I just want to tell you that I’m newbie to blogs and definitely loved this web site. 파워볼커뮤니티
    Likely I’m planning to bookmark your blog . You really come with superb stories. Thanks a bunch for revealing your website page.

  6. I am in fact thankful to the owner of this site who has shared this impressive
    piece of writing at here.

    Also visit my site: Bonuses

  7. Thanks a bunch for sharing this with all people you really understand what you’re
    speaking about! Bookmarked. Kindly additionally
    talk over with my website =). We may have a link change arrangement between us

    Here is my web-site – Visit this page

  8. Quality content is the important to attract the visitors to
    visit the site, that’s what this web site is providing.

  9. You have made some good points there. I checked on the internet to learn more about the issue and found most people will go along with your views on this site.

  10. A fascinating discussion is worth comment. There’s
    no doubt that that you need to publish more on this issue, it may not be a
    taboo subject but usually folks don’t talk about
    such issues. To the next! All the best!!

  11. I really like looking through a post that will make men and
    women think. Also, many thanks for allowing for me to comment!

  12. I’m more than happy to discover this great site.
    I need to to thank you for ones time due to this fantastic read!!

    I definitely enjoyed every little bit of it and
    i also have you saved as a favorite to look at
    new information on your blog.

  13. I blog quite often and I genuinely thank you for your information. Your article has truly peaked my
    interest. I’m going to take a note of your site and
    keep checking for new information about once a week.
    I opted in for your Feed too.

  14. With havin so much written content do you ever run into any issues of
    plagorism or copyright violation? My website has a lot
    of completely unique content I’ve either authored myself
    or outsourced but it looks like a lot of it is popping it up all over the internet without my permission. Do you know any techniques to
    help prevent content from being ripped off? I’d definitely appreciate

  15. Your mode of telling the whole thing in this piece of writing is
    genuinely fastidious, every one can without difficulty know it, Thanks a lot.

  16. Great beat ! I wish to apprentice even as you amend your site,
    how could i subscribe for a blog web site? The account
    helped me a appropriate deal. I were tiny bit acquainted of this your broadcast offered
    shiny transparent idea

  17. I am really impressed with your writing skills as well as with the layout on your blog.
    Is this a paid theme or did you modify it yourself?

    Anyway keep up the nice quality writing, it is rare to see
    a nice blog like this one nowadays.

  18. I’m not that much of a internet reader to be honest but
    your sites really nice, keep it up! I’ll go ahead and bookmark your website to come back
    in the future. All the best

  19. Valuable info. Fortunate me I found your web site by accident,
    and I am stunned why this accident did not happened earlier!
    I bookmarked it.

  20. Pretty great post. I just stumbled upon your blog and wanted to mention that I’ve really loved browsing your weblog posts.
    After all I’ll be subscribing on your rss feed and I’m hoping you write again very soon!

  21. You ought to take part in a contest for one of the most useful blogs on the net.
    I am going to highly recommend this web site!

  22. Pretty nice post. I simply stumbled upon your blog and wished to
    say that I’ve really loved browsing your weblog posts.
    In any case I’ll be subscribing in your rss feed and I am hoping you write again soon!

  23. Simply want to say your article is as astonishing. The clearness to your publish is just nice
    and i can think you’re knowledgeable in this subject.
    Fine together with your permission let me to snatch your feed to keep up to
    date with approaching post. Thanks 1,000,000 and please keep up
    the rewarding work.

  24. Definitely believe that which you said. Your favorite
    reason seemed to be on the net the easiest thing to be aware
    of. I say to you, I definitely get irked while people think about worries that they just don’t
    know about. You managed to hit the nail upon the top as well as
    defined out the whole thing without having side-effects , people could take a signal.
    Will likely be back to get more. Thanks

  25. Hey there! This is my first visit to your blog! We are a team of volunteers and starting a new initiative in a community
    in the same niche. Your blog provided us useful information to work
    on. You have done a marvellous job!

  26. This paragraph offers clear idea for the new visitors of blogging, that really how to
    do blogging and site-building.

  27. Excellent post. I certainly appreciate this website.
    Keep writing!

  28. Amazing blog! Do you have any tips and hints for aspiring writers?
    I’m planning to start my own blog soon but I’m a little lost on everything.
    Would you advise starting with a free platform like WordPress or go for a paid option? There are so many choices out there that I’m completely
    overwhelmed .. Any suggestions? Cheers!

  29. I like the helpful info you provide in your articles.

    I’ll bookmark your weblog and check again here frequently.
    I am quite sure I’ll learn many new stuff right here! Good luck for the next!

  30. It’s the best time to make some plans for the long run and it’s time to be happy.
    I have learn this put up and if I may I wish to suggest
    you some attention-grabbing things or suggestions.
    Perhaps you could write subsequent articles regarding this article.
    I desire to learn even more things about it!

  31. I’m truly enjoying the design and layout of your site. It’s a very easy on the eyes which makes it much more pleasant for me to come here and visit
    more often. Did you hire out a designer to create your theme?
    Superb work!

  32. Attractive section of content. I just stumbled upon your blog and
    in accession capital to assert that I get actually enjoyed
    account your blog posts. Any way I will be subscribing to your augment and
    even I achievement you access consistently fast.

  33. It’s amazing in favor of me to have a website,
    which is helpful in favor of my know-how. thanks admin

  34. It’s going to be ending of mine day, but before finish I
    am reading this impressive post to increase my know-how.

  35. Someone necessarily assist to make severely posts I would state.
    This is the first time I frequented your web page and to
    this point? I surprised with the research you made to make this particular post extraordinary.
    Wonderful job!

  36. Hey! I know this is somewhat off topic but I was wondering which blog platform are you using for this website?
    I’m getting fed up of WordPress because I’ve had issues with
    hackers and I’m looking at alternatives for another platform.
    I would be fantastic if you could point me in the direction of
    a good platform.

  37. I am not sure where you’re getting your information, but good topic.

    I needs to spend some time learning much more or understanding more.
    Thanks for magnificent info I was looking for this info for my mission.

  38. Excellent post. I was checking constantly this weblog and
    I’m impressed! Very useful information specially the remaining part 🙂 I care for such information a lot.

    I was looking for this certain info for a very lengthy time.
    Thank you and good luck.

  39. It’s going to be finish of mine day, but before end I am reading this enormous
    article to increase my know-how.

  40. I like the valuable information you supply to your
    articles. I will bookmark your weblog and check once more here regularly.
    I am rather certain I’ll be informed lots of new stuff right here!
    Best of luck for the following!

  41. A fascinating discussion is definitely worth comment.

    I think that you ought to write more on this subject, it may not
    be a taboo subject but generally people do not speak about such subjects.
    To the next! Cheers!!

  42. It’s perfect time to make some plans for the future and it is time to be happy.

    I’ve read this post and if I could I want to suggest you some interesting things or suggestions.
    Perhaps you could write next articles referring to this article.

    I wish to read even more things about it!

  43. For the reason that the admin of this site is working, no doubt very rapidly
    it will be renowned, due to its quality contents.

  44. Pretty component of content. I simply stumbled upon your weblog
    and in accession capital to say that I get in fact loved account your
    blog posts. Any way I’ll be subscribing in your feeds or even I fulfillment you access constantly quickly.

  45. Hello to all, the contents present at this website are in fact amazing for people knowledge, well, keep up the nice work fellows.

  46. We absolutely love your blog and find the majority of
    your post’s to be exactly what I’m looking for. Does one offer guest writers to write content for yourself?

    I wouldn’t mind producing a post or elaborating on some
    of the subjects you write regarding here. Again, awesome weblog!

  47. Great article! This is the type off information that
    are supposed too be shared around the internet.
    Disgrace on thhe search engines for no longer positioning this put uup higher!
    Come on ovger and discuss with mmy website . Thanks =)
    web page

  48. I enjoy, cause I found just what I was looking for. You have ended my four day
    long hunt! God Bless you man. Have a great day. Bye

  49. I am really loving the theme/design of your weblog.
    Do you ever run into any browser compatibility problems? A
    handful of my blog visitors have complained about my blog not working correctly in Explorer
    but looks great in Chrome. Do you have any advice to help fix this problem?

  50. Howdy! This is kind of off topic but I need some guidance from
    an established blog. Is it very hard to set up your own blog?
    I’m not very techincal but I can figure things out
    pretty quick. I’m thinking about making my own but I’m not sure where to begin. Do you have any points or suggestions?


  51. Hi, i believe that i noticed you visited
    my blog so i came to go back the choose?.I am attempting
    to find things to improve my site!I guess its ok to use a
    few of your concepts!!

  52. Very good article! We are linking to this particularly
    great post on our site. Keep up the good writing.

  53. Hi there mates, how is everything, and what you would like to say regarding
    this article, in my view its really remarkable designed for me.

  54. Hey there would you mid letting me know which webhost
    you’re working with? I’ve loaded your blog inn 3 different
    web browsers and I must say tthis blog loads a lot
    quicker then most. Can you suggest a giod hosting provider at a fair price?
    Kudos, I appreciate it!
    web page

  55. Howdy! I know this is kinda off topic however I’d figured I’d ask.
    Would you be interested in exchanging links or maybe
    guest authoring a blog article or vice-versa?
    My site discusses a lot of the same subjects as yours and I feel we could greatly
    benefit from each other. If you might be interested feel free to send
    me an e-mail. I look forward to hearing from you! Wonderful blog by the way!

    My page: crank out windows

  56. I was wondering if you ever considered changing the structure of your website?
    Its very well written; I love what youve got to say. But maybe you could a little
    more in the way of content so people could connect with it better.
    Youve got an awful lot of text for only having one or two pictures.
    Maybe you could space it out better?

  57. I’ve been exploring for a little bit for any high quality articles or weblog posts in this kind
    of house . Exploring in Yahoo I at last stumbled upon this site.
    Studying this information So i’m happy to convey that I’ve a very excellent uncanny
    feeling I came upon just what I needed. I such a lot indubitably will make certain to don?t put out
    of your mind this web site and provides it a look on a continuing basis.

  58. Hi there! I could have sworn I’ve been to this blog before but
    after reading through some of the post I realized it’s new to me.

    Anyhow, I’m definitely happy I found it and I’ll be bookmarking and checking back

  59. My brother suggested I might like this website. He was totally right.
    This post actually made my day. You cann’t imagine simply how much time I had
    spent for this information! Thanks!

  60. Good day! I could have sworn I’ve been to your blog before but after browsing through some of the posts I realized it’s new to me.
    Regardless, I’m certainly pleased I stumbled upon it and I’ll be
    bookmarking it and checking back often!

  61. I constantly spent my half an hour to read this webpage’s articles
    or reviews every day along with a mug of coffee.

  62. What’s up everybody, here every person is sharing these
    kinds of familiarity, therefore it’s fastidious to read this weblog,
    and I used to visit this blog all the time.

  63. Hey! I realize this is somewhat off-topic however I had to ask.
    Does managing a well-established blog such as yours require
    a large amount of work? I’m completely new to running a blog but I do write in my journal on a daily
    basis. I’d like to start a blog so I will be able to
    share my personal experience and feelings online.
    Please let me know if you have any kind of suggestions or tips for
    new aspiring bloggers. Appreciate it!

  64. Very good website you have here but I was wondering if you knew of any forums that cover the same topics discussed in this article?
    I’d really like to be a part of community where I can get feedback
    from other experienced individuals that share the same interest.
    If you have any recommendations, please let me know. Cheers!

  65. Hello, i think that i saw you visited my weblog thus i came to “return the favor”.I am trying to find things to improve my site!I suppose its ok to use some of your ideas!!

  66. Hello, I enjoy reading all of your article post. I wanted
    to write a little comment to support you.

  67. I think this is among the most significant information for me.
    And i’m glad reading your article. But want to remark on some general things,
    The site style is ideal, the articles is really great : D.
    Good job, cheers

  68. Hi, I do think this is an excellent website. I stumbledupon it
    😉 I will revisit once again since i have book marked it.
    Money and freedom is the best way to change, may
    you be rich and continue to help other people.

  69. Architecture entirely to your diligent generic cialis 5mg online update the ED: alprostadil (Caverject) avanafil (Stendra) sildenafil (Viagra) tadalafil (Cialis) instrumentation (Androderm) vardenafil (Levitra) As a service to some men, advanced in years residents may transfer hit the deck ED. online casino games real money best online casino

  70. Aw, this was an extremely nice post. Finding the time and actual effort to produce a really good article… but what
    can I say… I procrastinate a lot and don’t seem to get
    anything done.

  71. This is a topic which is close to my heart…
    Thank you! Exactly where are your contact details though?

  72. I am really enjoying the theme/design of your blog. Do you ever run into any browser compatibility problems?
    A number of my blog visitors have complained about my site not
    operating correctly in Explorer but looks great in Opera.
    Do you have any tips to help fix this problem?

  73. 右玉县降雨偏少,右玉县通过加强水库管理,科学调度水源,

  74. Having read this I believed it was rather enlightening.
    I appreciate you taking the time and energy to put this article together.
    I once again find myself spending a lot of time both
    reading and posting comments. But so what, it was still worth it!

  75. Wonderful beat ! I would like to apprentice whilst
    you amend your web site, how could i subscribe for a weblog website?
    The account aided me a acceptable deal. I have been tiny bit
    acquainted of this your broadcast offered vibrant transparent

  76. Normally I don’t read article on blogs, however I wish to say that this write-up very pressured me to check out and do so!
    Your writing style has been amazed me. Thanks, quite nice post.

  77. These are really enormous ideas in about blogging. You have touched some fastidious
    things here. Any way keep up wrinting.

  78. Amazing! Its in fact remarkable post, I have got much clear idea on the topic of from this post.

  79. Hmm it appears like your website ate my first comment
    (it was super long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your
    blog. I as well am an aspiring blog blogger but I’m still new to everything.
    Do you have any recommendations for newbie
    blog writers? I’d genuinely appreciate it.

  80. La mayoría de los casos pueden tratarse satisfactoriamente mediante la
    pronta administración de sales de rehidratación oral.

    Necesitan mucha ayuda para hacer sus deberes escolares,
    van muy atrasados en la escuela la mayoría de los niños.
    Una de las grandes e incompresibles dificultades para los niños en Tanzania es que la enseñanza primaria se imparte en swahili y después la escuela secundaria
    en Inglés, por este motivo es muy importante voluntarios tengan un nivel de ingles suficiente para apoyar
    en las clases y poder ayudar a los los niños de la escuela primaria para que
    puedan acceder a la escuela secundaria en inglés y seguir así
    su formación. También deberás tener en cuenta que, si obtienes este visado, solo podrás ingresar al país a través
    de los siguientes pasos fronterizos: Aeropuerto
    Internacional Julius Nyerere (JNIA), Aeropuerto Internacional Kilimanjaro (KIA), Aeropuerto Internacional Abeid Amani
    Karume (AAKIA), cruce de frontera Namanga o cruce de frontera Tunduma. https://evisatanzania.org/

  81. Howdy! This post could not be written any better! Reading through this post reminds
    me of my old room mate! He always kept chatting about this.
    I will forward this article to him. Fairly certain he will have a good read.

    Many thanks for sharing!

  82. This article will assist the internet users for creating new weblog or
    even a weblog from start to end.

  83. It’s great that you are getting ideas from this post as well as from
    our discussion made at this time.

  84. Hi friends, fastidious post and good urging commented at this place,
    I aam in faact enjoying by these.

  85. Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates.
    I’ve been looking for a plug-in like this for quite some time and was
    hoping maybe you would have some experience with something
    like this. Please let me know if you run into anything.

    I truly enjoy reading your blog and I look forward to your new updates.

  86. I have read a few just right stuff here. Certainly worth bookmarking for revisiting.
    I wonder how much effort you put to create any such excellent informative site.

  87. Hi there would you mind sharing which blog platform you’re working with?
    I’m going to start my own blog soon but I’m having a difficult time
    deciding between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your design seems different then most blogs and I’m looking for something completely
    unique. P.S My apologies for getting off-topic but I had to ask!

  88. Just wish to say your article is as surprising. The clarity for your post
    is simply cool and i could assume you’re an expert on this
    subject. Well with your permission let me to seize your RSS feed
    to stay updated with coming near near post. Thank you 1,000,000 and please carry on the
    enjoyable work.

  89. You actually make it seem so easy with your presentation but
    I find this topic to be actually something which I think I would never understand.
    It seems too complex and very broad for me. I am looking forward for your next post, I will try to get
    the hang of it!

  90. I’m not sire where you are getting your info, but great topic.

    I needs to spend some time learning more or understanding more.

    Thanks for magnificent info I was looking for this info foor my mission.
    web page

  91. Wonderful article! That is the kind of information that are supposed to be shared across the internet.
    Disgrace on Google for now not positioning this post higher!
    Come on over and seek advice from my web site .
    Thanks =)