“People in standard finance are rational. People in behavioural finance are normal.”

-Meir Statman, Ph.D. Machine Learning, Santa Clara University


The revolutionary work of psychologists Daniel Kahneman and Amos Tversky in the 1970s-1980s, and their research conducted over the last three decades have revealed striking insights into the intricate ways the human mind operates. This research identifies prevalent, deep-seeded, subconscious biases and heuristics present in the human decision-making process, and reveals an entirely new perspective on why we behave the way we do. This body of work, and subsequent work by other researchers, represents an entirely new field of endeavour, referred to as behavioral finance and economics. In this series of blogs, I aim to shed light on the various investor biases in behavioral finance and explain how Machine Learning techniques can help to deal with them, effectively.

The applications of behavioral science to finance are now fairly comprehensive and well established. They encompass activities such as spending, investing, trading, financial planning, portfolio management and business commerce. In order to understand why economists began studying real people to assess the validity of rational economic theories, it is necessary to understand the concept of indifference curves.

The objective of indifference curve analysis is to demonstrate, mathematically, the basis on which a rational consumer substitutes a certain quantity of one good for another. An indifference curve is a line that depicts all of the possible combinations of two goods between which a person is indifferent; that is, consuming any bundle on the indifference curve yields the same level of utility.

Biases and Heuristics: Investment Behaviour

The biases relate to how we process information to reach decisions and the preferences we have (Shefrin, 2000). In spite of the considerable funds at stake, financial decision-making i.e. investment activity is one of many domains of human activity to be affected by cognitive biases. What investors believe are valid judgements may in fact be the result of effort-saving mechanisms by the brain.

A heuristics and biases framework can be intended as a counterpart to standard finance theory’s asset pricing model. When a decision maker faced with huge amounts of data and information, and an array of decision problems, people are incapable of doing the complex optimization calculations that drives standard finance theory. Instead, they rely on a limited number of cognitive strategies or heuristics to simplify the complex events while making decisions. Some of the most common biases are confirmation bias (the phenomenon of seeking selective information to support one’s own opinions or to interpret the facts in a way that suits our own world view), myopic loss aversion (investors fear losses more than the appreciation of profits), disposition effect (gains are realized too early and losses too late), and framing bias (decisions are based largely on how facts are depicted).   

Behavioural finance theory has identified three main types of biases that explain many central facts about the markets, including average returns, time series, predictability, momentum and reversals, bubbles, and trading volume:

1.We tend to over-extrapolate the past, putting too much weight on the recent past (such as recent performance) while making decisions and forming beliefs about the future.

2. We tend to be over-confident in our own ability over other people’s, as well as in the accuracy of our beliefs. According to Nicholas Barberis, Professor of Finance at Yale School of Management, “The irrationally high trading volume we observe in financial markets is one piece of evidence for over-confidence in our own ability where those on both sides of a trade have to believe they have superior knowledge of the future prospects of the asset than the other.”

3. We tend to be more sensitive to the prospect of loss over gain, and to give too much weight to low-probability outcomes.

According to Marcos Lopez De Prado (author of Advances in Financial Machine Learning), “Firms are increasingly using textual sentiment analysis to model overconfidence of market participants and meta-labelling to help size bets, and identifying exponential patterns to monitor or respond to greed and fear. The United States Flash Crash of May 2010 is a perfect example of the latter being put into action, with unbiased machines managing to stabilise the market shortly after humans panicked.” This is a testament to the fact that humans get trapped by narratives and their biases.

Conclusion and Final Insights

As Daniel Kahneman in his book Thinking, Fast and Slow states: “We (humans) are pattern seekers.” However, it is tough to apply an individual and context-based approach on a massive scale. Machine learning can significantly resolve this challenge by detecting patterns and searching in a large set of data for variables that are influential in shaping the patterns. And thus, Machine Learning in Finance is not only about prediction but also interpretation of data both qualitative and quantitative to find out which input variables are more crucial in predicting that target variable (including accounting for or deriving benefit from behavioral biases that may exist).

In the next blog, I will explain some economic and psychological models which can be used to predict people’s decisions using Machine Learning. So, stay tuned for the next blog.


de Prado, M. L. (2018). Advances in Financial Machine Learning. Wiley. Retrieved from https://books.google.co.in/books?id=oU9KDwAAQBAJ

Guszcza Jim. (2015). How Machine Learning and behavioral economics can work together | Deloitte Insights. Retrieved from https://www2.deloitte.com/insights/us/en/deloitte-review/issue-16/behavioral-economics-predictive-analytics.html

Pompian, M. M. (2011). Behavioral Finance and Wealth Management: How to Build Optimal Portfolios That Account for Investor Biases. Wiley. Retrieved from https://books.google.co.in/books?id=mC2q9tIJcCoC

Additional Reading:

A.Agarwal, E. Hazan, S. Kale, and R. E. Schapire. Algorithms for portfolio management based on the newton method. In Proceedings of the 23rd International Conference on Machine Learning, pages 9–16, 2006.

Delgado M., Amorim D., Do we need Hundreds of Classifiers to Solve Real World Classification Problems?, Journal of Machine Learning Research, 2014, 15, 3133-3181.

Demir, N. (2016), Ensemble methods: Elegant techniques to produce improved machine learning results, KDnuggets.

Patel, J., Shah, S., Thakkar, P. and Kotecha, K. (2015), Predicting stock market index using fusion of machine learning techniques, Expert Systems with Applications 42(4), 2162–2172.

Witten, I. H., Frank, E. and Hall, M. A. (2011), Data Mining: Practical Machine Learning Tools and Techniques, The Morgan Kaufmann Series in Data Management Systems, third edition, Morgan Kaufmann, Burlington, MA

Leave a Reply

Your email address will not be published. Required fields are marked *