For as long as people have traded in financial instruments, they have used a range of tools and mathematical models to forecast markets, assess risk and guide investment decisions. The goal has always been the same: to anticipate where markets might move, so that investors can maximise their returns by selling and buying assets at prices that lock in the best possible returns.
But forecasting the future direction of any asset class or even a single stock has never been an exact science. Prices are shaped by countless variables, many of them unpredictable, making even the most sophisticated models imperfect guides to what comes next. History is full of examples of where individual and institutional investors have got it badly wrong when depending on predictions about where an asset price will go next.
In 2007, for example, quantitative equity funds in the US racked up billions of dollars in losses in an event called the quant quake. These firms used mathematical models to identify relationships and patterns in stock prices rather than relying on human analysis. Researchers concluded that their losses were triggered when several funds that used similar signals for trading tried to simultaneously exit the same positions.
Nearly 20 years later, investors and traders are still trying to use mathematical models and historical data to try to beat the markets. Prediction markets such as Polymarket and online bookies, where participants bet real money on events like elections and central bank decisions, offer a real-time window into the probabilities of outcomes like a particular presidential candidate winning or the Fed hiking interest rates.
AI transforms the art of forecasting – but there are limitations
Intuitively, it would seem that the art of forecasting should be more accurate today than in 2007. There is more real-time data at our fingertips, thanks to social media and the proliferation of digital platforms. And advances in computing power, machine learning and AI enables us to analyse vast quantities of information nearly instantly.
Large language models can digest news, filings, research papers, earnings transcripts and social media data in seconds, allowing market participants to form probabilities based on the latest information. Today’s tools can surface historical analogues and subtle relationships that humans can easily overlook. AI is especially powerful in forecasting structured, data-rich events where historical patterns exist, but it is not able to access closed-system proprietary research on individual securities or economics that is not available on open-source platforms.
If real-time data and powerful AI are available to all, prices should adjust faster, and traders will have less time and room to capitalise on any unique insight they have. The better the market becomes at incorporating information, the harder it becomes for any individual investor to gain an edge from that information. Uncertainty, unexpected events or novel conditions are what create both risk and potential reward.
Right about the event, wrong about the market
Of course, in reality, we have numerous recent examples of low-probability and completely unforeseen events rocking financial markets. In 2016, betting markets, like pollsters, suggested that the chances of Britain voting to leave the European Union and of Donald Trump beating Hilary Clinton in the US presidential election were small. And in 2020, we were all blindsided by the Covid-19 pandemic.
These events are outliers – betting and prediction markets are right more often than they are wrong. But the consequences of getting it wrong can be disastrous. Perhaps even more importantly, correctly predicting an event is different from and more difficult than forecasting how markets react. There are several examples in recent years of markets reacting to events in ways that contradict how asset prices would be expected to move. (Refer to our July article entitled ‘Unpredictable outcomes from transformative events’.)
During the Covid-19 pandemic, equities crashed in March 2020 as nations imposed lockdowns and stay-at-home orders. But global stock markets still finished the year significantly higher, largely because monetary and fiscal policy overwhelmed short-term economic data. When Russia invaded Ukraine in 2022, investors expected pressure on emerging-market currencies. However, the rand strengthened on the back of higher commodity prices. And this year, gold fell, despite expectations that the war in the Middle East would cause it to rally.
What these examples illustrate is that markets tend to look forward rather than backwards. Once a widely anticipated event happens, markets have priced it in. Investors will only be able to achieve an upside if the outcome differs from what was widely expected. If traders are already buying gold as a hedge against geopolitical uncertainty, that expectation is reflected in the price. Should that negative event materialise, gold may remain flat or even fall if it is no worse than investors had anticipated.
Consider interest rates as another example. Traders position their portfolios in anticipation of rate hikes or cuts ahead of Fed meetings. If the Fed holds, reduces or increases rates in line with expectations, equities prices are not likely to respond. But if the Fed surprises markets by moving rates differently than expected, asset prices can move sharply as investors adjust their portfolios to reflect a new outlook for inflation and interest rates.
Many of the largest investment errors come from overconfidence in forecasts
Models can help investors manage risk and understand context, but they cannot necessarily predict whether an asset will fall or rise tomorrow. A 2023 study testing machine-learning techniques across more than 100 securities in 10 international markets found their next-day forecasts were not much better than flipping a coin. Other research shows slight predictive edges, but these can disappear once fees and transaction costs are considered.
To profit from these strategies, firms need to be able to exploit tiny inefficiencies thousands of times a day, backed by heavy risk management, rather than calling the market’s direction. And as the quant quake example shows, the risks of a wrong call can be significant. Market prices reflect the views, incentives and information of millions of participants. Consistently beating that aggregation is extraordinarily difficult.
Since forecasting is unreliable, many of the largest investment errors come from overconfidence in forecasts rather than ignorance of the facts. Rather than betting on forecasts, the most successful investors will focus on managing uncertainty, probabilities and risk. Long-term outcomes depend more on asset allocation, diversification, fees, and investor behaviour than on macroeconomic predictions.
Personal finance author Morgan Housel makes the point that success belongs less to those who foresee the future than to those who are still invested after a future nobody predicted has arrived. Wise investors understand that black swan events such as the 9/11 attacks or global pandemics can catch them by surprise, and that the impact of such events on the market might not be what one would expect.
At Dynasty, we utilise AI as an efficiency tool in our information gathering and synthesising process, but place a far greater emphasis on proprietary research – be this economic, fund manager/instrument related, or stock specific. In a real-time, open-source, but complex world, we place less reliance on prediction models that base forecasts on models, however frequently they may be updated.
