Predicting directions of S&P 500 using AI & ML models

Abstract

The challenge of predicting (financial) market movements has long been based on quantitative models. These models are based on statistical analysis of historical data, which is used to identify patterns and relationships that are used to predict market behavior. Artificial intelligence (AI) and machine learning (ML) models are now extensively researched and applied in the domain of Financial Markets. The development of Python packages that encapsulate the theoretical underpinnings of these models has also taken place at the same time. Many teams in the finance industry have mastered and exploited these packages and report promising results in asset allocation and risk control. In our investigation of applying AI/ML and quant models we have used Python packages for the software realization of Random Forest, Gaussian Naive Bayes, Decision Tree, Artificial Neural Net (ANN) and Logistic Regression models. The central aim of our investigation has been to generate next day market movement predictions specifically for S&P 500. The study employs a comprehensive dataset spanning five years, from 2017 to 2022, comprising closing prices of the S&P 500 index, VIX index, Gold, Oil, and 10-year U.S. Treasury yields as features. Technical indicators derived from these asset prices are utilized as additional features to enhance the predictive power of the models. Initially, the individual models are evaluated, with accuracy scores ranging from 50.90% (Random Forest) to 53.42% (Gaussian Naive Bayes), highlighting the inherent challenges of stock market prediction. To improve prediction performance, an ensemble modeling approach is adopted, where the final prediction is based on the majority voting of the individual models. The results demonstrate that the ensemble modeling technique generally improves prediction accuracy, with the highest overall accuracy of 58.90% achieved when all models agree on the market direction. While the ensemble modeling approach shows promise, the achieved accuracy levels emphasize the complexity of stock market prediction tasks and the need for further enhancements. This research serves as a foundation for developing stock trading strategies by leveraging the ensemble model predictions for directional trading, risk management, portfolio optimization, and other comprehensive trading strategies.