AI methods help predict the emergence of ‘gazelles’ and other high-growth firms, but challenges remain

Predicting whether or not companies will be successful is crucial for guiding investment decisions and designing effective economic policies. However, past research on high-growth firms—enterprises thought to be key for driving economic development—has typically shown low predictive accuracy, suggesting that growth may be largely random. Does this assumption still hold in the AI era, in which vast amounts of data and advanced analytical methods are now available? Can AI techniques overcome difficulties in predicting high-growth firms? These questions were raised in a chapter I co-authored in the De Gruyter Handbook of SME Entrepreneurship, which reviewed scientific contributions on firm growth prediction with AI methods.

According to the Eurostat-OECD (Organization for Economic Cooperation and Development) definition, high-growth firms are businesses with at least 10 employees in the initial growth period and “average annualized growth greater than 20% per annum, over a three-year period.” Growth can be measured by the firm’s number of employees or by its turnover. A subset of high-growth firms, known as “gazelles”, are young businesses—typically start-ups—that are up to five years old and experience fast growth.

High-growth firms drive development, innovation and job creation. Identifying firms with high-growth potential enables investors, start-up incubators, accelerators, large companies and policymakers to spot potential opportunities for investment, strategic partnerships and resource allocation at an early stage. Forecasting outcomes for start-ups is more challenging than doing so for large companies due to limited historical data, high uncertainty, and reliance on qualitative factors like founder experience and market fit.

How random is firm growth?

Accurate growth forecasting is especially crucial given the high failure rate of start-ups. One in five start-ups fail in their first year, and two-thirds fail within 10 years. Some start-ups can also contribute significantly to job creation: research analyzing data from Spanish and Russian firms between 2010 and 2018 has shown that while “gazelles” represented only about 1%–2% of all businesses in both countries, they were responsible for approximately 14% of employment growth in Russia and 9% in Spain.

High-growth firms are “widely considered essential for stimulating economic growth and employment” but are difficult to identify. Stakeholders need accurate growth predictions to help optimize decision-making and minimize risks by identifying firms with the highest potential for success.

In an effort to understand why some firms grow faster than others, researchers have looked into various factors including the personality of entrepreneurs, competitive strategy, available resources, market conditions and macroeconomic environment. These factors, however, only explained a small portion of the variation in firm growth and were limited in their practical application. This led to the suggestion that predicting the growth of new businesses is like playing a game of chance. Another viewpoint argued that the problem of growth prediction might stem from the methods employed, suggesting an “illusion of randomness.”

As firm growth is a complex, diverse, dynamic and non-linear process, adopting a new set of methods and approaches, such as those driven by big data and AI, can shed new light on the growth debate and forecasting.

AI offers new opportunities for predicting high-growth firms

AI methods are being increasingly adopted to forecast firm growth. For example, 70% of venture capital firms are adopting AI to increase internal productivity and facilitate and speed up sourcing, screening, classifying and monitoring start-ups with high potential. Crunchbase, a company data platform, claims that internal testing has shown that its AI models can predict start-up success with “95% precision” by analyzing thousands of signals. These developments promise to fundamentally change how investors and businesses approach decision-making in private markets.

The advantages of AI techniques lie in their ability to process a far greater volume, variety and velocity of data about businesses and their environments compared to traditional statistical methods. For example, machine learning methods such as random forest (RF) and least absolute shrinkage and selection operator (LASSO) help identify key variables affecting business outcomes in datasets with a large number of predictors. A “fused” large language model has been shown to predict start-up success using both structured (organized in tables) fundamental information and unstructured (unorganized and more complex) textual descriptions. AI techniques help enhance the accuracy of firm growth predictions, identify the most important growth factors and minimize human biases. As some scholars have noted, the improved prediction indicates that perhaps firm growth is less random than previously thought. Furthermore, the ability to capture data in real time is especially valuable in fast-paced, dynamic environments, such as high-technology industries.

Challenges remain

Despite AI’s rapid progress, there is still considerable potential for advancement. Although the prediction of high-growth firms has been improved with modern AI techniques, studies indicate that it continues to be a challenge. For instance, start-up success often depends on rapidly changing and intangible factors that are not easily captured by data. Further methodological advances, such as incorporating a broader range of predictors, diverse data sources and more sophisticated algorithms, are recommended.

One of the main challenges for AI methods is their ability to offer explanations for the predictions they make. Predictions generated by complex deep learning models resemble a “black box,” with the causal mechanisms that transform input into output remaining unclear. Producing more explainable AI has become one of the key objectives set by the research community. Understanding what is explainable and what is not (yet) explainable with the use of AI methods can better guide practitioners in identifying and supporting high-growth firms.

While start-ups offer the potential for significant investment returns, they carry considerable risks, making careful selection and accurate prediction crucial. As AI models evolve, they will increasingly integrate diverse and unstructured data sources and real-time market signals to detect early indicators of potential success. Advancements are expected to further enhance the scalability, accuracy, speed and transparency of AI-driven predictions, reshaping how high-growth firms are identified and supported.

This article is republished from The Conversation under a Creative Commons license. Read the original article.