How accurate are your sales forecasts?  What can you learn from the leading research in the field?  Mark Blessington and Karl Hellman share some new insights.

Big data is a hot topic for most marketers. They are eager to assemble and analyze their vast stores of buyer and customer data. Increasingly greater levels of sophistication are being brought to bear on the opportunity … at extraordinary cost. But have these tremendous technical advancements impacted the discipline of sales forecasting in the marketing department? As best we can tell, the short answer is no. The most common method used by product managers to establish baseline forecasts is rather primitive: next year’s growth will equal last year’s growth. This is a far cry from the latest and greatest analytics being applied to big data.

Product Planning Forecasts

Sales forecasts play a major role in product planning. Marketers often make product investment decisions that rely heavily on growth predictions. For example, a classic marketing strategy is to invest aggressively in high growth products (e.g., increase advertising and promotion expenditures), cautiously invest in moderate growth products (e.g., make small, targeted investments that offer a quick return), and harvest high growth products (e.g., increase price, reduce cost of goods sold).

The most common forecasting technique used by marketers revolves around percent change: year-to-year change, quarter-to-quarter change, and so on. These statistics are simple to calculate. Once a baseline is established, marketers then apply judgment to reach a final forecast. For example, next year’s growth rate will be the same, higher or lower than this year’s growth rate due to specific competitive trends or shifts in product strategy.

But there are other techniques for predicting future growth: simple linear regression, exponential smoothing, and so on. A considerable body of research has accumulated over the last several decades on forecasting. As far as we know, this research has never claimed percent growth to be a top forecasting method. Our concern is that marketers have embraced a naïve forecasting method and that other methods might be more accurate. We wonder: “Does percent change yield accurate forecast, and do marketers limit their effectiveness by not using better forecasting methods?”

Several major forecasting studies and contests show that exponential smoothing often beats other forecasting methods[i]. So a more specific question deserves attention: “Is exponential smoothing more accurate than percent change?”

80 Product Forecasts

To compare forecast accuracy between percent change and exponential smoothing, we took several steps. First we established three ranges of annual product forecasts: (1) Growth—over 10%, (2) Moderate Change—between -10% and +10%, and (3) Decline—below -10%.

Next we found 80 products to forecast from a well-studied dataset.[ii] Then we used a classic forecast test design: we treated the last year of sales for each product as if it had not yet occurred. In other words, we generated forecasts without looking at the last year of sales. For the percent change forecasts, we used average percent change for the last four quarters. The exponential smoothing forecasts were generated using a software package[iii].

Once our forecasts were made, we compared them to the last year of sales and assessed forecast accuracy using a Bayesian probability analysis [iv]. This was done by sorting each forecast into one of the three growth categories. Then we used actual sales growth to determine whether the sort was correct or not. For example, if a forecast was +20% and actual was +12%, we declared a victory. On the other hand, if a forecast was +20% and actual was +8%, we recorded a failure.

Accuracy Findings

Our comparison of percent change versus exponential smoothing is summarized in the table. In every category exponential smoothing is more accurate, and the probability of success is higher by 10 to 17 points. So our first conclusion is that exponential smoothing is more accurate than percent change.

Our second conclusion is that forecast accuracy is surprisingly low: the highest probability was only 55%. This is a sobering finding. It means that forecasts must be treated as “educated guesses.” While statistical methods are better than naïve methods such as percent change, it is wise to anticipate a considerable amount of forecast error. Marketers are well-advised to guard against treating forecasts as certainties, and should surround their forecasts with confidence intervals.

Predicting Growth is Hard!

We were curious about why growth predictions were the least accurate. Why did this happen? About the same number of products were forecasted for the growth (25% of total) and decline (29% of total) categories, so prevalence does not explain the finding. In other words, since these two categories had about the same number of forecasts, a low number of observations did not create the low Bayesian probability.

Instead, it seems that the high growth category is especially volatile. Perhaps competition in this segment is more intense because everyone invests in growth products, which leads to high volatility in the category. Regardless of the theoretical explanation, there is a clear phenomenon at play among these 80 products: high growth is very hard to predict accurately.

The implication for marketing managers is: Whenever you see a forecast for high growth, be especially careful. Guard against the risk by running “what if growth is lower” scenarios.

Implications

As Yogi Berra is credited with saying: “It’s tough to make predictions, especially about the future.” In other words, product planners need all the help they can get. When they use percent growth to create a baseline forecast, they hamper themselves.

Simple percent change calculations are incapable of detecting subtle data patterns; they are a blunt instrument. In contrast, exponential smoothing adds insight to the product forecasting exercise. It detects nuances that are missed by the naked eye and naïve statistics. Further, its broad availability and ease of use eliminates most excuses for not using it. Forecasts grounded in exponential smoothing do not eliminate the need for caution and sound judgment, but they remove an important portion of the guesswork.

The last conclusion we reach is this: forecast accuracy measurement is an imperative. Our experience tells us that few product management departments study forecast accuracy. They are so busy with campaign design and implementation that they don’t pause to assess forecast accuracy and compare alternative methods. This study proves that incremental gains in forecasting sophistication can deliver substantial improvements to the decision-making process that surrounds product planning.

Mark Blessington is a Partner at Consentric Marketing. 

Dr. Karl Hellman is Managing Director of Consentric Marketing.

NOTES

[i] Perhaps the most famous forecast accuracy study is the M3 Competition: Spyros Makridakis and Michele Hibon, The M3-Competition: Results, Conclusions and Implications, International Journal of Forecasting, 2000.

[ii] The 80 time series used in this study come from the M3 competition referenced above and are monthly micro datasets with 68, 69, or 126 months of data.

[iii] We made exponential smoothing calculations with Stata (version 14 SE). The most accurate of four methods was selected (based on mean absolute scaled error) for each product: single exponential smoothing, double exponential smoothing, Holt-Winters nonseasonal smoothing, and Holt-Winters seasonal smoothing.

[iv] For more details on applying Bayesian probabilities to forecasting and other ways to assess forecast accuracy, see Sales Forecasting—A Practical Guide by Mark Blessington, 2015