A large international retail chain approached us to develop an advanced demand forecasting system to optimize their supply chain operations. The project’s primary goal was to create a reliable, data-driven system for predicting product demand across a national subsidiary’s 200 stores.
The focus was on the most crucial ultra-fresh product category (for example: fruits and vegetables), which was the most prone to error in demand vs supply because even a few days of unsold inventory results in complete loss.
The goal of the pilot was to improve their existing 37% error rate benchmark, which was successful, as our final model achieved 26% error rate across the test set.
Project Scope and Technological Implementation
Our project encompassed the development of a comprehensive demand forecasting system that would handle predictions for thousands of products across hundreds of stores. The system needed to process and predict demand while accounting for various factors such as product characteristics, store locations, seasonality, and external factors.
Key Challenges Addressed
Data Quality and Integration
In retail forecasting, the main challenge is dealing with incomplete and inconsistent data from multiple sources rather than making accurate predictions. Differences in formats, update schedules, and reliability can create more problems than the prediction task itself.
Our system needed to integrate with the clients data practices, handle multiple data sources, each with its own complexities:
- Shelf life tracking for perishable goods – critical for fresh produce and prepared foods
- Promotion data with multiple variables (discount or promotion type)
- Advertisement data (product was advertised on different platforms and in different quality)
- Store Placement data (how visible was the spot where the product was placed in the given store)
- Store-specific losses and waste data – important for understanding true demand
- RTC (Reduced to Clear) transactions – price reductions applied to products approaching their sell-by date to prevent waste
The target variable was predicting the sales of ordered products. The image clearly shows that sales across product categories are far from constant, however certain patterns are visible, suggesting the presence of underlying factors.
Business Complexity
Retail operations aren’t uniform across locations and periods – understanding and accounting for these differences was crucial:
- Regional Differences: Our analysis revealed significant variations between:
- Urban vs. rural stores – different shopping patterns and sensitivities
- Store formats and sizes – from convenience stores to hypermarkets
- Customer behavior patterns – frequency of visits, basket sizes
- Price and promotion sensitivity – varying by region and store type
- Seasonal Patterns of Product Sales: Analysis of product sales revealed distinct pattern types that required different forecasting approaches – while stationary products could rely more on recent sales history, seasonal products needed longer historical windows and external factors for accurate prediction.
- Stationary Products – Items like bananas and tomatoes showed relatively stable year-round demand with mainly weekly patterns
- Seasonal Products – Items such as strawberries (summer peak) and root vegetables (winter peak) exhibited strong seasonal patterns
- Event-Driven Products – Certain vegetables like lettuce showed spikes during holiday periods
- Weather-Sensitive Products – Items like watermelon demonstrated strong correlation with weather
Technical Requirements
The system needed to balance accuracy with practical operational needs:
- Performance Demands:
- Multiple prediction horizons – from 7-day tactical to 8-week strategic forecasts
- Store-group level predictions with individual store breakdowns
- Integration with existing systems – including legacy infrastructure
- Real-time adjustment capabilities for rapid response to changes
Implementation Approach
Data Engineering Innovations
In forecasting, data quality often trumps model complexity. A simple model with well-engineered features can outperform a complex one trained on messy or incomplete data. Good features highlight real patterns while bad ones create noise that even the best algorithms struggle to make sense of.
We developed several advanced features that significantly improved model performance. These included external factor modeling, business-driven metrics, and time-series features. By incorporating these signals, the model could better capture demand fluctuations, leading to more reliable forecast:
- External Factor Integration: Understanding how external factors affect demand:
- Weather impact modeling with location-specific sensitivity – different products react differently to weather changes
- Fuel price correlation analysis – discovered 2x higher impact in rural areas
- Holiday effect modeling with regional variations – capturing different celebration and holiday patterns
- Advanced Business Metrics: Converting business knowledge into quantitative features:
- Product turnover velocity calculations – how quickly products move through the system
- Absolute and relative promotion impact scoring – measuring promotional effectiveness
- Store clustering based on opening hours and customer patterns
- Hierarchical category relationships – capturing product similarities and substitutions
- Time-Series Features: Capturing temporal patterns at multiple scales:
- Calendar-based patterns at multiple granularities – daily, weekly, monthly patterns
- Rolling window aggregations – capturing recent trends
- Lag features with varying time horizons – incorporating historical patterns
- Anomaly detection using Holt-Winters method – identifying and handling unusual periods
Model Architecture
Instead of relying on a single model, we combined multiple specialized models, each focusing on a different aspect of the forecast:
- Base Models: Each bringing different strengths:
- H2O AutoML for automated feature selection and model optimization
- Facebook Prophet for capturing seasonal patterns and holiday effects
- XGBoost for handling complex feature interactions
- Ensemble Layer: Intelligent combination of base models:
- Custom weighting mechanism for model combination for each product category, weight were determined during the training phase on
- Adaptive to different product categories and store types
- Capable of producing both higher and lower predictions than individual models
Results and Performance Metrics
The system demonstrated significant improvements over baseline performance:
- Overall Accuracy:
- Improved from 37% (benchmark) to 25.6% Mean Absolute Error – a 11.6 % improvement
- Consistent performance across different product categories
- Better handling of promotion periods – traditionally difficult to forecast
- Regional Performance: our accuracy of urban store were 2% better than in rural areas
This is a forecast utilizing Prophet library for testing an early phase training dataset.
Final predictive error (MAE) for each subgroups of products.
More Ways a Data-First Approach Pays Off
Along the way, we pulled key insights from the data and helped the client see what was really driving their numbers:
- Regional Sensitivity: Location matters more than initially expected:
- Rural stores showed 2x higher sensitivity to fuel prices – likely due to travel costs
- Holiday effects were 1.5x stronger in rural areas – different shopping patterns
- Store clustering crucial for accurate predictions – similar stores behave similarly
- Interestingly, adverse weather conditions had less impact on rural shopping patterns, suggesting that customers preferred consolidated trips to larger stores rather than frequent visits to local convenience stores
- Data Engineering Impact: The power of good feature engineering:
- Feature engineering contributed more to accuracy than model sophistication
- External factors (weather, fuel prices) provided significant predictive power
- Hierarchical approach to categories improved model stability – especially for sparse data
- Product consumption patterns showed strong interdependencies, with the demand for certain items predicting others through complementary (purchased together) or substitution (purchased instead) effects
- Current stock levels emerged as a crucial predictor of future demand (people don’t like to buy the last remaining items)
- Operational Insights: Store characteristics matter:
- Store size and format significantly influence demand patterns
- Opening hours create distinct customer behavior clusters
- Promotion effectiveness varies significantly by region – requiring localized strategies
- School holidays had a positive effect on fruit sales, suggesting it is a popular choice among children
- These promotional patterns had less influence on key allocation forecasting than initially expected
What We Learned
The project didn’t just boost forecast accuracy—it uncovered deep insights into regional differences and customer behavior, adding unexpected business value along the way.
But the takeaways go beyond retail. It’s a reminder that understanding the bigger business context, building the right data pipeline, and using flexible, adaptive models are what really make AI work in the real world.
If you’re looking to enhance your forecasting, optimize your data strategy, or leverage AI for real-world impact, we can help. Contact us to discuss how we can tailor a solution to your needs!
