Accurate Solar Power Prediction: A Real-Time Nowcasting Solution

In Hungary, the challenge of predicting solar power generation accurately is critical as the country taps into its photovoltaic potential of 1750 PJ per year. With solar power already making up 25% of the total grid demand, reliable short-term forecasts are needed to manage the variability in energy production.

Our project developed an AI-based prediction system that forecasts solar output up to 2.5 hours ahead, using real-time meteorological satellite data. The goal was to create a “nowcasting” tool to help grid operators adjust to rapid changes in solar power production, improving operational response times and overall grid management.

What Is Nowcasting?

Nowcasting refers to the process of generating highly localized, short-term weather forecasts, typically within a few hours. In our case, it involves using satellite weather data to predict solar energy output. Unlike traditional forecasting, which can span days or weeks, nowcasting focuses on real-time data to anticipate quick shifts in solar production. This enables grid operators to make adjustments rapidly, especially during periods of sudden solar fluctuations.

Key Challenges Addressed

• Data Source Integration:

• Processing complex MSG (Meteosat Second Generation) satellite data across 12 spectral bands
• Handling 15-minute update intervals with varying latencies
• Managing 3-5 km spatial resolution data for the Hungarian region

• Operational Requirements:

• Need for predictions within critical 1-3 hour window
• Handling data latency from satellite observations (5-15 minutes)
• Managing computational requirements for real-time processing
• Predictions for power plants at any location

• Weather Variability:

• Handling rapid changes in cloud cover and atmospheric conditions
• Accounting for seasonal variations in solar radiation
• Managing prediction accuracy during critical weather transitions

Behind the Scenes: Our Approach to Training the Model

Training Data

The system integrated multiple data sources used for training:

1. MSG-SEVIRI Satellite Data:
   • 12 spectral bands from 0.6 µm to 14 µm wavelength
   • 15-minute temporal resolution
   • ~5 km spatial resolution over Hungary
   • 5 selected bands (VIS 0.6, VIS 0.8, IR 3.9, IR 10.8, IR 12.0) based on information content
   • 5-15 minute data latency
2. Numerical Weather Prediction Data:
   • HIRLAM regional weather model output
   • 3-hour update frequency
   • ~3km spatial resolution
   • Used in baseline ensemble model
   • Includes radiation and cloud cover parameters
3. Solar Production Data:
   • 1kW residential installation in Szentendre region
   • 15-minute resolution measurements
   • Static panel configuration with optimized tilt
   • Used for model training and validation
4. Auxiliary Data:
   • Sun position parameters (elevation, azimuth)
   • Time-based features (day/year periodicities)
   • Clear-sky radiation estimates
   • Geographical coordinates

Data Processing Challenges:

• Integration of multiple meteorological data formats (BUFR, NetCDF, GRIB)
• Handling missing or corrupted satellite data
• Alignment of different spatial and temporal resolutions
• Real-time data processing pipeline development
• Management of data latency in an operational context

Model Development and Training

Model Selection and Optimization:

• Initial feature selection through correlation analysis and linear regression
• Hyperparameter optimization of the model using random search across 100 configurations
• Final model architecture with 1.5M trainable parameters

Computational Requirements:

• Training performed on an NVIDIA GPU with 8GB VRAM
• Training time on the order of hours per configuration
• Complex data preprocessing pipeline required approximately one week of computation time per 6 months of data
• Implementation in the PyTorch framework for efficient GPU utilization

Dataset Configuration:

• Training data: Q3 2018
• Test data: Q3 2019
• Validation approach ensured seasonal consistency in evaluation

Evaluated Deep Learning Models

The project evaluated several approaches before selecting the final implementation:

1. Scaled Persistence Model:
   • Traditional approach using current production scaled by clear-sky radiation curve
   • Strong performance in very short-term predictions (under 1 hour)
   • Simple implementation with no external data dependencies
   • Used as a baseline for model performance evaluation
2. NWP-Optical Flow Ensemble:
   • Combined Numerical Weather Prediction models with optical flow techniques
   • Utilized HIRLAM regional weather model data, one of the best short-term models
   • Incorporated MSG-SEVIRI satellite imagery for motion tracking, providing the ultra-short term prediction component missing from the computationally expensive numerical weather models
   • Required significant computational resources but provided robust predictions
3. MetNet-based Deep Learning Architecture:
   • Selected as the final implementation
   • Adapted from Google’s precipitation forecasting approach
   • Modified for solar production prediction context
   • Balanced accuracy with computational efficiency

Final Model Architecture

The final implementation utilized the sophisticated deep learning architecture of Google’s MetNet model. The implementation featured four main components:

• Convolution Network (CNN) based spatial downsampling for efficient processing
• Convolution-LSTM networks for temporal feature encoding
• Spatial-aggregator using Axial attention mechanisms for efficient spatial information processing in satellite imagery
• Final Convolution Network (CNN) with a single Fully Connected (FC) layer to convert 2D data into a single final output

Visualization of the operation and structure of the used model with its individual components, starting from the input data,

up to the production forecast.

Significant Gains Over Traditional Approaches

The system demonstrated significant improvements over baseline approaches:

• Maintained consistent performance across different weather conditions
• Outperformed traditional numerical weather prediction methods after 75-150 minutes
• Achieved a 7.72% normalized MAE at the 150-minute horizon (versus 11.40% for the Numerical Weather Prediction model + Optical-Flow Ensemble baseline)
• Showed particular strength in predicting rapid weather changes

Sample from the model’s prediction (pred) and actual data(gt), this was predicted 1 hour into the future. 

Key Takeaways from Model Development and Data Integration

1. Data Processing Impact:

• Critical importance of satellite data quality and preprocessing
• Significant influence of data latency on short-term predictions
• Need for robust handling of missing or corrupted data

2. Model Architecture:

• Effectiveness of axial attention for spatial feature extraction
• Importance of balancing model complexity with operational requirements
• Value of ensemble approaches for different weather conditions

3. Operational Considerations:

• Critical importance of real-time data processing capabilities
• Need for handling various weather condition scenarios
• Importance of prediction reliability during rapid weather changes

Key Takeaways from the Project

The project successfully demonstrated the viability of AI-driven nowcasting for solar power prediction, achieving superior performance compared to traditional methods while maintaining operational efficiency. The implementation provides a foundation for improved grid management and renewable energy integration.

Opportunities for Future Improvement

Potential areas for future enhancement include:

• Integration with ground-based sensors (sky imagery)
• Incorporation of real-time power production feedback using an ensemble approach
• Extension of prediction horizons beyond 2.5 hours
• Enhancement of prediction accuracy during extreme weather events

Beyond these, the fully machine learning-based methodology opens up further opportunities. The model’s flexibility could allow it to predict not just solar power production, but also other high-value, weather-dependent metrics. For instance:

• Estimating the total production of the ~4,000 residential solar installations in Hungary and predicting their impact on local energy consumption
• Predicting residential or industrial energy consumption for heating, cooling, or other weather-dependent needs
• Expanding the model’s scope to forecast the effects of weather on the broader energy market, including energy distribution and grid load management

These developments could offer even more precise forecasting for energy systems, supporting smarter grid management and a more responsive energy market.

Interested in a custom AI-driven solution for your project? Reach out to discuss how we can help you build a solution tailored to your needs!