A Comparison of MEX Guidance vs. Ensemble-based Temperature Forecasts

Introduction

Ensemble-based forecasting has become a useful tool for forecasters, especially for extended forecasts. A study was conducted between June 15, 2002, and December 31, 2003, to evaluate the use of GFS and Canadian ensembles on which to base temperature forecasts. The objective of the study was to obtain some idea of how well temperature forecasts using the GFS and Canadian ensembles, as well as GFS ensemble mean MOS, fared against using GFS MOS (MEX) alone. 

Procedure

The study consisted of deriving temperature forecasts for the Chattanooga, Tennessee (CHA), Knoxville, Tennessee (TYS), and Tri Cities, Tennessee (TRI), forecast points for days 3, 4, 5, 6, and 7. The temperature forecasts were derived by using the base MEX temperature projections and adjusting these forecast temperatures to reflect trends in the 500 mb heights in the ensembles (“spaghetti plots” - available at https://www.cdc.noaa.gov/map/images/ens/spag_usbg_animation.html and/or https://eyewall.met.psu.edu/ensembles/) vs. the 500 mb heights reflected in the operational GFS. For example, if the perceived ensemble 500 mb heights mean was “higher” than the operational GFS 500 mb heights over East Tennessee, then warmer temperatures were forecast vs. the MEX projections. Some subjectivity was necessary in judging the degree to which the perceived ensemble heights mean differed from the operational GFS heights at each site, i.e., whether the operational GFS heights were significantly “higher” or “lower” than the perceived ensemble mean as estimated by “eyeballing” the 23 members of the NCEP 500 mb “spaghetti plots”. Several degrees were added to/subtracted from MEX temperature guidance where it appeared the operational GFS 500 mb heights were dramatically “higher/lower” than the perceived ensemble mean while only a degree or so was added to/subtracted from the MEX data when there were only modest differences in the heights. No adjustments were made where the perceived differences were non-existent or minimal. Adjustments based on the 500 mb heights of the Canadian ensemble mean (available at https://weatheroffice.ec.gc.ca/ensemble/index_e.html - “GZ 500 maps”) were also included, using the same scheme of perceived height differences between the Canadian ensemble mean and the operational GFS heights.

The MEX temperature data, the temperature forecasts adjusted by the respective ensembles, and the GFS ensemble mean MOS (available at https://eyewall.met.psu.edu/mos/weather/index.html) were input into a QuattroPro database. Separate databases were maintained for each of the three sites. Daily actual maximum and minimum temperatures obtained from the daily climate reports for each site were also input into the database. Most, but not all, of the projected/derived temperature data were included in the database during the study time frame. Data were generally not derived/included on the author’s scheduled days off. There were a few instances where data were not available for one or more data sources for some days (e.g., “spaghetti plots” not updated). All of the official climate data were input from June 15, 2002 through January 6, 2004, inclusive. QuattroPro was used to calculate the mean absolute errors and temperature bias between the projected/derived temperatures and the actual official temperatures at each site.

Results

Using the aforementioned procedure and based on the mean absolute error (MAE), the following results were obtained for each forecast point:

                               MEX MAE*                              GFS Ensemble MAE*

Day 7                          5.87^o                                                   5.76^o

Day 6                          5.87^o                                                   5.74^o

Day 5                          5.63^o                                                   5.51^o

Day 4                          4.39^o                                                   4.38^o

Day 3                          3.52^o                                                   3.48^o

                   Canadian Ensemble MAE*                GFS Ensemble MOS MAE*

Day 7                          5.90^o                                                   6.02^o

Day 6                          5.85^o                                                   5.94^o

Day 5                          5.65^o                                                   5.49^o

Day 4                          4.58^o                                                   4.40^o

Day 3                          3.69^o                                                   3.64^o

 

                               MEX MAE*                              GFS Ensemble MAE*

Day 7                          6.01^o                                                   5.95^o

Day 6                          5.90^o                                                   5.73^o

Day 5                          5.69^o                                                   5.52^o

Day 4                          4.41^o                                                   4.75^o

Day 3                          3.40^o                                                   3.44^o

                   Canadian Ensemble MAE*                GFS Ensemble MOS MAE*

Day 7                          5.93^o                                                   6.14^o

Day 6                          5.83^o                                                   5.97^o

Day 5                          5.66^o                                                   5.62^o

Day 4                          4.87^o                                                   4.81^o

Day 3                          3.51^o                                                   3.56^o

 

                               MEX MAE*                              GFS Ensemble MAE*

Day 7                          6.18^o                                                   6.14^o

Day 6                          6.09^o                                                   5.97^o

Day 5                          6.07^o                                                   6.01^o

Day 4                          4.65^o                                                   4.69^o

Day 3                          3.47^o                                                   3.52^o

                   Canadian Ensemble MAE*                GFS Ensemble MOS MAE*

Day 7                          6.13^o                                                   6.31^o

Day 6                          6.00^o                                                   6.28^o

Day 5                          6.05^o                                                   5.88^o

Day 4                          4.80^o                                                   4.75^o

Day 3                          3.65^o                                                   3.62^o

* = Mean Absolute Error
BOLD indicates best performance (least MAE) for specific day

Conclusion

Based on this study, forecast temperatures using the ensemble-based adjustments were generally improved slightly over using MEX alone, especially for days 5-7. In fact, the GFS ensemble adjustment was an improvement (i.e., less mean absolute error) over MEX for all three forecast points for days 5-7. It was the best performer in all cases for days 5-7, with the exception of day 7 for TYS where Canadian ensemble adjustment turned out to be the best performer, and day 5 for TRI where the GFS ensemble MOS was the best performer. Actual improvements of mean absolute error for temperatures based on GFS ensemble adjustments vs. MEX temperature projections were 0.12^o F, 0.13^o F, and 0.07^o F, for CHA, TYS and TRI, respectively, for days 5-7.

The Canadian ensemble adjustment was generally improved over MEX by a lesser degree than the GFS ensemble adjustment ( < 0.01^o F, 0.06^o F, and 0.05^o F, for CHA, TYS and TRI, respectively, for days 5-7). Temperatures based on the GSF ensemble MOS generally degraded the quality of the temperature forecasts. For days 3 and 4, the MEX alone actually did somewhat better than the adjusted values for TYS and TRI. There was also no clear bias noted (negative temperature bias at CHA; positive temperature bias at TYS and TRI).

In actuality, the forecast improvement of the temperature forecasts from adjustments based on the GFS ensembles were generally small (only a few tenth of a degree or less), and the mean absolute error for the far extended periods was still quite large (near six degrees F). Nevertheless, ensembles, especially the 23-member GFS ensemble runs, can be used to minimize forecast error for the far extended (i.e., day 5-7) over the long run.