Precipitation Forecasts and Verification
Max Marchand
Fuelberg Lab at FSU

Glossary/Methodology   |   Forecasts   |   Verification


Glossary/Methodology for Real Time Forecast Section:

The plotted values for the forecasts are fractional probabilities of exceeding the specified accumulation threshold during the specified 6 hour forecast interval. Fractions include all grid cells within a 60 km radius as demonstrated in the figure below from Schwartz et al. (2009). Despite a different pattern of precipitation exceedance in panel a) and b) (grey shaded cells indicate exceedance), both exhibit a fractional probability of 38% (8/21: 8 shaded cells and 21 total cells within radius r) for the center grid point.

Blended: Equally weighted average of fractional probabilities of all 12 models;
HRRR: High-Resolution Rapid Refresh (~3km horizontal grid spacing);
HiRes-ARW: NCEP High-Resolution Window using ARW core (~4.2km);
HiRes-NMMB: NCEP High-Resolution Window using NMM-B core (~3.6km);
HiRes NAM: High-Resolution North American Mesoscale Model (~5km);
RAP: Rapid Refresh (~13km);
NAM: North American Mesoscale Model (~12km);
GFS: Global Forecast System (0.25°; run finer at ~13 km);
GFS ENS: Global Ensemble Forecast System (0.5°; run finer at ~33 km) including 1 control and 20 perturbation members whose fractional probabilities are averaged with an equal weighting;
SREF-ARW: Short Range Ensemble Forecast using ARW-core (~16 km) including 1 control and 12 perturbation members with fractional probabilities computed as with GFS Ensemble (GFS ENS);
SREF-NMMB: Short Range Ensemble Forecast using NMMB-core (~16 km) including 1 control and 12 perturbation members with fractional probabilities computed as with GFS Ensemble (GFS ENS);
CMC ENS: Canadian Meteorological Centre Ensemble (1.0°; run finer at ~50 km) including 1 control and 20 perturbation members with fractional probabilities computed as with GFS Ensemble (GFS ENS);
FNMO ENS: Fleet Numerical Meteorological and Oceanographic Center Ensemble (1.0°; run at ~90 km) including 1 control and 20 perturbation members with fractional probabilities computed as with GFS Ensemble (GFS ENS);

Glossary/Methodology for Verification Section:

All verification is against NCEP Stage IV radar and rain-gauge observations which is on an ~4km horizontal grid covering the contiguous United States. Due to mountaineous terrain and poor radar coverage, Stage IV observations are of poorer quality over the western United States (Nelson et al. 2016). Consequently, verification over this region (e.g., west of 104°W) should be used cautiously.

FSS: fractions skill score computed using a 60km radius of influence for 6 hour precipitation accumulation intervals from various models verified against NCEP Stage IV observations for given precipitation thresholds and model initiation time (i.e., 0000 UTC in left column, 1200 UTC in right column). Note that the verification grid is a 1/16th° lat-lon grid. Verification points are primarily over land (red shading seen here). Only a limited number of water points are used for computing the fractions (red shading seen here). FSS is aggregated according to the verification time of the forecasts for monthly or multi-monthly periods and so that days with large areas of simulated and rainfall receive more weight. FSS values range from 0.0-1.0 with 1.0 representing a perfect score. FSS is a variation of the fractions Brier score (FBS) which utilizes the fractional probabilities (PF) plotted in the real time forecast section and described above. The observed fractional probabilities are denoted PO.



More detailed description and demonstration of FSS can be found in Roberts and Lean (2008) and Schwartz et al. (2009).

Bias: bias score, or frequency bias, is the sum of forecasted fractional probabilities divided by the sum of observed fractional probabilities (using NCEP Stage IV as observation). Consequently, a value of 1.0 is a forecast with no bias relative to observation. Values less than 1.0 represent underprediction and values greater than 1.0 represent overprediction.

bld: composite of the 11 different models using an equal weight to compute the fractional probability;
hrb: composite of the 4 high-resolution models run at convective-permitting scales (<5km horizontal grid spacing): hrr, arw, nmm, and nhr;
hrr: High-Resolution Rapid Refresh (~3km; WRF-ARW core);
arw: NCEP High-Resolution Window using ARW core (~4.2km);
nmm: NCEP High-Resolution Window using NMM-B core (~3.6km);
nhr: High-Resolution North American Mesoscale Model (~5km; WRF-NMMB core);
nam: North American Mesoscale Model (~12km; WRF-NMMB core);
rap: Rapid Refresh (~13km; WRF-ARW core);
gfs: Global Forecast System (0.25°; run finer at ~13 km);
gen: Global Ensemble Forecast System (0.5°; run finer at ~33 km; prior to Dec. 2015: 1.0°; run finer at ~55 km) including 1 control and 20 perturbation members whose fractions are averaged with an equal weighting so that this FSS represents a hybrid of the fraction skill score and Brier skill score;
sra: Short Range Ensemble Forecast using ARW-core (~16 km) including 1 control and 12 perturbation members (see configurations here) with fractional probabilities computed as with GFS Ensemble (gen);
srn: Short Range Ensemble Forecast using NMM-core (~16 km) including 1 control and 6 perturbation members with fractional probabilities computed as with GFS Ensemble (gen), decommissioned in Oct. 2015;
srb: Short Range Ensemble Forecast using NMMB-core (~16 km) including 1 control and 12 perturbation members with fractional probabilities computed as with GFS Ensemble (gen);
sre: Short Range Ensemble Forecast using all 26 members (~16 km) with fractional probabilities computed as with GFS Ensemble (gen).
Also note that the first 3 h of sra, srn, srb, and sre are ignored since the model system initializes at 03, 09, 15, and 21 UTC cycles. Consequently, the 09 UTC initializations are used for the 12 UTC cycle figures and 21 UTC is used for the 00 UTC cycle figures, and 0-6 forecast hour accumulations are actually 3-9 forecast hour accumulations, etc.
cmc: Canadian Meteorological Centre Ensemble (1.0°; run finer at ~50 km) including 1 control and 20 perturbation members with fractional probabilities computed as with GFS Ensemble (gen), verified since Nov. 2015;
fnm: Fleet Numerical Meteorological and Oceanographic Center Ensemble (1.0°; run at ~90 km) including 1 control and 20 perturbation members with fractional probabilities computed as with GFS Ensemble (gen), verified since Nov. 2015.
See this PDF or the model websites for more details about the model configurations.

For further questions or comments, please send email to mrm06j at my.fsu.edu