Forecast Verification Measures
How can the performance of method "Lev" 2.0 be measured? The flare forecast based on the levels of LYRA, GOES and sunspot number, between 01 Feb 2010 and 31 Dec 2014 was tested on these days. Forecasted and realized maximum daily flare strengths (or their probabilities) were compared.
One can measure the difference between the order of magnitude of the realized flare strength and the median of the estimated probability distribution of the flare strengths with the root mean squared error:
"Lev" for same day RMSE=0.55
"Lev" for next day RMSE=0.57
Persistence method RMSE=0.62
Constantly C1.5(e.g.) RMSE=0.79
One can take a skill score defined as SkSc=1-MSE/MSE(ref), where "MSE" is the mean squared error of the various methods, and "ref" is a reference method, in this case the persistence method ("tomorrow will be like today"):
"Lev" for same day SkSc= 0.22
"Lev" for next day SkSc= 0.15
Persistence method SkSc= 0.00
Constantly C1.5(e.g.) SkSc=-0.62
One can measure the performance with a contingency table for binary events (either-or decisions), comparing the forecasted and the observed event, which leads to four numbers: "a" (hits), "b" (false alarms), "c" (misses), "d" (correct rejections). Taking a skill score (True Skill Statistic) defined as TSS=(ad-bc)/((a+c)*(b+d)) for the event that "There will be a flare of class C or higher":
"Lev" for same day TSS=0.62
"Lev" for next day TSS=0.57
Persistence method TSS=0.59
Constantly C1.5(e.g.) TSS=0.00
Or for the event that "There will be a flare of class M or higher":
"Lev" for same day TSS=0.00
"Lev" for next day TSS=0.00
Persistence method TSS=0.39
Constantly C1.5(e.g.) TSS=0.00
One can actually take the various percentages of estimated flare probabilities into account, and demonstrate for X, M, C, and smaller flares whether the probability predicted on certain days corresponds to the realized probability on these days. These "reliability functions" show if a method is biased or not.
IED, 05 Feb 2014