Page 1 of 1

Forecast evaluation output

Posted: Tue Oct 09, 2018 6:23 am
by Maglin
Hi!

I use the forecast evaluation "fcasteval" but the tabel is puzzling. In the output table showing evaluation statistics there is no constant relationship between RMSE and TheilU2.

I use the following command where "gr_f" contains around 50 forecasts for PC_GDP. (I use Eviews10)

PC_GDP.fcasteval(evalsmpl=%wfend-!NRF %wfend, mean, trmean, median, trim=15) gr_f

What am I missing?

Re: Forecast evaluation output

Posted: Tue Oct 09, 2018 12:45 pm
by EViews Gareth
viewtopic.php?t=17447

The last few posts.

Re: Forecast evaluation output

Posted: Tue Oct 09, 2018 1:43 pm
by Maglin
Sorry, I still don’t get it. If you refer to the discussion about the sample size I don’t see how that could have different impact on TheilU2 for different series (and all my forecast series have the same sample). The forecast rank should still be the same according to RMSE and TheilU2.

Re: Forecast evaluation output

Posted: Tue Oct 09, 2018 3:08 pm
by EViews Gareth
Sorry, I'm not sure I understand what you're asking.

I don't believe there it is the case, using the definition of U2 given, that U2 and RMSE have to give the same ranking of forecasts.

Re: Forecast evaluation output

Posted: Wed Oct 10, 2018 6:08 am
by Maglin
Perhaps I'm missing something in the previous topic. Here´s an example to explain my problem:

Series a = actual series
Series f = forecast of a
series fpe = (f-a)/a
series ape = (a-a(-1))/a
genr sq_fpe = (fpe)^2
genr sq_ape = (ape)^2
scalar fpe2 = @sum(sq_fpe)
scalar ape2 = @sum(sq_ape)
RMSE_fpe = @sqrt(fpe2)
RMSE_ape = @sqrt(ape2)
U2= RMSE_fpe / RMSE_ape

Since RMSE_ape does not change for a given sample, RMSE and U2 must give the same ranking of the forecasts. But this is not the case in the output table.

Re: Forecast evaluation output

Posted: Wed Oct 10, 2018 1:15 pm
by EViews Gareth

Code: Select all

create u 100
rndseed 10
Series a = nrnd
series f1 = nrnd
series f2 = nrnd

series e1 = f1-a
series e2 = f2-a
series fpe1 = e1/a(-1)
series fpe2 = e2/a(-1)
series ape = (a(-1)-a)/a(-1)
series sq_fpe1 = (e1/a(-1))^2
series sq_fpe2 = (e2/a(-1))^2
genr sq_ape = ((a(-1)-a)/a(-1))^2
series sq_e1 = e1^2
series sq_e2 = e2^2
smpl 90 @last
scalar fpe21 = @sum(sq_fpe1)
scalar fpe22 = @sum(sq_fpe2)
scalar ape2 = @sum(sq_ape)
scalar U21 = @sqrt(fpe21/ape2)
scalar U22 = @sqrt(fpe22/ape2)

smpl 89 @last

scalar rmse1 = @sqrt(@sum(sq_e1)/12)
scalar rmse2 = @sqrt(@sum(sq_e2)/12)

a.fcasteval f1 f2

Re: Forecast evaluation output

Posted: Thu Oct 11, 2018 7:20 am
by Maglin
Ok, thanks for this clarification. Googled TheilU2 and there seems to be different definitions, not sure which one is the orginal.