In sample and out of sample test
Posted: Sat Aug 27, 2011 8:00 am
by EViews Gareth
You'll have to be more specific. What are you trying to test?
Re: In sample and out of sample test
Posted: Sun Aug 28, 2011 7:39 am
by sweivE
Let's say your data sample is 1995q1 2010q4
If I want to test a model used for forecasting, I would estimate it for a sub-sample that leaves me with enough out-of-sample observations, such as 1995q1 2006q4.
For in-sample test, estimate the equation, and check the statistics like RMSE, MAE, etc, that compares the fitted to the actual.
For out-of-sample, you need to use the estimated equation to produce n steps ahead forecasts, and compare them to the actuals.
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an out-of-sample RMSE, MAE, etc.
The technical way of doing this is by writing a recursive rolling regression which adds one observation at a time creats a one step ahead forecast and stores it to a series
I have prepared a code for what I wrote above. It is modified for my uses, but hopefully you will be able to get the point.
Re: In sample and out of sample test
Posted: Tue Sep 20, 2011 6:52 am
by hichbenn
Let's say your data sample is 1995q1 2010q4
If I want to test a model used for forecasting, I would estimate it for a sub-sample that leaves me with enough out-of-sample observations, such as 1995q1 2006q4.
For in-sample test, estimate the equation, and check the statistics like RMSE, MAE, etc, that compares the fitted to the actual.
For out-of-sample, you need to use the estimated equation to produce n steps ahead forecasts, and compare them to the actuals.
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an out-of-sample RMSE, MAE, etc.
The technical way of doing this is by writing a recursive rolling regression which adds one observation at a time creats a one step ahead forecast and stores it to a series
I have prepared a code for what I wrote above. It is modified for my uses, but hopefully you will be able to get the point.
My friend,
its exactly what im looking for. i would like to estimate a var model for every subsample and calculate the forecasting for h horizons, then calculate the RMSE for each horizon.
Is it what your program do?
thanks alot
Re: In sample and out of sample test
Posted: Mon Oct 03, 2011 6:23 am
by dongyadu
Hi SweivE
I am trying to calculate the mean squared prediction error for the one month step ahead out of sample forecasting with rolling window forecasting and recursive window forecasting scheme. But, I don't know how to write the code following with below program. Please can you help me? Many Thanks
Code: Select all
'create some data
create u 100
series y=nrnd
series x1=nrnd
series x2=nrnd
series z=nrnd
'-------------------------------------------------------------------------------------
'run rolling regression
' set window size
!window = 20
' set step size
!step = 1
' get size of workfile
!length = @obsrange
' declare equation for estimation
equation eq1
'calculate number of rolls
!nrolls = @floor((!length-!window)/!step)
'matrix to store coefficient estimates
matrix(3,!nrolls) coefmat ' where 3 is the number of coefficients
'series to store forecast estimates
series fcast
'redundant series for catching start and end points
series ser = 1
%start = @otod(@ifirst(ser))
%end = @otod(@ilast(ser))
'variable keeping track of how many rolls we've done
!j=0
' move sample !step obs at a time
for !i = 1 to !length-!window+1-!step step !step
!j=!j+1
' set sample for estimation period
%first = @otod(@dtoo(%start)+!i-1)
%last = @otod(@dtoo(%start)+!i+!window-2)
smpl {%first} {%last}
' estimate equation - where the equation is y=c(1) + c(2)*x1 + c(3)*x2
eq1.ls y c x1 x2
' store coefficients
colplace(coefmat,eq1.@coefs,!j)
' 4-period-ahead forecast
%4pers = @otod(@dtoo(%start)+!i+!window-1) 'start point
%4pere = @otod(@dtoo(%start)+!i+!window+2) 'end point: %4pere - %4pers +1 = 4
if {%end} < {%4pere} then 'check whether the forecast end point is greater than the workfile end point
return
endif
' set smpl for forecasting period
smpl {%4pers} {%4pere}
' forecast with command *forecast* (see also *fit*)
eq1.forecast(f=na) yf
' set sampl to obtain the 4th period observation
smpl {%4pere} {%4pere}
' store forecasts
fcast = yf
next
smpl @all
show coefmat
show fcast.line
d(noerr) ser
Dongya
Re: In sample and out of sample test
Posted: Sun Apr 10, 2016 11:30 pm
by Faqeer1
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an out-of-sample RMSE, MAE, etc.????
==
www.solitairecardgame.info ==