In sample and out of sample test
Moderators: EViews Gareth, EViews Moderator

 Fe ddaethom, fe welon, fe amcangyfrifon
 Posts: 11409
 Joined: Tue Sep 16, 2008 5:38 pm
In sample and out of sample test
You'll have to be more specific. What are you trying to test?
Follow us on Twitter @IHSEViews
Re: In sample and out of sample test
Let's say your data sample is 1995q1 2010q4
If I want to test a model used for forecasting, I would estimate it for a subsample that leaves me with enough outofsample observations, such as 1995q1 2006q4.
For insample test, estimate the equation, and check the statistics like RMSE, MAE, etc, that compares the fitted to the actual.
For outofsample, you need to use the estimated equation to produce n steps ahead forecasts, and compare them to the actuals.
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an outofsample RMSE, MAE, etc.
The technical way of doing this is by writing a recursive rolling regression which adds one observation at a time creats a one step ahead forecast and stores it to a series
I have prepared a code for what I wrote above. It is modified for my uses, but hopefully you will be able to get the point.
If I want to test a model used for forecasting, I would estimate it for a subsample that leaves me with enough outofsample observations, such as 1995q1 2006q4.
For insample test, estimate the equation, and check the statistics like RMSE, MAE, etc, that compares the fitted to the actual.
For outofsample, you need to use the estimated equation to produce n steps ahead forecasts, and compare them to the actuals.
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an outofsample RMSE, MAE, etc.
The technical way of doing this is by writing a recursive rolling regression which adds one observation at a time creats a one step ahead forecast and stores it to a series
I have prepared a code for what I wrote above. It is modified for my uses, but hopefully you will be able to get the point.
 Attachments

 rolling.prg
 (4.58 KiB) Downloaded 2231 times
Re: In sample and out of sample test
sweivE wrote:Let's say your data sample is 1995q1 2010q4
If I want to test a model used for forecasting, I would estimate it for a subsample that leaves me with enough outofsample observations, such as 1995q1 2006q4.
For insample test, estimate the equation, and check the statistics like RMSE, MAE, etc, that compares the fitted to the actual.
For outofsample, you need to use the estimated equation to produce n steps ahead forecasts, and compare them to the actuals.
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an outofsample RMSE, MAE, etc.
The technical way of doing this is by writing a recursive rolling regression which adds one observation at a time creats a one step ahead forecast and stores it to a series
I have prepared a code for what I wrote above. It is modified for my uses, but hopefully you will be able to get the point.
My friend,
its exactly what im looking for. i would like to estimate a var model for every subsample and calculate the forecasting for h horizons, then calculate the RMSE for each horizon.
Is it what your program do?
thanks alot
Re: In sample and out of sample test
Hi SweivE
I am trying to calculate the mean squared prediction error for the one month step ahead out of sample forecasting with rolling window forecasting and recursive window forecasting scheme. But, I don't know how to write the code following with below program. Please can you help me? Many Thanks
Dongya
I am trying to calculate the mean squared prediction error for the one month step ahead out of sample forecasting with rolling window forecasting and recursive window forecasting scheme. But, I don't know how to write the code following with below program. Please can you help me? Many Thanks
Code: Select all
'create some data
create u 100
series y=nrnd
series x1=nrnd
series x2=nrnd
series z=nrnd
'
'run rolling regression
' set window size
!window = 20
' set step size
!step = 1
' get size of workfile
!length = @obsrange
' declare equation for estimation
equation eq1
'calculate number of rolls
!nrolls = @floor((!length!window)/!step)
'matrix to store coefficient estimates
matrix(3,!nrolls) coefmat ' where 3 is the number of coefficients
'series to store forecast estimates
series fcast
'redundant series for catching start and end points
series ser = 1
%start = @otod(@ifirst(ser))
%end = @otod(@ilast(ser))
'variable keeping track of how many rolls we've done
!j=0
' move sample !step obs at a time
for !i = 1 to !length!window+1!step step !step
!j=!j+1
' set sample for estimation period
%first = @otod(@dtoo(%start)+!i1)
%last = @otod(@dtoo(%start)+!i+!window2)
smpl {%first} {%last}
' estimate equation  where the equation is y=c(1) + c(2)*x1 + c(3)*x2
eq1.ls y c x1 x2
' store coefficients
colplace(coefmat,eq1.@coefs,!j)
' 4periodahead forecast
%4pers = @otod(@dtoo(%start)+!i+!window1) 'start point
%4pere = @otod(@dtoo(%start)+!i+!window+2) 'end point: %4pere  %4pers +1 = 4
if {%end} < {%4pere} then 'check whether the forecast end point is greater than the workfile end point
return
endif
' set smpl for forecasting period
smpl {%4pers} {%4pere}
' forecast with command *forecast* (see also *fit*)
eq1.forecast(f=na) yf
' set sampl to obtain the 4th period observation
smpl {%4pere} {%4pere}
' store forecasts
fcast = yf
next
smpl @all
show coefmat
show fcast.line
d(noerr) ser
Dongya
Re: In sample and out of sample test
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an outofsample RMSE, MAE, etc.????
== www.solitairecardgame.info ==
== www.solitairecardgame.info ==
Return to “Suggestions and Requests”
Who is online
Users browsing this forum: No registered users and 2 guests