In sample and out of sample test

[EViews Gareth]

You'll have to be more specific. What are you trying to test?

[sweivE]

Let's say your data sample is 1995q1 2010q4
If I want to test a model used for forecasting, I would estimate it for a sub-sample that leaves me with enough out-of-sample observations, such as 1995q1 2006q4.

For in-sample test, estimate the equation, and check the statistics like RMSE, MAE, etc, that compares the fitted to the actual.
For out-of-sample, you need to use the estimated equation to produce n steps ahead forecasts, and compare them to the actuals.
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an out-of-sample RMSE, MAE, etc.

The technical way of doing this is by writing a recursive rolling regression which adds one observation at a time creats a one step ahead forecast and stores it to a series

I have prepared a code for what I wrote above. It is modified for my uses, but hopefully you will be able to get the point.

[hichbenn]

Let's say your data sample is 1995q1 2010q4
If I want to test a model used for forecasting, I would estimate it for a sub-sample that leaves me with enough out-of-sample observations, such as 1995q1 2006q4.

For in-sample test, estimate the equation, and check the statistics like RMSE, MAE, etc, that compares the fitted to the actual.
For out-of-sample, you need to use the estimated equation to produce n steps ahead forecasts, and compare them to the actuals.
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an out-of-sample RMSE, MAE, etc.

The technical way of doing this is by writing a recursive rolling regression which adds one observation at a time creats a one step ahead forecast and stores it to a series

I have prepared a code for what I wrote above. It is modified for my uses, but hopefully you will be able to get the point.

My friend,
its exactly what im looking for. i would like to estimate a var model for every subsample and calculate the forecasting for h horizons, then calculate the RMSE for each horizon.
Is it what your program do?
thanks alot

[dongyadu]

Hi SweivE

I am trying to calculate the mean squared prediction error for the one month step ahead out of sample forecasting with rolling window forecasting and recursive window forecasting scheme. But, I don't know how to write the code following with below program. Please can you help me? Many Thanks

Code: Select all

'create some data create u 100 series y=nrnd series x1=nrnd series x2=nrnd series z=nrnd '------------------------------------------------------------------------------------- 'run rolling regression ' set window size !window = 20 ' set step size !step = 1 ' get size of workfile !length = @obsrange ' declare equation for estimation equation eq1 'calculate number of rolls !nrolls = @floor((!length-!window)/!step) 'matrix to store coefficient estimates matrix(3,!nrolls) coefmat ' where 3 is the number of coefficients 'series to store forecast estimates series fcast 'redundant series for catching start and end points series ser = 1 %start = @otod(@ifirst(ser)) %end = @otod(@ilast(ser)) 'variable keeping track of how many rolls we've done !j=0 ' move sample !step obs at a time for !i = 1 to !length-!window+1-!step step !step !j=!j+1 ' set sample for estimation period %first = @otod(@dtoo(%start)+!i-1) %last = @otod(@dtoo(%start)+!i+!window-2) smpl {%first} {%last} ' estimate equation - where the equation is y=c(1) + c(2)*x1 + c(3)*x2 eq1.ls y c x1 x2 ' store coefficients colplace(coefmat,eq1.@coefs,!j) ' 4-period-ahead forecast %4pers = @otod(@dtoo(%start)+!i+!window-1) 'start point %4pere = @otod(@dtoo(%start)+!i+!window+2) 'end point: %4pere - %4pers +1 = 4 if {%end} < {%4pere} then 'check whether the forecast end point is greater than the workfile end point return endif ' set smpl for forecasting period smpl {%4pers} {%4pere} ' forecast with command *forecast* (see also *fit*) eq1.forecast(f=na) yf ' set sampl to obtain the 4th period observation smpl {%4pere} {%4pere} ' store forecasts fcast = yf next smpl @all show coefmat show fcast.line d(noerr) ser

Dongya

[Faqeer1]

For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an out-of-sample RMSE, MAE, etc.????


== www.solitairecardgame.info ==