## In sample and out of sample test

For making suggestions and/or requests for new features you'd like added to EViews.

Moderators: EViews Gareth, EViews Moderator

EViews Gareth
Fe ddaethom, fe welon, fe amcangyfrifon
Posts: 11776
Joined: Tue Sep 16, 2008 5:38 pm

### In sample and out of sample test

You'll have to be more specific. What are you trying to test?

sweivE
Posts: 18
Joined: Thu Mar 03, 2011 5:21 am

### Re: In sample and out of sample test

Let's say your data sample is 1995q1 2010q4
If I want to test a model used for forecasting, I would estimate it for a sub-sample that leaves me with enough out-of-sample observations, such as 1995q1 2006q4.

For in-sample test, estimate the equation, and check the statistics like RMSE, MAE, etc, that compares the fitted to the actual.
For out-of-sample, you need to use the estimated equation to produce n steps ahead forecasts, and compare them to the actuals.
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an out-of-sample RMSE, MAE, etc.

The technical way of doing this is by writing a recursive rolling regression which adds one observation at a time creats a one step ahead forecast and stores it to a series

I have prepared a code for what I wrote above. It is modified for my uses, but hopefully you will be able to get the point.
Attachments
rolling.prg

hichbenn
Posts: 5
Joined: Tue Sep 20, 2011 3:43 am

### Re: In sample and out of sample test

sweivE wrote:Let's say your data sample is 1995q1 2010q4
If I want to test a model used for forecasting, I would estimate it for a sub-sample that leaves me with enough out-of-sample observations, such as 1995q1 2006q4.

For in-sample test, estimate the equation, and check the statistics like RMSE, MAE, etc, that compares the fitted to the actual.
For out-of-sample, you need to use the estimated equation to produce n steps ahead forecasts, and compare them to the actuals.
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an out-of-sample RMSE, MAE, etc.

The technical way of doing this is by writing a recursive rolling regression which adds one observation at a time creats a one step ahead forecast and stores it to a series

I have prepared a code for what I wrote above. It is modified for my uses, but hopefully you will be able to get the point.

My friend,
its exactly what im looking for. i would like to estimate a var model for every subsample and calculate the forecasting for h horizons, then calculate the RMSE for each horizon.
Is it what your program do?
thanks alot

Posts: 12
Joined: Mon Sep 05, 2011 7:43 pm

### Re: In sample and out of sample test

Hi SweivE

I am trying to calculate the mean squared prediction error for the one month step ahead out of sample forecasting with rolling window forecasting and recursive window forecasting scheme. But, I don't know how to write the code following with below program. Please can you help me? Many Thanks

Code: Select all

`    'create some data    create u 100    series y=nrnd    series x1=nrnd    series x2=nrnd    series z=nrnd    '-------------------------------------------------------------------------------------    'run rolling regression    ' set window size    !window = 20    ' set step size    !step = 1    ' get size of workfile    !length = @obsrange    ' declare equation for estimation    equation eq1    'calculate number of rolls    !nrolls = @floor((!length-!window)/!step)    'matrix to store coefficient estimates    matrix(3,!nrolls) coefmat ' where 3 is the number of coefficients    'series to store forecast estimates    series fcast    'redundant series for catching start and end points    series ser = 1       %start = @otod(@ifirst(ser))       %end = @otod(@ilast(ser))           'variable keeping track of how many rolls we've done    !j=0    ' move sample !step obs at a time    for !i = 1  to  !length-!window+1-!step step !step       !j=!j+1                 ' set sample for estimation period                %first = @otod(@dtoo(%start)+!i-1)       %last = @otod(@dtoo(%start)+!i+!window-2)       smpl {%first} {%last}             ' estimate equation - where the equation is y=c(1) + c(2)*x1 + c(3)*x2       eq1.ls y c x1 x2              ' store coefficients       colplace(coefmat,eq1.@coefs,!j)              ' 4-period-ahead forecast       %4pers = @otod(@dtoo(%start)+!i+!window-1)      'start point       %4pere = @otod(@dtoo(%start)+!i+!window+2)   'end point: %4pere - %4pers +1 = 4       if {%end} < {%4pere} then   'check whether the forecast end point is greater than the workfile end point          return       endif              ' set smpl for forecasting period       smpl {%4pers} {%4pere}                 ' forecast with command *forecast* (see also *fit*)       eq1.forecast(f=na) yf                    ' set sampl to obtain the 4th period observation       smpl {%4pere} {%4pere}                 ' store forecasts       fcast = yf    next    smpl @all    show coefmat    show fcast.line    d(noerr) ser`

Dongya

Faqeer1
Posts: 1
Joined: Sun Apr 10, 2016 11:24 pm

### Re: In sample and out of sample test

For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an out-of-sample RMSE, MAE, etc.????

== www.solitairecardgame.info ==