In sample and out of sample test

For making suggestions and/or requests for new features you'd like added to EViews.

Moderators: EViews Gareth, EViews Moderator

EViews Gareth
Fe ddaethom, fe welon, fe amcangyfrifon
Posts: 13307
Joined: Tue Sep 16, 2008 5:38 pm

In sample and out of sample test

Postby EViews Gareth » Sat Aug 27, 2011 8:00 am

You'll have to be more specific. What are you trying to test?
Follow us on Twitter @IHSEViews

sweivE
Posts: 18
Joined: Thu Mar 03, 2011 5:21 am

Re: In sample and out of sample test

Postby sweivE » Sun Aug 28, 2011 7:39 am

Let's say your data sample is 1995q1 2010q4
If I want to test a model used for forecasting, I would estimate it for a sub-sample that leaves me with enough out-of-sample observations, such as 1995q1 2006q4.

For in-sample test, estimate the equation, and check the statistics like RMSE, MAE, etc, that compares the fitted to the actual.
For out-of-sample, you need to use the estimated equation to produce n steps ahead forecasts, and compare them to the actuals.
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an out-of-sample RMSE, MAE, etc.

The technical way of doing this is by writing a recursive rolling regression which adds one observation at a time creats a one step ahead forecast and stores it to a series

I have prepared a code for what I wrote above. It is modified for my uses, but hopefully you will be able to get the point.
Attachments
rolling.prg
(4.58 KiB) Downloaded 3163 times

hichbenn
Posts: 5
Joined: Tue Sep 20, 2011 3:43 am

Re: In sample and out of sample test

Postby hichbenn » Tue Sep 20, 2011 6:52 am

sweivE wrote:Let's say your data sample is 1995q1 2010q4
If I want to test a model used for forecasting, I would estimate it for a sub-sample that leaves me with enough out-of-sample observations, such as 1995q1 2006q4.

For in-sample test, estimate the equation, and check the statistics like RMSE, MAE, etc, that compares the fitted to the actual.
For out-of-sample, you need to use the estimated equation to produce n steps ahead forecasts, and compare them to the actuals.
For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an out-of-sample RMSE, MAE, etc.

The technical way of doing this is by writing a recursive rolling regression which adds one observation at a time creats a one step ahead forecast and stores it to a series

I have prepared a code for what I wrote above. It is modified for my uses, but hopefully you will be able to get the point.


My friend,
its exactly what im looking for. i would like to estimate a var model for every subsample and calculate the forecasting for h horizons, then calculate the RMSE for each horizon.
Is it what your program do?
thanks alot

dongyadu
Posts: 12
Joined: Mon Sep 05, 2011 7:43 pm

Re: In sample and out of sample test

Postby dongyadu » Mon Oct 03, 2011 6:23 am

Hi SweivE

I am trying to calculate the mean squared prediction error for the one month step ahead out of sample forecasting with rolling window forecasting and recursive window forecasting scheme. But, I don't know how to write the code following with below program. Please can you help me? Many Thanks

Code: Select all

    'create some data
    create u 100

    series y=nrnd
    series x1=nrnd
    series x2=nrnd
    series z=nrnd

    '-------------------------------------------------------------------------------------
    'run rolling regression

    ' set window size
    !window = 20

    ' set step size
    !step = 1

    ' get size of workfile
    !length = @obsrange

    ' declare equation for estimation
    equation eq1

    'calculate number of rolls
    !nrolls = @floor((!length-!window)/!step)

    'matrix to store coefficient estimates
    matrix(3,!nrolls) coefmat ' where 3 is the number of coefficients

    'series to store forecast estimates
    series fcast

    'redundant series for catching start and end points
    series ser = 1   
    %start = @otod(@ifirst(ser))   
    %end = @otod(@ilast(ser))
       
    'variable keeping track of how many rolls we've done
    !j=0

    ' move sample !step obs at a time
    for !i = 1  to  !length-!window+1-!step step !step
       !j=!j+1
         
       ' set sample for estimation period         
       %first = @otod(@dtoo(%start)+!i-1)
       %last = @otod(@dtoo(%start)+!i+!window-2)
       smpl {%first} {%last}     

       ' estimate equation - where the equation is y=c(1) + c(2)*x1 + c(3)*x2
       eq1.ls y c x1 x2
       
       ' store coefficients
       colplace(coefmat,eq1.@coefs,!j)
       
       ' 4-period-ahead forecast
       %4pers = @otod(@dtoo(%start)+!i+!window-1)      'start point
       %4pere = @otod(@dtoo(%start)+!i+!window+2)   'end point: %4pere - %4pers +1 = 4
       if {%end} < {%4pere} then   'check whether the forecast end point is greater than the workfile end point
          return
       endif
       
       ' set smpl for forecasting period
       smpl {%4pers} {%4pere}   
       
       ' forecast with command *forecast* (see also *fit*)
       eq1.forecast(f=na) yf     
       
       ' set sampl to obtain the 4th period observation
       smpl {%4pere} {%4pere}   
       
       ' store forecasts
       fcast = yf
    next

    smpl @all
    show coefmat
    show fcast.line

    d(noerr) ser




Dongya

Faqeer1
Posts: 1
Joined: Sun Apr 10, 2016 11:24 pm

Re: In sample and out of sample test

Postby Faqeer1 » Sun Apr 10, 2016 11:30 pm

For example, for a 1 step ahead forecast you will run the regression 20 times (20 quarters) and get 20 forecasts. Once you have a series of 20 forecasts you can use it and the actuals to compute an out-of-sample RMSE, MAE, etc.????


== www.solitairecardgame.info ==


Return to “Suggestions and Requests”

Who is online

Users browsing this forum: No registered users and 17 guests