Page 1 of 1

Kalman filter state vector unmatch

Posted: Tue Apr 12, 2011 2:19 am
by watabehj
Hello. First time posting.

I was trying to make migration from TSP to Eviews. However, I couldn't reproduce the result of the original TSP program using Kalman filter. I found that the state vectors calculated in TSP and Eviews are different.
Attached please find the Excel file which contains both programs.
Kalmantest.xls
Excel file concerning Kalman filter state vector unmatch
(27 KiB) Downloaded 301 times

Sheet "TSP PGM" contains TSP version, and Sheet "Eviews PGM" contains Eviews version. (Note: This is simplified version of original program, so it contains only one observation.) The state vector calculated by TSP is (-0.06, -0.02, -0.02, 0.44), whereas that of Eviews is (0.2, 0, 0, -0.1). I also calculated the state vector using formula written in wikipedia, and that result matched TSP (see Sheet "Check by Excel").

My Eviews is version 7.1. I would appreciate any information on this matter.

Re: Kalman filter state vector unmatch

Posted: Tue Apr 12, 2011 11:10 am
by EViews Glenn
Hi, welcome to EViews, glad to have you join us...

Two things going on here...

First, the EViews specification doesn't quite match the specification you appear to be using in your calculation. In particular, you've set the signal variances to zero, which doesn't match the R signal variance matrix (which is a diagonal matrix with unit variances) that you used in your spreadsheet. You'll want to modify the EViews signal equations to add the variances.

Second, as noted in the manual, the EViews initial state vector and variance must be specified as P(1|0) and s(1|0), not as P(0|0) and s(0|0) as you appear to be doing. For s(0|0) this doesn't matter in your specification since applying the one-step-ahead update yields the same vector, but this difference does matter for P(1|0). Translating from P(0|0) to P(1|0) using the expression

P(1|0) = F*P(0|0)*F' + Q

and then entering those values in as the initial conditions solves the second problem. Note that the invariance of the state vector to the prediction update is not a general result and typically one would have to obtain s(1|0).

Modifying your program, we have

Code: Select all

wfcreate biwatest m 2011:02 2011:02            
series Y1            
Y1.fill(o=2011M2) 0.1            
series Y2            
Y2.fill(o=2011M2) 0.2            
smpl 2011M2 2011:02            
sspace ss1            
  ss1.append @signal Y1=sv1+sv4 + [var=1]         
  ss1.append @signal Y2=sv1 + [var=1]            
  ss1.append @state sv1= sv1(-1) + sv2(-1) + [var=1]            
  ss1.append @state sv2= sv1(-1)          
  ss1.append @state sv3= sv2(-1)            
  ss1.append @state sv4= sv4(-1) + [var=1]            
vector(4) svec0            
svec0.fill 0,0,0,1            
sym svar0 = @identity(4)            
svar0(1,1) = 3
svar0(1,3) = 1
svar0(2,1) = 1
svar0(3,1) = 1
svar0(4,4) = 2   
ss1.append @mprior svec0            
ss1.append @vprior svar0            
ss1.ml(showopts,m=1000,c=1e-9)            
ss1.makestate(t=filt) f*

which yields the desired results. I've specified each of the individual elements of the modified svar0, but you could, if you want, specify matrices for F and Q instead, and then use an expression to modify your original svar0.

Hope this answers your question.

(Incidentally, answering this question was made a lot easier by all of the information you provided in the spreadsheet. In particular, it would have taken me forever to find the missing R signal variances had you not been so explicit about what you were trying to do and had I only had the TSP code to go by... Thanks...)

Re: Kalman filter state vector unmatch

Posted: Tue Apr 12, 2011 8:59 pm
by watabehj
Thank you very much! It perfectly solved my problem. :D

Re: Kalman filter state vector unmatch

Posted: Fri Jan 31, 2020 12:51 am
by watabehj
The above program Mr. EViews Glenn posted at Wed Apr 13, 2011 4:10 am used to yield the state vector (-0.06, -0.02, -0.02, 0.44) as intended. But when run on the newest version of EViews, it yields the state vector (-0.08, -0.06, -0.02, 0.44). I'd like to know why this change occurred. I would appreciate any information on this matter.

Re: Kalman filter state vector unmatch

Posted: Fri Jan 31, 2020 1:39 pm
by EViews Glenn
Could you please repost the spreadsheet if possible?

Re: Kalman filter state vector unmatch

Posted: Sun Feb 02, 2020 5:21 pm
by watabehj
Attached please find the spreadsheet I previously posted at Tue Apr 12, 2011 7:19 pm.

Re: Kalman filter state vector unmatch

Posted: Tue Feb 04, 2020 11:26 am
by EViews Glenn
If you change your option to smoothing, you match the TSP results (note that your TSP command smooths).

ss1.makestate(t=smooth) f*

[edit: However, in this case the smooths and the filtered should align. All of the reported one-step ahead and smoothed look fine and all of our test cases report that there is no issue there, but I think there is an issue with our output of the filtered. I'm investigating further.]

Re: Kalman filter state vector unmatch

Posted: Tue Feb 04, 2020 7:51 pm
by watabehj
Thank you for your reply. I see that "makestate(t=filt)" and "makestate(t=smooth)" used to yield the same state vector for my one-period sample, but now they yield different state vectors. I presume that change occurred at November, 2019 patch, which describes "Fix for kalman filter contemporaneous updates for output of saved series." Unfortunately, we are now using Kalman filter without smoothing, so the change is fatal for us.

It seems that "makestate(t=filt)" command used to yield "the filtered (contemporaneous) state vector" noted as alpha_t in the User's Guide Chapter "State Space Models and the Kalman Filter", but now it yields "the final state vector" noted as alpha_T in the User's Guide. However, "Make State Series" section in the User's Guide says it saves alpha_t, not alpha_T. So I suspect that "makestate(t=filt)" command now does not match the description of the User's Guide. I'd like to know if there is a way to save alpha_t in the newest version of EViews.

Re: Kalman filter state vector unmatch

Posted: Tue Feb 04, 2020 7:58 pm
by EViews Glenn
Thanks for the additional info. I'll get back to you as soon as possible.

Re: Kalman filter state vector unmatch

Posted: Wed Feb 05, 2020 12:02 pm
by EViews Glenn
Apologies. Found the issue and we will get a fix out shortly. I'm very sorry for the inconvenience. Thanks again for your help.

Re: Kalman filter state vector unmatch

Posted: Thu Feb 06, 2020 2:24 pm
by EViews Glenn
The patch has been released.

Re: Kalman filter state vector unmatch

Posted: Wed Feb 12, 2020 5:15 pm
by watabehj
We've downloaded the patch and it seems OK. Thank you for the quick fix.