State space model with nothing to estimate, "near-singular matrix" error

For technical questions regarding estimation of single equations, systems, VARs, Factor analysis and State Space Models in EViews. General econometric questions and advice should go in the Econometric Discussions forum.

Moderators: EViews Gareth, EViews Moderator

dman
Posts: 11
Joined: Mon Jul 18, 2016 10:50 am

State space model with nothing to estimate, "near-singular matrix" error

Postby dman » Mon Feb 05, 2018 1:46 pm

Hello.
Would appreciate some help figuring out what's causing the errors.

I'm trying to put together a state-space model with all coefficients/variances already estimated, in order to use Eviews Kalman filter/smoother.
For some (but not all) combinations of my 200-300 signal variables I get "matrix near singular" errors while trying to initialize/estimate the state-space model.

I'm not sure what it's trying to estimate other than the final state vector, which I can try to estimate on my own using Kalman filter formulas and it works w/o a hitch (I do have to use @pinverse though, due to signal idiosyncratic error covariance matrix being rank-deficient).

A attach a small example workfile.
Estimating statespace 'now' on a sample of 1993m1 1993m1 yields the error.
Running the code below to calculate the final state works fine (and produces something fairly close to Eviews calculations for specifications where now.ml does not fail)

Code: Select all

'--------------------Estimate a couple of Kalman filter steps------------------.
'               Filter notation:
'               ksi(t+1)=F*ksi(t)+v(t+1)
'               y(t)=H'*ksi(t)+w(t)
'               var(v(t))=Q
'               var(w(t))=R
'               
matrix F=a1
Matrix H=@transpose(lambda)
scalar dim=@columns(H)
sym Q=sigmah
sym R=sig_ksi
vector(2) ksi1_0=0   'initial state
sym P1_0=@unvec(@inverse(@identity(4)-@kronecker(a1,a1))*@vec(sigmah),2) 'initial state MSE
matrix(dim,dim) K1=F*P1_0*H*@pinverse(@transpose(H)*P1_0*H+R) 'Kalman gain period 1
now.makeendog endog
sample testsmpl 1993m1 1993m1
stomna(endog,y1,testsmpl)   'y1-vector of signals on 1993m1
vector(2) ksi2_1=F*ksi1_0+K1*(@transpose(y1)-@transpose(H)*ksi1_0) 'state forecast on 1993m2
sym P2_1=(F-K1*@transpose(H))*P1_0*(@transpose(F)-H*@transpose(K1))+K1*R*@transpose(K1)+Q 'state MSE on 1993m2



Thanks in advance to anyone who can help.

PS: my signal idiosyncratic error covariance matrix is rank-deficient in most specifications, even when now.ml goes w/o error.
Attachments
diag.wf1
Workfile with a statespace model "now"
(69.07 KiB) Downloaded 23 times

EViews Glenn
EViews Developer
Posts: 2597
Joined: Wed Oct 15, 2008 9:17 am

Re: State space model with nothing to estimate, "near-singular matrix" error

Postby EViews Glenn » Fri Feb 09, 2018 4:24 pm

All that estimate does in a model without coefficients is to run the filter to initialize internal values.

I haven't thought about the KF in a long time, but my first instinct was that you having to use @pinverse in some case was likely to be related to the issue. A quick check of the internals confirms that we are detecting singularity when computing the gain and that @transpose(H)*P1_0*H+R is the problematic term.

dman
Posts: 11
Joined: Mon Jul 18, 2016 10:50 am

Re: State space model with nothing to estimate, "near-singular matrix" error

Postby dman » Tue Feb 13, 2018 4:06 pm

Thanks for a reply.
I can post a file with pretty much the SS model above, dropping just one variable, and the 'estimation' would run just fine despite the same matrix still being rank deficient. Which tells me you're using some version of @pinverse code while computing the first step of the filter, but perhaps it is coded differently from the @pinverse code, cause the latter works.

dman
Posts: 11
Joined: Mon Jul 18, 2016 10:50 am

Re: State space model with nothing to estimate, "near-singular matrix" error

Postby dman » Thu Feb 15, 2018 9:10 am

Here's a slightly different file, with a rank-deficient idiosyncratic covariance matrix where Eviews estimation works.

@inverse(@transpose(H)*P1_0*H+R) would give a 'near-singular' matrix error. The code above (with @pinverse) and the Eviews now.ml produce results differing in the 4th decimal.

I suspect the Kalman state filtering formulas use somewhat different implementation of the pseudo-inverse.
Attachments
diag_v2.wf1
forgot to kill extra sheets
(64.21 KiB) Downloaded 16 times

EViews Glenn
EViews Developer
Posts: 2597
Joined: Wed Oct 15, 2008 9:17 am

Re: State space model with nothing to estimate, "near-singular matrix" error

Postby EViews Glenn » Tue Feb 20, 2018 11:18 am

I'm afraid that I wasn't clear in the earlier post. Our implementation of the kfilter does not use a pinverse.

dman
Posts: 11
Joined: Mon Jul 18, 2016 10:50 am

Re: State space model with nothing to estimate, "near-singular matrix" error

Postby dman » Tue Feb 20, 2018 5:59 pm

Is there a reference on how it works w/o it? I'm trying to understand why it works in some rank-deficient signal variance cases and fails in others, so that I could possibly prevent failures early.

EViews Glenn
EViews Developer
Posts: 2597
Joined: Wed Oct 15, 2008 9:17 am

Re: State space model with nothing to estimate, "near-singular matrix" error

Postby EViews Glenn » Tue Mar 06, 2018 2:19 pm

We use a standard inverse. My guess is that the rank deficiencies are not numeric in the other cases.


Return to “Estimation”

Who is online

Users browsing this forum: No registered users and 10 guests