## State space model with nothing to estimate, "near-singular matrix" error

For technical questions regarding estimation of single equations, systems, VARs, Factor analysis and State Space Models in EViews. General econometric questions and advice should go in the Econometric Discussions forum.

Moderators: EViews Gareth, EViews Moderator

dman
Posts: 11
Joined: Mon Jul 18, 2016 10:50 am

### State space model with nothing to estimate, "near-singular matrix" error

Hello.
Would appreciate some help figuring out what's causing the errors.

I'm trying to put together a state-space model with all coefficients/variances already estimated, in order to use Eviews Kalman filter/smoother.
For some (but not all) combinations of my 200-300 signal variables I get "matrix near singular" errors while trying to initialize/estimate the state-space model.

I'm not sure what it's trying to estimate other than the final state vector, which I can try to estimate on my own using Kalman filter formulas and it works w/o a hitch (I do have to use @pinverse though, due to signal idiosyncratic error covariance matrix being rank-deficient).

A attach a small example workfile.
Estimating statespace 'now' on a sample of 1993m1 1993m1 yields the error.
Running the code below to calculate the final state works fine (and produces something fairly close to Eviews calculations for specifications where now.ml does not fail)

Code: Select all

`'--------------------Estimate a couple of Kalman filter steps------------------.'               Filter notation:'               ksi(t+1)=F*ksi(t)+v(t+1)'               y(t)=H'*ksi(t)+w(t)'               var(v(t))=Q'               var(w(t))=R'               matrix F=a1Matrix H=@transpose(lambda)scalar dim=@columns(H)sym Q=sigmahsym R=sig_ksivector(2) ksi1_0=0   'initial statesym P1_0=@unvec(@inverse(@identity(4)-@kronecker(a1,a1))*@vec(sigmah),2) 'initial state MSEmatrix(dim,dim) K1=F*P1_0*H*@pinverse(@transpose(H)*P1_0*H+R) 'Kalman gain period 1now.makeendog endogsample testsmpl 1993m1 1993m1stomna(endog,y1,testsmpl)   'y1-vector of signals on 1993m1vector(2) ksi2_1=F*ksi1_0+K1*(@transpose(y1)-@transpose(H)*ksi1_0) 'state forecast on 1993m2sym P2_1=(F-K1*@transpose(H))*P1_0*(@transpose(F)-H*@transpose(K1))+K1*R*@transpose(K1)+Q 'state MSE on 1993m2`

Thanks in advance to anyone who can help.

PS: my signal idiosyncratic error covariance matrix is rank-deficient in most specifications, even when now.ml goes w/o error.
Attachments
diag.wf1
Workfile with a statespace model "now"

EViews Glenn
EViews Developer
Posts: 2600
Joined: Wed Oct 15, 2008 9:17 am

### Re: State space model with nothing to estimate, "near-singular matrix" error

All that estimate does in a model without coefficients is to run the filter to initialize internal values.

I haven't thought about the KF in a long time, but my first instinct was that you having to use @pinverse in some case was likely to be related to the issue. A quick check of the internals confirms that we are detecting singularity when computing the gain and that @transpose(H)*P1_0*H+R is the problematic term.

dman
Posts: 11
Joined: Mon Jul 18, 2016 10:50 am

### Re: State space model with nothing to estimate, "near-singular matrix" error

Thanks for a reply.
I can post a file with pretty much the SS model above, dropping just one variable, and the 'estimation' would run just fine despite the same matrix still being rank deficient. Which tells me you're using some version of @pinverse code while computing the first step of the filter, but perhaps it is coded differently from the @pinverse code, cause the latter works.

dman
Posts: 11
Joined: Mon Jul 18, 2016 10:50 am

### Re: State space model with nothing to estimate, "near-singular matrix" error

Here's a slightly different file, with a rank-deficient idiosyncratic covariance matrix where Eviews estimation works.

@inverse(@transpose(H)*P1_0*H+R) would give a 'near-singular' matrix error. The code above (with @pinverse) and the Eviews now.ml produce results differing in the 4th decimal.

I suspect the Kalman state filtering formulas use somewhat different implementation of the pseudo-inverse.
Attachments
diag_v2.wf1
forgot to kill extra sheets

EViews Glenn
EViews Developer
Posts: 2600
Joined: Wed Oct 15, 2008 9:17 am

### Re: State space model with nothing to estimate, "near-singular matrix" error

I'm afraid that I wasn't clear in the earlier post. Our implementation of the kfilter does not use a pinverse.

dman
Posts: 11
Joined: Mon Jul 18, 2016 10:50 am

### Re: State space model with nothing to estimate, "near-singular matrix" error

Is there a reference on how it works w/o it? I'm trying to understand why it works in some rank-deficient signal variance cases and fails in others, so that I could possibly prevent failures early.

EViews Glenn
EViews Developer
Posts: 2600
Joined: Wed Oct 15, 2008 9:17 am

### Re: State space model with nothing to estimate, "near-singular matrix" error

We use a standard inverse. My guess is that the rank deficiencies are not numeric in the other cases.