EViews.com

I do not understand what R^2 is in the WLS output.
In the Weighted Least Squares output, the manual states the statistics are from the actual estimated equation.
If that means R^2 is computed as 1 minus Residual SS / Total SS , and we use the output, the R^2 computed that
way is not the reported R^2.
Example:
Dependent Variable: LRATMUR
Method: Least Squares
Date: 09/13/13 Time: 13:55
Sample: 1 703
Included observations: 703
Weighting series: 1/DENSITY
Weight type: Inverse standard deviation (no scaling)

Variable Coefficient Std. Error t-Statistic Prob.

C 6.941144 1.473009 4.712220 0.0000
LOG(RPCPI) 0.243077 0.167021 1.455373 0.1460
LOG(ARRMURD) -0.720749 0.030417 -23.69553 0.0000
LOG(CONVMURD) -0.665553 0.030671 -21.69945 0.0000
LOG(DENSITY) -0.320790 0.030445 -10.53667 0.0000
LOG(PPB) 0.002043 0.023552 0.086740 0.9309

Weighted Statistics

R-squared 0.615787 Mean dependent var 0.057614
Adjusted R-squared 0.613031 S.D. dependent var 0.150715 R^2 = 1 - 0.686284/(0.150715*(703-6))=1-0.686284/15.94545=.956951
S.E. of regression 0.031379 Akaike info criterion -4.076874
Sum squared resid 0.686284 Schwarz criterion -4.037995
Log likelihood 1439.021 Hannan-Quinn criter. -4.061848
F-statistic 223.4198 Durbin-Watson stat 1.678864
Prob(F-statistic) 0.000000 Weighted mean dep. 2.490813

Unweighted Statistics

R-squared -0.073278 Mean dependent var 1.691755
Adjusted R-squared -0.080977 S.D. dependent var 0.841445
S.E. of regression 0.874851 Sum squared resid 533.4590
Durbin-Watson stat 0.796153

And that agrees with the R^2 calculated in a least squares with weighted variables
Dependent Variable: LRATMUR/DENSITY
Method: Least Squares
Date: 09/14/13 Time: 10:15
Sample: 1 703
Included observations: 703

Variable Coefficient Std. Error t-Statistic Prob.

1/DENSITY 6.941144 1.473009 4.712220 0.0000
LOG(RPCPI)/DENSITY 0.243077 0.167021 1.455373 0.1460
LOG(ARRMURD)/DENSITY -0.720749 0.030417 -23.69553 0.0000
LOG(CONVMURD)/DENSITY -0.665553 0.030671 -21.69945 0.0000
LOG(DENSITY)/DENSITY -0.320790 0.030445 -10.53667 0.0000
LOG(PPB)/DENSITY 0.002043 0.023552 0.086740 0.9309

R-squared 0.956962 Mean dependent var 0.057614
Adjusted R-squared 0.956653 S.D. dependent var 0.150715
S.E. of regression 0.031379 Akaike info criterion -4.076874
Sum squared resid 0.686284 Schwarz criterion -4.037995
Log likelihood 1439.021 Hannan-Quinn criter. -4.061848
Durbin-Watson stat 1.678864

Another interpretation (and the original) is the correlation squared between the actual and the predicted.
The correlation is .978249 which squared is .956971 -

So what is 0.615787???

Additionally, what is the unweighted NEGATIVE R^2? Yes it is calculated as 1 - RSS/TSS from
R-squared -0.07327801 Mean dependent var 1.691754524
Adjusted R-squared -0.080977278 S.D. dependent var 0.841445199
S.E. of regression 0.874851054 Sum squared resid 533.4589635
Durbin-Watson stat 0.796153391

but what do those numbers mean (what are their exact formulas)

Bob

Sorry, I meant to post the data set.
Bob

For the weighted R2 you want to compare the sum-of-squares from the weighted model with the sum-of-squares from the weighted intercept only model For the unweighted R2 (which I don't think is a particularly interesting statistic), you want to do the computations in the unweighted data space. Since the computations are done in unweighted space, but the estimation is in weighted space, it is possible to get negative R2s: Note that SSR1A and SSR1B are two ways of computing the same thing.

Thanks so much for your reply.

Can you tell me how test.@ssr is calculated. Its value is 1.786208 . For an intercept only model, the SSResidual is exactly the SSTotal. The SS of LRATMUR/DENSITY is 15.94595 as I calculated from both the regression output and from descriptive statistics. The SS of LRATMUR is 497.0371. How is test.@ssr equal to 1.786208? I will delay arguments about the unweighted statistics.
Bob

Thank you. Your R^2 appears to be the Generalized R^2 by Anderson-Sprecher, Am Stat, Vol 48 No 2 (May 1994) which is defined as GR^2 = 1 – (RSS(full)/RSS(reduced)). If RSS(full)<RSS(reduced) the measure is between 0 and 1. If the estimates minimize RSS(full) and RSS(reduced) the measure is between 0 and 1. You stated "you want to compare the sum-of-squares from the weighted model with the sum-of-squares from the weighted intercept only model". I am not sure that is what I want but given the discussion in Anderson-Sprecher, I know why others might want that measure. The manual needs to state that. The manual states "The Weighted Statistics show statistics corresponding to the actual estimated equation." What is the actual estimated equation?

Consider the two equations: Both models contain an intercept, both models have the same coefficient estimates, and the same S.E. of regression, Sum squared resid, Log likelihood, Mean dependent var, S.D. dependent var, Akaike info criterion, Schwarz criterion, Hannan-Quinn criter., and Durbin-Watson stat. I would argue that both outputs should produce identical R^2 values. eq08_handi R^2 is the explained variation of lratmur/density about the mean of lratmur/density while eq08_wlsi is the variation of lratmur/density about 1/density. To me it is not clear which is a better measure.

Unweighted statististics: You state 'Since the computations are done in unweighted space, but the estimation is in weighted space, it is possible to get negative R2s:' . Yes by your formula. Your calculation is Willett and Singer's (Amstat Vol 42 No 3 August 1988) pseudo R^2. They do not discuss (and maybe they do not realize) the measure can be negative. The residuals of non-ols coefficient estimates are not orthogonal to the predicteds. So sumsq(residuals_nonOLS) can and will be larger than sumsq(residuals_OLS) often producing a negative measure.

R^2 is supposed to be a measure (percent) of explained variation. The coefficient estimates from the WLS equation are not OLS and hence the percent explained by those coefficients must be less than the OLS explanation. Because the WLS coefficients are not OLS, the usual orthogonal decomposition of sums of squares does not work. But what I want is to know a percent explained variation from the WLS coefficients to compare to the OLS percent. Using RSS does not yield such a measure. The correlation^2 between the actual and predicted does.

Maybe you could do both, your GR^2 and the correlation squared between predicted and actual, in both the weighted and unweighted output.

I wasn't familiar with the Anderson-Sprecher term for this R2--thanks for the reference. I was simply taught that this is the way to think of R2 in a weighted model, without a specific name or reference.

By actual estimation, I meant an equation that is equivalent to the weighted least squares specification: obtained by transforming the specification so that the errors are homoskedastic. You are correct that we should be clearer about what this means for the R2 computation--your reference will help us to do so.

You write:

Both models contain an intercept, both models have the same coefficient estimates, and the same S.E. of regression, Sum squared resid, Log likelihood, Mean dependent var, S.D. dependent var, Akaike info criterion, Schwarz criterion, Hannan-Quinn criter., and Durbin-Watson stat. I would argue that both outputs should produce identical R^2 values. eq08_handi R^2 is the explained variation of lratmur/density about the mean of lratmur/density while eq08_wlsi is the variation of lratmur/density about 1/density. To me it is not clear which is a better measure.

Fair enough. But our view is that the measures we produce are correct for the context in which the estimator is defined. If one views R2 as a measure of the quality of fit of the model against an agnostic specification, which we do, then both computations are defined as one would want. In our view, the problem with defining the WLS R2 as you wish is that the base is not a candidate specification that can be estimated within the context of WLS.

You write

R^2 is supposed to be a measure (percent) of explained variation. The coefficient estimates from the WLS equation are not OLS and hence the percent explained by those coefficients must be less than the OLS explanation. Because the WLS coefficients are not OLS, the usual orthogonal decomposition of sums of squares does not work. But what I want is to know a percent explained variation from the WLS coefficients to compare to the OLS percent.

Your second sentence is not entirely true. WLS coefficients are OLS in the weighted space, which is why we believe that the weighted R2 we use makes the most sense.

As to the last sentence, to be honest, I'm still not certain that I understand what you mean by "percent explained variation from the WLS coefficients compared to the OLS percent".

I was simply taught that this is the way to think of R2 in a weighted model, without a specific name or reference.

And I was taught R^2 was the correlation^2 between the actual and predicted because most problems have those (and of course angles between the vectors and ...).

If one views R2 as a measure of the quality of fit of the model against an agnostic specification, which we do, then both computations are defined as one would want. In our view, the problem with defining the WLS R2 as you wish is that the base is not a candidate specification that can be estimated within the context of WLS.

If correlation squared between the actual and predicted is taken as the indication of quality of the model (which it is), we do not have to discuss the 'base' model. The problem with 'base' models, IMHO, is one is usually thinking about subset models and WLS is not in that class so far as GR^2 is concerned. I don't have my Stigler history of statistics handy (lent it to a grad student) we might be able to settle what should be called R^2, GR^2, coefficient of determination, etc in non-LS models.

Your second sentence is not entirely true. WLS coefficients are OLS in the weighted space, which is why we believe that the weighted R2 we use makes the most sense

What I meant was the WLS coefficients are not OLS in the unweighted space. In the unweighted space, I would like to know how much of the VAR(dependent) is explained by the WLS coefficients. The correlation^2 measure has good properties, including never being negative.

As to the last sentence, to be honest, I'm still not certain that I understand what you mean by "percent explained variation from the WLS coefficients compared to the OLS percent".

If you take the correlation^2, that is the corr^2 between Y (the actual) and Y_hat (the predicted). Use the WLS coefficients (or any others you want) to produce Y_hat. The correlation^2 between Y_hat(wls coefficients) and Y is comparable to the R^2 of the original equation. I would, for one, like to know if I use the WLS coefficients, how much worse the model is in the unweighted data space.

Continuing with this interesting conversation...

And I was taught R^2 was the correlation^2 between the actual and predicted because most problems have those (and of course angles between the vectors and ...).

I agree that there are different interpretations of R2 in the texbook setting (including projections and the like). But that begs the question of what interpretation one wishes to use in nonstandard settings. If you look at, for example, pseudo-R2 measures for limited dependent variable models you'll see various definitions involving different views of the appropriate metric, the most common of which is the percent explained versus a reference model. In a number of cases, these measures are defined without reference to actual and predicted values [edit] (notably, McFadden's R2).

(I also want to point out that in settings where a simple regression model is estimated without an intercept, the correlation^2 no longer equals the conventional definitions of R2).

The problem with 'base' models, IMHO, is one is usually thinking about subset models and WLS is not in that class so far as GR^2 is concerned.

You claim that WLS doesn't have a subset model for the GR2. I'm not sure why that is. The intercept only model is, I would argue, the natural subset model for examining the percent of the variation explained by the additional regressors. (Though I may be misunderstanding what the GR2 is...)

What I meant was the WLS coefficients are not OLS in the unweighted space. In the unweighted space, I would like to know how much of the VAR(dependent) is explained by the WLS coefficients. The correlation^2 measure has good properties, including never being negative.

We agree on the properties of WLS coefficients in the unweighted space. As to the latter part, I now see what you want, and it's an interesting approach to the problem that has considerable merit. [edit] (Though I'm not sure it can be thought of as how much can be explained by the WLS coefficients in this space since there's no actual decomposition, it has merit as a measure of fit correlation.) We'll give it some more thought and consider whether it might fit into our standard output.

For now, the easiest way to get what you want is I'm in a rush and I've got to run so my remarks have been hurried. Apologies if I've mangled the interpretation of any of your comments.

But that begs the question of what interpretation one wishes to use in nonstandard settings. If you look at, for example, pseudo-R2 measures for limited dependent variable models you'll see various definitions involving different views of the appropriate metric, the most common of which is the percent explained versus a reference model. In a number of cases, these measures are defined without reference to actual and predicted values.

They all are defined based on actual and predicted values. E.g., your use of @RSS means residuals and residuals references predicteds and actuals. Even R^2 based on likelihood ratios use predicteds and actuals. A nice psuedo R^2 display is http://www.ats.ucla.edu/stat/mult_pkg/f ... uareds.htm and rather than choosing for the user, present them all.

(I also want to point out that in settings where a simple regression model is estimated without an intercept, the correlation^2 no longer equals the conventional definitions of R2).

YEP! Arguments for denominator of SUMSQ(Y) or SUMSQ(Y-MEAN(Y)) and similarly for the numerator because without an intercept SUM(residuals) is not 0 and the MEAN(predicted) not equal MEAN(actual).
I argue that the correlation squared between actual and predicted is better than figuring out what you are explaining - well, it forces you to explain the VAR(Y).

You claim that WLS doesn't have a subset model for the GR2. I'm not sure why that is. The intercept only model is, I would argue, the natural subset model for examining the percent of the variation explained by the additional regressors. (Though I may be misunderstanding what the GR2 is...)

Sorry, what I meant was that the estimates from WLS are not a subset model of the actual (versus weighted) data space. In the weighted data space, there are subset models.

We agree on the properties of WLS coefficients in the unweighted space. As to the latter part, I now see what you want, and it's an interesting approach to the problem that has merit. We'll give it some more thought and consider whether it might fit into our standard output.

Oh my gosh I have opened a can of worms that I might have to eat. I have an Excel sheet that takes the standard output and makes it into a column. When you added the H-Q statistic, well....

Maybe, like Confidence Intervals, one could add a 'view' to the equation with lots of other R^2 such as the list from IDRE.

Thanks for this discussion - I have learned.

Bob

Think we're close to a end on this...

They all are defined based on actual and predicted values. E.g., your use of @RSS means residuals and residuals references predicteds and actuals. Even R^2 based on likelihood ratios use predicteds and actuals.

It looks like I edited my comments as you were writing. As I noted in the edited form, some pseudo-R2s don't use fitted values at all (as in the McFadden-R2, which is likelihood based).

I argue that the correlation squared between actual and predicted is better than figuring out what you are explaining - well, it forces you to explain the VAR(Y).

I don't disagree with most of this statement, the one thing I take issue with is the idea that you are explaining the var(y) since that implies decomposition which it really isn't (I come at this from a testing point of view). I have no problems with the notion that the correlation between the actuals and fits squared is of independent interest as a measure of association (and happens to equal the R2 in the standard setting).

Oh my gosh I have opened a can of worms that I might have to eat. I have an Excel sheet that takes the standard output and makes it into a column. When you added the H-Q statistic, well....

Maybe, like Confidence Intervals, one could add a 'view' to the equation with lots of other R^2 such as the list from IDRE.

It's also possible that we could throw together an add-in which would do this.

Thanks for this discussion - I have learned.

Same here. I haven't thought this much about R2 for many-a-day. We had a lengthy discussion about the issue this morning and it's possible that we'll force you to update your spreadsheets (be careful what you ask for :))The add-in is also a possibility.

some pseudo-R2s don't use fitted values at all (as in the McFadden-R2, which is likelihood based).

Likelihoods are based on the model, X Beta, which, in many if not all cases, can be or is interpreted as the predicted. E.g., in Eviews help logl the likelihood is a function of res1 which is a function of the actuals and predicteds.

Another psuedo R^2 not mentioned on the IDRE page is Tjur, T. (2009) “Coefficients of determination in logistic regression models—A new proposal: The coefficient of discrimination.” The American Statistician 63: 366-372.

I would vote strongly for the add-in because the user can add or modify if she wishes.

Bob

The likelihood to which you refer is the standard normal likelihood which is of course a function of actuals and predicted. My original point was that it need not be a function of predicteds (as in all likelihoods are not functions of predicteds).

We'll keep the vote for an add-in in mind. Of course, we'd be encourage any user to put one together on their own.

OK, understood about non-predictive models. I guess I would say R^2 by any definition is mysterious in non-predictive models (but then I don't do those so I really don't know).

OMG Pseudo R^2 is already an add-in
Package Name: PSEUDOR2
Author: Quantitative Micro Software
Date: 2010/04/28

Add-in Type: Equation
Default Proc Name: pseudor2
Default Menu Text: Pseudo R-squareds

Description: This add-in calculates the Mcfadden, Efron, Cox & Snell, and Nagelkerke pseudo R-squareds, as defined by:
http://www.ats.ucla.edu/stat/mult_pkg/f ... uareds.htm

Interestingly it only produces McFadden and Adjusted McFadden on the murder data set while it produces all four for a time series data set. And my guess is that the program was designed for binary dependent variable estimation. I have not looked at the code.

Yes it was targeted at limited dependent variables. After following our discussion for a bit, Gareth immediately suggested extending the add-in to cover additional cases. It shouldn't be particularly difficult.