ROC curve

[eddylle]

Hello,

I'd like to know how a ROC curve can be performed under Eviews7 ?

Thank you in advance

[EViews Gareth]

Never heard of them. Perhaps you could describe what they are.

[eddylle]

For a logit model with Defaulted firm (1) or not (0) as dependent variable :

ROC curve illustrates the performance of a binary classifier system as its discrimination treshold is varied. It plots the fraction of true positive out of positives against the fraction of false positives out of the negatives. The curve thus shows the ability for the model to classify failing firms in the failing group in comparison with firms forecasted to fail but who have not. The null model has a diagonal ROC Curve, corresponding to an AUC of 0.5.

[eddylle]

Anyone?

[eddylle]

Up

[EViews Glenn]

Sorry missed this one.

It's not built-in to EViews right now. You could create one with a bit of work, either by looping over the existing fit table routine with different input values and grabbing the results from there, or, as I'd probably recommend, just doing the computations directly and plotting the result. Have you written EViews programs before?

[eddylle]

No never.. I guess it's gonna be a bit tough for me to compute it then? Or can you help me?

Thanks

[EViews Glenn]

I did a quick bit of coding for you.The following code produces a ROC curve from a binary equation. It's not really production quality code, but should hold you over for now. Open the workfile containing the equation of interest. Open a program window by selecting File/New/Program. Paste the code below into the program window

Code: Select all

%dep = "grade" %eq = "eq_probit" ' find valid observations {%eq}.makeresid _xxe ' get number of 0 and 1 smpl if _xxe<>na and {%dep} = 0 !n0 = @obs({%dep}) smpl if _xxe<>na and {%dep} = 1 !n1 = @obs({%dep}) ' get fittted values smpl if _xxe<>na {%eq}.forecast _xxyf group a {%dep} _xxyf stom(a, b) stom(_xxyf, _xxprob) ' sort in descending order so we have threshold of infinity which is "off" vector _xxpranks = @ranks(_xxprob, "d", "i") matrix _xxsdata = @capplyranks(b, _xxpranks) ' get unique fitted values vector _xxuniq = @uniquevals(pranks) !nunique = @rows(_xxuniq) delete _xxuniq ' create output ROC matrix matrix(!nunique+2, 4) _xxroc ' we start at threshold of infinity which is "off" !ntype1 = 0 ' number of correct 1 predictions !ntype0 = !n0 ' number of correct 0 predictions !j = 1 !i = 1 !n = @rows(_xxsdata) roc(1, 1) = 1 - (!ntype0 / !n0) ' 1 - specificity roc(1, 2) = !ntype1 / !n1 ' sensitivity roc(1, 3) = 0 roc(1, 4) = 0 while !j <= !nunique !target = _xxsdata(!i, 2) !val = !target while !val = !target !response = sdata(!i, 1) if (!response = 0) then !ntype0 = !ntype0 - 1 endif if (!response = 1) then !ntype1 = !ntype1 + 1 endif !i = !i+1 if (!i > !n) then !val = 0 else !val = sdata(!i, 2) endif wend _xxroc(!j+1, 1) = 1 - (!ntype0 / !n0) _xxroc(!j+1, 2) = !ntype1 / !n1 _xxroc(!j+1, 3) = !j / !nunique _xxroc(!j+1, 4) = !j / !nunique !j = !j + 1 wend _xxroc(!j+1, 1) = 1 - (!ntype0 / !n0) _xxroc(!j+1, 2) = !ntype1 / !n1 delete(noerr) roc_graph freeze(roc_graph) _xxroc.xypair roc_graph.options linepat -legend size(4,4) roc_graph.setelem(1) linewidth(1.15) roc_graph.setelem(2) linepattern(8) %title = "ROC graph for" %title = %title + " " + @upper(%eq) roc_graph.addtext(t, font("Ariel", 12)) %title roc_graph.addtext(l) "True Positive Rate" roc_graph.addtext(b) "False Positive Rate" show roc_graph delete _xx*

I assume in the program that you don't have anything named ROC_GRAPH in the workfile and that you don't have variables that begin with the characters "__XX".

Next edit the program to replace GRADE and EQ_PROBIT with the variable and equation of interest. Click on run.

The results will be in the graph object ROC_GRAPH. If you want the numbers, we'll have to add a line to the program to rename the results matrix before I cleanup the temps.

I've done some quick checks of the results against Fawcett's (2006) paper, but obviously haven't done extensive testing, especially for cases where there are repeats in the classifier series.

Let me know how it works for you.

[eddylle]

Thank you for your quick answer.

I did what you told me to do, and I get an error message after having clicked on "run". When I want to click on OK with the following window (that asks me to run the program with different choices of runtime erros) I get this : @UNIQUEVALS is an illegal or reserved name in "VECTOR_XXUNIQ = @UNIQUEVQLS(PRANKS)

Moreover, that would be awesome if I could get the figures related to the graph.

Thanks again

[EViews Glenn]

Oh yes. Its an EViews 8 function. I'm away from a computer right now but let me see what I can do later.

[eddylle]

Ok, look forward to hearing from you.
Thanks

[EViews Glenn]

Rewritten for EViews 7 and fixed some other things that I introduced when I moved to temp matrices and series.

Code: Select all

%dep = "grade" %eq = "eq_probit" ' find valid observations {%eq}.makeresid _xxe ' get number of 0 and 1 smpl if _xxe<>na and {%dep} = 0 !n0 = @obs({%dep}) smpl if _xxe<>na and {%dep} = 1 !n1 = @obs({%dep}) ' get fittted values smpl if _xxe<>na {%eq}.forecast _xxyf group _xxa {%dep} _xxyf stom(_xxa, _xxb) stom(_xxyf, _xxprob) ' sort in descending order so we have threshold of infinity which is "off" vector _xxpranks = @ranks(_xxprob, "d", "i") matrix _xxsdata = @capplyranks(_xxb, _xxpranks) ' get unique fitted values !nunique = 1 for !i=2 to @rows(_xxsdata) if _xxsdata(!i, 2) < _xxsdata(!i-1, 2) then !nunique = !nunique+1 endif next ' create output ROC matrix matrix(!nunique+2, 4) _xxroc ' we start at threshold of infinity which is "off" !ntype1 = 0 ' number of correct 1 predictions !ntype0 = !n0 ' number of correct 0 predictions !j = 1 !i = 1 !n = @rows(_xxsdata) _xxroc(1, 1) = 1 - (!ntype0 / !n0) ' 1 - specificity _xxroc(1, 2) = !ntype1 / !n1 ' sensitivity _xxroc(1, 3) = 0 _xxroc(1, 4) = 0 while !j <= !nunique !target = _xxsdata(!i, 2) !val = !target while !val = !target !response = _xxsdata(!i, 1) if (!response = 0) then !ntype0 = !ntype0 - 1 endif if (!response = 1) then !ntype1 = !ntype1 + 1 endif !i = !i+1 if (!i > !n) then !val = 0 else !val = _xxsdata(!i, 2) endif wend _xxroc(!j+1, 1) = 1 - (!ntype0 / !n0) _xxroc(!j+1, 2) = !ntype1 / !n1 _xxroc(!j+1, 3) = !j / !nunique _xxroc(!j+1, 4) = !j / !nunique !j = !j + 1 wend _xxroc(!j+1, 1) = 1 - (!ntype0 / !n0) _xxroc(!j+1, 2) = !ntype1 / !n1 delete(noerr) roc_graph freeze(roc_graph) _xxroc.xypair roc_graph.options linepat -legend size(4,4) roc_graph.setelem(1) linewidth(1.15) roc_graph.setelem(2) linepattern(8) %title = "ROC graph for" %title = %title + " " + @upper(%eq) roc_graph.addtext(t, font("Ariel", 12)) %title roc_graph.addtext(l) "True Positive Rate" roc_graph.addtext(b) "False Positive Rate" show roc_graph delete _xx*

[eddylle]

Wonderful! Thank you very much!
One last thing, is it possible to compute the aera under the curve ? it is supposed to be a number bounded between 0 and 1, to know how accurate the model is for predicting

[EViews Glenn]

Rather than me writing a program from scracth for this, you could grab the nonparametric version of the AUC from the Mann-Whitney statistic and manipulate the statistic to get the desired value.
1. Produce a forecast from the binary equation for the estimation sample and save it in a series, say Y_F.
2. Open Y_F and select View/Equality Tests by Classification...
3. Put the original variable in the Variable to Classify edit field, and select Median test. Click on OK to compute the Mann-Whitney statistic.
4. Find the value of the Wilcoxon/Mann-Whitney statistic in the table and the number of 0 and 1 values. Make a note of these values.

The following assumes that the ordinary and tie-adjusted values are the same.

Code: Select all

!n1 = 21 !n2 = 11 !unorm = 3.531162 !mu = !n1*!n2/2 !sigma = sqr(!n1*!n2*(!n1+!n2+1)/12) !u = !unorm*!sigma + !mu delete(noerr) rocarea scalar rocarea = !u/(!n1*!n2)

The program snippet above computes the AUC. Replace !n1, !n2, and !unorm, with the numbers of 0 and 1 values, and the value of the Mann-Whitney statistic. The scalar ROCAREA should give you the value.

As with the earlier code, this hasn't been tested much (almost at all). I did a quick calculation for the BINARY workfile that EViews provides and it seems to work, but use at your own risk.

[eddylle]

The answer seems to be consistent with what I expected.

Thank you very much for all your explanations.