ROC curve
Moderators: EViews Gareth, EViews Moderator

 Fe ddaethom, fe welon, fe amcangyfrifon
 Posts: 12420
 Joined: Tue Sep 16, 2008 5:38 pm
Re: ROC curve
Never heard of them. Perhaps you could describe what they are.
Follow us on Twitter @IHSEViews
Re: ROC curve
For a logit model with Defaulted firm (1) or not (0) as dependent variable :
ROC curve illustrates the performance of a binary classifier system as its discrimination treshold is varied. It plots the fraction of true positive out of positives against the fraction of false positives out of the negatives. The curve thus shows the ability for the model to classify failing firms in the failing group in comparison with firms forecasted to fail but who have not. The null model has a diagonal ROC Curve, corresponding to an AUC of 0.5.
ROC curve illustrates the performance of a binary classifier system as its discrimination treshold is varied. It plots the fraction of true positive out of positives against the fraction of false positives out of the negatives. The curve thus shows the ability for the model to classify failing firms in the failing group in comparison with firms forecasted to fail but who have not. The null model has a diagonal ROC Curve, corresponding to an AUC of 0.5.

 EViews Developer
 Posts: 2627
 Joined: Wed Oct 15, 2008 9:17 am
Re: ROC curve
Sorry missed this one.
It's not builtin to EViews right now. You could create one with a bit of work, either by looping over the existing fit table routine with different input values and grabbing the results from there, or, as I'd probably recommend, just doing the computations directly and plotting the result. Have you written EViews programs before?
It's not builtin to EViews right now. You could create one with a bit of work, either by looping over the existing fit table routine with different input values and grabbing the results from there, or, as I'd probably recommend, just doing the computations directly and plotting the result. Have you written EViews programs before?
Re: ROC curve
No never.. I guess it's gonna be a bit tough for me to compute it then? Or can you help me?
Thanks
Thanks

 EViews Developer
 Posts: 2627
 Joined: Wed Oct 15, 2008 9:17 am
Re: ROC curve
I did a quick bit of coding for you.The following code produces a ROC curve from a binary equation. It's not really production quality code, but should hold you over for now. Open the workfile containing the equation of interest. Open a program window by selecting File/New/Program. Paste the code below into the program window
I assume in the program that you don't have anything named ROC_GRAPH in the workfile and that you don't have variables that begin with the characters "__XX".
Next edit the program to replace GRADE and EQ_PROBIT with the variable and equation of interest. Click on run.
The results will be in the graph object ROC_GRAPH. If you want the numbers, we'll have to add a line to the program to rename the results matrix before I cleanup the temps.
I've done some quick checks of the results against Fawcett's (2006) paper, but obviously haven't done extensive testing, especially for cases where there are repeats in the classifier series.
Let me know how it works for you.
Code: Select all
%dep = "grade"
%eq = "eq_probit"
' find valid observations
{%eq}.makeresid _xxe
' get number of 0 and 1
smpl if _xxe<>na and {%dep} = 0
!n0 = @obs({%dep})
smpl if _xxe<>na and {%dep} = 1
!n1 = @obs({%dep})
' get fittted values
smpl if _xxe<>na
{%eq}.forecast _xxyf
group a {%dep} _xxyf
stom(a, b)
stom(_xxyf, _xxprob)
' sort in descending order so we have threshold of infinity which is "off"
vector _xxpranks = @ranks(_xxprob, "d", "i")
matrix _xxsdata = @capplyranks(b, _xxpranks)
' get unique fitted values
vector _xxuniq = @uniquevals(pranks)
!nunique = @rows(_xxuniq)
delete _xxuniq
' create output ROC matrix
matrix(!nunique+2, 4) _xxroc
' we start at threshold of infinity which is "off"
!ntype1 = 0 ' number of correct 1 predictions
!ntype0 = !n0 ' number of correct 0 predictions
!j = 1
!i = 1
!n = @rows(_xxsdata)
roc(1, 1) = 1  (!ntype0 / !n0) ' 1  specificity
roc(1, 2) = !ntype1 / !n1 ' sensitivity
roc(1, 3) = 0
roc(1, 4) = 0
while !j <= !nunique
!target = _xxsdata(!i, 2)
!val = !target
while !val = !target
!response = sdata(!i, 1)
if (!response = 0) then
!ntype0 = !ntype0  1
endif
if (!response = 1) then
!ntype1 = !ntype1 + 1
endif
!i = !i+1
if (!i > !n) then
!val = 0
else
!val = sdata(!i, 2)
endif
wend
_xxroc(!j+1, 1) = 1  (!ntype0 / !n0)
_xxroc(!j+1, 2) = !ntype1 / !n1
_xxroc(!j+1, 3) = !j / !nunique
_xxroc(!j+1, 4) = !j / !nunique
!j = !j + 1
wend
_xxroc(!j+1, 1) = 1  (!ntype0 / !n0)
_xxroc(!j+1, 2) = !ntype1 / !n1
delete(noerr) roc_graph
freeze(roc_graph) _xxroc.xypair
roc_graph.options linepat legend size(4,4)
roc_graph.setelem(1) linewidth(1.15)
roc_graph.setelem(2) linepattern(8)
%title = "ROC graph for"
%title = %title + " " + @upper(%eq)
roc_graph.addtext(t, font("Ariel", 12)) %title
roc_graph.addtext(l) "True Positive Rate"
roc_graph.addtext(b) "False Positive Rate"
show roc_graph
delete _xx*
I assume in the program that you don't have anything named ROC_GRAPH in the workfile and that you don't have variables that begin with the characters "__XX".
Next edit the program to replace GRADE and EQ_PROBIT with the variable and equation of interest. Click on run.
The results will be in the graph object ROC_GRAPH. If you want the numbers, we'll have to add a line to the program to rename the results matrix before I cleanup the temps.
I've done some quick checks of the results against Fawcett's (2006) paper, but obviously haven't done extensive testing, especially for cases where there are repeats in the classifier series.
Let me know how it works for you.
Re: ROC curve
Thank you for your quick answer.
I did what you told me to do, and I get an error message after having clicked on "run". When I want to click on OK with the following window (that asks me to run the program with different choices of runtime erros) I get this : @UNIQUEVALS is an illegal or reserved name in "VECTOR_XXUNIQ = @UNIQUEVQLS(PRANKS)
Moreover, that would be awesome if I could get the figures related to the graph.
Thanks again
I did what you told me to do, and I get an error message after having clicked on "run". When I want to click on OK with the following window (that asks me to run the program with different choices of runtime erros) I get this : @UNIQUEVALS is an illegal or reserved name in "VECTOR_XXUNIQ = @UNIQUEVQLS(PRANKS)
Moreover, that would be awesome if I could get the figures related to the graph.
Thanks again

 EViews Developer
 Posts: 2627
 Joined: Wed Oct 15, 2008 9:17 am
Re: ROC curve
Oh yes. Its an EViews 8 function. I'm away from a computer right now but let me see what I can do later.
Re: ROC curve
Ok, look forward to hearing from you.
Thanks
Thanks

 EViews Developer
 Posts: 2627
 Joined: Wed Oct 15, 2008 9:17 am
Re: ROC curve
Rewritten for EViews 7 and fixed some other things that I introduced when I moved to temp matrices and series.
Code: Select all
%dep = "grade"
%eq = "eq_probit"
' find valid observations
{%eq}.makeresid _xxe
' get number of 0 and 1
smpl if _xxe<>na and {%dep} = 0
!n0 = @obs({%dep})
smpl if _xxe<>na and {%dep} = 1
!n1 = @obs({%dep})
' get fittted values
smpl if _xxe<>na
{%eq}.forecast _xxyf
group _xxa {%dep} _xxyf
stom(_xxa, _xxb)
stom(_xxyf, _xxprob)
' sort in descending order so we have threshold of infinity which is "off"
vector _xxpranks = @ranks(_xxprob, "d", "i")
matrix _xxsdata = @capplyranks(_xxb, _xxpranks)
' get unique fitted values
!nunique = 1
for !i=2 to @rows(_xxsdata)
if _xxsdata(!i, 2) < _xxsdata(!i1, 2) then
!nunique = !nunique+1
endif
next
' create output ROC matrix
matrix(!nunique+2, 4) _xxroc
' we start at threshold of infinity which is "off"
!ntype1 = 0 ' number of correct 1 predictions
!ntype0 = !n0 ' number of correct 0 predictions
!j = 1
!i = 1
!n = @rows(_xxsdata)
_xxroc(1, 1) = 1  (!ntype0 / !n0) ' 1  specificity
_xxroc(1, 2) = !ntype1 / !n1 ' sensitivity
_xxroc(1, 3) = 0
_xxroc(1, 4) = 0
while !j <= !nunique
!target = _xxsdata(!i, 2)
!val = !target
while !val = !target
!response = _xxsdata(!i, 1)
if (!response = 0) then
!ntype0 = !ntype0  1
endif
if (!response = 1) then
!ntype1 = !ntype1 + 1
endif
!i = !i+1
if (!i > !n) then
!val = 0
else
!val = _xxsdata(!i, 2)
endif
wend
_xxroc(!j+1, 1) = 1  (!ntype0 / !n0)
_xxroc(!j+1, 2) = !ntype1 / !n1
_xxroc(!j+1, 3) = !j / !nunique
_xxroc(!j+1, 4) = !j / !nunique
!j = !j + 1
wend
_xxroc(!j+1, 1) = 1  (!ntype0 / !n0)
_xxroc(!j+1, 2) = !ntype1 / !n1
delete(noerr) roc_graph
freeze(roc_graph) _xxroc.xypair
roc_graph.options linepat legend size(4,4)
roc_graph.setelem(1) linewidth(1.15)
roc_graph.setelem(2) linepattern(8)
%title = "ROC graph for"
%title = %title + " " + @upper(%eq)
roc_graph.addtext(t, font("Ariel", 12)) %title
roc_graph.addtext(l) "True Positive Rate"
roc_graph.addtext(b) "False Positive Rate"
show roc_graph
delete _xx*
Re: ROC curve
Wonderful! Thank you very much!
One last thing, is it possible to compute the aera under the curve ? it is supposed to be a number bounded between 0 and 1, to know how accurate the model is for predicting
One last thing, is it possible to compute the aera under the curve ? it is supposed to be a number bounded between 0 and 1, to know how accurate the model is for predicting

 EViews Developer
 Posts: 2627
 Joined: Wed Oct 15, 2008 9:17 am
Re: ROC curve
Rather than me writing a program from scracth for this, you could grab the nonparametric version of the AUC from the MannWhitney statistic and manipulate the statistic to get the desired value.
1. Produce a forecast from the binary equation for the estimation sample and save it in a series, say Y_F.
2. Open Y_F and select View/Equality Tests by Classification...
3. Put the original variable in the Variable to Classify edit field, and select Median test. Click on OK to compute the MannWhitney statistic.
4. Find the value of the Wilcoxon/MannWhitney statistic in the table and the number of 0 and 1 values. Make a note of these values.
The following assumes that the ordinary and tieadjusted values are the same.
The program snippet above computes the AUC. Replace !n1, !n2, and !unorm, with the numbers of 0 and 1 values, and the value of the MannWhitney statistic. The scalar ROCAREA should give you the value.
As with the earlier code, this hasn't been tested much (almost at all). I did a quick calculation for the BINARY workfile that EViews provides and it seems to work, but use at your own risk.
1. Produce a forecast from the binary equation for the estimation sample and save it in a series, say Y_F.
2. Open Y_F and select View/Equality Tests by Classification...
3. Put the original variable in the Variable to Classify edit field, and select Median test. Click on OK to compute the MannWhitney statistic.
4. Find the value of the Wilcoxon/MannWhitney statistic in the table and the number of 0 and 1 values. Make a note of these values.
The following assumes that the ordinary and tieadjusted values are the same.
Code: Select all
!n1 = 21
!n2 = 11
!unorm = 3.531162
!mu = !n1*!n2/2
!sigma = sqr(!n1*!n2*(!n1+!n2+1)/12)
!u = !unorm*!sigma + !mu
delete(noerr) rocarea
scalar rocarea = !u/(!n1*!n2)
The program snippet above computes the AUC. Replace !n1, !n2, and !unorm, with the numbers of 0 and 1 values, and the value of the MannWhitney statistic. The scalar ROCAREA should give you the value.
As with the earlier code, this hasn't been tested much (almost at all). I did a quick calculation for the BINARY workfile that EViews provides and it seems to work, but use at your own risk.
Re: ROC curve
The answer seems to be consistent with what I expected.
Thank you very much for all your explanations.
Thank you very much for all your explanations.
Who is online
Users browsing this forum: No registered users and 7 guests