Getting the index of the first observation in sample

[elnoch]

Hello,

I am trying to run a Hill estimator on a rolling window of observations, however I am having trouble to get the sub-sample for each window. So far what I have is this:

Code: Select all

'Move sample !step at a time for !i=1 to !length-!window+1-!step step !step !j=!j+1 for !upper=1 to 2 if !upper=1 then smpl @first+!i-1 @first+!i+!window-2 'Setting sample to each window sort(d) y 'Sorting the data because I just want to use the biggest 10% of the observations in the window 'I know the following lines do not work/make sense, but hopefully you can understand what I am trying to do !start= @first+!i-1 !end = @first+!i+!window-2 !prop=0.1 !truncnumber = !start+@floor((!prop)*(!end-start) smpl !start !truncnumber 'This is where I want to set the sample to the original first observation of the window and biggest 10% of that window 'Do stuff Here

I would appreciate any suggestions.

Thanks.

[EViews Matt]

Hello,

Your logic is in terms of observation numbers, but I assume you have dated data, yes? If you think in terms of offsets from @first, then you can keep the spirit of your logic while only using @first in the context of smpl. For example,

Code: Select all

!start = !i-1 !end = !i+!window-2 !prop = 0.1 !truncnumber = !start+@floor(!prop*(!end-!start)) smpl @first+!start @first+!truncnumber

By recognizing that !end-!start is constant within the loop, you can make further refinements and simplifications. And FYI, I believe your loop has the potential to terminate one iteration too soon for some combinations of !length, !window, and !step.

[elnoch]

Hello,

Thank you very much for your answer, I understand the logic of it, however I have a question: you are right, the original data are dated, but the thing is that I need to select the 10% highest/lowest values of each window, which is why I have to sort it...this will make the dated structure disappear, right? I mean, if I understand your code, with the

Code: Select all

smpl @first+!start @first+!truncnumber

I would get the first 10% observations of each window, no matter the value of the variable.

What I had in mind is to create a trend variable at the beginning re-sort the data according to this variable at the end of each iteration to "move on through time", is there a way to do this without losing the dated structure?

Thanks again.

[EViews Gareth]

I can't figure out what you're trying to do.

Can you explain in words?

[elnoch]

I can't figure out what you're trying to do.

Can you explain in words?

Hello,

Yeah, sure. So I have daily dated data and I need to calculate and estimator on a rolling window of observations. So for each window (for example, of 1000 observations), I need a fixed proportion (10%) of the highest/lowest values (let's say lowest) of a variable Y, so for instance, I think that the code proposed by Math would give me the first 10% observations (on a chronological basis) for each window, but what I need is the first 10% observations based on the value of a variable. So in words, the approach I took is the following:
- generate a trend variable (OBS_ID) before starting the loop that will help me re-order the series chronologically at beginning of each iteration
- Start the loop for the rolling window
- Set the sample for the first window
- Sort the sample according to the value of Y (this is where I lose the time series structure)
- Use only the lowest 10% values of Y to calculate the estimator
- Perform the calculation I need
- Sort the sample using OBS_ID to recover the "time structure"
- End the iteration here and move on to the next window of observations.

Thanks.

[EViews Gareth]

Still isn't clear.

Each window is the lowest 10% values of all of Y.

Surely each window will contain the same values? (unless inside the window you are changing the values of Y?)

[elnoch]

Each window is the lowest 10% values of all of Y.

No, for example each window contains a time period (say March), I want to use ONLY the 3 lowest values from March, perform the calculation, then move on to April, get the lowest 3 values from April, perform calculation, then May and so on... So in each window I want to use the lowest 10% values of Y for that window, not for all Y.

Surely each window will contain the same values? (unless inside the window you are changing the values of Y?)

No, each window will not contain the same values because each window would be a different time period.

[EViews Gareth]

After doing the first set sample, and then the sort, you could use the @otods function to get the obsid of the first observation in the sample and the third (or whatever proportion you want), then set the sample again to be between those obids

[elnoch]

After doing the first set sample, and then the sort, you could use the @otods function to get the obsid of the first observation in the sample and the third (or whatever proportion you want), then set the sample again to be between those obids

Correct me if I am mistaken please because I think I don't follow your suggestion. From what I understand, what you propose would do this:
- Set the sample to March for example (assume that the 3 lowest values occur on the 2nd, the 5th and the 7th).
- Sort the data so the first 3 values would be the ones from the 2nd, 5th and 7th
- Use @otods(1) and @otods(3), which would return March 2nd and March 7th respectively
- Set the sample to be between March 2nd and March 7th

But I would still have observations that I don't need, wouldn't it? (the ones from the 3rd, 4th and 6th?)

[EViews Gareth]

Once you have done a sort, all workfile structure is removed, so it would tell you the observation number in current workfile space, allowing you to set the sample.

You might actually be better off just moving the data to a new page with each iteration of the loop.

[elnoch]

Once you have done a sort, all workfile structure is removed, so it would tell you the observation number in current workfile space, allowing you to set the sample.

You might actually be better off just moving the data to a new page with each iteration of the loop.

Oh ok I understand, thanks!

[EViews Gareth]

Or, once you have set the sample to the month, you could convert the series into a vector and sort the vector, and grab first 3 rows.

[elnoch]

Or, once you have set the sample to the month, you could convert the series into a vector and sort the vector, and grab first 3 rows.

Ok thanks, I like that idea! I was also thinking that another approach would be that once I set the sample to the month, I could use the @quantile function to create a scalar alpha and set the sample to "smpl MONTH if Y<alpha"? Do you know if one procedure is more efficient than the other? Because here I used months to explain what I wanted to do, but if the step of each iteration is only one day, then I would take a while to finish.

[EViews Gareth]

That idea would work pretty efficiently.

[elnoch]

That idea would work pretty efficiently.

Thanks, I really appreciate your help.