Managing resources when running a lot of regression

[andybossie]

I have some questions about resource management when EViews workfiles become big, using EViews 14. I am using the <autoarma> function to automate the time series part of an overall automated model selection for combinations of exogenous variables.

As you can guess, this spits out a lot of regressions. For the set of variables I have now I need to run autoarma about 430k times. Autoarma produces a very useful spool and the original iteration of my program saved all the spools produced along with other information and saved a backup every 10k regressions. These files became quite large quickly and saving the files seemed to add significantly more time to the unsaved baseline runtime.

To save disk space--and I had hoped speed things up--I now save only a table with enough identifying information to rerun autoarma for the top N regressions selected to produce the final output for humans. What I am noticing is that when the number of regressions is small (10k-20k) the process takes about 20 minutes. To go from 390k->400k, however, it took about two and a half hours.

The reason I am here is because I am trying to get a handle on what is happening. Eviews is pretty consistently taking up up about 26% or27% of the CPU, I assume that is because it is only running on one core. I don’t believe this matters here?

Right now at >400k regressions Eviews is taking up about 400mb of RAM and the file size at 400k regressions is 201mb. None of this strikes me as exceptional. Anyway, I’m at a loss and looking for alternative ways of thinking about how to speed my program up. I expected this program to take a while, but I did not expect the exponential increase in runtimes. I assume, from what I said above, that it has to do with saving the file?
I have done all the things suggested about slow runtimes elsewhere in the forum. I am working off a server drive, but when I tested it on a local drive runtimes were not that different. I also turned off automatic snapshots.

A related auxiliary question: Is it possible to delete workfiles from within EViews? If so, does deleting files also slow the process down in the same way saving files does?

[EViews Gareth]

Are you sure the time taken is exponentially slower? You say 20k regressions takes 20 minutes. That's 1k regressions per minute. You'd expect 400k regressions to take 400 minutes, or 6.5 hours? Yet you take it takes 2.5 hours. That seems like it gets faster, not slower?

You can use wfclose to close a workfile.

You also might want to store things in a database instead of saving to workfiles. That might speed things up a little.

[andybossie]

Sorry, I was not clear: The times I gave were each for running 10k regression as I have time stamps for saves every 10k regressions.

To make it more concrete: 10k total regressions saved at 8:05am then 20k total regressions saved at 8:22am (17minutes between saves) compared to 390k total regressions saved at 10:35am then 400k total regressions saved at 1:00pm (2:25 between saves).

[andybossie]

Also, I may not have been clear about deleting workfiles. When savings backups every 10k regressions I would like to delete the previous saved workfile off the hard drive after I save a new one.

I could avoid this problem by just by overwriting the old file, obviously, but I like to have the number of regressions run in the name of the workfile as an easy way to check on progress while the program is running. Anyway, this is a minor issue now that I am not saving the spools from every regression because the workfile sizes are more manageable. I am just curious if I have missed a way to delete workfiles off the hard drive from within EViews.

[EViews Gareth]

You could use the spawn or shell commands to issue a windows/dos command to delete files.


I see what you mean about the time now. Yes, as the workfile becomes populated with more and more objects, the time taken to use that workfile will increase exponentially. Using a database instead of a workfile would alleviate this.