EViews.com

Hello,

my program opens an existing workfile (which contains one page) named "germany" and then does some data manipulation, etc. In the end the workfile contains a second page and I want to save the workfile under a different name (wfsave germany5x5.wf1).

Then the following error message appears: Error writing file G:\Data\germany5x5.wf1 in "wfsave germany5x5".

Also when I try to save the workfile manually via the Save As Option, the error message appears ("Error writing file G:\Data\germany5x5.wf1.")

Interestingly the name of the open workfile changes from germany to germany5x5.


I have another issue when I want to delete certain types of objects. For example, I have created a certain amount of scalars and I want to delete them after they were used for some calculations. It takes more than 10 minutes to delete them, which is ridiculous as the whole computation is done under a minute.


Hopefully someone can help me with this issue.

Not sure why you're getting an error with calling WFSAVE with a new name. Try doing it on your C drive instead to see if that resolves the issue. Perhaps your G drive is full or you're having strange network permissions issues?

About your delete objects issue - how are you deleting the objects? Are you performing a WFSAVE operation between each delete?

No, as those objects have a specific name (each start with sca_) I just use " delete sca_* ". I guess the problem is the amount of objects (over 100k).

I think thats one of my biggest problems in general, the vast amount of objects. My program needs 3 hours to compute everything. As im a newbie: are there any tips in order to enhance the performance? I've got a normal 8 GB RAM, so maybe it helps to outsource the whole process in our university network ?! (I'm a finance guy and dont have any clue about real informatics, but I know for a fact that our physics department uses the network for their computations, but dont know which language they use).

Using the c drive helped solving the saving issue.

Is there a reason you are creating and deleting that many scalar objects in the workfile, rather than just using program variables (! variables)?

I thought about that, but yes there's a reason. Those objects are not just scalars, could also be series oder equation objects.

Fair enough. It is going to be slow going with that many. Is there a way to structure your code so that you are not holding on to so many at the same time?

I think there is some room for optimization, thank you. So far I deleted all obsolete objects after a specific step in my program, but part of this process could be done in before.

Further ideas to enhance computational power? Powering up my computer or using other RAM (cloud, network)? I guess there exists nothing like a paralleization, right?

No, they won't make much difference.

Another thing worth trying is not using wildcards. You said that you were deleting with a wildcard expression. I would be willing to bet that it is the querying of the wildcard expression that is taking a fair bit of time.


Sent from my iPhone using Tapatalk

Question for Gareth: Will saving workfile after object deletion and then loading back saved version speed up subsequent calculations by speeding up internal table searches?

No.

Okay. Does it make a difference how many objects my wildcard contains?

For example: instead of using " delete sca_* " I should use for loops (in my case for each month exists a certain amount of objects I want to delete):

for !year = 1991 to 2016
for !month = 1 to 12

delete sca_{!year}{!month)_*

next
next

I doubt it, but worth a shot.

Interesting

Thank you, this is pretty interesting and could make a huge difference regarding my issue. The unsolved question is: why is that? It makes no sense at first glance.

I came across some curious performance behavior in one of my programs, so I looked into the phenomenon and it turned out to have the same root cause. Using a modified version of Gareth's script, I observed the following runtimes (in seconds) for deleting various numbers of scalars:
I'm guessing that the results for (2) and (4) are the most surprising. I suspect that most people would have the intuition that deleting objects in bigger "chunks" would always be faster. Obviously, that's not the case, so let me illuminate some of EViews' inner workings...

Deletion techniques (1-3) must translate the wildcard expressions into a list of matching object names. The observed differences in runtime primarily reflect a tradeoff between two aspects of that process, 1) going through EViews' master list of objects to find objects with matching names, and 2) building the secondary list containing those matching names. For example, in the 100k scenario, (1) will go through the master list once and build a large list of 100k names. Contrast this with (2), which will go through the master list 100 times and build 100 lists of 1k names each (noting that the master list shrinks after each intermediate deletion of 1k objects). (3) is even more extreme, going through the (ever shrinking) master list 10,000 times, yet building small lists of only 10 names.

Going through the master list of objects so many times, even as it's slowly shrinking, is slow. Going though fewer times, with the list shrinking more quickly, as (2) does, is better. Why then is (1) not the best? It turns out that EViews' algorithm for building the secondary list of matching names slows down as that list becomes large. In computer science, we'd categorize the temporal behavior of the algorithm as being O(n^2), which means that the work the algorithm must perform, and thus the time it takes to execute, grows quadratically with the size of the list. In other words, if the size of the list doubles, the time it takes to build quadruples. In our scenario, (1) builds a list 100 times larger than (2), which takes 100^2 = 10,000 times as long. Even considering that (2) is going to build 100 lists, (1) still spends 100 times the effort (and time) on list-building. This significant increase in work causes (1)'s runtime exceed (2)'s. The is a similar effort differential between (2) and (3), but it's completely outweighed by the time (3) spends repeatedly traversing the master list. In a way, (2) represents a performance "sweet spot" among these three techniques. (2) doesn't build lists large enough cause slowdowns, as (1) does, nor does it traverse the master list enough times to cause slowdowns, as (3) does.

(4) doesn't go through the wildcard resolution process, so it skips all the work I've outlined above. That (4) is still a little bit slower than (2) mostly stems from the fact that (4) goes through 100k iterations while (2) only goes through 100. There is overhead in executing an EViews program and all those extra iterations cost time.