Error wfsave command & delete function

For notifying us of what you believe are bugs or errors in EViews.
Please ensure your copy of EViews is up-to-date before posting.

Moderators: EViews Gareth, EViews Moderator

weu39843
Posts: 10
Joined: Fri Oct 21, 2016 1:05 am

Error wfsave command & delete function

Postby weu39843 » Fri Jan 27, 2017 4:47 am

Hello,

my program opens an existing workfile (which contains one page) named "germany" and then does some data manipulation, etc. In the end the workfile contains a second page and I want to save the workfile under a different name (wfsave germany5x5.wf1).

Then the following error message appears: Error writing file G:\Data\germany5x5.wf1 in "wfsave germany5x5".

Also when I try to save the workfile manually via the Save As Option, the error message appears ("Error writing file G:\Data\germany5x5.wf1.")

Interestingly the name of the open workfile changes from germany to germany5x5.


I have another issue when I want to delete certain types of objects. For example, I have created a certain amount of scalars and I want to delete them after they were used for some calculations. It takes more than 10 minutes to delete them, which is ridiculous as the whole computation is done under a minute.


Hopefully someone can help me with this issue.

EViews Steve
EViews Developer
Posts: 788
Joined: Tue Sep 16, 2008 3:00 pm
Location: Irvine, CA

Re: Error wfsave command & delete function

Postby EViews Steve » Mon Jan 30, 2017 10:24 am

Not sure why you're getting an error with calling WFSAVE with a new name. Try doing it on your C drive instead to see if that resolves the issue. Perhaps your G drive is full or you're having strange network permissions issues?

About your delete objects issue - how are you deleting the objects? Are you performing a WFSAVE operation between each delete?

weu39843
Posts: 10
Joined: Fri Oct 21, 2016 1:05 am

Re: Error wfsave command & delete function

Postby weu39843 » Tue Feb 14, 2017 5:57 am

No, as those objects have a specific name (each start with sca_) I just use " delete sca_* ". I guess the problem is the amount of objects (over 100k).

I think thats one of my biggest problems in general, the vast amount of objects. My program needs 3 hours to compute everything. As im a newbie: are there any tips in order to enhance the performance? I've got a normal 8 GB RAM, so maybe it helps to outsource the whole process in our university network ?! (I'm a finance guy and dont have any clue about real informatics, but I know for a fact that our physics department uses the network for their computations, but dont know which language they use).

Using the c drive helped solving the saving issue.

EViews Gareth
Fe ddaethom, fe welon, fe amcangyfrifon
Posts: 13307
Joined: Tue Sep 16, 2008 5:38 pm

Re: Error wfsave command & delete function

Postby EViews Gareth » Tue Feb 14, 2017 6:44 am

Is there a reason you are creating and deleting that many scalar objects in the workfile, rather than just using program variables (! variables)?
Follow us on Twitter @IHSEViews

weu39843
Posts: 10
Joined: Fri Oct 21, 2016 1:05 am

Re: Error wfsave command & delete function

Postby weu39843 » Tue Feb 14, 2017 7:22 am

I thought about that, but yes there's a reason. Those objects are not just scalars, could also be series oder equation objects.

EViews Gareth
Fe ddaethom, fe welon, fe amcangyfrifon
Posts: 13307
Joined: Tue Sep 16, 2008 5:38 pm

Re: Error wfsave command & delete function

Postby EViews Gareth » Tue Feb 14, 2017 8:02 am

Fair enough. It is going to be slow going with that many. Is there a way to structure your code so that you are not holding on to so many at the same time?
Follow us on Twitter @IHSEViews

weu39843
Posts: 10
Joined: Fri Oct 21, 2016 1:05 am

Re: Error wfsave command & delete function

Postby weu39843 » Tue Feb 14, 2017 9:36 am

I think there is some room for optimization, thank you. So far I deleted all obsolete objects after a specific step in my program, but part of this process could be done in before.

Further ideas to enhance computational power? Powering up my computer or using other RAM (cloud, network)? I guess there exists nothing like a paralleization, right?

EViews Gareth
Fe ddaethom, fe welon, fe amcangyfrifon
Posts: 13307
Joined: Tue Sep 16, 2008 5:38 pm

Re: Error wfsave command & delete function

Postby EViews Gareth » Tue Feb 14, 2017 3:02 pm

No, they won't make much difference.

Another thing worth trying is not using wildcards. You said that you were deleting with a wildcard expression. I would be willing to bet that it is the querying of the wildcard expression that is taking a fair bit of time.


Sent from my iPhone using Tapatalk
Follow us on Twitter @IHSEViews

startz
Non-normality and collinearity are NOT problems!
Posts: 3775
Joined: Wed Sep 17, 2008 2:25 pm

Re: Error wfsave command & delete function

Postby startz » Tue Feb 14, 2017 3:31 pm

Question for Gareth: Will saving workfile after object deletion and then loading back saved version speed up subsequent calculations by speeding up internal table searches?

EViews Gareth
Fe ddaethom, fe welon, fe amcangyfrifon
Posts: 13307
Joined: Tue Sep 16, 2008 5:38 pm

Re: Error wfsave command & delete function

Postby EViews Gareth » Tue Feb 14, 2017 4:28 pm

No.
Follow us on Twitter @IHSEViews

weu39843
Posts: 10
Joined: Fri Oct 21, 2016 1:05 am

Re: Error wfsave command & delete function

Postby weu39843 » Wed Feb 15, 2017 4:51 am

Okay. Does it make a difference how many objects my wildcard contains?

For example: instead of using " delete sca_* " I should use for loops (in my case for each month exists a certain amount of objects I want to delete):

for !year = 1991 to 2016
for !month = 1 to 12

delete sca_{!year}{!month)_*

next
next

EViews Gareth
Fe ddaethom, fe welon, fe amcangyfrifon
Posts: 13307
Joined: Tue Sep 16, 2008 5:38 pm

Re: Error wfsave command & delete function

Postby EViews Gareth » Wed Feb 15, 2017 6:20 am

I doubt it, but worth a shot.
Follow us on Twitter @IHSEViews

EViews Gareth
Fe ddaethom, fe welon, fe amcangyfrifon
Posts: 13307
Joined: Tue Sep 16, 2008 5:38 pm

Re: Error wfsave command & delete function

Postby EViews Gareth » Wed Feb 15, 2017 6:35 am

Code: Select all

create u 100
table times
for !i=1 to 100
   for !j=1 to 100
      for !k = 1 to 10
         scalar x_!i_!j_!k = 3
         scalar y_!i_!j_!k = 4
      next
   next
next

tic
d x*
!t1 = @toc
times(1,1) = "x*"
times(1,2) = !t1
d y*

for !i=1 to 100
   for !j=1 to 100
      for !k = 1 to 10
         scalar x_!i_!j_!k = 3
         scalar y_!i_!j_!k = 4
      next
   next
next
tic
for !i=1 to 100
   d x_!i*
next
!t1 = @toc
times(2,1) = "x_!i*"
times(2,2) = !t1
d y*

for !i=1 to 100
   for !j=1 to 100
      for !k = 1 to 10
         scalar x_!i_!j_!k = 3
         scalar y_!i_!j_!k = 4
      next
   next
next
tic
for !i=1 to 100
   for !j=1 to 100
      d x_!i_!j*
   next
next
!t1 = @toc
times(3,1) = "x_!i_!j*"
times(3,2) = !t1
d y*

for !i=1 to 100
   for !j=1 to 100
      for !k = 1 to 10
         scalar x_!i_!j_!k = 3
         scalar y_!i_!j_!k = 4
      next
   next
next
tic
for !i=1 to 100
   for !j=1 to 100
      for !k = 1 to 10
         d x_!i_!j_!k
      next
   next
next
!t1 = @toc
times(4,1) = "x_!i_!j_!z"
times(4,2) = !t1
d y*

show times


Interesting
Follow us on Twitter @IHSEViews

weu39843
Posts: 10
Joined: Fri Oct 21, 2016 1:05 am

Re: Error wfsave command & delete function

Postby weu39843 » Mon Feb 20, 2017 1:20 am

Thank you, this is pretty interesting and could make a huge difference regarding my issue. The unsolved question is: why is that? It makes no sense at first glance.

EViews Matt
EViews Developer
Posts: 560
Joined: Thu Apr 25, 2013 7:48 pm

Re: Error wfsave command & delete function

Postby EViews Matt » Thu Apr 13, 2017 10:18 am

I came across some curious performance behavior in one of my programs, so I looked into the phenomenon and it turned out to have the same root cause. Using a modified version of Gareth's script, I observed the following runtimes (in seconds) for deleting various numbers of scalars:

Code: Select all

                   100k     200k      300k      400k
(1) x*           2.297    8.625    20.117    34.947
(2) x_!i_*       1.578    5.547    12.238    21.298
(3) x_!i_!j_*   29.405   68.358   108.006   150.844
(4) x_!i_!j_!k   2.656    7.833    15.285    25.445

I'm guessing that the results for (2) and (4) are the most surprising. I suspect that most people would have the intuition that deleting objects in bigger "chunks" would always be faster. Obviously, that's not the case, so let me illuminate some of EViews' inner workings...

Deletion techniques (1-3) must translate the wildcard expressions into a list of matching object names. The observed differences in runtime primarily reflect a tradeoff between two aspects of that process, 1) going through EViews' master list of objects to find objects with matching names, and 2) building the secondary list containing those matching names. For example, in the 100k scenario, (1) will go through the master list once and build a large list of 100k names. Contrast this with (2), which will go through the master list 100 times and build 100 lists of 1k names each (noting that the master list shrinks after each intermediate deletion of 1k objects). (3) is even more extreme, going through the (ever shrinking) master list 10,000 times, yet building small lists of only 10 names.

Going through the master list of objects so many times, even as it's slowly shrinking, is slow. Going though fewer times, with the list shrinking more quickly, as (2) does, is better. Why then is (1) not the best? It turns out that EViews' algorithm for building the secondary list of matching names slows down as that list becomes large. In computer science, we'd categorize the temporal behavior of the algorithm as being O(n^2), which means that the work the algorithm must perform, and thus the time it takes to execute, grows quadratically with the size of the list. In other words, if the size of the list doubles, the time it takes to build quadruples. In our scenario, (1) builds a list 100 times larger than (2), which takes 100^2 = 10,000 times as long. Even considering that (2) is going to build 100 lists, (1) still spends 100 times the effort (and time) on list-building. This significant increase in work causes (1)'s runtime exceed (2)'s. The is a similar effort differential between (2) and (3), but it's completely outweighed by the time (3) spends repeatedly traversing the master list. In a way, (2) represents a performance "sweet spot" among these three techniques. (2) doesn't build lists large enough cause slowdowns, as (1) does, nor does it traverse the master list enough times to cause slowdowns, as (3) does.

(4) doesn't go through the wildcard resolution process, so it skips all the work I've outlined above. That (4) is still a little bit slower than (2) mostly stems from the fact that (4) goes through 100k iterations while (2) only goes through 100. There is overhead in executing an EViews program and all those extra iterations cost time.


Return to “Bug Reports”

Who is online

Users browsing this forum: Google [Bot] and 16 guests