A Loop Which Iterations Take Increasing Amount of Time

[Ecovi]

Is there any reason why this loop's iterations,

Code: Select all

for !i=1 to 100 [do something] next

would require increasing amounts of time when there is no reason for that in the code? Perhaps due to some memory quirk?
Iteration 100 is a several orders of magnitude slower than the 1st iteration. The statements between iterations are completely independent.

If I run exactly the same code as follows, then the program runs fast at each iteration:

Code: Select all

for !i=1 to 100 exec <program> next

where the Eviews program <program> does exactly the same as [do something] above.

My results are exactly the same between the programs. The [do something] part consists of a large number of other loops with vector and matrix aggregation operations, and freezing model output for saving some of its cells. Run time should be around 10 minutes, but takes 4 hours with the former code organisation.

I can make a minimum working example if someone does not have an obvious (i.e. programmatic or technical) reason why this might occur.

[EViews Gareth]

We'd need to see an example.

[Ecovi]

Okay, it is going to take some time though to abridge the code and make up something presentable for you. I'll get back to this.

[Ecovi]

I did some more testing on this. It appears that the combination of creating objects together with performing some operation on them causes the process to take more and more time, if done in a single program.

The attached programs allow you to see the issue. The program iteratively creates new objects and performs a query on them (to get the Schwarz criterion). The whole process is structured with an inner loop to monitor the elapsed time for a number of these queries. This inner loop is called with an exec statement in the second version of the program.

Run the 'onefile' program to see that every iteration takes longer than the previous. Using the tic/toc functions the elapsed time for each iteration is measured and displayed on screen. The estimated remaning time is also graphed (blue line), along with the total calculated run time derived from this (red line).

When running the same code using the second version of the program, with the inner loop in another file, called using an exec statement, you can see that every iteration takes only small amount of time.

In my original program the process almost screaches to a halt, becoming so slow it it unusable.

[EViews Gareth]

That's a fantastic example program. Thanks!

The issue is with the creation and deletion of objects with different names. A single EViews program keeps track of all named objects that get created, even if they are deleted. You're creating tens of thousands of named objects (with different names) and then deleting them, but EViews has to keep track of them even though they're deleted.

(slight caveat - EViews only keeps track if you do something with the object, simply creating and immediately deleting is fine).

Execing fixes it because a second program is creating and deleting, and that program only has to keep track of 1,000 names.

The best solution (at least in this example, not sure if possible in your actual program) is to not use new names, but rather to use the same name over and over:

Code: Select all

wfcreate u 1 50 scalar starttime = @now*24*3600 'start time in seconds mode quiet logmode logmsg equation dummy series e = nrnd dummy.ls e c !updateinterval = 5 !meaniterationtime = 0.05 !elapsed_update = !updateinterval table(1,15) progresstable tic scalar iterationnumber scalar iterations = 100 for iterationnumber=1 to iterations logmsg Iteration {iterationnumber} progresstable.display for !r=1 to 999 copy dummy dummy1 scalar bic = dummy1.@sc delete dummy1 next !elapsed = @toc tic !meaniterationtime = ((iterationnumber-1) * !meaniterationtime + !elapsed) / iterationnumber !remaining = !meaniterationtime*(iterations - iterationnumber) matrix(iterations,2) progressmatrix progressmatrix(iterationnumber,1) = !remaining 'estimated remaining time in seconds progressmatrix(iterationnumber,2) = (@now*24*3600 + !remaining - starttime) 'total runtime estimate in seconds !elapsed_update = !elapsed_update + !elapsed if !elapsed_update > !updateinterval then progresstable(1,1) = "Seconds elapsed for last iteration (average): " + @str(!elapsed) + " (" + @str(!meaniterationtime) + ")" progressmatrix.line progresstable.display !elapsed_update = 0 endif next

[Ecovi]

That's a fantastic example program. Thanks!

The issue is with the creation and deletion of objects with different names. A single EViews program keeps track of all named objects that get created, even if they are deleted. You're creating tens of thousands of named objects (with different names) and then deleting them, but EViews has to keep track of them even though they're deleted.

(slight caveat - EViews only keeps track if you do something with the object, simply creating and immediately deleting is fine).

I'm glad you like it. Did you know about this problem before? It took me quite some time to find this out as well; I had to comment out lines and test the program lots of times before hunting this issue down to the combination of creating objects together with performing some operation on them. Indeed, the program runs fast if you do not get the Schwarz criterion.

This of course begs the question: can something be done about this in a future version of Eviews? Or is it---like quite some other inconveniences---so deeply entrenched within Eviews that it would require rewriting the software?

Execing fixes it because a second program is creating and deleting, and that program only has to keep track of 1,000 names.

The best solution (at least in this example, not sure if possible in your actual program) is to not use new names, but rather to use the same name over and over

Of course in my actual program, the equations are doing something useful, and are also all different. If at all possible, I like to keep objects around, performing operations on them only when necessary. This allows me to change the required operations or queries on my models after a (computationally intensive) estimation method (for thousands of models). This would not be possible if I would overwrite them.

Note that if you first create all the objects and only afterwards perform the operations on them, then at first these opertations are fast. So even if many objects already exist, Eviews 'keeps track of objects' only when it has first touched them. Since the program runs identical when using exec (albeit much faster), I fail to see why this problem exists at all, i.e. why would it need to keep track of objects if it would run perfectly without doing that?

As a workaround, it seems to me that some command that clears the 'objects tracking list' is all that is needed:
I didn't test this, but would periodically save and reopen a workfile help with this?
Also, would storing objects in a database circumvent the problem?

[EViews Gareth]

It is an issue that we knew about (but had forgotten about!). We've tried fixing it before, but all fixes other than re-writing the entire programming language from scratch result in other bad issues.

The tracking list is program specific. Changing workfile, or using a database won't help. Splitting into separate programs does. A quirk, but that's the way it is.

[Ecovi]

Strange that it is really unfixable in an easy way. I guess the 'used objects name list' is just some data structure that can be simply deleted. Maybe as a workaround only after it gets a thousand elements or something. This really might deserve some more attention, especially now that the 64-bit version of Eviews is here and advertises that the number of objects is "limited only by available RAM". This is true of course, but if opening any object takes seconds then users should be aware of this before they start programming something with so many objects.