Performance comparison between store and copy command for storing to Eviews databases on network drives

[melemen]

Hi there,

In a program I regularly run there are a quite a big number of time series (around 3500) stored from one workfile into two databases (~1.5MB and ~2.5MB), all located on a network drive. Over the recent months the performance of the storing process has suffered a lot (I just did a test from an HPC server and it took 13 minutes), and our IT people told us that this was due to high activity on the network. While they said they are looking into it, this might take a while until this gets better. So I was wondering if there is a better alternative to the store command. I don't have an alternative to doing this data management on a network drive as the series need to be accessible to a number of colleagues.
For a subset of the data I already tested the copy command and it seemed a lot faster. Is there a difference in how these two commands work? Or should the two commands be equally fast and this is only due to the fact that the copy operation kind of works in "quiet mode" while for store you see the series being copied? Is there any risk of changing my code to use the copy command? Are there similar performance differences between a fetch and a copy from database to workfile? Fetch works much faster than store, but still it takes a while when the network drive is busy...
I work in Eviews 9.5 (but I also had this performance issue in Eviews 8 already).

Basically what is currently used is the store command with a wildcard search.
%PEVDB refers to the folder path of the database location and %WV is the first part of the filename.

Code: Select all

PAGESELECT Q %filename = %PEVDB+%WV+"Q" STORE(D=%filename) {%R}_{%F}_Q_* PAGESELECT A %filename = %PEVDB+%WV+"A" STORE(D=%filename) {%R}_{%F}_A_*

I tried to reprogram the code to the below, but for the one database it gives me an error message saying that the database is not found, even though it exists.
Any ideas?

Code: Select all

PAGESELECT Q copy {%R}_{%F}_Q_* {%WV}Q:: PAGESELECT A copy {%R}_{%F}_A_* {%WV}A::

I just did a comparison test for the one database where it worked and the copy command actually was slower than the store command (in quiet mode) this time. Maybe copy works better for smaller numbers of series at a time?

Many thanks in advance for your help.

Best regards,
Alex

[EViews Steve]

melemen:

About the performance differences between STORE and COPY, I can tell you they should be minimal because both operations, at their core, will write the data the same way to the EViews database file. Any differences you see in performance is most likely due to differences in network congestion at the moment you ran the operation.

Writing to files on a network often does require much more time to complete. My suggestion is to make a local copy of your database files (.EDB, .E0, .E1A, and .E1B), perform the write to the local database, then copy them back to the network. This will always be much faster than trying to write directly to the network copy because of the sheer amount of overhead required for the network.

I believe you can use our SHELL command to call the MS-DOS COPY command during your program to perform these file copy operations.

Hope this helps...

Steve

[melemen]

Dear Steve,

Indeed this seems to work a lot faster! Many thanks for your help!

Best,
Alex

[melemen]

Would this also work with dbcopy instead of shell copy or would this slow down the process again?

[EViews Steve]

Yes, dbcopy should work too.


Sent from my iPhone using Tapatalk