Working with files in working memory

[johansamuelsson]

Hi

This is maybe a little bit odd suggestion/question and our problem has perhaps only something to do with our network. But i give it a try.

We using Eviews on network and our program runs relatively quickly. But when a program shall open or save a relativ big workfile, then it takes time, sometimes a lot of time. Eviews seems to read from (or write to) a workfile "bit by bit", i.e. eviews seems to have to communicate with the network many times during this kind of operation (i.e. wfopen or wfsave). The same phenomenon occurs when for exampel reading from or writing to a large excel files. The problem increases when using system of program.

We've made a test and create a workfile with 1000 observations and 10 000 objects (series objects), then saving the workfile, closing the workfile and finaly opening the same workfile. On the local drive (c:\) this takes 1.5 seconds and on the network it takes 117.5 seconds to do the operation. We can reduce the time on the network by temporarily move the workfile to and from c:\ by batch-command (shell-command). The run on the netwoork then reduces to 18.1 seconds. But this is a little inconvenient solution.

So if Eviews instead could save, for exampel, the workfile temporarely in the working memory or on c:\ and then moving the file to the correct direction, and when opening a workfile, first moving the file to working memory or on c:\ and then and then opening it (the same for other files). It would probably reduce the execution time.

We're fully aware that this ain't a quickfix, and maybe not feasible. Maybe we also have misunderstood how Eviews work. But on the other hand has the Eviewsteam a reputation of being overpowering hackers

Regards Johan (and coworkers)

[EViews Chris]

We'll look into it.

The operating system / networking layer is kind of meant to handle this for us (by bundling up little reads and writes into bigger ones) but there do appear to be cases where moving the entire file over to the local drive first is faster.

We've been looking into something like that for other forms of network storage, so we'll do some experiments and see if it looks a viable option.

[johansamuelsson]

OK, thanks

[johansamuelsson]

Hi

I've done the "speed test" again with a similar result. Ie it is still relatively slow to save / open on the network.

Do you have any thoughts on this?

Regards Johan

[EViews Glenn]

What version of EViews are you using?

[johansamuelsson]

Standard Edition - Jan 7 2016 (32 bit). We use Linux / Novell networks, or something like that (i.e. not windowsserver), maybe it matters?

Regards Johan

[EViews Steve]

Johan:

In EViews 9, we changed our code to use a buffered file reader class when opening and saving a workfile. The new class gave us significant improvement over our previous file reader. By default, this is turned on so you should already be using it.

For experimentation, we created an undocumented command named: USEFILEBUFFER.

To turn it on (i.e. use new buffered file reader), simply type in:

Code: Select all

USEFILEBUFFER

You should see a response message in the status bar at the bottom of the main window.

To turn it off (and use our old file reader), type in:

Code: Select all

USEFILEBUFFER(FALSE)

then try opening and saving the same workfile. Turning this setting off is only valid for the current instance of EViews. Once you restart, it gets reset back to ON.

Try playing with these settings and let me know if you see the expected difference in performance for both read and write.

Also, can you perform a test with a Windows file share to see if that changes the timings?

Steve

[johansamuelsson]

Hi Steve, here is some result

Eviews version 22 feb
workfile: range 1-1000, objects: 10 000, size 79,7 mb

1 Linux-server, network

Test 1.1 USEFILEBUFFER(TRUE)
A: save: 68 seconds
B: open: 3 seconds

Test 1.2 USEFILEBUFFER(FALSE)
A: save: 68 seconds
B: open: 40 seconds

2 local drive (c:\temp)

Test 2.1 USEFILEBUFFER(TRUE)
A: save: 1 seconds
B: open: 0 seconds

Test 2.2 USEFILEBUFFER(FALSE)
A: save: 1 seconds
B: open: 0 seconds

3 Linux-server, network and local drive

Test 3.1 USEFILEBUFFER(TRUE)
A: save on local drive, copy via batch from local drive to network: 4 seconds
B: copy via batch from network to local drive, open on local drive: 2 seconds

Test 3.2 USEFILEBUFFER(FALSE)
A: save on local drive, copy via batch from local drive to network: 4 seconds
B: copy via batch from network to local drive, open on local drive: 3 seconds

Comments

When using USEFILEBUFFER(TRUE) it's must faster to open file (compare 1.1B with 1.2B). The result in 1.1B is similar to the result in 3.1B and 3.2B

It's still slow to save file on netowork and diffrent options in USEFILEBUFFER does not seem to have any affect. It's must faster to save on local drive and then copy to network (see 3.1A and 3.2A)

It would be nice with another "file saving class" (or something), like the new "file reader class"

Regards Johan

[EViews Steve]

Johan:

Thanks for the info.

I've created a separate version of EViews 9 that uses a local save, then move to try and workaround this issue. Please email me at steve@eviews.com and refer to this post and I'll send you a link to download this version. Once confirmed it works for you, I'll release it in the next patch for EViews 9.

Steve