Hi
This is maybe a little bit odd suggestion/question and our problem has perhaps only something to do with our network. But i give it a try.
We using Eviews on network and our program runs relatively quickly. But when a program shall open or save a relativ big workfile, then it takes time, sometimes a lot of time. Eviews seems to read from (or write to) a workfile "bit by bit", i.e. eviews seems to have to communicate with the network many times during this kind of operation (i.e. wfopen or wfsave). The same phenomenon occurs when for exampel reading from or writing to a large excel files. The problem increases when using system of program.
We've made a test and create a workfile with 1000 observations and 10 000 objects (series objects), then saving the workfile, closing the workfile and finaly opening the same workfile. On the local drive (c:\) this takes 1.5 seconds and on the network it takes 117.5 seconds to do the operation. We can reduce the time on the network by temporarily move the workfile to and from c:\ by batch-command (shell-command). The run on the netwoork then reduces to 18.1 seconds. But this is a little inconvenient solution.
So if Eviews instead could save, for exampel, the workfile temporarely in the working memory or on c:\ and then moving the file to the correct direction, and when opening a workfile, first moving the file to working memory or on c:\ and then and then opening it (the same for other files). It would probably reduce the execution time.
We're fully aware that this ain't a quickfix, and maybe not feasible. Maybe we also have misunderstood how Eviews work. But on the other hand has the Eviewsteam a reputation of being overpowering hackers
Regards Johan (and coworkers)
Working with files in working memory
Moderators: EViews Gareth, EViews Moderator
-
- Posts: 160
- Joined: Thu Mar 03, 2011 10:07 pm
- Contact:
-
- EViews Developer
- Posts: 161
- Joined: Wed Sep 17, 2008 10:39 am
Re: Working with files in working memory
We'll look into it.
The operating system / networking layer is kind of meant to handle this for us (by bundling up little reads and writes into bigger ones) but there do appear to be cases where moving the entire file over to the local drive first is faster.
We've been looking into something like that for other forms of network storage, so we'll do some experiments and see if it looks a viable option.
The operating system / networking layer is kind of meant to handle this for us (by bundling up little reads and writes into bigger ones) but there do appear to be cases where moving the entire file over to the local drive first is faster.
We've been looking into something like that for other forms of network storage, so we'll do some experiments and see if it looks a viable option.
-
- Posts: 160
- Joined: Thu Mar 03, 2011 10:07 pm
- Contact:
Re: Working with files in working memory
OK, thanks
-
- Posts: 160
- Joined: Thu Mar 03, 2011 10:07 pm
- Contact:
Re: Working with files in working memory
Hi
I've done the "speed test" again with a similar result. Ie it is still relatively slow to save / open on the network.
Do you have any thoughts on this?
Regards Johan
I've done the "speed test" again with a similar result. Ie it is still relatively slow to save / open on the network.
Do you have any thoughts on this?
Regards Johan
-
- EViews Developer
- Posts: 2672
- Joined: Wed Oct 15, 2008 9:17 am
Re: Working with files in working memory
What version of EViews are you using?
-
- Posts: 160
- Joined: Thu Mar 03, 2011 10:07 pm
- Contact:
Re: Working with files in working memory
Standard Edition - Jan 7 2016 (32 bit). We use Linux / Novell networks, or something like that (i.e. not windowsserver), maybe it matters?
Regards Johan
Regards Johan
-
- EViews Developer
- Posts: 798
- Joined: Tue Sep 16, 2008 3:00 pm
- Location: Irvine, CA
Re: Working with files in working memory
Johan:
In EViews 9, we changed our code to use a buffered file reader class when opening and saving a workfile. The new class gave us significant improvement over our previous file reader. By default, this is turned on so you should already be using it.
For experimentation, we created an undocumented command named: USEFILEBUFFER.
To turn it on (i.e. use new buffered file reader), simply type in:
You should see a response message in the status bar at the bottom of the main window.
To turn it off (and use our old file reader), type in:
then try opening and saving the same workfile. Turning this setting off is only valid for the current instance of EViews. Once you restart, it gets reset back to ON.
Try playing with these settings and let me know if you see the expected difference in performance for both read and write.
Also, can you perform a test with a Windows file share to see if that changes the timings?
Steve
In EViews 9, we changed our code to use a buffered file reader class when opening and saving a workfile. The new class gave us significant improvement over our previous file reader. By default, this is turned on so you should already be using it.
For experimentation, we created an undocumented command named: USEFILEBUFFER.
To turn it on (i.e. use new buffered file reader), simply type in:
Code: Select all
USEFILEBUFFER
You should see a response message in the status bar at the bottom of the main window.
To turn it off (and use our old file reader), type in:
Code: Select all
USEFILEBUFFER(FALSE)
then try opening and saving the same workfile. Turning this setting off is only valid for the current instance of EViews. Once you restart, it gets reset back to ON.
Try playing with these settings and let me know if you see the expected difference in performance for both read and write.
Also, can you perform a test with a Windows file share to see if that changes the timings?
Steve
-
- Posts: 160
- Joined: Thu Mar 03, 2011 10:07 pm
- Contact:
Re: Working with files in working memory
Hi Steve, here is some result
Eviews version 22 feb
workfile: range 1-1000, objects: 10 000, size 79,7 mb
1 Linux-server, network
Test 1.1 USEFILEBUFFER(TRUE)
A: save: 68 seconds
B: open: 3 seconds
Test 1.2 USEFILEBUFFER(FALSE)
A: save: 68 seconds
B: open: 40 seconds
2 local drive (c:\temp)
Test 2.1 USEFILEBUFFER(TRUE)
A: save: 1 seconds
B: open: 0 seconds
Test 2.2 USEFILEBUFFER(FALSE)
A: save: 1 seconds
B: open: 0 seconds
3 Linux-server, network and local drive
Test 3.1 USEFILEBUFFER(TRUE)
A: save on local drive, copy via batch from local drive to network: 4 seconds
B: copy via batch from network to local drive, open on local drive: 2 seconds
Test 3.2 USEFILEBUFFER(FALSE)
A: save on local drive, copy via batch from local drive to network: 4 seconds
B: copy via batch from network to local drive, open on local drive: 3 seconds
Comments
When using USEFILEBUFFER(TRUE) it's must faster to open file (compare 1.1B with 1.2B). The result in 1.1B is similar to the result in 3.1B and 3.2B
It's still slow to save file on netowork and diffrent options in USEFILEBUFFER does not seem to have any affect. It's must faster to save on local drive and then copy to network (see 3.1A and 3.2A)
It would be nice with another "file saving class" (or something), like the new "file reader class"
Regards Johan
Eviews version 22 feb
workfile: range 1-1000, objects: 10 000, size 79,7 mb
1 Linux-server, network
Test 1.1 USEFILEBUFFER(TRUE)
A: save: 68 seconds
B: open: 3 seconds
Test 1.2 USEFILEBUFFER(FALSE)
A: save: 68 seconds
B: open: 40 seconds
2 local drive (c:\temp)
Test 2.1 USEFILEBUFFER(TRUE)
A: save: 1 seconds
B: open: 0 seconds
Test 2.2 USEFILEBUFFER(FALSE)
A: save: 1 seconds
B: open: 0 seconds
3 Linux-server, network and local drive
Test 3.1 USEFILEBUFFER(TRUE)
A: save on local drive, copy via batch from local drive to network: 4 seconds
B: copy via batch from network to local drive, open on local drive: 2 seconds
Test 3.2 USEFILEBUFFER(FALSE)
A: save on local drive, copy via batch from local drive to network: 4 seconds
B: copy via batch from network to local drive, open on local drive: 3 seconds
Comments
When using USEFILEBUFFER(TRUE) it's must faster to open file (compare 1.1B with 1.2B). The result in 1.1B is similar to the result in 3.1B and 3.2B
It's still slow to save file on netowork and diffrent options in USEFILEBUFFER does not seem to have any affect. It's must faster to save on local drive and then copy to network (see 3.1A and 3.2A)
It would be nice with another "file saving class" (or something), like the new "file reader class"
Regards Johan
-
- EViews Developer
- Posts: 798
- Joined: Tue Sep 16, 2008 3:00 pm
- Location: Irvine, CA
Re: Working with files in working memory
Johan:
Thanks for the info.
I've created a separate version of EViews 9 that uses a local save, then move to try and workaround this issue. Please email me at steve@eviews.com and refer to this post and I'll send you a link to download this version. Once confirmed it works for you, I'll release it in the next patch for EViews 9.
Steve
Thanks for the info.
I've created a separate version of EViews 9 that uses a local save, then move to try and workaround this issue. Please email me at steve@eviews.com and refer to this post and I'll send you a link to download this version. Once confirmed it works for you, I'll release it in the next patch for EViews 9.
Steve
Return to “Suggestions and Requests”
Who is online
Users browsing this forum: No registered users and 33 guests