Page 1 of 1

efficiently storing large amounts of data to server

Posted: Fri Jun 12, 2020 8:27 am
by capuchin
I sometimes store a very large amount of data to a server, about 4000 series with a few years of monthly data. The way I do this currently is to store the {string} of all the series that I am going to store, but it takes quite a while for the store command to finish running (20-30 minutes).

Is there a more efficient way to do this from a processing or bandwidth perspective? It takes long enough that my corporate-controlled computer goes into sleep mode (which I think pauses the store?) if I do not hang out to babysit it, which makes the long time a little more problematic. Would opening the database first give me access to a more efficient storing command? Would doing this as a program in Quiet mode save a significant amount of time? (To be honest, I often find the status bar to be reassuring on lengthy processing times.)

My upload speed is 5 mbps, which is not spectacular, but I figure that a bunch of numbers can't really be that many megabytes.

Re: efficiently storing large amounts of data to server

Posted: Fri Jun 12, 2020 9:43 am
by EViews Gareth
Could you provide a few more details - what type of file are you storing to? (EViews database - .edb, or some other kind?)

Re: efficiently storing large amounts of data to server

Posted: Fri Jun 12, 2020 9:59 am
by EViews Steve
Without knowing the details yet, I can say that storing to a local database/file first, and then pushing the database/file to the network is going to be significantly faster than pushing each object over the network one by one.

For example, save everything to a local CSV text file, then copy the CSV file to your network location.

Code: Select all

wfsave(type=txt) mydata.csv shell copy mydata.csv x:\network\folder\mydata.csv

Re: efficiently storing large amounts of data to server

Posted: Fri Jun 12, 2020 1:17 pm
by EViews Gareth
Although Steve's suggestion isn't going to work if you're updating a database.

Re: efficiently storing large amounts of data to server

Posted: Fri Jun 19, 2020 11:15 am
by capuchin
This is going to a FAME database on a server.

Re: efficiently storing large amounts of data to server

Posted: Mon Jun 22, 2020 8:07 am
by capuchin
Speaking of servers, my server just got changed to a long and cumbersome name. Is there a way to set the server option for the fetch/store command globally so that I can leave out the "server=..." part of the fetch/store commands? Right now, I write:

fetch(d=MYBANK,t=F,server=MYSERVER) MYSERIES

and I don't really want to change to

fetch(d=MYBANK,t=F,server=MYQWERTYOMGBBQ_#&*!@#SERVER) MYSERIES

Re: efficiently storing large amounts of data to server

Posted: Mon Jun 22, 2020 9:00 am
by EViews Gareth
Add it to your database registry