efficiently storing large amounts of data to server
Posted: Fri Jun 12, 2020 8:27 am
I sometimes store a very large amount of data to a server, about 4000 series with a few years of monthly data. The way I do this currently is to store the {string} of all the series that I am going to store, but it takes quite a while for the store command to finish running (20-30 minutes).
Is there a more efficient way to do this from a processing or bandwidth perspective? It takes long enough that my corporate-controlled computer goes into sleep mode (which I think pauses the store?) if I do not hang out to babysit it, which makes the long time a little more problematic. Would opening the database first give me access to a more efficient storing command? Would doing this as a program in Quiet mode save a significant amount of time? (To be honest, I often find the status bar to be reassuring on lengthy processing times.)
My upload speed is 5 mbps, which is not spectacular, but I figure that a bunch of numbers can't really be that many megabytes.
Is there a more efficient way to do this from a processing or bandwidth perspective? It takes long enough that my corporate-controlled computer goes into sleep mode (which I think pauses the store?) if I do not hang out to babysit it, which makes the long time a little more problematic. Would opening the database first give me access to a more efficient storing command? Would doing this as a program in Quiet mode save a significant amount of time? (To be honest, I often find the status bar to be reassuring on lengthy processing times.)
My upload speed is 5 mbps, which is not spectacular, but I figure that a bunch of numbers can't really be that many megabytes.