Hello, I would be very thankful if someone could give me a hand with my problem. I have Windows 7 Pro 64-bit operating system with Intel i7-2600K processor and 16 GB of RAM. I bought a new PC especially to be able to deal with a large database. Yet, it seems that Eviews is not capable of using all the memory provided. Perhaps I could receive some help here why I am receiving "out of memory" error while there is 10 GB of RAM unused.
I created using a "stack" command 3 large files. Each file contains 7 series each series having 5,5 million observations. They contain stacked data (by the year) because I want to create a panel. Every time I try to connect these 3 files by copying series from one file to another I receive "out of memory" error even though Eviews only uses 6 GB RAM - and I have 16 GB RAM available, so 10 GB of RAM is not being used. Can I do sth about it? Changing advanced options does not help at all.
Initially I had one file with many series for each variable and each year (but for many different companies) but I wasn't able to stack them together - therefore, I divided this initial file into 3 smaller parts which I was able to stack.
How to combine 3 large datasets
Moderators: EViews Gareth, EViews Steve, EViews Moderator, EViews Jason
-
EViews Chris
- EViews Developer
- Posts: 161
- Joined: Wed Sep 17, 2008 10:39 am
Re: How to combine 3 large datasets
The bad news is that EViews v7 is only a 32 bit application so it can't actually work with more than 4GB of data in RAM at any time.
That said, it sounds like your data should probably be able to fit within this. Are all the series numeric or do you also have character data in the workfiles?
For numeric data, the rough calculation is 8 bytes per data point * 5.5 million observations * 7 series * 3 files = 924 million bytes (if I'm not missing anything) which should be ok on a 64 bit operating system. (EViews has less space to work with on a 32 bit operating system because the operating system reserves address space for its own uses).
Could you give me a bit more detail as to what the structure of the three files are that you are combining? Some ways of moving the data around within EViews may require a bit less peak memory use than others.
That said, it sounds like your data should probably be able to fit within this. Are all the series numeric or do you also have character data in the workfiles?
For numeric data, the rough calculation is 8 bytes per data point * 5.5 million observations * 7 series * 3 files = 924 million bytes (if I'm not missing anything) which should be ok on a 64 bit operating system. (EViews has less space to work with on a 32 bit operating system because the operating system reserves address space for its own uses).
Could you give me a bit more detail as to what the structure of the three files are that you are combining? Some ways of moving the data around within EViews may require a bit less peak memory use than others.
Re: How to combine 3 large datasets
Thank you very much for an immediate answer to my question. I think, my fears are confirmed. I was wondering how a 32-bit application can handle more than 3 GB of RAM and now it is confirmed that it cannot (I was hoping for some clever way of going around a memory addressing problem).
Sir, you are correct that 7 series with 5,5 million observations each fits within 4 GB RAM, however, I have 3 such files, so in total I have 21 such series (each with 5,5 observations). So Eviews would have to use 12 GB of RAM to be able to merge these files into one big file.
Since it is not posible for a 32-bit program, I have to switch to Stata 64-bit to be able to work with the combined file or to find another application which is 64-bit. I will try with GRETL and Matlab.
Thank you once again for a quick and a clever response. And if I may ask just for my sinful curiosity - would Eviews Enterprise deal with this problem?
Or, perhaps, since neither Eviews nor Stata can estimate a model with an unbalanced panel (Stata would not accept such an input of data, Eviews skips any observation that has a NA) perhaps there is another way how I could go around this problem - is there a way how I could remove observations that have a NA in them? I would have to do it with data before I stack them, because a file that contains unstacked data is smaller than 4 GB (I don't have to divide it into 3 parts).
I have 800.000 companies and for each of these companies I have different financial ratios for years 2003 - 2010 (in the afternoon I can upload a file) - in Eviews it looks like this:
Eviews worksheet:
series with companies' names
series with roe ratios for 2003
series with roe ratios for 2004
......
series with gearing ratios for 2003
series with gearing ratios for 2004
....
etc, etc.
For a panel data I need to stack this data to combine in one series a ratio for all the years, so from 8 series: roe2010, roe2009, roe2008, .....roe2003, roe2002 I need to create one series 'roe' which contains roes for all the years.
But then my problems begin because I have 21 financial ratios for years 2002 - 2010 and when I use stack command I run out of memory. Therefore, I have to divide this file into 3 smaller each containing 7 financial ratios and then run stacking.
So after stacking I have 3 files each containing 7 stacked financial ratios for the same companies.
So I have to remove NAs before I begin stacking and I have to remove companies that have for any year, even only one year a NA. Is there any quick way to do it?
And one more question - by removing such companies I remove firms that went bankrupt so I loose a certain type of firms which some may consider as making the model I estimate biased. Is there any way to go around this problem and not to remove such firms (but Eviews would skip such firms anyway because they contain NAs). I do not think it is better to replace NAs with values from previous years...
Sir, you are correct that 7 series with 5,5 million observations each fits within 4 GB RAM, however, I have 3 such files, so in total I have 21 such series (each with 5,5 observations). So Eviews would have to use 12 GB of RAM to be able to merge these files into one big file.
Since it is not posible for a 32-bit program, I have to switch to Stata 64-bit to be able to work with the combined file or to find another application which is 64-bit. I will try with GRETL and Matlab.
Thank you once again for a quick and a clever response. And if I may ask just for my sinful curiosity - would Eviews Enterprise deal with this problem?
Or, perhaps, since neither Eviews nor Stata can estimate a model with an unbalanced panel (Stata would not accept such an input of data, Eviews skips any observation that has a NA) perhaps there is another way how I could go around this problem - is there a way how I could remove observations that have a NA in them? I would have to do it with data before I stack them, because a file that contains unstacked data is smaller than 4 GB (I don't have to divide it into 3 parts).
I have 800.000 companies and for each of these companies I have different financial ratios for years 2003 - 2010 (in the afternoon I can upload a file) - in Eviews it looks like this:
Eviews worksheet:
series with companies' names
series with roe ratios for 2003
series with roe ratios for 2004
......
series with gearing ratios for 2003
series with gearing ratios for 2004
....
etc, etc.
For a panel data I need to stack this data to combine in one series a ratio for all the years, so from 8 series: roe2010, roe2009, roe2008, .....roe2003, roe2002 I need to create one series 'roe' which contains roes for all the years.
But then my problems begin because I have 21 financial ratios for years 2002 - 2010 and when I use stack command I run out of memory. Therefore, I have to divide this file into 3 smaller each containing 7 financial ratios and then run stacking.
So after stacking I have 3 files each containing 7 stacked financial ratios for the same companies.
So I have to remove NAs before I begin stacking and I have to remove companies that have for any year, even only one year a NA. Is there any quick way to do it?
And one more question - by removing such companies I remove firms that went bankrupt so I loose a certain type of firms which some may consider as making the model I estimate biased. Is there any way to go around this problem and not to remove such firms (but Eviews would skip such firms anyway because they contain NAs). I do not think it is better to replace NAs with values from previous years...
-
EViews Chris
- EViews Developer
- Posts: 161
- Joined: Wed Sep 17, 2008 10:39 am
Re: How to combine 3 large datasets
Some comments - I'm not sure if I've coverred everything in your post.
- EViews v7 can address up to 4GB of RAM on a 64-bit version of Windows. There is no difference between Standard and Enterprise edition. We are expecting to release a 64-bit build of EViews along with v8, but I can't provide you much in the way of details at this point.
- Each series of numeric data with 5.5 million observations requires about 44 Megabytes, so you can store about 20 per Gigabyte. That sets a theoretical maximum somewhere around 80 series of that size in 4GB, but there's various reasons that you probably wouldn't be able to make it all the way up to that maximum in practice.
- The company names in your file may be chewing up a lot of memory. Series containing character data have the same number of bytes allocated to each observation in EViews, which means that the amount of memory used is based on the observation with the largest number of characters. You may want to consider truncating the names to a shorter length (eg. names = @left(names,12) to truncate to 12 characters) if that won't cause problems with the names no longer being unique. An alternative is to use a numeric identifier for the companies instead of a character label.
- Both EViews and Stata can cerrtinly work with unbalanced panels - not sure exactly what you have in mind here
- You can remove rows from an EViews workfile using Proc... Contract Current Page... and then setting an 'If' condition like "if x<>na" to drop any rows where x is not available (for multiple series you can use "if x<>na and y<>na and z<>na"
- EViews v7 can address up to 4GB of RAM on a 64-bit version of Windows. There is no difference between Standard and Enterprise edition. We are expecting to release a 64-bit build of EViews along with v8, but I can't provide you much in the way of details at this point.
- Each series of numeric data with 5.5 million observations requires about 44 Megabytes, so you can store about 20 per Gigabyte. That sets a theoretical maximum somewhere around 80 series of that size in 4GB, but there's various reasons that you probably wouldn't be able to make it all the way up to that maximum in practice.
- The company names in your file may be chewing up a lot of memory. Series containing character data have the same number of bytes allocated to each observation in EViews, which means that the amount of memory used is based on the observation with the largest number of characters. You may want to consider truncating the names to a shorter length (eg. names = @left(names,12) to truncate to 12 characters) if that won't cause problems with the names no longer being unique. An alternative is to use a numeric identifier for the companies instead of a character label.
- Both EViews and Stata can cerrtinly work with unbalanced panels - not sure exactly what you have in mind here
- You can remove rows from an EViews workfile using Proc... Contract Current Page... and then setting an 'If' condition like "if x<>na" to drop any rows where x is not available (for multiple series you can use "if x<>na and y<>na and z<>na"
Re: How to combine 3 large datasets
You have covered my problems brilliantly :). Two more questions:
- is there a way to set a condition removing NAs from many series at the same time without writing all the names of series I have?
- what does it mean that Eviews can work with an unbalanced panel - Stata requires that when I import data in a panel there must not be any missing values. So if a company went bankrupt I have to remove it from the sample or replace with historical values. I understood that Eviews accepts import of data with missing values but in estimation it ignores such observations. Do I get it right?
- is there a way to set a condition removing NAs from many series at the same time without writing all the names of series I have?
- what does it mean that Eviews can work with an unbalanced panel - Stata requires that when I import data in a panel there must not be any missing values. So if a company went bankrupt I have to remove it from the sample or replace with historical values. I understood that Eviews accepts import of data with missing values but in estimation it ignores such observations. Do I get it right?
-
EViews Chris
- EViews Developer
- Posts: 161
- Joined: Wed Sep 17, 2008 10:39 am
Re: How to combine 3 large datasets
One way of dealing with NAs in many series is to make a group from the series and then use the functions @robs() and/or @rnas() which return the number of observations or the number of missing values in each row of the group. For example:
group g * 'make a group containing all series
g.drop resid 'ignore the workfile residual series
smpl if @rnas(g)=0 'condition to exclude observations with any missing values
smpl if @robs(g)>0 'condition to exclude observations with all missing values
As far as unbalanced panels goes - how the missing values will be handled is going to depend on exactly what sort of procedures / estimations you are carrying out. But the general idea is that the estimators will use unbalanced moment matrices on whatever data is available.
group g * 'make a group containing all series
g.drop resid 'ignore the workfile residual series
smpl if @rnas(g)=0 'condition to exclude observations with any missing values
smpl if @robs(g)>0 'condition to exclude observations with all missing values
As far as unbalanced panels goes - how the missing values will be handled is going to depend on exactly what sort of procedures / estimations you are carrying out. But the general idea is that the estimators will use unbalanced moment matrices on whatever data is available.
Who is online
Users browsing this forum: No registered users and 2 guests