"Out of Memory" error

[fbrayton]

Hi:

My question concerns how large a matrix EViews can invert. I am using a PC with 2GB of ram (and 2 CPUs) with VISTA. Inversion of a 6000x6000 matrix worked, but inversion of an 8000x8000 matrix failed with an "out of memory" error. Without success, I tried changing the "memory reserved ..." option under "advanced system options" both down and up. I was surprised to see that Windows task manager indicated that only slightly more than 50 percent of total physical memory was in use immediately after the error occurred and that EViews 6 had only about 510k of memory. My understanding from reading the User's Guide is EViews can address up to 2GB of memory. Am I missing something?

[EViews Gareth]

I just inverted an 8,000x8,000 matrix, and it took up around 1GB of memory to do it.

You should really track how much memory is being used up to the point of the error - doing it after the error has occurred may not tell you much (as EViews will release the memory).

[fbrayton]

I've investigated this problem further since my initial post. My early post was based on running EViews on my home PC. Today, I have run some more tests on my work PC on which I had the memory increased from 2GB to 4GB this morning. Both PCs have 2 processors; the one at home runs Windows Vista; the one at work Windows XP. The EViews "Advanced System Options" settings on both PCs allow 4 million observations per series and 400MB of memory reserved for Windows, etc.

The results are the same on each PC: I can invert a 6200x6200 matrix but not one 6500x6500 or larger. For the successful 6200 matrix inversion, Task Manager shows that EViews is using 616000k of memory for most of the inversion process, but that memory use jumps to 907000k for a few seconds at the very end of the execution. It is at the memory-jump stage that the inversion of larger matrices fails (ie, at around 1 GB or higher).

The inversion failure takes two forms. In a simple test program that just inverts a matrix filled with random normal values, the "out of memory" error pops up. In the "real" application that I am working on, in which the inversion is part of a much longer sequence of commands, the error is "EViews has encountered an problem and needs to close." The first program is listed at the bottom of this post.

A pair of questions:
1. Why doesn't EViews doesn't switch to using virtual memory.
2. The following phrase appears in the EViews User's Guide vol. 1, p 770: "Reducing the size of the reserved space ... may lead to instability caused by out-of-memory conditions." Although I don't see that this is directly germane to my problem as I did try higher and lower amounts of the reserved memory in my tests at home, I nonetheless wonder whether there is something relevant here.




wfcreate junk q 2000q1 2010q4
!n = 1000
matrix(!n,!n) mmm

for !i = 1 to !n
for !j = 1 to !n
mmm(!i,!j) = @rnorm
next
next

tic
matrix inv = @inverse(mmm)
scalar elapsed = @toc
show elapsed

[EViews Chris]

Here's some background on what is going on.

There's actually two types of 'memory' that are important in the sort of tasks that you are working with.

The first type is actual storage capacity: physical RAM on the machine and any space on hard disks that has been allocated to be used as virtual memory. You are correct that for the operations that you are carrying out this probably isn't what is causing you problems. Each 6200*6200 matrix requires about 300 megabytes of storage (6200*6200*8/1024/1024), so a small number of these should probably be within the capacity of your machine. This is the sort of memory that you're seeing reported in Task Manager.

The second type of memory that matters to EViews is a thing called 'address space'. All memory that the operating system provides to EViews is through the use of address space. EViews doesn't actually know whether a piece of address space is associated with physical RAM or virtual memory on a hard drive. We just tell the operating system that we need to use some memory and it assigns us a piece of address space to work with. All the physical storage details are handled by the operating system.

On a 32-bit operating system, each application has a total address space of 4GB. This is further divided into space reserved by the operating system and space available to the application. On a typical 32 bit XP or Vista install, the operating system reserves 2GB of memory for itself, and allows the application to use the remaining 2GB. You can tweak this using something called the "3GB switch" so that the split is 3GB to the application and 1GB for the operating system. Finally, when a 32-bit application runs on 64 bit version of Windows, the entire 4GB is available to the application.

Ten years ago, the amount of address space available to a 32-bit application was so large relative to the physical memory of the machine that limitations on address space weren't really much of a concern. On a modern machine that has been loaded up with a decent amount of RAM, it is becoming more of a problem. The eventual solution will be for everyone to move to 64-bit operating systems running 64-bit applications, where the address space available is enormous, but we've got a few years to go yet before this becomes the new standard.

To get back to your problem, what is probably going on in your case is that EViews is finding it hard to allocate a large enough piece of address space to work with your large matrices. When EViews allocates a matrix, the elements are all held together in a single contiguous piece of address space. In the case of a 6200 by 6200 matrix, that requires about 300MB in a single chunk.

Needing a single big chunk can easily cause problems. As EViews runs, various bits of address space get used and discarded so that you can end up with what is called fragmentation. There can be lots of medium size bits of address space available but no single really big piece of address space that would be required to store a 6200 by 6200 matrix.

This generally isn't a problem when working with series in workfiles because each series is allocated separately, so that you never require a single enormous chunk of memory.

At the end of the day, there's no magic bullet to fix the problems that you're having, but here are some general tips.

Firstly, getting more address space from the operating system is probably going to help. The best you can do is to run EViews on a 64-bit version of Windows. The next best thing is to run a 32-bit version of Windows using the '/3gb' switch (google '/3gb' for details).

Secondly, you can try changing the global setting in EViews that you mention that reduces the amount of address space reserved for Windows, DLL files, device drivers. etc. to a lower amount. The instability that the manual mentions is most likely to occur if you are using ODBC drivers to read from relational databases or if you are using a very large number of small objects inside EViews. You can certainly try reducing this down to 200MB and see what happens. If you start seeing a lot of crashes that weren't there before, you'll have to change the setting back.

Lastly, EViews is going to have more chance of finding nice big chunks of address space when it has been freshly started up. You may be better off running your program up to the point where you need to invert your very large matrix, closing down and restarting EViews, then running the inversion and the remainder of your program. I know this is inconvenient, but if you're right up against the limits, you might be able to succeed in getting things to work this way when otherwise you would fail.

[EViews Gareth]

In addition to Chris' excellent description of the way memory works, I'll quickly point out that your program is quite inefficient in its generation of your matrix. In the future, you could replace:

Code: Select all

matrix(!n,!n) mmm for !i = 1 to !n for !j = 1 to !n mmm(!i,!j) = @rnorm next next

with:

Code: Select all

matrix(!n,!n) mmm nrnd(mmm)

If you want uniform randoms instead of normals you could do:

Code: Select all

matrix(!n,!n) mmm rnd(mmm)

[fbrayton]

Chris: Thanks a lot for your detailed response. It is much appreciated. I will try the "3gb switch" option, but it will be a few days before I'm back in my office, which is where the PC with 4GB of memory is located.

Gareth: I thought there must be a better way to load a matrix with random numbers. Thanks for pointing me to it.

[fbrayton]

Here's an update:

With 4GB of memory and the /3GB option turned on, I was able to invert an 8000x8000 matrix (but not a 9000x9000 matrix). Thanks again for the help.