Importing updated panel data from separate source

[DCFerral]

Hello,

How do I update a panel workfile with more recent data( with the same variables and cross-sections)? I thought this would be simple but can't get it working.

Details:
I download a database into an excel file. However, updating the whole database takes very long, hence I made two sets of the same spreadsheet:
-Data_Historical (all panel observations before January 2014), and
-Data_current (all panel observations from January 2014).
Both spreadsheets are the same except for the observations covering a different time period. Initially I tried the default import:

Code: Select all

wfopen(page=data) "data_historical" range=tab1
   pagestruct @date(reporting_period) crossid 
%filename= "Data_current"
import(page=data) %filename range=tab1 colhead=1 na="#N/A" @id crossid @date(reporting_period) @smpl @all

I was impressed that this deleted all of the data prior to 2014, leaving only the current data. Then I tried a 'mode' option to prevent overwriting missing observations:

Code: Select all

'after importing historical data:
%filename= "Data_current"
import(mode=ms,page=data) %filename range=tab1 colhead=1 na="#N/A" @id crossid @date(reporting_period) @smpl @all

This did not overwrite the old data, but it did not add the new observations either. So I tried adding the resize options to allow the sample to change:

Code: Select all

'after importing historical data:
%filename= "Data_current"
import(resize, mode=ms, page=data) %filename range=tab1 colhead=1 na="#N/A" @id crossid @date(reporting_period) @smpl @all

This still does not seem to bring in the new observations (it appears to have no effect at all). Next, I tried unstructuring the data to start with:

Code: Select all

   %filename = "data_historical"   
wfopen(page=data) %filename range=Tab1

%filename = "data_current" 

import(mode=ms) %filename range=Tab1

The data up to 2013 is imported, but the second appears not to do anything. What am I doing wrong?
Thank you for your time,
Diego

[EViews Steve]

The resize option on the second import should've worked, but I can understand why it doesn't for a dated panel -- resizing a dated panel can be very complex and all the data conditions have to be exactly correct on the second source file for it to theoretically work.

I don't know how many rows of data you're dealing with or what the performance of the following will be, but here's my suggestion:

First, import your historical data in UNSTRUCTURED format. This means just read in everything exactly as it appears in the source spreadsheet without any dated structure.

Code: Select all

wfopen(page=data) import_historical.xlsx @freq u

Now, you should save this workfile as your base. Whenever you want to include current data, you'll copy this base workfile as your starting point, and then perform the secondary import as an append (again UNSTRUCTURED), like this:

Code: Select all

import(resize, page=data) import_current.xlsx @append @freq u

(here the resize option works as expected)

Then finally, you can restructure this workfile as a dated panel:

Code: Select all

pagestruct(bal=m) crossid @date(reporting_period)

Again, I don't know what the performace will be like with your dataset.

As a side note, if the resize option did work properly with a dated panel, it would probably be very similar in performance to these manual steps because internally it would be doing something really similar (unstack the dated panel to unstructured first, append the new data, re-structure back to dated panel).

Steve

[DCFerral]

Thank you Steve,
This worked, and the updating process is much faster. The @append command is what I had missed. Thank you very much.