pytables - best practices / mem leaks

[py_genetic]

I have an H5 file with one group (off the root) and two large main
tables and I'm attempting to aggragate my data into 50+ new groups (off
the root) with two tables per sub group.

sys info:
PyTables version: 1.3.2
HDF5 version: 1.6.5
numarray version: 1.5.0
Zlib version: 1.2.3
BZIP2 version: 1.0.3 (15-Feb-2005)
Python version: 2.4.2 (#1, Jul 13 2006, 20:16:08)
[GCC 4.0.1 (Apple Computer, Inc. build 5250)]
Platform: darwin-Power Macintosh (v10.4.7)
Byte-ordering: big

Ran all pytables tests included with package and recieved an OK.
Using the following code I get one of three errors:

1. Illegal Instruction

2. Malloc(): trying to call free() twice

3. Bus Error

I believe all three stem from the same issue, involving a malloc()
memory problem in the pytable c libraries. I also believe this may be
due to how I'm attempting to write my sorting script.

The script executes fine and all goes well until I'm sorting about
group 20 to 30 and I throw one of the three above errors depending on
how/when I'm flush() close() the file. When I open the file after the
error using h5ls all tables are in perfact order up to the crash and if
I continue from the point every thing runs fine until python throws the
same error again after another 10 sorts or so. The somewhat random
crashing is what leads me to believe I have a memory leak or my method
of doing this is incorrect.

Is there a better way to aggragate data using pytables/python? Is there
a better way to be doing this? This seems strait forward enough.

Thanks,
Conor

#function to agg state data from main neg/pos tables into neg/pos state
tables

import string
import tables
def aggstate(state, h5file):

print state

class PosRecords(tables.IsDescription):
sic = tables.IntCol(0, 1, 4, 0, None, 0)
numsic = tables.IntCol(0, 1, 4, 0, None, 0)
empsiz = tables.StringCol(1, '?', 1, None, 0)
salvol = tables.StringCol(1, '?', 1, None, 0)
popcod = tables.StringCol(1, '?', 1, None, 0)
state = tables.StringCol(2, '?', 1, None, 0)
zip = tables.IntCol(0, 1, 4, 0, None, 1)

class NegRecords(tables.IsDescription):
sic = tables.IntCol(0, 1, 4, 0, None, 0)
numsic = tables.IntCol(0, 1, 4, 0, None, 0)
empsiz = tables.StringCol(1, '?', 1, None, 0)
salvol = tables.StringCol(1, '?', 1, None, 0)
popcod = tables.StringCol(1, '?', 1, None, 0)
state = tables.StringCol(2, '?', 1, None, 0)
zip = tables.IntCol(0, 1, 4, 0, None, 1)

group1 = h5file.createGroup("/", state+"_raw_records", state+" raw
records")

table1 = h5file.createTable(group1, "pos_records", PosRecords, state+"
raw pos record table")
table2 = h5file.createTable(group1, "neg_records", NegRecords, state+"
raw neg record table")

table = h5file.root.raw_records.pos_records
point = table1.row
for x in table.iterrows():
if x['state'] == state:
point['sic'] = x['sic']
point['numsic'] = x['numsic']
point['empsiz'] = x['empsiz']
point['salvol'] = x['salvol']
point['popcod'] = x['popcod']
point['state'] = x['state']
point['zip'] = x['zip']

point.append()

h5file.flush()

table = h5file.root.raw_records.neg_records
point = table2.row
for x in table.iterrows():
if x['state'] == state:
point['sic'] = x['sic']
point['numsic'] = x['numsic']
point['empsiz'] = x['empsiz']
point['salvol'] = x['salvol']
point['popcod'] = x['popcod']
point['state'] = x['state']
point['zip'] = x['zip']

point.append()
h5file.flush()

states =
['AL','AK','AZ','AR','CA','CO','CT','DC','DE','FL', 'GA','HI','ID','IL','IN','IA','KS','KY','LA','ME', 'MD','MA','MI','MN','MS','MO','MT','NE','NV','NH', 'NJ','NM','NY','NC','ND','OH','OK','OR','PA','RI', 'SC','SD','TN','TX','UT','VT','VA','WA','WV','WI', 'WY']

h5file = tables.openFile("200309_data.h5", mode = 'a')

for i in xrange(len(states)):
aggstate(states[i], h5file)

h5file.close()

[py_genetic]

py_genetic wrote:

I have an H5 file with one group (off the root) and two large main
tables and I'm attempting to aggragate my data into 50+ new groups (off
the root) with two tables per sub group.

sys info:
PyTables version: 1.3.2
HDF5 version: 1.6.5
numarray version: 1.5.0
Zlib version: 1.2.3
BZIP2 version: 1.0.3 (15-Feb-2005)
Python version: 2.4.2 (#1, Jul 13 2006, 20:16:08)
[GCC 4.0.1 (Apple Computer, Inc. build 5250)]
Platform: darwin-Power Macintosh (v10.4.7)
Byte-ordering: big

Ran all pytables tests included with package and recieved an OK.
Using the following code I get one of three errors:

1. Illegal Instruction

2. Malloc(): trying to call free() twice

3. Bus Error

I believe all three stem from the same issue, involving a malloc()
memory problem in the pytable c libraries. I also believe this may be
due to how I'm attempting to write my sorting script.

The script executes fine and all goes well until I'm sorting about
group 20 to 30 and I throw one of the three above errors depending on
how/when I'm flush() close() the file. When I open the file after the
error using h5ls all tables are in perfact order up to the crash and if
I continue from the point every thing runs fine until python throws the
same error again after another 10 sorts or so. The somewhat random
crashing is what leads me to believe I have a memory leak or my method
of doing this is incorrect.

Is there a better way to aggragate data using pytables/python? Is there
a better way to be doing this? This seems strait forward enough.

Thanks,
Conor

#function to agg state data from main neg/pos tables into neg/pos state
tables

import string
import tables
def aggstate(state, h5file):

print state

class PosRecords(tables.IsDescription):
sic = tables.IntCol(0, 1, 4, 0, None, 0)
numsic = tables.IntCol(0, 1, 4, 0, None, 0)
empsiz = tables.StringCol(1, '?', 1, None, 0)
salvol = tables.StringCol(1, '?', 1, None, 0)
popcod = tables.StringCol(1, '?', 1, None, 0)
state = tables.StringCol(2, '?', 1, None, 0)
zip = tables.IntCol(0, 1, 4, 0, None, 1)

class NegRecords(tables.IsDescription):
sic = tables.IntCol(0, 1, 4, 0, None, 0)
numsic = tables.IntCol(0, 1, 4, 0, None, 0)
empsiz = tables.StringCol(1, '?', 1, None, 0)
salvol = tables.StringCol(1, '?', 1, None, 0)
popcod = tables.StringCol(1, '?', 1, None, 0)
state = tables.StringCol(2, '?', 1, None, 0)
zip = tables.IntCol(0, 1, 4, 0, None, 1)

group1 = h5file.createGroup("/", state+"_raw_records", state+" raw
records")

table1 = h5file.createTable(group1, "pos_records", PosRecords, state+"
raw pos record table")
table2 = h5file.createTable(group1, "neg_records", NegRecords, state+"
raw neg record table")

table = h5file.root.raw_records.pos_records
point = table1.row
for x in table.iterrows():
if x['state'] == state:
point['sic'] = x['sic']
point['numsic'] = x['numsic']
point['empsiz'] = x['empsiz']
point['salvol'] = x['salvol']
point['popcod'] = x['popcod']
point['state'] = x['state']
point['zip'] = x['zip']

point.append()

h5file.flush()

table = h5file.root.raw_records.neg_records
point = table2.row
for x in table.iterrows():
if x['state'] == state:
point['sic'] = x['sic']
point['numsic'] = x['numsic']
point['empsiz'] = x['empsiz']
point['salvol'] = x['salvol']
point['popcod'] = x['popcod']
point['state'] = x['state']
point['zip'] = x['zip']

point.append()
h5file.flush()

states =
['AL','AK','AZ','AR','CA','CO','CT','DC','DE','FL', 'GA','HI','ID','IL','IN','IA','KS','KY','LA','ME', 'MD','MA','MI','MN','MS','MO','MT','NE','NV','NH', 'NJ','NM','NY','NC','ND','OH','OK','OR','PA','RI', 'SC','SD','TN','TX','UT','VT','VA','WA','WV','WI', 'WY']

h5file = tables.openFile("200309_data.h5", mode = 'a')

for i in xrange(len(states)):
aggstate(states[i], h5file)

h5file.close()

The problem with my above posting is that h5file.flush() should be
table.flush() (flush the table not the whole object) although
h5file.flush() is an actual method I don't believe it correctly writes
to the tables, it causes all types of issues as time goes on and I
think overlaps .close() causing more issues. I also flushed the table1
and table2 after I created the new group and table1 and table2 each
iteration, things are stable now, pytables is great.